Over the weekend, I was working on a personal Rust project when I ran into an excessive memory usage problem. After an evening of trial and error, I found a workaround to fix the memory usage, but I still didn’t understand how the issue was even possible, so I then spent another evening digging through the source code of the Rust standard library to understand the root cause.

… the story of how I identified the bug …

  • floofloof@lemmy.caEnglish
    9·
    10 months ago

    I just skimmed it but this seems to be the practical lesson for coders:

    As mentioned previously, adding a call to shrink_to_fit() prior to storing the edge lists in the final graph eliminated the memory leak issue. This is fine as a quick hack, but it is not the proper solution. The real fix is to call into_boxed_slice() instead.

    The basic problem is that Vecs are optimized for the case of a mutable list where you’re planning to add and remove elements in the future. However, people also commonly use them for fixed-length list data, even though this is not what Vecs are designed for, simply because Vec is the easiest and most intuitive way to store list data in Rust. The proper data structure for fixed-length lists is Box<[T]> rather than Vec<T>, but it takes a lot of Rust experience to even know about that as an option, and the syntax looks weirder.

    • quilan@lemmy.world
      6·
      10 months ago

      I actually came in here to post just that. It’s something that, while I’ve vaguely known, is not something I typically remember to do - I’m usually returning or collecting Vecs all over. I shall be now diligent & intentional from now on.

    • TehPers@beehaw.orgEnglish
      4·
      10 months ago

      For shared, immutable lists, you can also use Arc/Rc in place of Box! I guess what’s surprising is that [T] (and str/Path/OsStr/etc) are traditionally associated with shared borrows (like &amp;[T]), but they’re really just unsized types that can be used anywhere there’s a ?Sized bound.

      Also, overuse of lists (as opposed to other data structures) seems to be a common problem in all programming languages among less experienced devs since they’re simple enough to use in most places. Not saying that’s what happened to the author though - these types are way less obvious than something like a hash set or queue.

    • taladar@sh.itjust.works
      1·
      10 months ago

      Does Box&lt;[T]> have the relevant trait implementations to be used in all the places where people misuse Vec?