speice.io/_posts/2019-02-06-stacking-up.md

---
layout: post
title: "Fixed Memory: Stacking Up"
description: "We don't need no allocator."
category:
tags: [rust, understanding-allocations]
---

`const` and `static` are perfectly fine, but it's relatively rare that we know at compile-time about
either values or references that will be the same for the duration of our program. Put another way,
it's not often the case that either you or your compiler knows how much memory your entire program
will ever need.

However, there are still some optimizations the compiler can do if it knows how much memory
individual functions will need. Specifically, the compiler can make use of "stack" memory (as
opposed to "heap" memory) which can be managed far faster in both the short- and long-term. When
requesting memory, the [`push` instruction](http://www.cs.virginia.edu/~evans/cs216/guides/x86.html)
can typically complete in [1 or 2 cycles](https://agner.org/optimize/instruction_tables.ods) (<1
nanosecond on modern CPUs). Contrast that to heap memory which requires an allocator (specialized
software to track what memory is in use) to reserve space. When you're finished with stack memory,
the `pop` instruction runs in 1-3 cycles, as opposed to an allocator needing to worry about memory
fragmentation and other issues with the heap. All sorts of incredibly sophisticated techniques have
been used to design allocators:

- [Garbage Collection](<https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)>)
  strategies like [Tracing](https://en.wikipedia.org/wiki/Tracing_garbage_collection) (used in
  [Java](https://www.oracle.com/technetwork/java/javase/tech/g1-intro-jsp-135488.html)) and
  [Reference counting](https://en.wikipedia.org/wiki/Reference_counting) (used in
  [Python](https://docs.python.org/3/extending/extending.html#reference-counts))
- Thread-local structures to prevent locking the allocator in
  [tcmalloc](https://jamesgolick.com/2013/5/19/how-tcmalloc-works.html)
- Arena structures used in [jemalloc](http://jemalloc.net/), which
  [until recently](https://blog.rust-lang.org/2019/01/17/Rust-1.32.0.html#jemalloc-is-removed-by-default)
  was the primary allocator for Rust programs!

But no matter how fast your allocator is, the principle remains: the fastest allocator is the one
you never use. As such, we're not going to discuss how exactly the
[`push` and `pop` instructions work](http://www.cs.virginia.edu/~evans/cs216/guides/x86.html), but
we'll focus instead on the conditions that enable the Rust compiler to use faster stack-based
allocation for variables.

So, **how do we know when Rust will or will not use stack allocation for objects we create?**
Looking at other languages, it's often easy to delineate between stack and heap. Managed memory
languages (Python, Java,
[C#](https://blogs.msdn.microsoft.com/ericlippert/2010/09/30/the-truth-about-value-types/)) place
everything on the heap. JIT compilers ([PyPy](https://www.pypy.org/),
[HotSpot](https://www.oracle.com/technetwork/java/javase/tech/index-jsp-136373.html)) may optimize
some heap allocations away, but you should never assume it will happen. C makes things clear with
calls to special functions (like [malloc(3)](https://linux.die.net/man/3/malloc)) needed to access
heap memory. Old C++ has the [`new`](https://stackoverflow.com/a/655086/1454178) keyword, though
modern C++/C++11 is more complicated with [RAII](https://en.cppreference.com/w/cpp/language/raii).

For Rust, we can summarize as follows: **stack allocation will be used for everything that doesn't
involve "smart pointers" and collections**. We'll skip over a precise definition of the term "smart
pointer" for now, and instead discuss what we should watch for to understand when stack and heap
memory regions are used:

1. Stack manipulation instructions (`push`, `pop`, and `add`/`sub` of the `rsp` register) indicate
   allocation of stack memory:

   ```rust
   pub fn stack_alloc(x: u32) -> u32 {
       // Space for `y` is allocated by subtracting from `rsp`,
       // and then populated
       let y = [1u8, 2, 3, 4];
       // Space for `y` is deallocated by adding back to `rsp`
       x
   }
   ```

   -- [Compiler Explorer](https://godbolt.org/z/5WSgc9)

2. Tracking when exactly heap allocation calls occur is difficult. It's typically easier to watch
   for `call core::ptr::real_drop_in_place`, and infer that a heap allocation happened in the recent
   past:

   ```rust
   pub fn heap_alloc(x: usize) -> usize {
       // Space for elements in a vector has to be allocated
       // on the heap, and is then de-allocated once the
       // vector goes out of scope
       let y: Vec<u8> = Vec::with_capacity(x);
       x
   }
   ```

   -- [Compiler Explorer](https://godbolt.org/z/epfgoQ) (`real_drop_in_place` happens on line 1317)
   <span style="font-size: .8em">Note: While the
   [`Drop` trait](https://doc.rust-lang.org/std/ops/trait.Drop.html) is
   [called for stack-allocated objects](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=87edf374d8983816eb3d8cfeac657b46),
   the Rust standard library only defines `Drop` implementations for types that involve heap
   allocation.</span>

3. If you don't want to inspect the assembly, use a custom allocator that's able to track and alert
   when heap allocations occur. Crates like
   [`alloc_counter`](https://crates.io/crates/alloc_counter) are designed for exactly this purpose.

With all that in mind, let's talk about situations in which we're guaranteed to use stack memory:

- Structs are created on the stack.
- Function arguments are passed on the stack, meaning the
  [`#[inline]` attribute](https://doc.rust-lang.org/reference/attributes.html#inline-attribute) will
  not change the memory region used.
- Enums and unions are stack-allocated.
- [Arrays](https://doc.rust-lang.org/std/primitive.array.html) are always stack-allocated.
- Closures capture their arguments on the stack.
- Generics will use stack allocation, even with dynamic dispatch.
- [`Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html) types are guaranteed to be
  stack-allocated, and copying them will be done in stack memory.
- [`Iterator`s](https://doc.rust-lang.org/std/iter/trait.Iterator.html) in the standard library are
  stack-allocated even when iterating over heap-based collections.

# Structs

The simplest case comes first. When creating vanilla `struct` objects, we use stack memory to hold
their contents:

```rust
struct Point {
    x: u64,
    y: u64,
}

struct Line {
    a: Point,
    b: Point,
}

pub fn make_line() {
    // `origin` is stored in the first 16 bytes of memory
    // starting at location `rsp`
    let origin = Point { x: 0, y: 0 };
    // `point` makes up the next 16 bytes of memory
    let point = Point { x: 1, y: 2 };

    // When creating `ray`, we just move the content out of
    // `origin` and `point` into the next 32 bytes of memory
    let ray = Line { a: origin, b: point };
}
```

-- [Compiler Explorer](https://godbolt.org/z/vri9BE)

Note that while some extra-fancy instructions are used for memory manipulation in the assembly, the
`sub rsp, 64` instruction indicates we're still working with the stack.

# Function arguments

Have you ever wondered how functions communicate with each other? Like, once the variables are given
to you, everything's fine. But how do you "give" those variables to another function? How do you get
the results back afterward? The answer: the compiler arranges memory and assembly instructions using
a pre-determined [calling convention](http://llvm.org/docs/LangRef.html#calling-conventions). This
convention governs the rules around where arguments needed by a function will be located (either in
memory offsets relative to the stack pointer `rsp`, or in other registers), and where the results
can be found once the function has finished. And when multiple languages agree on what the calling
conventions are, you can do things like having [Go call Rust code](https://blog.filippo.io/rustgo/)!

Put simply: it's the compiler's job to figure out how to call other functions, and you can assume
that the compiler is good at its job.

We can see this in action using a simple example:

```rust
struct Point {
    x: i64,
    y: i64,
}

// We use integer division operations to keep
// the assembly clean, understanding the result
// isn't accurate.
fn distance(a: &Point, b: &Point) -> i64 {
    // Immediately subtract from `rsp` the bytes needed
    // to hold all the intermediate results - this is
    // the stack allocation step

    // The compiler used the `rdi` and `rsi` registers
    // to pass our arguments, so read them in
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    // Do the actual math work
    let x_pow = (x1 - x2) * (x1 - x2);
    let y_pow = (y1 - y2) * (y1 - y2);
    let squared = x_pow + y_pow;
    squared / squared

    // Our final result will be stored in the `rax` register
    // so that our caller knows where to retrieve it.
    // Finally, add back to `rsp` the stack memory that is
    // now ready to be used by other functions.
}

pub fn total_distance() {
    let start = Point { x: 1, y: 2 };
    let middle = Point { x: 3, y: 4 };
    let end = Point { x: 5, y: 6 };

    let _dist_1 = distance(&start, &middle);
    let _dist_2 = distance(&middle, &end);
}
```

-- [Compiler Explorer](https://godbolt.org/z/Qmx4ST)

As a consequence of function arguments never using heap memory, we can also infer that functions
using the `#[inline]` attributes also do not heap allocate. But better than inferring, we can look
at the assembly to prove it:

```rust
struct Point {
    x: i64,
    y: i64,
}

// Note that there is no `distance` function in the assembly output,
// and the total line count goes from 229 with inlining off
// to 306 with inline on. Even still, no heap allocations occur.
#[inline(always)]
fn distance(a: &Point, b: &Point) -> i64 {
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    let x_pow = (a.x - b.x) * (a.x - b.x);
    let y_pow = (a.y - b.y) * (a.y - b.y);
    let squared = x_pow + y_pow;
    squared / squared
}

pub fn total_distance() {
    let start = Point { x: 1, y: 2 };
    let middle = Point { x: 3, y: 4 };
    let end = Point { x: 5, y: 6 };

    let _dist_1 = distance(&start, &middle);
    let _dist_2 = distance(&middle, &end);
}
```

-- [Compiler Explorer](https://godbolt.org/z/30Sh66)

Finally, passing by value (arguments with type
[`Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html)) and passing by reference (either
moving ownership or passing a pointer) may have slightly different layouts in assembly, but will
still use either stack memory or CPU registers:

```rust
pub struct Point {
    x: i64,
    y: i64,
}

// Moving values
pub fn distance_moved(a: Point, b: Point) -> i64 {
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    let x_pow = (x1 - x2) * (x1 - x2);
    let y_pow = (y1 - y2) * (y1 - y2);
    let squared = x_pow + y_pow;
    squared / squared
}

// Borrowing values has two extra `mov` instructions on lines 21 and 22
pub fn distance_borrowed(a: &Point, b: &Point) -> i64 {
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    let x_pow = (x1 - x2) * (x1 - x2);
    let y_pow = (y1 - y2) * (y1 - y2);
    let squared = x_pow + y_pow;
    squared / squared
}
```

-- [Compiler Explorer](https://godbolt.org/z/06hGiv)

# Enums

If you've ever worried that wrapping your types in
[`Option`](https://doc.rust-lang.org/stable/core/option/enum.Option.html) or
[`Result`](https://doc.rust-lang.org/stable/core/result/enum.Result.html) would finally make them
large enough that Rust decides to use heap allocation instead, fear no longer: `enum` and union
types don't use heap allocation:

```rust
enum MyEnum {
    Small(u8),
    Large(u64)
}

struct MyStruct {
    x: MyEnum,
    y: MyEnum,
}

pub fn enum_compare() {
    let x = MyEnum::Small(0);
    let y = MyEnum::Large(0);

    let z = MyStruct { x, y };

    let opt = Option::Some(z);
}
```

-- [Compiler Explorer](https://godbolt.org/z/HK7zBx)

Because the size of an `enum` is the size of its largest element plus a flag, the compiler can
predict how much memory is used no matter which variant of an enum is currently stored in a
variable. Thus, enums and unions have no need of heap allocation. There's unfortunately not a great
way to show this in assembly, so I'll instead point you to the
[`core::mem::size_of`](https://doc.rust-lang.org/stable/core/mem/fn.size_of.html#size-of-enums)
documentation.

# Arrays

The array type is guaranteed to be stack allocated, which is why the array size must be declared.
Interestingly enough, this can be used to cause safe Rust programs to crash:

```rust
// 256 bytes
#[derive(Default)]
struct TwoFiftySix {
    _a: [u64; 32]
}

// 8 kilobytes
#[derive(Default)]
struct EightK {
    _a: [TwoFiftySix; 32]
}

// 256 kilobytes
#[derive(Default)]
struct TwoFiftySixK {
    _a: [EightK; 32]
}

// 8 megabytes - exceeds space typically provided for the stack,
// though the kernel can be instructed to allocate more.
// On Linux, you can check stack size using `ulimit -s`
#[derive(Default)]
struct EightM {
    _a: [TwoFiftySixK; 32]
}

fn main() {
    // Because we already have things in stack memory
    // (like the current function call stack), allocating another
    // eight megabytes of stack memory crashes the program
    let _x = EightM::default();
}
```

--
[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=587a6380a4914bcbcef4192c90c01dc4)

There aren't any security implications of this (no memory corruption occurs), but it's good to note
that the Rust compiler won't move arrays into heap memory even if they can be reasonably expected to
overflow the stack.

# Closures

Rules for how anonymous functions capture their arguments are typically language-specific. In Java,
[Lambda Expressions](https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html) are
actually objects created on the heap that capture local primitives by copying, and capture local
non-primitives as (`final`) references.
[Python](https://docs.python.org/3.7/reference/expressions.html#lambda) and
[JavaScript](https://javascriptweblog.wordpress.com/2010/10/25/understanding-javascript-closures/)
both bind _everything_ by reference normally, but Python can also
[capture values](https://stackoverflow.com/a/235764/1454178) and JavaScript has
[Arrow functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions).

In Rust, arguments to closures are the same as arguments to other functions; closures are simply
functions that don't have a declared name. Some weird ordering of the stack may be required to
handle them, but it's the compiler's responsiblity to figure that out.

Each example below has the same effect, but a different assembly implementation. In the simplest
case, we immediately run a closure returned by another function. Because we don't store a reference
to the closure, the stack memory needed to store the captured values is contiguous:

```rust
fn my_func() -> impl FnOnce() {
    let x = 24;
    // Note that this closure in assembly looks exactly like
    // any other function; you even use the `call` instruction
    // to start running it.
    move || { x; }
}

pub fn immediate() {
    my_func()();
    my_func()();
}
```

-- [Compiler Explorer](https://godbolt.org/z/mgJ2zl), 25 total assembly instructions

If we store a reference to the closure, the Rust compiler keeps values it needs in the stack memory
of the original function. Getting the details right is a bit harder, so the instruction count goes
up even though this code is functionally equivalent to our original example:

```rust
pub fn simple_reference() {
    let x = my_func();
    let y = my_func();
    y();
    x();
}
```

-- [Compiler Explorer](https://godbolt.org/z/K_dj5n), 55 total assembly instructions

Even things like variable order can make a difference in instruction count:

```rust
pub fn complex() {
    let x = my_func();
    let y = my_func();
    x();
    y();
}
```

-- [Compiler Explorer](https://godbolt.org/z/p37qFl), 70 total assembly instructions

In every circumstance though, the compiler ensured that no heap allocations were necessary.

# Generics

Traits in Rust come in two broad forms: static dispatch (monomorphization, `impl Trait`) and dynamic
dispatch (trait objects, `dyn Trait`). While dynamic dispatch is often _associated_ with trait
objects being stored in the heap, dynamic dispatch can be used with stack allocated objects as well:

```rust
trait GetInt {
    fn get_int(&self) -> u64;
}

// vtable stored at section L__unnamed_1
struct WhyNotU8 {
    x: u8
}
impl GetInt for WhyNotU8 {
    fn get_int(&self) -> u64 {
        self.x as u64
    }
}

// vtable stored at section L__unnamed_2
struct ActualU64 {
    x: u64
}
impl GetInt for ActualU64 {
    fn get_int(&self) -> u64 {
        self.x
    }
}

// `&dyn` declares that we want to use dynamic dispatch
// rather than monomorphization, so there is only one
// `retrieve_int` function that shows up in the final assembly.
// If we used generics, there would be one implementation of
// `retrieve_int` for each type that implements `GetInt`.
pub fn retrieve_int(u: &dyn GetInt) {
    // In the assembly, we just call an address given to us
    // in the `rsi` register and hope that it was set up
    // correctly when this function was invoked.
    let x = u.get_int();
}

pub fn do_call() {
    // Note that even though the vtable for `WhyNotU8` and
    // `ActualU64` includes a pointer to
    // `core::ptr::real_drop_in_place`, it is never invoked.
    let a = WhyNotU8 { x: 0 };
    let b = ActualU64 { x: 0 };

    retrieve_int(&a);
    retrieve_int(&b);
}
```

-- [Compiler Explorer](https://godbolt.org/z/u_yguS)

It's hard to imagine practical situations where dynamic dispatch would be used for objects that
aren't heap allocated, but it technically can be done.

# Copy types

Understanding move semantics and copy semantics in Rust is weird at first. The Rust docs
[go into detail](https://doc.rust-lang.org/stable/core/marker/trait.Copy.html) far better than can
be addressed here, so I'll leave them to do the job. From a memory perspective though, their
guideline is reasonable:
[if your type can implemement `Copy`, it should](https://doc.rust-lang.org/stable/core/marker/trait.Copy.html#when-should-my-type-be-copy).
While there are potential speed tradeoffs to _benchmark_ when discussing `Copy` (move semantics for
stack objects vs. copying stack pointers vs. copying stack `struct`s), _it's impossible for `Copy`
to introduce a heap allocation_.

But why is this the case? Fundamentally, it's because the language controls what `Copy` means -
["the behavior of `Copy` is not overloadable"](https://doc.rust-lang.org/std/marker/trait.Copy.html#whats-the-difference-between-copy-and-clone)
because it's a marker trait. From there we'll note that a type
[can implement `Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html#when-can-my-type-be-copy)
if (and only if) its components implement `Copy`, and that
[no heap-allocated types implement `Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html#implementors).
Thus, assignments involving heap types are always move semantics, and new heap allocations won't
occur because of implicit operator behavior.

```rust
#[derive(Clone)]
struct Cloneable {
    x: Box<u64>
}

// error[E0204]: the trait `Copy` may not be implemented for this type
#[derive(Copy, Clone)]
struct NotCopyable {
    x: Box<u64>
}
```

-- [Compiler Explorer](https://godbolt.org/z/VToRuK)

# Iterators

In managed memory languages (like
[Java](https://www.youtube.com/watch?v=bSkpMdDe4g4&feature=youtu.be&t=357)), there's a subtle
difference between these two code samples:

```java
public static int sum_for(List<Long> vals) {
    long sum = 0;
    // Regular for loop
    for (int i = 0; i < vals.length; i++) {
        sum += vals[i];
    }
    return sum;
}

public static int sum_foreach(List<Long> vals) {
    long sum = 0;
    // "Foreach" loop - uses iteration
    for (Long l : vals) {
        sum += l;
    }
    return sum;
}
```

In the `sum_for` function, nothing terribly interesting happens. In `sum_foreach`, an object of type
[`Iterator`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Iterator.html)
is allocated on the heap, and will eventually be garbage-collected. This isn't a great design;
iterators are often transient objects that you need during a function and can discard once the
function ends. Sounds exactly like the issue stack-allocated objects address, no?

In Rust, iterators are allocated on the stack. The objects to iterate over are almost certainly in
heap memory, but the iterator itself
([`Iter`](https://doc.rust-lang.org/std/slice/struct.Iter.html)) doesn't need to use the heap. In
each of the examples below we iterate over a collection, but never use heap allocation:

```rust
use std::collections::HashMap;
// There's a lot of assembly generated, but if you search in the text,
// there are no references to `real_drop_in_place` anywhere.

pub fn sum_vec(x: &Vec<u32>) {
    let mut s = 0;
    // Basic iteration over vectors doesn't need allocation
    for y in x {
        s += y;
    }
}

pub fn sum_enumerate(x: &Vec<u32>) {
    let mut s = 0;
    // More complex iterators are just fine too
    for (_i, y) in x.iter().enumerate() {
        s += y;
    }
}

pub fn sum_hm(x: &HashMap<u32, u32>) {
    let mut s = 0;
    // And it's not just Vec, all types will allocate the iterator
    // on stack memory
    for y in x.values() {
        s += y;
    }
}
```

-- [Compiler Explorer](https://godbolt.org/z/FTT3CT)