The project was soon derailed trying to sort out technical issues unrelated to the original purpose. Finding a resolution was a frustrating journey, and it's still not clear whether those problems were my fault. As a result, I'm writing this to try making sense of it, as a case study/reference material, and to salvage something from the process.

Starting strong​

The sole starting requirement was to write everything in TypeScript. Not because of project scale, but because guardrails help with unfamiliar territory. Keeping that in mind, the first question was: how does one start a new project? All I actually need is "compile TypeScript, show it in a browser."

Create React App (CRA) came to the rescue and the rest of that evening was a joy. My TypeScript/JavaScript skills were rusty, but the online documentation was helpful. I had never understood the appeal of JSX (why put a DOM in JavaScript?) until it made connecting an onEvent handler and a function easy.

Some quick dimensional analysis later and there was a sine wave oscillator playing A=440 through the speakers. I specifically remember thinking "modern browsers are magical."

Continuing on​

Now comes the first mistake: I began to worry about "scale" before encountering an actual problem. Rather than rendering audio in the main thread, why not use audio worklets and render in a background thread instead?

The first sign something was amiss came from the TypeScript compiler errors showing the audio worklet API was missing. After searching out Github issues and (unsuccessfully) tweaking the .tsconfig settings, I settled on installing a package and moving on.

The next problem came from actually using the API. Worklets must load from separate "modules," but it wasn't clear how to guarantee the worklet code stayed separate from the application. I saw recommendations to use new URL(<local path>, import.meta.url) and it worked! Well, kind of:

That file has the audio processor code, so why does it get served with Content-Type: video/mp2t?

Floundering about​

Now comes the second mistake: even though I didn't understand the error, I ignored recommendations to just use JavaScript and stuck by the original TypeScript requirement.

I tried different project structures. Moving the worklet code to a new folder didn't help, nor did setting up a monorepo and placing it in a new package.

I tried three different CRA tools - react-app-rewired, craco, customize-react-app - but got the same problem. Each has varying levels of compatibility with recent CRA versions, so it wasn't clear if I had the right solution but implemented it incorrectly. After attempting to eject the application and panicking after seeing the configuration, I abandoned that as well.

I tried changing the webpack configuration: using new loaders, setting asset rules, even changing how webpack detects worker resources. In hindsight, entry points may have been the answer. But because CRA actively resists attempts to change its webpack configuration, and I couldn't find audio worklet examples in any other framework, I gave up.

I tried so many application frameworks. Next.js looked like a good candidate, but added its own bespoke webpack complexity to the existing confusion. Astro had the best "getting started" experience, but I refuse to install an IDE-specific plugin. I first used Deno while exploring Lume, but it couldn't import the audio worklet types (maybe because of module compatibility?). Each framework was unique in its own way (shout-out to SvelteKit) but I couldn't figure out how to make them work.

Learning and reflecting​

I ended up using Vite and vite-plugin-react-pages to handle both "build the app" and "bundle worklets," but the specific tool choice isn't important. Instead, the focus should be on lessons learned.

For myself:

• I'm obsessed with tooling, to the point it can derail the original goal. While it comes from a good place (for example: "types are awesome"), it can get in the way of more important work
• I tend to reach for online resources right after seeing a new problem. While finding help online is often faster, spending time understanding the problem would have been more productive than cycling through (often outdated) blog posts

For the tools:

• Resource bundling is great and solves a genuine challenge. I've heard too many horror stories of developers writing modules by hand to believe this is unnecessary complexity
• Webpack is a build system and modern frameworks are deeply dependent on it (hence the "webpack industrial complex"). While this often saves users from unnecessary complexity, there's no path forward if something breaks
• There's little ability to mix and match tools across frameworks. Next.js and Gatsby let users extend webpack, but because each framework adds its own modules, changes aren't portable. After spending a week looking at webpack, I had an example running with parcel in thirty minutes, but couldn't integrate it

In the end, learning new systems is fun, but a focus on tools that "just work" can leave users out in the cold if they break down.

Still, wouldn't it be nice to have more than a single active interpreter thread? In an age of asynchronicity and M:N threading, Python seems lacking. The ideal scenario is to take advantage of both Python's productivity and the modern CPU's parallel capabilities.

Presented below are two strategies for releasing the GIL's icy grip without giving up on what makes Python a nice language to start with. Bear in mind: these are just the tools, no claim is made about whether it's a good idea to use them. Very often, unlocking the GIL is an XY problem; you want application performance, and the GIL seems like an obvious bottleneck. Remember that any gains from running code in parallel come at the expense of project complexity; messing with the GIL is ultimately messing with Python's memory model.

%load_ext Cython
from numba import jit

N = 1_000_000_000

Cython​

Put simply, Cython is a programming language that looks a lot like Python, gets transpiled to C/C++, and integrates well with the CPython API. It's great for building Python wrappers to C and C++ libraries, writing optimized code for numerical processing, and tons more. And when it comes to managing the GIL, there are two special features:

• The nogil function annotation asserts that a Cython function is safe to use without the GIL, and compilation will fail if it interacts with Python in an unsafe manner
• The with nogil context manager explicitly unlocks the CPython GIL while active

Whenever Cython code runs inside a with nogil block on a separate thread, the Python interpreter is unblocked and allowed to continue work elsewhere. We'll define a "busy work" function that demonstrates this principle in action:

%%cython

# Annotating a function with `nogil` indicates only that it is safe
# to call in a `with nogil` block. It *does not* release the GIL.
cdef unsigned long fibonacci(unsigned long n) nogil:
    if n <= 1:
        return n

    cdef unsigned long a = 0, b = 1, c = 0

    c = a + b
    for _i in range(2, n):
        a = b
        b = c
        c = a + b

    return c


def cython_nogil(unsigned long n):
    # Explicitly release the GIL while running `fibonacci`
    with nogil:
        value = fibonacci(n)

    return value


def cython_gil(unsigned long n):
    # Because the GIL is not explicitly released, it implicitly
    # remains acquired when running the `fibonacci` function
    return fibonacci(n)

First, let's time how long it takes Cython to calculate the billionth Fibonacci number:

%%time
_ = cython_gil(N);

CPU times: user 365 ms, sys: 0 ns, total: 365 ms
Wall time: 372 ms

%%time
_ = cython_nogil(N);

CPU times: user 381 ms, sys: 0 ns, total: 381 ms
Wall time: 388 ms

Both versions (with and without GIL) take effectively the same amount of time to run. Even when running this calculation in parallel on separate threads, it is expected that the run time will double because only one thread can be active at a time:

%%time
from threading import Thread

# Create the two threads to run on
t1 = Thread(target=cython_gil, args=[N])
t2 = Thread(target=cython_gil, args=[N])
# Start the threads
t1.start(); t2.start()
# Wait for the threads to finish
t1.join(); t2.join()

CPU times: user 641 ms, sys: 5.62 ms, total: 647 ms
Wall time: 645 ms

However, if the first thread releases the GIL, the second thread is free to acquire it and run in parallel:

%%time

t1 = Thread(target=cython_nogil, args=[N])
t2 = Thread(target=cython_gil, args=[N])
t1.start(); t2.start()
t1.join(); t2.join()

CPU times: user 717 ms, sys: 372 µs, total: 718 ms
Wall time: 358 ms

Because user time represents the sum of processing time on all threads, it doesn't change much. The "wall time" has been cut roughly in half because each function is running simultaneously.

Keep in mind that the order in which threads are started makes a difference!

%%time

# Note that the GIL-locked version is started first
t1 = Thread(target=cython_gil, args=[N])
t2 = Thread(target=cython_nogil, args=[N])
t1.start(); t2.start()
t1.join(); t2.join()

CPU times: user 667 ms, sys: 0 ns, total: 667 ms
Wall time: 672 ms

Even though the second thread releases the GIL while running, it can't start until the first has completed. Thus, the overall runtime is effectively the same as running two GIL-locked threads.

Finally, be aware that attempting to unlock the GIL from a thread that doesn't own it will crash the interpreter, not just the thread attempting the unlock:

%%cython

cdef int cython_recurse(int n) nogil:
    if n <= 0:
        return 0

    with nogil:
        return cython_recurse(n - 1)

cython_recurse(2)

Fatal Python error: PyEval_SaveThread: NULL tstate
Thread 0x00007f499effd700 (most recent call first):
File "/home/bspeice/.virtualenvs/release-the-gil/lib/python3.7/site-packages/ipykernel/parentpoller.py", line 39 in run
File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap

In practice, avoiding this issue is simple. First, nogil functions probably shouldn't contain with nogil blocks. Second, Cython can conditionally acquire/release the GIL, so these conditions can be used to synchronize access. Finally, Cython's documentation for external C code contains more detail on how to safely manage the GIL.

To conclude: use Cython's nogil annotation to assert that functions are safe for calling when the GIL is unlocked, and with nogil to actually unlock the GIL and run those functions.

Numba​

Like Cython, Numba is a "compiled Python." Where Cython works by compiling a Python-like language to C/C++, Numba compiles Python bytecode directly to machine code at runtime. Behavior is controlled with a special @jit decorator; calling a decorated function first compiles it to machine code before running. Calling the function a second time re-uses that machine code unless the argument types have changed.

Numba works best when a nopython=True argument is added to the @jit decorator; functions compiled in nopython mode avoid the CPython API and have performance comparable to C. Further, adding nogil=True to the @jit decorator unlocks the GIL while that function is running. Note that nogil and nopython are separate arguments; while it is necessary for code to be compiled in nopython mode in order to release the lock, the GIL will remain locked if nogil=False (the default).

Let's repeat the same experiment, this time using Numba instead of Cython:

# The `int` type annotation is only for humans and is ignored
# by Numba.
@jit(nopython=True, nogil=True)
def numba_nogil(n: int) -> int:
    if n <= 1:
        return n

    a = 0
    b = 1

    c = a + b
    for _i in range(2, n):
        a = b
        b = c
        c = a + b

    return c


# Run using `nopython` mode to receive a performance boost,
# but GIL remains locked due to `nogil=False` by default.
@jit(nopython=True)
def numba_gil(n: int) -> int:
    if n <= 1:
        return n

    a = 0
    b = 1

    c = a + b
    for _i in range(2, n):
        a = b
        b = c
        c = a + b

    return c


# Call each function once to force compilation; we don't want
# the timing statistics to include how long it takes to compile.
numba_nogil(N)
numba_gil(N);

We'll perform the same tests as above; first, figure out how long it takes the function to run:

%%time
_ = numba_gil(N)

CPU times: user 253 ms, sys: 258 µs, total: 253 ms
Wall time: 251 ms

Aside: it's not immediately clear why Numba takes ~20% less time to run than Cython for code that should be effectively identical after compilation.

When running two GIL-locked threads, the result (as expected) takes around twice as long to compute:

%%time
t1 = Thread(target=numba_gil, args=[N])
t2 = Thread(target=numba_gil, args=[N])
t1.start(); t2.start()
t1.join(); t2.join()

CPU times: user 541 ms, sys: 3.96 ms, total: 545 ms
Wall time: 541 ms

But if the GIL-unlocking thread starts first, both threads run in parallel:

%%time
t1 = Thread(target=numba_nogil, args=[N])
t2 = Thread(target=numba_gil, args=[N])
t1.start(); t2.start()
t1.join(); t2.join()

CPU times: user 551 ms, sys: 7.77 ms, total: 559 ms
Wall time: 279 ms

Just like Cython, starting the GIL-locked thread first leads to poor performance:

%%time
t1 = Thread(target=numba_gil, args=[N])
t2 = Thread(target=numba_nogil, args=[N])
t1.start(); t2.start()
t1.join(); t2.join()

CPU times: user 524 ms, sys: 0 ns, total: 524 ms
Wall time: 522 ms

Finally, unlike Cython, Numba will unlock the GIL if and only if it is currently acquired; recursively calling @jit(nogil=True) functions is perfectly safe:

from numba import jit

@jit(nopython=True, nogil=True)
def numba_recurse(n: int) -> int:
    if n <= 0:
        return 0

    return numba_recurse(n - 1)

numba_recurse(2);

Conclusion​

Before finishing, it's important to address pain points that will show up if these techniques are used in a more realistic project:

First, code running in a GIL-free context will likely also need non-trivial data structures; GIL-free functions aren't useful if they're constantly interacting with Python objects whose access requires the GIL. Cython provides extension types and Numba provides a @jitclass decorator to address this need.

Second, building and distributing applications that make use of Cython/Numba can be complicated. Cython packages require running the compiler, (potentially) linking/packaging external dependencies, and distributing a binary wheel. Numba is generally simpler because the code being distributed is pure Python, but can be tricky since errors aren't detected until runtime.

Finally, while unlocking the GIL is often a solution in search of a problem, both Cython and Numba provide tools to directly manage the GIL when appropriate. This enables true parallelism (not just concurrency) that is impossible in vanilla Python.

So let's say you're in need of a binary serialization format. Data will be going over the network, not just in memory, so having a schema document and code generation is a must. Performance is crucial, so formats that support zero-copy de/serialization are given priority. And the more languages supported, the better; I use Rust, but can't predict what other languages this could interact with.

Given these requirements, the candidates I could find were:

1. Cap'n Proto has been around the longest, and is the most established
2. Flatbuffers is the newest, and claims to have a simpler encoding
3. Simple Binary Encoding has the simplest encoding, but the Rust implementation is unmaintained

Any one of these will satisfy the project requirements: easy to transmit over a network, reasonably fast, and polyglot support. But how do you actually pick one? It's impossible to know what issues will follow that choice, so I tend to avoid commitment until the last possible moment.

Still, a choice must be made. Instead of worrying about which is "the best," I decided to build a small proof-of-concept system in each format and pit them against each other. All code can be found in the repository for this post.

We'll discuss more in detail, but a quick preview of the results:

• Cap'n Proto: Theoretically performs incredibly well, the implementation had issues
• Flatbuffers: Has some quirks, but largely lived up to its "zero-copy" promises
• SBE: Best median and worst-case performance, but the message structure has a limited feature set

Prologue: Binary Parsing with Nom​

Our benchmark system will be a simple data processor; given depth-of-book market data from IEX, serialize each message into the schema format, read it back, and calculate total size of stock traded and the lowest/highest quoted prices. This test isn't complex, but is representative of the project I need a binary format for.

But before we make it to that point, we have to actually read in the market data. To do so, I'm using a library called nom. Version 5.0 was recently released and brought some big changes, so this was an opportunity to build a non-trivial program and get familiar.

If you don't already know about nom, it's a "parser generator". By combining different smaller parsers, you can assemble a parser to handle complex structures without writing tedious code by hand. For example, when parsing PCAP files:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +---------------------------------------------------------------+
 0 |                    Block Type = 0x00000006                    |
   +---------------------------------------------------------------+
 4 |                      Block Total Length                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 8 |                         Interface ID                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
12 |                        Timestamp (High)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 |                        Timestamp (Low)                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
20 |                         Captured Len                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
24 |                          Packet Len                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                          Packet Data                          |
   |                              ...                              |

...you can build a parser in nom that looks like this:

const ENHANCED_PACKET: [u8; 4] = [0x06, 0x00, 0x00, 0x00];
pub fn enhanced_packet_block(input: &[u8]) -> IResult<&[u8], &[u8]> {
    let (
        remaining,
        (
            block_type,
            block_len,
            interface_id,
            timestamp_high,
            timestamp_low,
            captured_len,
            packet_len,
        ),
    ) = tuple((
        tag(ENHANCED_PACKET),
        le_u32,
        le_u32,
        le_u32,
        le_u32,
        le_u32,
        le_u32,
    ))(input)?;

    let (remaining, packet_data) = take(captured_len)(remaining)?;
    Ok((remaining, packet_data))
}

While this example isn't too interesting, more complex formats (like IEX market data) are where nom really shines.

Ultimately, because the nom code in this shootout was the same for all formats, we're not too interested in its performance. Still, it's worth mentioning that building the market data parser was actually fun; I didn't have to write tons of boring code by hand.

Cap'n Proto​

Now it's time to get into the meaty part of the story. Cap'n Proto was the first format I tried because of how long it has supported Rust (thanks to dwrensha for maintaining the Rust port since 2014!). However, I had a ton of performance concerns once I started using it.

To serialize new messages, Cap'n Proto uses a "builder" object. This builder allocates memory on the heap to hold the message content, but because builders can't be re-used, we have to allocate a new buffer for every single message. I was able to work around this with a special builder that could re-use the buffer, but it required reading through Cap'n Proto's benchmarks to find an example, and used std::mem::transmute to bypass Rust's borrow checker.

The process of reading messages was better, but still had issues. Cap'n Proto has two message encodings: a "packed" representation, and an "unpacked" version. When reading "packed" messages, we need a buffer to unpack the message into before we can use it; Cap'n Proto allocates a new buffer for each message we unpack, and I wasn't able to figure out a way around that. In contrast, the unpacked message format should be where Cap'n Proto shines; its main selling point is that there's no decoding step. However, accomplishing zero-copy deserialization required code in the private API (since fixed), and we allocate a vector on every read for the segment table.

In the end, I put in significant work to make Cap'n Proto as fast as possible, but there were too many issues for me to feel comfortable using it long-term.

Flatbuffers​

This is the new kid on the block. After a first attempt didn't pan out, official support was recently launched. Flatbuffers intends to address the same problems as Cap'n Proto: high-performance, polyglot, binary messaging. The difference is that Flatbuffers claims to have a simpler wire format and more flexibility.

On the whole, I enjoyed using Flatbuffers; the tooling is nice, and unlike Cap'n Proto, parsing messages was actually zero-copy and zero-allocation. However, there were still some issues.

First, Flatbuffers (at least in Rust) can't handle nested vectors. This is a problem for formats like the following:

table Message {
  symbol: string;
}
table MultiMessage {
  messages:[Message];
}

We want to create a MultiMessage which contains a vector of Message, and each Message itself contains a vector (the string type). I was able to work around this by caching Message elements in a SmallVec before building the final MultiMessage, but it was a painful process that I believe contributed to poor serialization performance.

Second, streaming support in Flatbuffers seems to be something of an afterthought. Where Cap'n Proto in Rust handles reading messages from a stream as part of the API, Flatbuffers just sticks a u32 at the front of each message to indicate the size. Not specifically a problem, but calculating message size without that tag is nigh on impossible.

Ultimately, I enjoyed using Flatbuffers, and had to do significantly less work to make it perform well.

Simple Binary Encoding​

Support for SBE was added by the author of one of my favorite Rust blog posts. I've talked previously about how important variance is in high-performance systems, so it was encouraging to read about a format that directly addressed my concerns. SBE has by far the simplest binary format, but it does make some tradeoffs.

Both Cap'n Proto and Flatbuffers use message offsets to handle variable-length data, unions, and various other features. In contrast, messages in SBE are essentially just structs; variable-length data is supported, but there's no union type.

As mentioned in the beginning, the Rust port of SBE works well, but is essentially unmaintained. However, if you don't need union types, and can accept that schemas are XML documents, it's still worth using. SBE's implementation had the best streaming support of all formats I tested, and doesn't trigger allocation during de/serialization.

Results​

After building a test harness for each format, it was time to actually take them for a spin. I used this script to run the benchmarks, and the raw results are here. All data reported below is the average of 10 runs on a single day of IEX data. Results were validated to make sure that each format parsed the data correctly.

Serialization​

This test measures, on a per-message basis, how long it takes to serialize the IEX message into the desired format and write to a pre-allocated buffer.

Deserialization​

This test measures, on a per-message basis, how long it takes to read the previously-serialized message and perform some basic aggregation. The aggregation code is the same for each format, so any performance differences are due solely to the format implementation.

Conclusion​

Building a benchmark turned out to be incredibly helpful in making a decision; because a "union" type isn't important to me, I can be confident that SBE best addresses my needs.

While SBE was the fastest in terms of both median and worst-case performance, its worst case performance was proportionately far higher than any other format. It seems to be that de/serialization time scales with message size, but I'll need to do some more research to understand what exactly is going on.

How I assumed HFT people learn their secret techniques

How else do you explain people working on systems that complete the round trip of market data in to orders out (a.k.a. tick-to-trade) consistently within 750-800 nanoseconds? In roughly the time it takes a computer to access main memory 8 times, trading systems are capable of reading the market data packets, deciding what orders to send, doing risk checks, creating new packets for exchange-specific protocols, and putting those packets on the wire.

Having now worked in the trading industry, I can confirm the developers aren't super-human; I've made some simple mistakes at the very least. Instead, what shows up in public discussions is that philosophy, not technique, separates high-performance systems from everything else. Performance-critical systems don't rely on "this one cool C++ optimization trick" to make code fast (though micro-optimizations have their place); there's a lot more to worry about than just the code written for the project.

The framework I'd propose is this: If you want to build high-performance systems, focus first on reducing performance variance (reducing the gap between the fastest and slowest runs of the same code), and only look at average latency once variance is at an acceptable level.

Don't get me wrong, I'm a much happier person when things are fast. Computer goes from booting in 20 seconds down to 10 because I installed a solid-state drive? Awesome. But if every fifth day it takes a full minute to boot because of corrupted sectors? Not so great. Average speed over the course of a week is the same in each situation, but you're painfully aware of that minute when it happens. When it comes to code, the principal is the same: speeding up a function by an average of 10 milliseconds doesn't mean much if there's a 100ms difference between your fastest and slowest runs. When performance matters, you need to respond quickly every time, not just in aggregate. High-performance systems should first optimize for time variance. Once you're consistent at the time scale you care about, then focus on improving average time.

This focus on variance shows up all the time in industry too (emphasis added in all quotes below):

• In marketing materials for NASDAQ's matching engine, the most performance-sensitive component of the exchange, dependability is highlighted in addition to instantaneous metrics:

  Able to consistently sustain an order rate of over 100,000 orders per second at sub-40 microsecond average latency

• The Aeron message bus has this to say about performance:

  Performance is the key focus. Aeron is designed to be the highest throughput with the lowest and most predictable latency possible of any messaging system

• The company PolySync, which is working on autonomous vehicles, mentions why they picked their specific messaging format:

  In general, high performance is almost always desirable for serialization. But in the world of autonomous vehicles, steady timing performance is even more important than peak throughput. This is because safe operation is sensitive to timing outliers. Nobody wants the system that decides when to slam on the brakes to occasionally take 100 times longer than usual to encode its commands.

• Solarflare, which makes highly-specialized network hardware, points out variance (jitter) as a big concern for electronic trading:

  The high stakes world of electronic trading, investment banks, market makers, hedge funds and exchanges demand the lowest possible latency and jitter while utilizing the highest bandwidth and return on their investment.

And to further clarify: we're not discussing total run-time, but variance of total run-time. There are situations where it's not reasonably possible to make things faster, and you'd much rather be consistent. For example, trading firms use wireless networks because the speed of light through air is faster than through fiber-optic cables. There's still at absolute minimum a ~33.76 millisecond delay required to send data between, say, Chicago and Tokyo. If a trading system in Chicago calls the function for "send order to Tokyo" and waits to see if a trade occurs, there's a physical limit to how long that will take. In this situation, the focus is on keeping variance of additional processing to a minimum, since speed of light is the limiting factor.

So how does one go about looking for and eliminating performance variance? To tell the truth, I don't think a systematic answer or flow-chart exists. There's no substitute for (A) building a deep understanding of the entire technology stack, and (B) actually measuring system performance (though (C) watching a lot of CppCon videos for inspiration never hurt). Even then, every project cares about performance to a different degree; you may need to build an entire replica production system to accurately benchmark at nanosecond precision, or you may be content to simply avoid garbage collection in your Java code.

Even though everyone has different needs, there are still common things to look for when trying to isolate and eliminate variance. In no particular order, these are my focus areas when thinking about high-performance systems:

Update 2019-09-21: Added notes on isolcpus and systemd affinity.

Language-specific​

Garbage Collection: How often does garbage collection happen? When is it triggered? What are the impacts?

• In Python, individual objects are collected if the reference count reaches 0, and each generation is collected if num_alloc - num_dealloc > gc_threshold whenever an allocation happens. The GIL is acquired for the duration of generational collection.
• Java has many different collection algorithms to choose from, each with different characteristics. The default algorithms (Parallel GC in Java 8, G1 in Java 9) freeze the JVM while collecting, while more recent algorithms (ZGC and Shenandoah) are designed to keep "stop the world" to a minimum by doing collection work in parallel.

Allocation: Every language has a different way of interacting with "heap" memory, but the principle is the same: running the allocator to allocate/deallocate memory takes time that can often be put to better use. Understanding when your language interacts with the allocator is crucial, and not always obvious. For example: C++ and Rust don't allocate heap memory for iterators, but Java does (meaning potential GC pauses). Take time to understand heap behavior (I made a a guide for Rust), and look into alternative allocators (jemalloc, tcmalloc) that might run faster than the operating system default.

Data Layout: How your data is arranged in memory matters; data-oriented design and cache locality can have huge impacts on performance. The C family of languages (C, value types in C#, C++) and Rust all have guarantees about the shape every object takes in memory that others (e.g. Java and Python) can't make. Cachegrind and kernel perf counters are both great for understanding how performance relates to memory layout.

Just-In-Time Compilation: Languages that are compiled on the fly (LuaJIT, C#, Java, PyPy) are great because they optimize your program for how it's actually being used, rather than how a compiler expects it to be used. However, there's a variance problem if the program stops executing while waiting for translation from VM bytecode to native code. As a remedy, many languages support ahead-of-time compilation in addition to the JIT versions (CoreRT in C# and GraalVM in Java). On the other hand, LLVM supports Profile Guided Optimization, which theoretically brings JIT benefits to non-JIT languages. Finally, be careful to avoid comparing apples and oranges during benchmarks; you don't want your code to suddenly speed up because the JIT compiler kicked in.

Programming Tricks: These won't make or break performance, but can be useful in specific circumstances. For example, C++ can use templates instead of branches in critical sections.

Kernel​

Code you wrote is almost certainly not the only code running on your hardware. There are many ways the operating system interacts with your program, from interrupts to system calls, that are important to watch for. These are written from a Linux perspective, but Windows does typically have equivalent functionality.

Scheduling: The kernel is normally free to schedule any process on any core, so it's important to reserve CPU cores exclusively for the important programs. There are a few parts to this: first, limit the CPU cores that non-critical processes are allowed to run on by excluding cores from scheduling (isolcpus kernel command-line option), or by setting the init process CPU affinity (systemd example). Second, set critical processes to run on the isolated cores by setting the processor affinity using taskset. Finally, use NO_HZ or chrt to disable scheduling interrupts. Turning off hyper-threading is also likely beneficial.

System calls: Reading from a UNIX socket? Writing to a file? In addition to not knowing how long the I/O operation takes, these all trigger expensive system calls (syscalls). To handle these, the CPU must context switch to the kernel, let the kernel operation complete, then context switch back to your program. We'd rather keep these to a minimum (see timestamp 18:20). Strace is your friend for understanding when and where syscalls happen.

Signal Handling: Far less likely to be an issue, but signals do trigger a context switch if your code has a handler registered. This will be highly dependent on the application, but you can block signals if it's an issue.

Interrupts: System interrupts are how devices connected to your computer notify the CPU that something has happened. The CPU will then choose a processor core to pause and context switch to the OS to handle the interrupt. Make sure that SMP affinity is set so that interrupts are handled on a CPU core not running the program you care about.

NUMA: While NUMA is good at making multi-cell systems transparent, there are variance implications; if the kernel moves a process across nodes, future memory accesses must wait for the controller on the original node. Use numactl to handle memory-/cpu-cell pinning so this doesn't happen.

Hardware​

CPU Pipelining/Speculation: Speculative execution in modern processors gave us vulnerabilities like Spectre, but it also gave us performance improvements like branch prediction. And if the CPU mis-speculates your code, there's variance associated with rewind and replay. While the compiler knows a lot about how your CPU pipelines instructions, code can be structured to help the branch predictor.

Paging: For most systems, virtual memory is incredible. Applications live in their own worlds, and the CPU/MMU figures out the details. However, there's a variance penalty associated with memory paging and caching; if you access more memory pages than the TLB can store, you'll have to wait for the page walk. Kernel perf tools are necessary to figure out if this is an issue, but using huge pages can reduce TLB burdens. Alternately, running applications in a hypervisor like Jailhouse allows one to skip virtual memory entirely, but this is probably more work than the benefits are worth.

Network Interfaces: When more than one computer is involved, variance can go up dramatically. Tuning kernel network parameters may be helpful, but modern systems more frequently opt to skip the kernel altogether with a technique called kernel bypass. This typically requires specialized hardware and drivers, but even industries like telecom are finding the benefits.

Networks​

Routing: There's a reason financial firms are willing to pay millions of euros for rights to a small plot of land - having a straight-line connection from point A to point B means the path their data takes is the shortest possible. In contrast, there are currently 6 computers in between me and Google, but that may change at any moment if my ISP realizes a more efficient route is available. Whether it's using research-quality equipment for shortwave radio, or just making sure there's no data inadvertently going between data centers, routing matters.

Protocol: TCP as a network protocol is awesome: guaranteed and in-order delivery, flow control, and congestion control all built in. But these attributes make the most sense when networking infrastructure is lossy; for systems that expect nearly all packets to be delivered correctly, the setup handshaking and packet acknowledgment are just overhead. Using UDP (unicast or multicast) may make sense in these contexts as it avoids the chatter needed to track connection state, and gap-fill strategies can handle the rest.

Switching: Many routers/switches handle packets using "store-and-forward" behavior: wait for the whole packet, validate checksums, and then send to the next device. In variance terms, the time needed to move data between two nodes is proportional to the size of that data; the switch must "store" all data before it can calculate checksums and "forward" to the next node. With "cut-through" designs, switches will begin forwarding data as soon as they know where the destination is, checksums be damned. This means there's a fixed cost (at the switch) for network traffic, no matter the size.

Final Thoughts​

High-performance systems, regardless of industry, are not magical. They do require extreme precision and attention to detail, but they're designed, built, and operated by regular people, using a lot of tools that are publicly available. Interested in seeing how context switching affects performance of your benchmarks? taskset should be installed in all modern Linux distributions, and can be used to make sure the OS never migrates your process. Curious how often garbage collection triggers during a crucial operation? Your language of choice will typically expose details of its operations (Python, Java). Want to know how hard your program is stressing the TLB? Use perf record and look for dtlb_load_misses.miss_causes_a_walk.

Two final guiding questions, then: first, before attempting to apply some of the technology above to your own systems, can you first identify where/when you care about "high-performance"? As an example, if parts of a system rely on humans pushing buttons, CPU pinning won't have any measurable effect. Humans are already far too slow to react in time. Second, if you're using benchmarks, are they being designed in a way that's actually helpful? Tools like Criterion (also in Rust) and Google's Benchmark output not only average run time, but variance as well; your benchmarking environment is subject to the same concerns your production environment is.

Finally, I believe high-performance systems are a matter of philosophy, not necessarily technique. Rigorous focus on variance is the first step, and there are plenty of ways to measure and mitigate it; once that's at an acceptable level, then optimize for speed.

Either way, I'm baking a little bit again, and figured it was worth taking a quick break to focus on some lighter material. I recently learned two critically important lessons: first, the temperature of the dough when you put the yeast in makes a huge difference.

Previously, when I wasn't paying attention to dough temperature:

Compared with what happens when I put the dough in the microwave for a defrost cycle because the water I used wasn't warm enough:

I mean, just look at the bubbles!

After shaping the dough, I've got two loaves ready:

Now, the recipe normally calls for a Dutch Oven to bake the bread because it keeps the dough from drying out in the oven. Because I don't own a Dutch Oven, I typically put a casserole dish on the bottom rack and fill it with water so there's still some moisture in the oven. This time, I forgot to add the water and learned my second lesson: never add room-temperature water to a glass dish that's currently at 500 degrees.

Needless to say, trying to pull out sharp glass from an incredibly hot oven is not what I expected to be doing during my garden leave.

In the end, the bread crust wasn't great, but the bread itself turned out pretty alright:

I've been writing a lot more during this break, so I'm looking forward to sharing that in the future. In the mean-time, I'm planning on making a sandwich.

Global Allocation:

• const is a fixed value; the compiler is allowed to copy it wherever useful.
• static is a fixed reference; the compiler will guarantee it is unique.

Stack Allocation:

• Everything not using a smart pointer will be allocated on the stack.
• Structs, enums, iterators, arrays, and closures are all stack allocated.
• Cell types (RefCell) behave like smart pointers, but are stack-allocated.
• Inlining (#[inline]) will not affect allocation behavior for better or worse.
• Types that are marked Copy are guaranteed to have their contents stack-allocated.

Heap Allocation:

• Smart pointers (Box, Rc, Mutex, etc.) allocate their contents in heap memory.
• Collections (HashMap, Vec, String, etc.) allocate their contents in heap memory.
• Some smart pointers in the standard library have counterparts in other crates that don't need heap memory. If possible, use those.

-- Raph Levien

Throughout the series so far, we've put a handicap on the code. In the name of consistent and understandable results, we've asked the compiler to pretty please leave the training wheels on. Now is the time where we throw out all the rules and take off the kid gloves. As it turns out, both the Rust compiler and the LLVM optimizers are incredibly sophisticated, and we'll step back and let them do their job.

Similar to "What Has My Compiler Done For Me Lately?", we're focusing on interesting things the Rust language (and LLVM!) can do with memory management. We'll still be looking at assembly code to understand what's going on, but it's important to mention again: please use automated tools like alloc-counter to double-check memory behavior if it's something you care about. It's far too easy to mis-read assembly in large code sections, you should always verify behavior if you care about memory usage.

The guiding principal as we move forward is this: optimizing compilers won't produce worse programs than we started with. There won't be any situations where stack allocations get moved to heap allocations. There will, however, be an opera of optimization.

Update 2019-02-10: When debugging a related issue, it was discovered that the original code worked because LLVM optimized out the entire function, rather than just the allocation segments. The code has been updated with proper use of read_volatile, and a previous section on vector capacity has been removed.

The Case of the Disappearing Box​

Our first optimization comes when LLVM can reason that the lifetime of an object is sufficiently short that heap allocations aren't necessary. In these cases, LLVM will move the allocation to the stack instead! The way this interacts with #[inline] attributes is a bit opaque, but the important part is that LLVM can sometimes do better than the baseline Rust language:

use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicBool, Ordering};

pub fn cmp(x: u32) {
    // Turn on panicking if we allocate on the heap
    DO_PANIC.store(true, Ordering::SeqCst);

    // The compiler is able to see through the constant `Box`
    // and directly compare `x` to 24 - assembly line 73
    let y = Box::new(24);
    let equals = x == *y;

    // This call to drop is eliminated
    drop(y);

    // Need to mark the comparison result as volatile so that
    // LLVM doesn't strip out all the code. If `y` is marked
    // volatile instead, allocation will be forced.
    unsafe { std::ptr::read_volatile(&equals) };

    // Turn off panicking, as there are some deallocations
    // when we exit main.
    DO_PANIC.store(false, Ordering::SeqCst);
}

fn main() {
    cmp(12)
}

#[global_allocator]
static A: PanicAllocator = PanicAllocator;
static DO_PANIC: AtomicBool = AtomicBool::new(false);
struct PanicAllocator;

unsafe impl GlobalAlloc for PanicAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        if DO_PANIC.load(Ordering::SeqCst) {
            panic!("Unexpected allocation.");
        }
        System.alloc(layout)
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        if DO_PANIC.load(Ordering::SeqCst) {
            panic!("Unexpected deallocation.");
        }
        System.dealloc(ptr, layout);
    }
}

-- Compiler Explorer

-- Rust Playground

Dr. Array or: how I learned to love the optimizer​

Finally, this isn't so much about LLVM figuring out different memory behavior, but LLVM stripping out code that doesn't do anything. Optimizations of this type have a lot of nuance to them; if you're not careful, they can make your benchmarks look impossibly good. In Rust, the black_box function (implemented in both libtest and criterion) will tell the compiler to disable this kind of optimization. But if you let LLVM remove unnecessary code, you can end up running programs that previously caused errors:

#[derive(Default)]
struct TwoFiftySix {
    _a: [u64; 32]
}

#[derive(Default)]
struct EightK {
    _a: [TwoFiftySix; 32]
}

#[derive(Default)]
struct TwoFiftySixK {
    _a: [EightK; 32]
}

#[derive(Default)]
struct EightM {
    _a: [TwoFiftySixK; 32]
}

pub fn main() {
    // Normally this blows up because we can't reserve size on stack
    // for the `EightM` struct. But because the compiler notices we
    // never do anything with `_x`, it optimizes out the stack storage
    // and the program completes successfully.
    let _x = EightM::default();
}

-- Compiler Explorer

-- Rust Playground

The heap is used in two situations; when the compiler is unable to predict either the total size of memory needed, or how long the memory is needed for, it allocates space in the heap.

This happens pretty frequently; if you want to download the Google home page, you won't know how large it is until your program runs. And when you're finished with Google, we deallocate the memory so it can be used to store other webpages. If you're interested in a slightly longer explanation of the heap, check out The Stack and the Heap in Rust's documentation.

We won't go into detail on how the heap is managed; the ownership documentation does a phenomenal job explaining both the "why" and "how" of memory management. Instead, we're going to focus on understanding "when" heap allocations occur in Rust.

To start off, take a guess for how many allocations happen in the program below:

fn main() {}

It's obviously a trick question; while no heap allocations occur as a result of that code, the setup needed to call main does allocate on the heap. Here's a way to show it:

#![feature(integer_atomics)]
use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicU64, Ordering};

static ALLOCATION_COUNT: AtomicU64 = AtomicU64::new(0);

struct CountingAllocator;

unsafe impl GlobalAlloc for CountingAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        ALLOCATION_COUNT.fetch_add(1, Ordering::SeqCst);
        System.alloc(layout)
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        System.dealloc(ptr, layout);
    }
}

#[global_allocator]
static A: CountingAllocator = CountingAllocator;

fn main() {
    let x = ALLOCATION_COUNT.fetch_add(0, Ordering::SeqCst);
    println!("There were {} allocations before calling main!", x);
}

-- Rust Playground

As of the time of writing, there are five allocations that happen before main is ever called.

But when we want to understand more practically where heap allocation happens, we'll follow this guide:

• Smart pointers hold their contents in the heap
• Collections are smart pointers for many objects at a time, and reallocate when they need to grow

Finally, there are two "addendum" issues that are important to address when discussing Rust and the heap:

• Non-heap alternatives to many standard library types are available.
• Special allocators to track memory behavior should be used to benchmark code.

Smart pointers​

The first thing to note are the "smart pointer" types. When you have data that must outlive the scope in which it is declared, or your data is of unknown or dynamic size, you'll make use of these types.

The term smart pointer comes from C++, and while it's closely linked to a general design pattern of "Resource Acquisition Is Initialization", we'll use it here specifically to describe objects that are responsible for managing ownership of data allocated on the heap. The smart pointers available in the alloc crate should look mostly familiar:

• Box
• Rc
• Arc
• Cow

The standard library also defines some smart pointers to manage heap objects, though more than can be covered here. Some examples are:

• RwLock
• Mutex

Finally, there is one "gotcha": cell types (like RefCell) look and behave similarly, but don't involve heap allocation. The core::cell docs have more information.

When a smart pointer is created, the data it is given is placed in heap memory and the location of that data is recorded in the smart pointer. Once the smart pointer has determined it's safe to deallocate that memory (when a Box has gone out of scope or a reference count goes to zero), the heap space is reclaimed. We can prove these types use heap memory by looking at code:

use std::rc::Rc;
use std::sync::Arc;
use std::borrow::Cow;

pub fn my_box() {
    // Drop at assembly line 1640
    Box::new(0);
}

pub fn my_rc() {
    // Drop at assembly line 1650
    Rc::new(0);
}

pub fn my_arc() {
    // Drop at assembly line 1660
    Arc::new(0);
}

pub fn my_cow() {
    // Drop at assembly line 1672
    Cow::from("drop");
}

-- Compiler Explorer

Collections​

Collection types use heap memory because their contents have dynamic size; they will request more memory when needed, and can release memory when it's no longer necessary. This dynamic property forces Rust to heap allocate everything they contain. In a way, collections are smart pointers for many objects at a time. Common types that fall under this umbrella are Vec, HashMap, and String (not str).

While collections store the objects they own in heap memory, creating new collections will not allocate on the heap. This is a bit weird; if we call Vec::new(), the assembly shows a corresponding call to real_drop_in_place:

pub fn my_vec() {
    // Drop in place at line 481
    Vec::<u8>::new();
}

-- Compiler Explorer

But because the vector has no elements to manage, no calls to the allocator will ever be dispatched:

use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicBool, Ordering};

fn main() {
    // Turn on panicking if we allocate on the heap
    DO_PANIC.store(true, Ordering::SeqCst);

    // Interesting bit happens here
    let x: Vec<u8> = Vec::new();
    drop(x);

    // Turn panicking back off, some deallocations occur
    // after main as well.
    DO_PANIC.store(false, Ordering::SeqCst);
}

#[global_allocator]
static A: PanicAllocator = PanicAllocator;
static DO_PANIC: AtomicBool = AtomicBool::new(false);
struct PanicAllocator;

unsafe impl GlobalAlloc for PanicAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        if DO_PANIC.load(Ordering::SeqCst) {
            panic!("Unexpected allocation.");
        }
        System.alloc(layout)
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        if DO_PANIC.load(Ordering::SeqCst) {
            panic!("Unexpected deallocation.");
        }
        System.dealloc(ptr, layout);
    }
}

-- Rust Playground

Other standard library types follow the same behavior; make sure to check out HashMap::new(), and String::new().

Heap Alternatives​

While it is a bit strange to speak of the stack after spending time with the heap, it's worth pointing out that some heap-allocated objects in Rust have stack-based counterparts provided by other crates. If you have need of the functionality, but want to avoid allocating, there are typically alternatives available.

When it comes to some standard library smart pointers (RwLock and Mutex), stack-based alternatives are provided in crates like parking_lot and spin. You can check out lock_api::RwLock, lock_api::Mutex, and spin::Once if you're in need of synchronization primitives.

thread_id may be necessary if you're implementing an allocator because thread::current().id() uses a thread_local! structure that needs heap allocation.

Tracing Allocators​

When writing performance-sensitive code, there's no alternative to measuring your code. If you didn't write a benchmark, you don't care about it's performance You should never rely on your instincts when a microsecond is an eternity.

Similarly, there's great work going on in Rust with allocators that keep track of what they're doing (like alloc_counter). When it comes to tracking heap behavior, it's easy to make mistakes; please write tests and make sure you have tools to guard against future issues.

However, there are still some optimizations the compiler can do if it knows how much memory individual functions will need. Specifically, the compiler can make use of "stack" memory (as opposed to "heap" memory) which can be managed far faster in both the short- and long-term.

When requesting memory, the push instruction can typically complete in 1 or 2 cycles (<1ns on modern CPUs). Contrast that to heap memory which requires an allocator (specialized software to track what memory is in use) to reserve space. When you're finished with stack memory, the pop instruction runs in 1-3 cycles, as opposed to an allocator needing to worry about memory fragmentation and other issues with the heap. All sorts of incredibly sophisticated techniques have been used to design allocators:

• Garbage Collection strategies like Tracing (used in Java) and Reference counting (used in Python)
• Thread-local structures to prevent locking the allocator in tcmalloc
• Arena structures used in jemalloc, which until recently was the primary allocator for Rust programs!

But no matter how fast your allocator is, the principle remains: the fastest allocator is the one you never use. As such, we're not going to discuss how exactly the push and pop instructions work, but we'll focus instead on the conditions that enable the Rust compiler to use faster stack-based allocation for variables.

So, how do we know when Rust will or will not use stack allocation for objects we create? Looking at other languages, it's often easy to delineate between stack and heap. Managed memory languages (Python, Java, C#) place everything on the heap. JIT compilers (PyPy, HotSpot) may optimize some heap allocations away, but you should never assume it will happen. C makes things clear with calls to special functions (like malloc(3)) needed to access heap memory. Old C++ has the new keyword, though modern C++/C++11 is more complicated with RAII.

For Rust, we can summarize as follows: stack allocation will be used for everything that doesn't involve "smart pointers" and collections. We'll skip over a precise definition of the term "smart pointer" for now, and instead discuss what we should watch for to understand when stack and heap memory regions are used:

1. Stack manipulation instructions (push, pop, and add/sub of the rsp register) indicate allocation of stack memory:

   pub fn stack_alloc(x: u32) -> u32 {
       // Space for `y` is allocated by subtracting from `rsp`,
       // and then populated
       let y = [1u8, 2, 3, 4];
       // Space for `y` is deallocated by adding back to `rsp`
       x
   }
   

   -- Compiler Explorer

2. Tracking when exactly heap allocation calls occur is difficult. It's typically easier to watch for call core::ptr::real_drop_in_place, and infer that a heap allocation happened in the recent past:

   pub fn heap_alloc(x: usize) -> usize {
       // Space for elements in a vector has to be allocated
       // on the heap, and is then de-allocated once the
       // vector goes out of scope
       let y: Vec<u8> = Vec::with_capacity(x);
       x
   }
   

   -- Compiler Explorer (real_drop_in_place happens on line 1317) Note: While the Drop trait is called for stack-allocated objects, the Rust standard library only defines Drop implementations for types that involve heap allocation.

3. If you don't want to inspect the assembly, use a custom allocator that's able to track and alert when heap allocations occur. Crates like alloc_counter are designed for exactly this purpose.

With all that in mind, let's talk about situations in which we're guaranteed to use stack memory:

• Structs are created on the stack.
• Function arguments are passed on the stack, meaning the #[inline] attribute will not change the memory region used.
• Enums and unions are stack-allocated.
• Arrays are always stack-allocated.
• Closures capture their arguments on the stack.
• Generics will use stack allocation, even with dynamic dispatch.
• Copy types are guaranteed to be stack-allocated, and copying them will be done in stack memory.
• Iterators in the standard library are stack-allocated even when iterating over heap-based collections.

Structs​

The simplest case comes first. When creating vanilla struct objects, we use stack memory to hold their contents:

struct Point {
    x: u64,
    y: u64,
}

struct Line {
    a: Point,
    b: Point,
}

pub fn make_line() {
    // `origin` is stored in the first 16 bytes of memory
    // starting at location `rsp`
    let origin = Point { x: 0, y: 0 };
    // `point` makes up the next 16 bytes of memory
    let point = Point { x: 1, y: 2 };

    // When creating `ray`, we just move the content out of
    // `origin` and `point` into the next 32 bytes of memory
    let ray = Line { a: origin, b: point };
}

-- Compiler Explorer

Note that while some extra-fancy instructions are used for memory manipulation in the assembly, the sub rsp, 64 instruction indicates we're still working with the stack.

Function arguments​

Have you ever wondered how functions communicate with each other? Like, once the variables are given to you, everything's fine. But how do you "give" those variables to another function? How do you get the results back afterward? The answer: the compiler arranges memory and assembly instructions using a pre-determined calling convention. This convention governs the rules around where arguments needed by a function will be located (either in memory offsets relative to the stack pointer rsp, or in other registers), and where the results can be found once the function has finished. And when multiple languages agree on what the calling conventions are, you can do things like having Go call Rust code!

Put simply: it's the compiler's job to figure out how to call other functions, and you can assume that the compiler is good at its job.

We can see this in action using a simple example:

struct Point {
    x: i64,
    y: i64,
}

// We use integer division operations to keep
// the assembly clean, understanding the result
// isn't accurate.
fn distance(a: &Point, b: &Point) -> i64 {
    // Immediately subtract from `rsp` the bytes needed
    // to hold all the intermediate results - this is
    // the stack allocation step

    // The compiler used the `rdi` and `rsi` registers
    // to pass our arguments, so read them in
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    // Do the actual math work
    let x_pow = (x1 - x2) * (x1 - x2);
    let y_pow = (y1 - y2) * (y1 - y2);
    let squared = x_pow + y_pow;
    squared / squared

    // Our final result will be stored in the `rax` register
    // so that our caller knows where to retrieve it.
    // Finally, add back to `rsp` the stack memory that is
    // now ready to be used by other functions.
}

pub fn total_distance() {
    let start = Point { x: 1, y: 2 };
    let middle = Point { x: 3, y: 4 };
    let end = Point { x: 5, y: 6 };

    let _dist_1 = distance(&start, &middle);
    let _dist_2 = distance(&middle, &end);
}

-- Compiler Explorer

As a consequence of function arguments never using heap memory, we can also infer that functions using the #[inline] attributes also do not heap allocate. But better than inferring, we can look at the assembly to prove it:

struct Point {
    x: i64,
    y: i64,
}

// Note that there is no `distance` function in the assembly output,
// and the total line count goes from 229 with inlining off
// to 306 with inline on. Even still, no heap allocations occur.
#[inline(always)]
fn distance(a: &Point, b: &Point) -> i64 {
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    let x_pow = (a.x - b.x) * (a.x - b.x);
    let y_pow = (a.y - b.y) * (a.y - b.y);
    let squared = x_pow + y_pow;
    squared / squared
}

pub fn total_distance() {
    let start = Point { x: 1, y: 2 };
    let middle = Point { x: 3, y: 4 };
    let end = Point { x: 5, y: 6 };

    let _dist_1 = distance(&start, &middle);
    let _dist_2 = distance(&middle, &end);
}

-- Compiler Explorer

Finally, passing by value (arguments with type Copy) and passing by reference (either moving ownership or passing a pointer) may have slightly different layouts in assembly, but will still use either stack memory or CPU registers:

pub struct Point {
    x: i64,
    y: i64,
}

// Moving values
pub fn distance_moved(a: Point, b: Point) -> i64 {
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    let x_pow = (x1 - x2) * (x1 - x2);
    let y_pow = (y1 - y2) * (y1 - y2);
    let squared = x_pow + y_pow;
    squared / squared
}

// Borrowing values has two extra `mov` instructions on lines 21 and 22
pub fn distance_borrowed(a: &Point, b: &Point) -> i64 {
    let x1 = a.x;
    let x2 = b.x;
    let y1 = a.y;
    let y2 = b.y;

    let x_pow = (x1 - x2) * (x1 - x2);
    let y_pow = (y1 - y2) * (y1 - y2);
    let squared = x_pow + y_pow;
    squared / squared
}

-- Compiler Explorer

Enums​

If you've ever worried that wrapping your types in Option or Result would finally make them large enough that Rust decides to use heap allocation instead, fear no longer: enum and union types don't use heap allocation:

enum MyEnum {
    Small(u8),
    Large(u64)
}

struct MyStruct {
    x: MyEnum,
    y: MyEnum,
}

pub fn enum_compare() {
    let x = MyEnum::Small(0);
    let y = MyEnum::Large(0);

    let z = MyStruct { x, y };

    let opt = Option::Some(z);
}

-- Compiler Explorer

Because the size of an enum is the size of its largest element plus a flag, the compiler can predict how much memory is used no matter which variant of an enum is currently stored in a variable. Thus, enums and unions have no need of heap allocation. There's unfortunately not a great way to show this in assembly, so I'll instead point you to the core::mem::size_of documentation.

Arrays​

The array type is guaranteed to be stack allocated, which is why the array size must be declared. Interestingly enough, this can be used to cause safe Rust programs to crash:

// 256 bytes
#[derive(Default)]
struct TwoFiftySix {
    _a: [u64; 32]
}

// 8 kilobytes
#[derive(Default)]
struct EightK {
    _a: [TwoFiftySix; 32]
}

// 256 kilobytes
#[derive(Default)]
struct TwoFiftySixK {
    _a: [EightK; 32]
}

// 8 megabytes - exceeds space typically provided for the stack,
// though the kernel can be instructed to allocate more.
// On Linux, you can check stack size using `ulimit -s`
#[derive(Default)]
struct EightM {
    _a: [TwoFiftySixK; 32]
}

fn main() {
    // Because we already have things in stack memory
    // (like the current function call stack), allocating another
    // eight megabytes of stack memory crashes the program
    let _x = EightM::default();
}

-- Rust Playground

There aren't any security implications of this (no memory corruption occurs), but it's good to note that the Rust compiler won't move arrays into heap memory even if they can be reasonably expected to overflow the stack.

Closures​

Rules for how anonymous functions capture their arguments are typically language-specific. In Java, Lambda Expressions are actually objects created on the heap that capture local primitives by copying, and capture local non-primitives as (final) references. Python and JavaScript both bind everything by reference normally, but Python can also capture values and JavaScript has Arrow functions.

In Rust, arguments to closures are the same as arguments to other functions; closures are simply functions that don't have a declared name. Some weird ordering of the stack may be required to handle them, but it's the compiler's responsiblity to figure that out.

Each example below has the same effect, but a different assembly implementation. In the simplest case, we immediately run a closure returned by another function. Because we don't store a reference to the closure, the stack memory needed to store the captured values is contiguous:

fn my_func() -> impl FnOnce() {
    let x = 24;
    // Note that this closure in assembly looks exactly like
    // any other function; you even use the `call` instruction
    // to start running it.
    move || { x; }
}

pub fn immediate() {
    my_func()();
    my_func()();
}

-- Compiler Explorer, 25 total assembly instructions

If we store a reference to the closure, the Rust compiler keeps values it needs in the stack memory of the original function. Getting the details right is a bit harder, so the instruction count goes up even though this code is functionally equivalent to our original example:

pub fn simple_reference() {
    let x = my_func();
    let y = my_func();
    y();
    x();
}

-- Compiler Explorer, 55 total assembly instructions

Even things like variable order can make a difference in instruction count:

pub fn complex() {
    let x = my_func();
    let y = my_func();
    x();
    y();
}

-- Compiler Explorer, 70 total assembly instructions

In every circumstance though, the compiler ensured that no heap allocations were necessary.

Generics​

Traits in Rust come in two broad forms: static dispatch (monomorphization, impl Trait) and dynamic dispatch (trait objects, dyn Trait). While dynamic dispatch is often associated with trait objects being stored in the heap, dynamic dispatch can be used with stack allocated objects as well:

trait GetInt {
    fn get_int(&self) -> u64;
}

// vtable stored at section L__unnamed_1
struct WhyNotU8 {
    x: u8
}
impl GetInt for WhyNotU8 {
    fn get_int(&self) -> u64 {
        self.x as u64
    }
}

// vtable stored at section L__unnamed_2
struct ActualU64 {
    x: u64
}
impl GetInt for ActualU64 {
    fn get_int(&self) -> u64 {
        self.x
    }
}

// `&dyn` declares that we want to use dynamic dispatch
// rather than monomorphization, so there is only one
// `retrieve_int` function that shows up in the final assembly.
// If we used generics, there would be one implementation of
// `retrieve_int` for each type that implements `GetInt`.
pub fn retrieve_int(u: &dyn GetInt) {
    // In the assembly, we just call an address given to us
    // in the `rsi` register and hope that it was set up
    // correctly when this function was invoked.
    let x = u.get_int();
}

pub fn do_call() {
    // Note that even though the vtable for `WhyNotU8` and
    // `ActualU64` includes a pointer to
    // `core::ptr::real_drop_in_place`, it is never invoked.
    let a = WhyNotU8 { x: 0 };
    let b = ActualU64 { x: 0 };

    retrieve_int(&a);
    retrieve_int(&b);
}

-- Compiler Explorer

It's hard to imagine practical situations where dynamic dispatch would be used for objects that aren't heap allocated, but it technically can be done.

Copy types​

Understanding move semantics and copy semantics in Rust is weird at first. The Rust docs go into detail far better than can be addressed here, so I'll leave them to do the job. From a memory perspective though, their guideline is reasonable: if your type can implemement Copy, it should. While there are potential speed tradeoffs to benchmark when discussing Copy (move semantics for stack objects vs. copying stack pointers vs. copying stack structs), it's impossible for Copy to introduce a heap allocation.

But why is this the case? Fundamentally, it's because the language controls what Copy means - "the behavior of Copy is not overloadable" because it's a marker trait. From there we'll note that a type can implement Copy if (and only if) its components implement Copy, and that no heap-allocated types implement Copy. Thus, assignments involving heap types are always move semantics, and new heap allocations won't occur because of implicit operator behavior.

#[derive(Clone)]
struct Cloneable {
    x: Box<u64>
}

// error[E0204]: the trait `Copy` may not be implemented for this type
#[derive(Copy, Clone)]
struct NotCopyable {
    x: Box<u64>
}

-- Compiler Explorer

Iterators​

In managed memory languages (like Java), there's a subtle difference between these two code samples:

public static int sum_for(List<Long> vals) {
    long sum = 0;
    // Regular for loop
    for (int i = 0; i < vals.length; i++) {
        sum += vals[i];
    }
    return sum;
}

public static int sum_foreach(List<Long> vals) {
    long sum = 0;
    // "Foreach" loop - uses iteration
    for (Long l : vals) {
        sum += l;
    }
    return sum;
}

In the sum_for function, nothing terribly interesting happens. In sum_foreach, an object of type Iterator is allocated on the heap, and will eventually be garbage-collected. This isn't a great design; iterators are often transient objects that you need during a function and can discard once the function ends. Sounds exactly like the issue stack-allocated objects address, no?

In Rust, iterators are allocated on the stack. The objects to iterate over are almost certainly in heap memory, but the iterator itself (Iter) doesn't need to use the heap. In each of the examples below we iterate over a collection, but never use heap allocation:

use std::collections::HashMap;
// There's a lot of assembly generated, but if you search in the text,
// there are no references to `real_drop_in_place` anywhere.

pub fn sum_vec(x: &Vec<u32>) {
    let mut s = 0;
    // Basic iteration over vectors doesn't need allocation
    for y in x {
        s += y;
    }
}

pub fn sum_enumerate(x: &Vec<u32>) {
    let mut s = 0;
    // More complex iterators are just fine too
    for (_i, y) in x.iter().enumerate() {
        s += y;
    }
}

pub fn sum_hm(x: &HashMap<u32, u32>) {
    let mut s = 0;
    // And it's not just Vec, all types will allocate the iterator
    // on stack memory
    for y in x.values() {
        s += y;
    }
}

-- Compiler Explorer

Understanding the value/reference distinction is important for reasons we'll go into below, and while the full specification for these two keywords is available, we'll take a hands-on approach to the topic.

const values​

When a value is guaranteed to be unchanging in your program (where "value" may be scalars, structs, etc.), you can declare it const. This tells the compiler that it's safe to treat the value as never changing, and enables some interesting optimizations; not only is there no initialization cost to creating the value (it is loaded at the same time as the executable parts of your program), but the compiler can also copy the value around if it speeds up the code.

The points we need to address when talking about const are:

• Const values are stored in read-only memory - it's impossible to modify.
• Values resulting from calling a const fn are materialized at compile-time.
• The compiler may (or may not) copy const values wherever it chooses.

Read-Only​

The first point is a bit strange - "read-only memory." The Rust book mentions in a couple places that using mut with constants is illegal, but it's also important to demonstrate just how immutable they are. Typically in Rust you can use interior mutability to modify things that aren't declared mut. RefCell provides an example of this pattern in action:

use std::cell::RefCell;

fn my_mutator(cell: &RefCell<u8>) {
    // Even though we're given an immutable reference,
    // the `replace` method allows us to modify the inner value.
    cell.replace(14);
}

fn main() {
    let cell = RefCell::new(25);
    // Prints out 25
    println!("Cell: {:?}", cell);
    my_mutator(&cell);
    // Prints out 14
    println!("Cell: {:?}", cell);
}

-- Rust Playground

When const is involved though, interior mutability is impossible:

use std::cell::RefCell;

const CELL: RefCell<u8> = RefCell::new(25);

fn my_mutator(cell: &RefCell<u8>) {
    cell.replace(14);
}

fn main() {
    // First line prints 25 as expected
    println!("Cell: {:?}", &CELL);
    my_mutator(&CELL);
    // Second line *still* prints 25
    println!("Cell: {:?}", &CELL);
}

-- Rust Playground

And a second example using Once:

use std::sync::Once;

const SURPRISE: Once = Once::new();

fn main() {
    // This is how `Once` is supposed to be used
    SURPRISE.call_once(|| println!("Initializing..."));
    // Because `Once` is a `const` value, we never record it
    // having been initialized the first time, and this closure
    // will also execute.
    SURPRISE.call_once(|| println!("Initializing again???"));
}

-- Rust Playground

When the const specification refers to "rvalues", this behavior is what they refer to. Clippy will treat this as an error, but it's still something to be aware of.

Initialization​

The next thing to mention is that const values are loaded into memory as part of your program binary. Because of this, any const values declared in your program will be "realized" at compile-time; accessing them may trigger a main-memory lookup (with a fixed address, so your CPU may be able to prefetch the value), but that's it.

use std::cell::RefCell;

const CELL: RefCell<u32> = RefCell::new(24);

pub fn multiply(value: u32) -> u32 {
    // CELL is stored at `.L__unnamed_1`
    value * (*CELL.get_mut())
}

-- Compiler Explorer

The compiler creates one RefCell, uses it everywhere, and never needs to call the RefCell::new function.

Copying​

If it's helpful though, the compiler can choose to copy const values.

const FACTOR: u32 = 1000;

pub fn multiply(value: u32) -> u32 {
    // See assembly line 4 for the `mov edi, 1000` instruction
    value * FACTOR
}

pub fn multiply_twice(value: u32) -> u32 {
    // See assembly lines 22 and 29 for `mov edi, 1000` instructions
    value * FACTOR * FACTOR
}

-- Compiler Explorer

In this example, the FACTOR value is turned into the mov edi, 1000 instruction in both the multiply and multiply_twice functions; the "1000" value is never "stored" anywhere, as it's small enough to inline into the assembly instructions.

Finally, getting the address of a const value is possible, but not guaranteed to be unique (because the compiler can choose to copy values). I was unable to get non-unique pointers in my testing (even using different crates), but the specifications are clear enough: don't rely on pointers to const values being consistent. To be frank, caring about locations for const values is almost certainly a code smell.

static values​

Static variables are related to const variables, but take a slightly different approach. When we declare that a reference is unique for the life of a program, you have a static variable (unrelated to the 'static lifetime). Because of the reference/value distinction with const/static, static variables behave much more like typical "global" variables.

But to understand static, here's what we'll look at:

• static variables are globally unique locations in memory.
• Like const, static variables are loaded at the same time as your program being read into memory.
• All static variables must implement the Sync marker trait.
• Interior mutability is safe and acceptable when using static variables.

Memory Uniqueness​

The single biggest difference between const and static is the guarantees provided about uniqueness. Where const variables may or may not be copied in code, static variables are guarantee to be unique. If we take a previous const example and change it to static, the difference should be clear:

static FACTOR: u32 = 1000;

pub fn multiply(value: u32) -> u32 {
    // The assembly to `mul dword ptr [rip + example::FACTOR]` is how FACTOR gets used
    value * FACTOR
}

pub fn multiply_twice(value: u32) -> u32 {
    // The assembly to `mul dword ptr [rip + example::FACTOR]` is how FACTOR gets used
    value * FACTOR * FACTOR
}

-- Compiler Explorer

Where previously there were plenty of references to multiplying by 1000, the new assembly refers to FACTOR as a named memory location instead. No initialization work needs to be done, but the compiler can no longer prove the value never changes during execution.

Initialization​

Next, let's talk about initialization. The simplest case is initializing static variables with either scalar or struct notation:

#[derive(Debug)]
struct MyStruct {
    x: u32
}

static MY_STRUCT: MyStruct = MyStruct {
    // You can even reference other statics
    // declared later
    x: MY_VAL
};

static MY_VAL: u32 = 24;

fn main() {
    println!("Static MyStruct: {:?}", MY_STRUCT);
}