"use strict";(self.webpackChunkspeice_io=self.webpackChunkspeice_io||[]).push([["1225"],{6760:function(e){e.exports=JSON.parse('{"archive":{"blogPosts":[{"id":"2011/11/webpack-industrial-complex","metadata":{"permalink":"/2011/11/webpack-industrial-complex","source":"@site/blog/2022-11-20-webpack-industrial-complex/index.mdx","title":"The webpack industrial complex","description":"This started because I wanted to build a synthesizer. Setting a goal of \\"digital DX7\\" was ambitious, but I needed something unrelated to the day job. Beyond that, working with audio seemed like a good challenge. I enjoy performance-focused code, and performance problems in audio are conspicuous. Building a web project was an obvious choice because of the web audio API documentation and independence from a large Digital Audio Workstation (DAW).","date":"2022-11-20T12:00:00.000Z","tags":[],"readingTime":4.51,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2011/11/webpack-industrial-complex","title":"The webpack industrial complex","date":"2022-11-20T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731274898000,"nextItem":{"title":"Release the GIL","permalink":"/2019/12/release-the-gil"}},"content":"This started because I wanted to build a synthesizer. Setting a goal of \\"digital DX7\\" was ambitious, but I needed something unrelated to the day job. Beyond that, working with audio seemed like a good challenge. I enjoy performance-focused code, and performance problems in audio are conspicuous. Building a web project was an obvious choice because of the web audio API documentation and independence from a large Digital Audio Workstation (DAW).\\n\\nThe project was soon derailed trying to sort out technical issues unrelated to the original purpose. Finding a resolution was a frustrating journey, and it\'s still not clear whether those problems were my fault. As a result, I\'m writing this to try making sense of it, as a case study/reference material, and to salvage something from the process.\\n\\n\x3c!-- truncate --\x3e\\n\\n## Starting strong\\n\\nThe sole starting requirement was to write everything in TypeScript. Not because of project scale, but because guardrails help with unfamiliar territory. Keeping that in mind, the first question was: how does one start a new project? All I actually need is \\"compile TypeScript, show it in a browser.\\"\\n\\nCreate React App (CRA) came to the rescue and the rest of that evening was a joy. My TypeScript/JavaScript skills were rusty, but the online documentation was helpful. I had never understood the appeal of JSX (why put a DOM in JavaScript?) until it made connecting an `onEvent` handler and a function easy.\\n\\nSome quick dimensional analysis later and there was a sine wave oscillator playing A=440 through the speakers. I specifically remember thinking \\"modern browsers are magical.\\"\\n\\n## Continuing on\\n\\nNow comes the first mistake: I began to worry about \\"scale\\" before encountering an actual problem. Rather than rendering audio in the main thread, why not use audio worklets and render in a background thread instead?\\n\\nThe first sign something was amiss came from the TypeScript compiler errors showing the audio worklet API [was missing](https://github.com/microsoft/TypeScript/issues/28308). After searching out Github issues and (unsuccessfully) tweaking the `.tsconfig` settings, I settled on installing a package and moving on.\\n\\nThe next problem came from actually using the API. Worklets must load from separate \\"modules,\\" but it wasn\'t clear how to guarantee the worklet code stayed separate from the application. I saw recommendations to use `new URL(<local path>, import.meta.url)` and it worked! Well, kind of:\\n\\n![Browser error](./video_mp2t.png)\\n\\nThat file has the audio processor code, so why does it get served with `Content-Type: video/mp2t`?\\n\\n## Floundering about\\n\\nNow comes the second mistake: even though I didn\'t understand the error, I ignored recommendations to [just use JavaScript](https://hackernoon.com/implementing-audioworklets-with-react-8a80a470474) and stuck by the original TypeScript requirement.\\n\\nI tried different project structures. Moving the worklet code to a new folder didn\'t help, nor did setting up a monorepo and placing it in a new package.\\n\\nI tried three different CRA tools - `react-app-rewired`, `craco`, `customize-react-app` - but got the same problem. Each has varying levels of compatibility with recent CRA versions, so it wasn\'t clear if I had the right solution but implemented it incorrectly. After attempting to eject the application and panicking after seeing the configuration, I abandoned that as well.\\n\\nI tried changing the webpack configuration: using [new](https://github.com/webpack/webpack/issues/11543#issuecomment-917673256) [loaders](https://github.com/popelenkow/worker-url), setting [asset rules](https://github.com/webpack/webpack/discussions/14093#discussioncomment-1257149), even [changing how webpack detects worker resources](https://github.com/webpack/webpack/issues/11543#issuecomment-826897590). In hindsight, entry points may have been the answer. But because CRA actively resists attempts to change its webpack configuration, and I couldn\'t find audio worklet examples in any other framework, I gave up.\\n\\nI tried so many application frameworks. Next.js looked like a good candidate, but added its own [bespoke webpack complexity](https://github.com/vercel/next.js/issues/24907) to the existing confusion. Astro had the best \\"getting started\\" experience, but I refuse to install an IDE-specific plugin. I first used Deno while exploring Lume, but it couldn\'t import the audio worklet types (maybe because of module compatibility?). Each framework was unique in its own way (shout-out to SvelteKit) but I couldn\'t figure out how to make them work.\\n\\n## Learning and reflecting\\n\\nI ended up using Vite and vite-plugin-react-pages to handle both \\"build the app\\" and \\"bundle worklets,\\" but the specific tool choice isn\'t important. Instead, the focus should be on lessons learned.\\n\\nFor myself:\\n\\n- I\'m obsessed with tooling, to the point it can derail the original goal. While it comes from a good place (for example: \\"types are awesome\\"), it can get in the way of more important work\\n- I tend to reach for online resources right after seeing a new problem. While finding help online is often faster, spending time understanding the problem would have been more productive than cycling through (often outdated) blog posts\\n\\nFor the tools:\\n\\n- Resource bundling is great and solves a genuine challenge. I\'ve heard too many horror stories of developers writing modules by hand to believe this is unnecessary complexity\\n- Webpack is a build system and modern frameworks are deeply dependent on it (hence the \\"webpack industrial complex\\"). While this often saves users from unnecessary complexity, there\'s no path forward if something breaks\\n- There\'s little ability to mix and match tools across frameworks. Next.js and Gatsby let users extend webpack, but because each framework adds its own modules, changes aren\'t portable. After spending a week looking at webpack, I had an example running with parcel in thirty minutes, but couldn\'t integrate it\\n\\nIn the end, learning new systems is fun, but a focus on tools that \\"just work\\" can leave users out in the cold if they break down."},{"id":"2019/12/release-the-gil","metadata":{"permalink":"/2019/12/release-the-gil","source":"@site/blog/2019-12-14-release-the-gil/index.mdx","title":"Release the GIL","description":"Complaining about the Global Interpreter Lock","date":"2019-12-14T12:00:00.000Z","tags":[],"readingTime":8.58,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/12/release-the-gil","title":"Release the GIL","date":"2019-12-14T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731207983000,"prevItem":{"title":"The webpack industrial complex","permalink":"/2011/11/webpack-industrial-complex"},"nextItem":{"title":"Binary format shootout","permalink":"/2019/09/binary-format-shootout"}},"content":"Complaining about the [Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock)\\n(GIL) seems like a rite of passage for Python developers. It\'s easy to criticize a design decision\\nmade before multi-core CPU\'s were widely available, but the fact that it\'s still around indicates\\nthat it generally works [Good](https://wiki.c2.com/?PrematureOptimization)\\n[Enough](https://wiki.c2.com/?YouArentGonnaNeedIt). Besides, there are simple and effective\\nworkarounds; it\'s not hard to start a\\n[new process](https://docs.python.org/3/library/multiprocessing.html) and use message passing to\\nsynchronize code running in parallel.\\n\\nStill, wouldn\'t it be nice to have more than a single active interpreter thread? In an age of\\nasynchronicity and _M:N_ threading, Python seems lacking. The ideal scenario is to take advantage of\\nboth Python\'s productivity and the modern CPU\'s parallel capabilities.\\n\\n\x3c!-- truncate --\x3e\\n\\nPresented below are two strategies for releasing the GIL\'s icy grip without giving up on what makes\\nPython a nice language to start with. Bear in mind: these are just the tools, no claim is made about\\nwhether it\'s a good idea to use them. Very often, unlocking the GIL is an\\n[XY problem](https://en.wikipedia.org/wiki/XY_problem); you want application performance, and the\\nGIL seems like an obvious bottleneck. Remember that any gains from running code in parallel come at\\nthe expense of project complexity; messing with the GIL is ultimately messing with Python\'s memory\\nmodel.\\n\\n```python\\n%load_ext Cython\\nfrom numba import jit\\n\\nN = 1_000_000_000\\n```\\n\\n## Cython\\n\\nPut simply, [Cython](https://cython.org/) is a programming language that looks a lot like Python,\\ngets [transpiled](https://en.wikipedia.org/wiki/Source-to-source_compiler) to C/C++, and integrates\\nwell with the [CPython](https://en.wikipedia.org/wiki/CPython) API. It\'s great for building Python\\nwrappers to C and C++ libraries, writing optimized code for numerical processing, and tons more. And\\nwhen it comes to managing the GIL, there are two special features:\\n\\n- The `nogil`\\n  [function annotation](https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#declaring-a-function-as-callable-without-the-gil)\\n  asserts that a Cython function is safe to use without the GIL, and compilation will fail if it\\n  interacts with Python in an unsafe manner\\n- The `with nogil`\\n  [context manager](https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#releasing-the-gil)\\n  explicitly unlocks the CPython GIL while active\\n\\nWhenever Cython code runs inside a `with nogil` block on a separate thread, the Python interpreter\\nis unblocked and allowed to continue work elsewhere. We\'ll define a \\"busy work\\" function that\\ndemonstrates this principle in action:\\n\\n```python\\n%%cython\\n\\n# Annotating a function with `nogil` indicates only that it is safe\\n# to call in a `with nogil` block. It *does not* release the GIL.\\ncdef unsigned long fibonacci(unsigned long n) nogil:\\n    if n <= 1:\\n        return n\\n\\n    cdef unsigned long a = 0, b = 1, c = 0\\n\\n    c = a + b\\n    for _i in range(2, n):\\n        a = b\\n        b = c\\n        c = a + b\\n\\n    return c\\n\\n\\ndef cython_nogil(unsigned long n):\\n    # Explicitly release the GIL while running `fibonacci`\\n    with nogil:\\n        value = fibonacci(n)\\n\\n    return value\\n\\n\\ndef cython_gil(unsigned long n):\\n    # Because the GIL is not explicitly released, it implicitly\\n    # remains acquired when running the `fibonacci` function\\n    return fibonacci(n)\\n```\\n\\nFirst, let\'s time how long it takes Cython to calculate the billionth Fibonacci number:\\n\\n```python\\n%%time\\n_ = cython_gil(N);\\n```\\n\\n> <pre>\\n> CPU times: user 365 ms, sys: 0 ns, total: 365 ms\\n> Wall time: 372 ms\\n> </pre>\\n\\n```python\\n%%time\\n_ = cython_nogil(N);\\n```\\n\\n> <pre>\\n> CPU times: user 381 ms, sys: 0 ns, total: 381 ms\\n> Wall time: 388 ms\\n> </pre>\\n\\nBoth versions (with and without GIL) take effectively the same amount of time to run. Even when\\nrunning this calculation in parallel on separate threads, it is expected that the run time will\\ndouble because only one thread can be active at a time:\\n\\n```python\\n%%time\\nfrom threading import Thread\\n\\n# Create the two threads to run on\\nt1 = Thread(target=cython_gil, args=[N])\\nt2 = Thread(target=cython_gil, args=[N])\\n# Start the threads\\nt1.start(); t2.start()\\n# Wait for the threads to finish\\nt1.join(); t2.join()\\n```\\n\\n> <pre>\\n> CPU times: user 641 ms, sys: 5.62 ms, total: 647 ms\\n> Wall time: 645 ms\\n> </pre>\\n\\nHowever, if the first thread releases the GIL, the second thread is free to acquire it and run in\\nparallel:\\n\\n```python\\n%%time\\n\\nt1 = Thread(target=cython_nogil, args=[N])\\nt2 = Thread(target=cython_gil, args=[N])\\nt1.start(); t2.start()\\nt1.join(); t2.join()\\n```\\n\\n> <pre>\\n> CPU times: user 717 ms, sys: 372 \xb5s, total: 718 ms\\n> Wall time: 358 ms\\n> </pre>\\n\\nBecause `user` time represents the sum of processing time on all threads, it doesn\'t change much.\\nThe [\\"wall time\\"](https://en.wikipedia.org/wiki/Elapsed_real_time) has been cut roughly in half\\nbecause each function is running simultaneously.\\n\\nKeep in mind that the **order in which threads are started** makes a difference!\\n\\n```python\\n%%time\\n\\n# Note that the GIL-locked version is started first\\nt1 = Thread(target=cython_gil, args=[N])\\nt2 = Thread(target=cython_nogil, args=[N])\\nt1.start(); t2.start()\\nt1.join(); t2.join()\\n```\\n\\n> <pre>\\n> CPU times: user 667 ms, sys: 0 ns, total: 667 ms\\n> Wall time: 672 ms\\n> </pre>\\n\\nEven though the second thread releases the GIL while running, it can\'t start until the first has\\ncompleted. Thus, the overall runtime is effectively the same as running two GIL-locked threads.\\n\\nFinally, be aware that attempting to unlock the GIL from a thread that doesn\'t own it will crash the\\n**interpreter**, not just the thread attempting the unlock:\\n\\n```python\\n%%cython\\n\\ncdef int cython_recurse(int n) nogil:\\n    if n <= 0:\\n        return 0\\n\\n    with nogil:\\n        return cython_recurse(n - 1)\\n\\ncython_recurse(2)\\n```\\n\\n> <pre>\\n> Fatal Python error: PyEval_SaveThread: NULL tstate\\n> \\n> Thread 0x00007f499effd700 (most recent call first):\\n>   File \\"/home/bspeice/.virtualenvs/release-the-gil/lib/python3.7/site-packages/ipykernel/parentpoller.py\\", line 39 in run\\n>   File \\"/usr/lib/python3.7/threading.py\\", line 926 in _bootstrap_inner\\n>   File \\"/usr/lib/python3.7/threading.py\\", line 890 in _bootstrap\\n> </pre>\\n\\nIn practice, avoiding this issue is simple. First, `nogil` functions probably shouldn\'t contain\\n`with nogil` blocks. Second, Cython can\\n[conditionally acquire/release](https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#conditional-acquiring-releasing-the-gil)\\nthe GIL, so these conditions can be used to synchronize access. Finally, Cython\'s documentation for\\n[external C code](https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#acquiring-and-releasing-the-gil)\\ncontains more detail on how to safely manage the GIL.\\n\\nTo conclude: use Cython\'s `nogil` annotation to assert that functions are safe for calling when the\\nGIL is unlocked, and `with nogil` to actually unlock the GIL and run those functions.\\n\\n## Numba\\n\\nLike Cython, [Numba](https://numba.pydata.org/) is a \\"compiled Python.\\" Where Cython works by\\ncompiling a Python-like language to C/C++, Numba compiles Python bytecode _directly to machine code_\\nat runtime. Behavior is controlled with a special `@jit` decorator; calling a decorated function\\nfirst compiles it to machine code before running. Calling the function a second time re-uses that\\nmachine code unless the argument types have changed.\\n\\nNumba works best when a `nopython=True` argument is added to the `@jit` decorator; functions\\ncompiled in [`nopython`](http://numba.pydata.org/numba-doc/latest/user/jit.html?#nopython) mode\\navoid the CPython API and have performance comparable to C. Further, adding `nogil=True` to the\\n`@jit` decorator unlocks the GIL while that function is running. Note that `nogil` and `nopython`\\nare separate arguments; while it is necessary for code to be compiled in `nopython` mode in order to\\nrelease the lock, the GIL will remain locked if `nogil=False` (the default).\\n\\nLet\'s repeat the same experiment, this time using Numba instead of Cython:\\n\\n```python\\n# The `int` type annotation is only for humans and is ignored\\n# by Numba.\\n@jit(nopython=True, nogil=True)\\ndef numba_nogil(n: int) -> int:\\n    if n <= 1:\\n        return n\\n\\n    a = 0\\n    b = 1\\n\\n    c = a + b\\n    for _i in range(2, n):\\n        a = b\\n        b = c\\n        c = a + b\\n\\n    return c\\n\\n\\n# Run using `nopython` mode to receive a performance boost,\\n# but GIL remains locked due to `nogil=False` by default.\\n@jit(nopython=True)\\ndef numba_gil(n: int) -> int:\\n    if n <= 1:\\n        return n\\n\\n    a = 0\\n    b = 1\\n\\n    c = a + b\\n    for _i in range(2, n):\\n        a = b\\n        b = c\\n        c = a + b\\n\\n    return c\\n\\n\\n# Call each function once to force compilation; we don\'t want\\n# the timing statistics to include how long it takes to compile.\\nnumba_nogil(N)\\nnumba_gil(N);\\n```\\n\\nWe\'ll perform the same tests as above; first, figure out how long it takes the function to run:\\n\\n```python\\n%%time\\n_ = numba_gil(N)\\n```\\n\\n> <pre>\\n> CPU times: user 253 ms, sys: 258 \xb5s, total: 253 ms\\n> Wall time: 251 ms\\n> </pre>\\n\\n<small>\\nAside: it\'s not immediately clear why Numba takes ~20% less time to run than Cython for code that should be\\neffectively identical after compilation.\\n</small>\\n\\nWhen running two GIL-locked threads, the result (as expected) takes around twice as long to compute:\\n\\n```python\\n%%time\\nt1 = Thread(target=numba_gil, args=[N])\\nt2 = Thread(target=numba_gil, args=[N])\\nt1.start(); t2.start()\\nt1.join(); t2.join()\\n```\\n\\n> <pre>\\n> CPU times: user 541 ms, sys: 3.96 ms, total: 545 ms\\n> Wall time: 541 ms\\n> </pre>\\n\\nBut if the GIL-unlocking thread starts first, both threads run in parallel:\\n\\n```python\\n%%time\\nt1 = Thread(target=numba_nogil, args=[N])\\nt2 = Thread(target=numba_gil, args=[N])\\nt1.start(); t2.start()\\nt1.join(); t2.join()\\n```\\n\\n> <pre>\\n> CPU times: user 551 ms, sys: 7.77 ms, total: 559 ms\\n> Wall time: 279 ms\\n> </pre>\\n\\nJust like Cython, starting the GIL-locked thread first leads to poor performance:\\n\\n```python\\n%%time\\nt1 = Thread(target=numba_gil, args=[N])\\nt2 = Thread(target=numba_nogil, args=[N])\\nt1.start(); t2.start()\\nt1.join(); t2.join()\\n```\\n\\n> <pre>\\n> CPU times: user 524 ms, sys: 0 ns, total: 524 ms\\n> Wall time: 522 ms\\n> </pre>\\n\\nFinally, unlike Cython, Numba will unlock the GIL if and only if it is currently acquired;\\nrecursively calling `@jit(nogil=True)` functions is perfectly safe:\\n\\n```python\\nfrom numba import jit\\n\\n@jit(nopython=True, nogil=True)\\ndef numba_recurse(n: int) -> int:\\n    if n <= 0:\\n        return 0\\n\\n    return numba_recurse(n - 1)\\n\\nnumba_recurse(2);\\n```\\n\\n## Conclusion\\n\\nBefore finishing, it\'s important to address pain points that will show up if these techniques are\\nused in a more realistic project:\\n\\nFirst, code running in a GIL-free context will likely also need non-trivial data structures;\\nGIL-free functions aren\'t useful if they\'re constantly interacting with Python objects whose access\\nrequires the GIL. Cython provides\\n[extension types](http://docs.cython.org/en/latest/src/tutorial/cdef_classes.html) and Numba\\nprovides a [`@jitclass`](https://numba.pydata.org/numba-doc/dev/user/jitclass.html) decorator to\\naddress this need.\\n\\nSecond, building and distributing applications that make use of Cython/Numba can be complicated.\\nCython packages require running the compiler, (potentially) linking/packaging external dependencies,\\nand distributing a binary wheel. Numba is generally simpler because the code being distributed is\\npure Python, but can be tricky since errors aren\'t detected until runtime.\\n\\nFinally, while unlocking the GIL is often a solution in search of a problem, both Cython and Numba\\nprovide tools to directly manage the GIL when appropriate. This enables true parallelism (not just\\n[concurrency](https://stackoverflow.com/a/1050257)) that is impossible in vanilla Python."},{"id":"2019/09/binary-format-shootout","metadata":{"permalink":"/2019/09/binary-format-shootout","source":"@site/blog/2019-09-28-binary-format-shootout/index.mdx","title":"Binary format shootout","description":"I\'ve found that in many personal projects,","date":"2019-09-28T12:00:00.000Z","tags":[],"readingTime":8.37,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/09/binary-format-shootout","title":"Binary format shootout","date":"2019-09-28T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731207983000,"prevItem":{"title":"Release the GIL","permalink":"/2019/12/release-the-gil"},"nextItem":{"title":"On building high performance systems","permalink":"/2019/06/high-performance-systems"}},"content":"I\'ve found that in many personal projects,\\n[analysis paralysis](https://en.wikipedia.org/wiki/Analysis_paralysis) is particularly deadly.\\nMaking good decisions in the beginning avoids pain and suffering later; if extra research prevents\\nfuture problems, I\'m happy to continue ~~procrastinating~~ researching indefinitely.\\n\\nSo let\'s say you\'re in need of a binary serialization format. Data will be going over the network,\\nnot just in memory, so having a schema document and code generation is a must. Performance is\\ncrucial, so formats that support zero-copy de/serialization are given priority. And the more\\nlanguages supported, the better; I use Rust, but can\'t predict what other languages this could\\ninteract with.\\n\\nGiven these requirements, the candidates I could find were:\\n\\n\x3c!-- truncate --\x3e\\n\\n1. [Cap\'n Proto](https://capnproto.org/) has been around the longest, and is the most established\\n2. [Flatbuffers](https://google.github.io/flatbuffers/) is the newest, and claims to have a simpler\\n   encoding\\n3. [Simple Binary Encoding](https://github.com/real-logic/simple-binary-encoding) has the simplest\\n   encoding, but the Rust implementation is unmaintained\\n\\nAny one of these will satisfy the project requirements: easy to transmit over a network, reasonably\\nfast, and polyglot support. But how do you actually pick one? It\'s impossible to know what issues\\nwill follow that choice, so I tend to avoid commitment until the last possible moment.\\n\\nStill, a choice must be made. Instead of worrying about which is \\"the best,\\" I decided to build a\\nsmall proof-of-concept system in each format and pit them against each other. All code can be found\\nin the [repository](https://github.com/speice-io/marketdata-shootout) for this post.\\n\\nWe\'ll discuss more in detail, but a quick preview of the results:\\n\\n- Cap\'n Proto: Theoretically performs incredibly well, the implementation had issues\\n- Flatbuffers: Has some quirks, but largely lived up to its \\"zero-copy\\" promises\\n- SBE: Best median and worst-case performance, but the message structure has a limited feature set\\n\\n## Prologue: Binary Parsing with Nom\\n\\nOur benchmark system will be a simple data processor; given depth-of-book market data from\\n[IEX](https://iextrading.com/trading/market-data/#deep), serialize each message into the schema\\nformat, read it back, and calculate total size of stock traded and the lowest/highest quoted prices.\\nThis test isn\'t complex, but is representative of the project I need a binary format for.\\n\\nBut before we make it to that point, we have to actually read in the market data. To do so, I\'m\\nusing a library called [`nom`](https://github.com/Geal/nom). Version 5.0 was recently released and\\nbrought some big changes, so this was an opportunity to build a non-trivial program and get\\nfamiliar.\\n\\nIf you don\'t already know about `nom`, it\'s a \\"parser generator\\". By combining different smaller\\nparsers, you can assemble a parser to handle complex structures without writing tedious code by\\nhand. For example, when parsing\\n[PCAP files](https://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html#rfc.section.3.3):\\n\\n```\\n   0                   1                   2                   3\\n   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1\\n   +---------------------------------------------------------------+\\n 0 |                    Block Type = 0x00000006                    |\\n   +---------------------------------------------------------------+\\n 4 |                      Block Total Length                       |\\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\\n 8 |                         Interface ID                          |\\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\\n12 |                        Timestamp (High)                       |\\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\\n16 |                        Timestamp (Low)                        |\\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\\n20 |                         Captured Len                          |\\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\\n24 |                          Packet Len                           |\\n   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\\n   |                          Packet Data                          |\\n   |                              ...                              |\\n```\\n\\n...you can build a parser in `nom` that looks like\\n[this](https://github.com/speice-io/marketdata-shootout/blob/369613843d39cfdc728e1003123bf87f79422497/src/parsers.rs#L59-L93):\\n\\n```rust\\nconst ENHANCED_PACKET: [u8; 4] = [0x06, 0x00, 0x00, 0x00];\\npub fn enhanced_packet_block(input: &[u8]) -> IResult<&[u8], &[u8]> {\\n    let (\\n        remaining,\\n        (\\n            block_type,\\n            block_len,\\n            interface_id,\\n            timestamp_high,\\n            timestamp_low,\\n            captured_len,\\n            packet_len,\\n        ),\\n    ) = tuple((\\n        tag(ENHANCED_PACKET),\\n        le_u32,\\n        le_u32,\\n        le_u32,\\n        le_u32,\\n        le_u32,\\n        le_u32,\\n    ))(input)?;\\n\\n    let (remaining, packet_data) = take(captured_len)(remaining)?;\\n    Ok((remaining, packet_data))\\n}\\n```\\n\\nWhile this example isn\'t too interesting, more complex formats (like IEX market data) are where\\n[`nom` really shines](https://github.com/speice-io/marketdata-shootout/blob/369613843d39cfdc728e1003123bf87f79422497/src/iex.rs).\\n\\nUltimately, because the `nom` code in this shootout was the same for all formats, we\'re not too\\ninterested in its performance. Still, it\'s worth mentioning that building the market data parser was\\nactually fun; I didn\'t have to write tons of boring code by hand.\\n\\n## Cap\'n Proto\\n\\nNow it\'s time to get into the meaty part of the story. Cap\'n Proto was the first format I tried\\nbecause of how long it has supported Rust (thanks to [dwrensha](https://github.com/dwrensha) for\\nmaintaining the Rust port since\\n[2014!](https://github.com/capnproto/capnproto-rust/releases/tag/rustc-0.10)). However, I had a ton\\nof performance concerns once I started using it.\\n\\nTo serialize new messages, Cap\'n Proto uses a \\"builder\\" object. This builder allocates memory on the\\nheap to hold the message content, but because builders\\n[can\'t be re-used](https://github.com/capnproto/capnproto-rust/issues/111), we have to allocate a\\nnew buffer for every single message. I was able to work around this with a\\n[special builder](https://github.com/speice-io/marketdata-shootout/blob/369613843d39cfdc728e1003123bf87f79422497/src/capnp_runner.rs#L17-L51)\\nthat could re-use the buffer, but it required reading through Cap\'n Proto\'s\\n[benchmarks](https://github.com/capnproto/capnproto-rust/blob/master/benchmark/benchmark.rs#L124-L156)\\nto find an example, and used\\n[`std::mem::transmute`](https://doc.rust-lang.org/std/mem/fn.transmute.html) to bypass Rust\'s borrow\\nchecker.\\n\\nThe process of reading messages was better, but still had issues. Cap\'n Proto has two message\\nencodings: a [\\"packed\\"](https://capnproto.org/encoding.html#packing) representation, and an\\n\\"unpacked\\" version. When reading \\"packed\\" messages, we need a buffer to unpack the message into\\nbefore we can use it; Cap\'n Proto allocates a new buffer for each message we unpack, and I wasn\'t\\nable to figure out a way around that. In contrast, the unpacked message format should be where Cap\'n\\nProto shines; its main selling point is that there\'s [no decoding step](https://capnproto.org/).\\nHowever, accomplishing zero-copy deserialization required code in the private API\\n([since fixed](https://github.com/capnproto/capnproto-rust/issues/148)), and we allocate a vector on\\nevery read for the segment table.\\n\\nIn the end, I put in significant work to make Cap\'n Proto as fast as possible, but there were too\\nmany issues for me to feel comfortable using it long-term.\\n\\n## Flatbuffers\\n\\nThis is the new kid on the block. After a\\n[first attempt](https://github.com/google/flatbuffers/pull/3894) didn\'t pan out, official support\\nwas [recently launched](https://github.com/google/flatbuffers/pull/4898). Flatbuffers intends to\\naddress the same problems as Cap\'n Proto: high-performance, polyglot, binary messaging. The\\ndifference is that Flatbuffers claims to have a simpler wire format and\\n[more flexibility](https://google.github.io/flatbuffers/flatbuffers_benchmarks.html).\\n\\nOn the whole, I enjoyed using Flatbuffers; the [tooling](https://crates.io/crates/flatc-rust) is\\nnice, and unlike Cap\'n Proto, parsing messages was actually zero-copy and zero-allocation. However,\\nthere were still some issues.\\n\\nFirst, Flatbuffers (at least in Rust) can\'t handle nested vectors. This is a problem for formats\\nlike the following:\\n\\n```\\ntable Message {\\n  symbol: string;\\n}\\ntable MultiMessage {\\n  messages:[Message];\\n}\\n```\\n\\nWe want to create a `MultiMessage` which contains a vector of `Message`, and each `Message` itself\\ncontains a vector (the `string` type). I was able to work around this by\\n[caching `Message` elements](https://github.com/speice-io/marketdata-shootout/blob/e9d07d148bf36a211a6f86802b313c4918377d1b/src/flatbuffers_runner.rs#L83)\\nin a `SmallVec` before building the final `MultiMessage`, but it was a painful process that I\\nbelieve contributed to poor serialization performance.\\n\\nSecond, streaming support in Flatbuffers seems to be something of an\\n[afterthought](https://github.com/google/flatbuffers/issues/3898). Where Cap\'n Proto in Rust handles\\nreading messages from a stream as part of the API, Flatbuffers just sticks a `u32` at the front of\\neach message to indicate the size. Not specifically a problem, but calculating message size without\\nthat tag is nigh on impossible.\\n\\nUltimately, I enjoyed using Flatbuffers, and had to do significantly less work to make it perform\\nwell.\\n\\n## Simple Binary Encoding\\n\\nSupport for SBE was added by the author of one of my favorite\\n[Rust blog posts](https://web.archive.org/web/20190427124806/https://polysync.io/blog/session-types-for-hearty-codecs/).\\nI\'ve [talked previously](/2019/06/high-performance-systems) about how important\\nvariance is in high-performance systems, so it was encouraging to read about a format that\\n[directly addressed](https://github.com/real-logic/simple-binary-encoding/wiki/Why-Low-Latency) my\\nconcerns. SBE has by far the simplest binary format, but it does make some tradeoffs.\\n\\nBoth Cap\'n Proto and Flatbuffers use [message offsets](https://capnproto.org/encoding.html#structs)\\nto handle variable-length data, [unions](https://capnproto.org/language.html#unions), and various\\nother features. In contrast, messages in SBE are essentially\\n[just structs](https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-samples/src/main/resources/example-schema.xml);\\nvariable-length data is supported, but there\'s no union type.\\n\\nAs mentioned in the beginning, the Rust port of SBE works well, but is\\n[essentially unmaintained](https://users.rust-lang.org/t/zero-cost-abstraction-frontier-no-copy-low-allocation-ordered-decoding/11515/9).\\nHowever, if you don\'t need union types, and can accept that schemas are XML documents, it\'s still\\nworth using. SBE\'s implementation had the best streaming support of all formats I tested, and\\ndoesn\'t trigger allocation during de/serialization.\\n\\n## Results\\n\\nAfter building a test harness\\n[for](https://github.com/speice-io/marketdata-shootout/blob/master/src/capnp_runner.rs)\\n[each](https://github.com/speice-io/marketdata-shootout/blob/master/src/flatbuffers_runner.rs)\\n[format](https://github.com/speice-io/marketdata-shootout/blob/master/src/sbe_runner.rs), it was\\ntime to actually take them for a spin. I used\\n[this script](https://github.com/speice-io/marketdata-shootout/blob/master/run_shootout.sh) to run\\nthe benchmarks, and the raw results are\\n[here](https://github.com/speice-io/marketdata-shootout/blob/master/shootout.csv). All data reported\\nbelow is the average of 10 runs on a single day of IEX data. Results were validated to make sure\\nthat each format parsed the data correctly.\\n\\n### Serialization\\n\\nThis test measures, on a\\n[per-message basis](https://github.com/speice-io/marketdata-shootout/blob/master/src/main.rs#L268-L272),\\nhow long it takes to serialize the IEX message into the desired format and write to a pre-allocated\\nbuffer.\\n\\n| Schema               | Median | 99th Pctl | 99.9th Pctl | Total  |\\n| :------------------- | :----- | :-------- | :---------- | :----- |\\n| Cap\'n Proto Packed   | 413ns  | 1751ns    | 2943ns      | 14.80s |\\n| Cap\'n Proto Unpacked | 273ns  | 1828ns    | 2836ns      | 10.65s |\\n| Flatbuffers          | 355ns  | 2185ns    | 3497ns      | 14.31s |\\n| SBE                  | 91ns   | 1535ns    | 2423ns      | 3.91s  |\\n\\n### Deserialization\\n\\nThis test measures, on a\\n[per-message basis](https://github.com/speice-io/marketdata-shootout/blob/master/src/main.rs#L294-L298),\\nhow long it takes to read the previously-serialized message and perform some basic aggregation. The\\naggregation code is the same for each format, so any performance differences are due solely to the\\nformat implementation.\\n\\n| Schema               | Median | 99th Pctl | 99.9th Pctl | Total  |\\n| :------------------- | :----- | :-------- | :---------- | :----- |\\n| Cap\'n Proto Packed   | 539ns  | 1216ns    | 2599ns      | 18.92s |\\n| Cap\'n Proto Unpacked | 366ns  | 737ns     | 1583ns      | 12.32s |\\n| Flatbuffers          | 173ns  | 421ns     | 1007ns      | 6.00s  |\\n| SBE                  | 116ns  | 286ns     | 659ns       | 4.05s  |\\n\\n## Conclusion\\n\\nBuilding a benchmark turned out to be incredibly helpful in making a decision; because a \\"union\\"\\ntype isn\'t important to me, I can be confident that SBE best addresses my needs.\\n\\nWhile SBE was the fastest in terms of both median and worst-case performance, its worst case\\nperformance was proportionately far higher than any other format. It seems to be that\\nde/serialization time scales with message size, but I\'ll need to do some more research to understand\\nwhat exactly is going on."},{"id":"2019/06/high-performance-systems","metadata":{"permalink":"/2019/06/high-performance-systems","source":"@site/blog/2019-06-31-high-performance-systems/index.mdx","title":"On building high performance systems","description":"Prior to working in the trading industry, my assumption was that High Frequency Trading (HFT) is","date":"2019-07-01T12:00:00.000Z","tags":[],"readingTime":12.175,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/06/high-performance-systems","title":"On building high performance systems","date":"2019-07-01T12:00:00.000Z","last_updated":{"date":"2019-09-21T12:00:00.000Z"},"authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731207625000,"prevItem":{"title":"Binary format shootout","permalink":"/2019/09/binary-format-shootout"},"nextItem":{"title":"Making bread","permalink":"/2019/05/making-bread"}},"content":"Prior to working in the trading industry, my assumption was that High Frequency Trading (HFT) is\\nmade up of people who have access to secret techniques mortal developers could only dream of. There\\nhad to be some secret art that could only be learned if one had an appropriately tragic backstory.\\n\\n\x3c!-- truncate --\x3e\\n\\n![Kung Fu fight](./kung-fu.webp)\\n\\n> How I assumed HFT people learn their secret techniques\\n\\nHow else do you explain people working on systems that complete the round trip of market data in to\\norders out (a.k.a. tick-to-trade) consistently within\\n[750-800 nanoseconds](https://stackoverflow.com/a/22082528/1454178)? In roughly the time it takes a\\ncomputer to access\\n[main memory 8 times](https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html),\\ntrading systems are capable of reading the market data packets, deciding what orders to send, doing\\nrisk checks, creating new packets for exchange-specific protocols, and putting those packets on the\\nwire.\\n\\nHaving now worked in the trading industry, I can confirm the developers aren\'t super-human; I\'ve\\nmade some simple mistakes at the very least. Instead, what shows up in public discussions is that\\nphilosophy, not technique, separates high-performance systems from everything else.\\nPerformance-critical systems don\'t rely on \\"this one cool C++ optimization trick\\" to make code fast\\n(though micro-optimizations have their place); there\'s a lot more to worry about than just the code\\nwritten for the project.\\n\\nThe framework I\'d propose is this: **If you want to build high-performance systems, focus first on\\nreducing performance variance** (reducing the gap between the fastest and slowest runs of the same\\ncode), **and only look at average latency once variance is at an acceptable level**.\\n\\nDon\'t get me wrong, I\'m a much happier person when things are fast. Computer goes from booting in 20\\nseconds down to 10 because I installed a solid-state drive? Awesome. But if every fifth day it takes\\na full minute to boot because of corrupted sectors? Not so great. Average speed over the course of a\\nweek is the same in each situation, but you\'re painfully aware of that minute when it happens. When\\nit comes to code, the principal is the same: speeding up a function by an average of 10 milliseconds\\ndoesn\'t mean much if there\'s a 100ms difference between your fastest and slowest runs. When\\nperformance matters, you need to respond quickly _every time_, not just in aggregate.\\nHigh-performance systems should first optimize for time variance. Once you\'re consistent at the time\\nscale you care about, then focus on improving average time.\\n\\nThis focus on variance shows up all the time in industry too (emphasis added in all quotes below):\\n\\n- In [marketing materials](https://business.nasdaq.com/market-tech/marketplaces/trading) for\\n  NASDAQ\'s matching engine, the most performance-sensitive component of the exchange, dependability\\n  is highlighted in addition to instantaneous metrics:\\n\\n  > Able to **consistently sustain** an order rate of over 100,000 orders per second at sub-40\\n  > microsecond average latency\\n\\n- The [Aeron](https://github.com/real-logic/aeron) message bus has this to say about performance:\\n\\n  > Performance is the key focus. Aeron is designed to be the highest throughput with the lowest and\\n  > **most predictable latency possible** of any messaging system\\n\\n- The company PolySync, which is working on autonomous vehicles,\\n  [mentions why](https://polysync.io/blog/session-types-for-hearty-codecs/) they picked their\\n  specific messaging format:\\n\\n  > In general, high performance is almost always desirable for serialization. But in the world of\\n  > autonomous vehicles, **steady timing performance is even more important** than peak throughput.\\n  > This is because safe operation is sensitive to timing outliers. Nobody wants the system that\\n  > decides when to slam on the brakes to occasionally take 100 times longer than usual to encode\\n  > its commands.\\n\\n- [Solarflare](https://solarflare.com/), which makes highly-specialized network hardware, points out\\n  variance (jitter) as a big concern for\\n  [electronic trading](https://solarflare.com/electronic-trading/):\\n  > The high stakes world of electronic trading, investment banks, market makers, hedge funds and\\n  > exchanges demand the **lowest possible latency and jitter** while utilizing the highest\\n  > bandwidth and return on their investment.\\n\\nAnd to further clarify: we\'re not discussing _total run-time_, but variance of total run-time. There\\nare situations where it\'s not reasonably possible to make things faster, and you\'d much rather be\\nconsistent. For example, trading firms use\\n[wireless networks](https://sniperinmahwah.wordpress.com/2017/06/07/network-effects-part-i/) because\\nthe speed of light through air is faster than through fiber-optic cables. There\'s still at _absolute\\nminimum_ a [~33.76 millisecond](http://tinyurl.com/y2vd7tn8) delay required to send data between,\\nsay,\\n[Chicago and Tokyo](https://www.theice.com/market-data/connectivity-and-feeds/wireless/tokyo-chicago).\\nIf a trading system in Chicago calls the function for \\"send order to Tokyo\\" and waits to see if a\\ntrade occurs, there\'s a physical limit to how long that will take. In this situation, the focus is\\non keeping variance of _additional processing_ to a minimum, since speed of light is the limiting\\nfactor.\\n\\nSo how does one go about looking for and eliminating performance variance? To tell the truth, I\\ndon\'t think a systematic answer or flow-chart exists. There\'s no substitute for (A) building a deep\\nunderstanding of the entire technology stack, and (B) actually measuring system performance (though\\n(C) watching a lot of [CppCon](https://www.youtube.com/channel/UCMlGfpWw-RUdWX_JbLCukXg) videos for\\ninspiration never hurt). Even then, every project cares about performance to a different degree; you\\nmay need to build an entire\\n[replica production system](https://www.youtube.com/watch?v=NH1Tta7purM&feature=youtu.be&t=3015) to\\naccurately benchmark at nanosecond precision, or you may be content to simply\\n[avoid garbage collection](https://www.youtube.com/watch?v=BD9cRbxWQx8&feature=youtu.be&t=1335) in\\nyour Java code.\\n\\nEven though everyone has different needs, there are still common things to look for when trying to\\nisolate and eliminate variance. In no particular order, these are my focus areas when thinking about\\nhigh-performance systems:\\n\\n**Update 2019-09-21**: Added notes on `isolcpus` and `systemd` affinity.\\n\\n## Language-specific\\n\\n**Garbage Collection**: How often does garbage collection happen? When is it triggered? What are the\\nimpacts?\\n\\n- [In Python](https://rushter.com/blog/python-garbage-collector/), individual objects are collected\\n  if the reference count reaches 0, and each generation is collected if\\n  `num_alloc - num_dealloc > gc_threshold` whenever an allocation happens. The GIL is acquired for\\n  the duration of generational collection.\\n- Java has\\n  [many](https://docs.oracle.com/en/java/javase/12/gctuning/parallel-collector1.html#GUID-DCDD6E46-0406-41D1-AB49-FB96A50EB9CE)\\n  [different](https://docs.oracle.com/en/java/javase/12/gctuning/garbage-first-garbage-collector.html#GUID-ED3AB6D3-FD9B-4447-9EDF-983ED2F7A573)\\n  [collection](https://docs.oracle.com/en/java/javase/12/gctuning/garbage-first-garbage-collector-tuning.html#GUID-90E30ACA-8040-432E-B3A0-1E0440AB556A)\\n  [algorithms](https://docs.oracle.com/en/java/javase/12/gctuning/z-garbage-collector1.html#GUID-A5A42691-095E-47BA-B6DC-FB4E5FAA43D0)\\n  to choose from, each with different characteristics. The default algorithms (Parallel GC in Java\\n  8, G1 in Java 9) freeze the JVM while collecting, while more recent algorithms\\n  ([ZGC](https://wiki.openjdk.java.net/display/zgc) and\\n  [Shenandoah](https://wiki.openjdk.java.net/display/shenandoah)) are designed to keep \\"stop the\\n  world\\" to a minimum by doing collection work in parallel.\\n\\n**Allocation**: Every language has a different way of interacting with \\"heap\\" memory, but the\\nprinciple is the same: running the allocator to allocate/deallocate memory takes time that can often\\nbe put to better use. Understanding when your language interacts with the allocator is crucial, and\\nnot always obvious. For example: C++ and Rust don\'t allocate heap memory for iterators, but Java\\ndoes (meaning potential GC pauses). Take time to understand heap behavior (I made a\\n[a guide for Rust](/2019/02/understanding-allocations-in-rust)), and look into alternative\\nallocators ([jemalloc](http://jemalloc.net/),\\n[tcmalloc](https://gperftools.github.io/gperftools/tcmalloc.html)) that might run faster than the\\noperating system default.\\n\\n**Data Layout**: How your data is arranged in memory matters;\\n[data-oriented design](https://www.youtube.com/watch?v=yy8jQgmhbAU) and\\n[cache locality](https://www.youtube.com/watch?v=2EWejmkKlxs&feature=youtu.be&t=1185) can have huge\\nimpacts on performance. The C family of languages (C, value types in C#, C++) and Rust all have\\nguarantees about the shape every object takes in memory that others (e.g. Java and Python) can\'t\\nmake. [Cachegrind](http://valgrind.org/docs/manual/cg-manual.html) and kernel\\n[perf](https://perf.wiki.kernel.org/index.php/Main_Page) counters are both great for understanding\\nhow performance relates to memory layout.\\n\\n**Just-In-Time Compilation**: Languages that are compiled on the fly (LuaJIT, C#, Java, PyPy) are\\ngreat because they optimize your program for how it\'s actually being used, rather than how a\\ncompiler expects it to be used. However, there\'s a variance problem if the program stops executing\\nwhile waiting for translation from VM bytecode to native code. As a remedy, many languages support\\nahead-of-time compilation in addition to the JIT versions\\n([CoreRT](https://github.com/dotnet/corert) in C# and [GraalVM](https://www.graalvm.org/) in Java).\\nOn the other hand, LLVM supports\\n[Profile Guided Optimization](https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization),\\nwhich theoretically brings JIT benefits to non-JIT languages. Finally, be careful to avoid comparing\\napples and oranges during benchmarks; you don\'t want your code to suddenly speed up because the JIT\\ncompiler kicked in.\\n\\n**Programming Tricks**: These won\'t make or break performance, but can be useful in specific\\ncircumstances. For example, C++ can use\\n[templates instead of branches](https://www.youtube.com/watch?v=NH1Tta7purM&feature=youtu.be&t=1206)\\nin critical sections.\\n\\n## Kernel\\n\\nCode you wrote is almost certainly not the _only_ code running on your hardware. There are many ways\\nthe operating system interacts with your program, from interrupts to system calls, that are\\nimportant to watch for. These are written from a Linux perspective, but Windows does typically have\\nequivalent functionality.\\n\\n**Scheduling**: The kernel is normally free to schedule any process on any core, so it\'s important\\nto reserve CPU cores exclusively for the important programs. There are a few parts to this: first,\\nlimit the CPU cores that non-critical processes are allowed to run on by excluding cores from\\nscheduling\\n([`isolcpus`](https://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/re46.html)\\nkernel command-line option), or by setting the `init` process CPU affinity\\n([`systemd` example](https://access.redhat.com/solutions/2884991)). Second, set critical processes\\nto run on the isolated cores by setting the\\n[processor affinity](https://en.wikipedia.org/wiki/Processor_affinity) using\\n[taskset](https://linux.die.net/man/1/taskset). Finally, use\\n[`NO_HZ`](https://github.com/torvalds/linux/blob/master/Documentation/timers/NO_HZ.txt) or\\n[`chrt`](https://linux.die.net/man/1/chrt) to disable scheduling interrupts. Turning off\\nhyper-threading is also likely beneficial.\\n\\n**System calls**: Reading from a UNIX socket? Writing to a file? In addition to not knowing how long\\nthe I/O operation takes, these all trigger expensive\\n[system calls (syscalls)](https://en.wikipedia.org/wiki/System_call). To handle these, the CPU must\\n[context switch](https://en.wikipedia.org/wiki/Context_switch) to the kernel, let the kernel\\noperation complete, then context switch back to your program. We\'d rather keep these\\n[to a minimum](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript) (see\\ntimestamp 18:20). [Strace](https://linux.die.net/man/1/strace) is your friend for understanding when\\nand where syscalls happen.\\n\\n**Signal Handling**: Far less likely to be an issue, but signals do trigger a context switch if your\\ncode has a handler registered. This will be highly dependent on the application, but you can\\n[block signals](https://www.linuxprogrammingblog.com/all-about-linux-signals?page=show#Blocking_signals)\\nif it\'s an issue.\\n\\n**Interrupts**: System interrupts are how devices connected to your computer notify the CPU that\\nsomething has happened. The CPU will then choose a processor core to pause and context switch to the\\nOS to handle the interrupt. Make sure that\\n[SMP affinity](http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux) is\\nset so that interrupts are handled on a CPU core not running the program you care about.\\n\\n**[NUMA](https://www.kernel.org/doc/html/latest/vm/numa.html)**: While NUMA is good at making\\nmulti-cell systems transparent, there are variance implications; if the kernel moves a process\\nacross nodes, future memory accesses must wait for the controller on the original node. Use\\n[numactl](https://linux.die.net/man/8/numactl) to handle memory-/cpu-cell pinning so this doesn\'t\\nhappen.\\n\\n## Hardware\\n\\n**CPU Pipelining/Speculation**: Speculative execution in modern processors gave us vulnerabilities\\nlike Spectre, but it also gave us performance improvements like\\n[branch prediction](https://stackoverflow.com/a/11227902/1454178). And if the CPU mis-speculates\\nyour code, there\'s variance associated with rewind and replay. While the compiler knows a lot about\\nhow your CPU [pipelines instructions](https://youtu.be/nAbCKa0FzjQ?t=4467), code can be\\n[structured to help](https://www.youtube.com/watch?v=NH1Tta7purM&feature=youtu.be&t=755) the branch\\npredictor.\\n\\n**Paging**: For most systems, virtual memory is incredible. Applications live in their own worlds,\\nand the CPU/[MMU](https://en.wikipedia.org/wiki/Memory_management_unit) figures out the details.\\nHowever, there\'s a variance penalty associated with memory paging and caching; if you access more\\nmemory pages than the [TLB](https://en.wikipedia.org/wiki/Translation_lookaside_buffer) can store,\\nyou\'ll have to wait for the page walk. Kernel perf tools are necessary to figure out if this is an\\nissue, but using [huge pages](https://blog.pythian.com/performance-tuning-hugepages-in-linux/) can\\nreduce TLB burdens. Alternately, running applications in a hypervisor like\\n[Jailhouse](https://github.com/siemens/jailhouse) allows one to skip virtual memory entirely, but\\nthis is probably more work than the benefits are worth.\\n\\n**Network Interfaces**: When more than one computer is involved, variance can go up dramatically.\\nTuning kernel\\n[network parameters](https://github.com/leandromoreira/linux-network-performance-parameters) may be\\nhelpful, but modern systems more frequently opt to skip the kernel altogether with a technique\\ncalled [kernel bypass](https://blog.cloudflare.com/kernel-bypass/). This typically requires\\nspecialized hardware and [drivers](https://www.openonload.org/), but even industries like\\n[telecom](https://www.bbc.co.uk/rd/blog/2018-04-high-speed-networking-open-source-kernel-bypass) are\\nfinding the benefits.\\n\\n## Networks\\n\\n**Routing**: There\'s a reason financial firms are willing to pay\\n[millions of euros](https://sniperinmahwah.wordpress.com/2019/03/26/4-les-moeres-english-version/)\\nfor rights to a small plot of land - having a straight-line connection from point A to point B means\\nthe path their data takes is the shortest possible. In contrast, there are currently 6 computers in\\nbetween me and Google, but that may change at any moment if my ISP realizes a\\n[more efficient route](https://en.wikipedia.org/wiki/Border_Gateway_Protocol) is available. Whether\\nit\'s using\\n[research-quality equipment](https://sniperinmahwah.wordpress.com/2018/05/07/shortwave-trading-part-i-the-west-chicago-tower-mystery/)\\nfor shortwave radio, or just making sure there\'s no data inadvertently going between data centers,\\nrouting matters.\\n\\n**Protocol**: TCP as a network protocol is awesome: guaranteed and in-order delivery, flow control,\\nand congestion control all built in. But these attributes make the most sense when networking\\ninfrastructure is lossy; for systems that expect nearly all packets to be delivered correctly, the\\nsetup handshaking and packet acknowledgment are just overhead. Using UDP (unicast or multicast) may\\nmake sense in these contexts as it avoids the chatter needed to track connection state, and\\n[gap-fill](https://iextrading.com/docs/IEX%20Transport%20Specification.pdf)\\n[strategies](http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf)\\ncan handle the rest.\\n\\n**Switching**: Many routers/switches handle packets using \\"store-and-forward\\" behavior: wait for the\\nwhole packet, validate checksums, and then send to the next device. In variance terms, the time\\nneeded to move data between two nodes is proportional to the size of that data; the switch must\\n\\"store\\" all data before it can calculate checksums and \\"forward\\" to the next node. With\\n[\\"cut-through\\"](https://www.networkworld.com/article/2241573/latency-and-jitter--cut-through-design-pays-off-for-arista--blade.html)\\ndesigns, switches will begin forwarding data as soon as they know where the destination is,\\nchecksums be damned. This means there\'s a fixed cost (at the switch) for network traffic, no matter\\nthe size.\\n\\n## Final Thoughts\\n\\nHigh-performance systems, regardless of industry, are not magical. They do require extreme precision\\nand attention to detail, but they\'re designed, built, and operated by regular people, using a lot of\\ntools that are publicly available. Interested in seeing how context switching affects performance of\\nyour benchmarks? `taskset` should be installed in all modern Linux distributions, and can be used to\\nmake sure the OS never migrates your process. Curious how often garbage collection triggers during a\\ncrucial operation? Your language of choice will typically expose details of its operations\\n([Python](https://docs.python.org/3/library/gc.html),\\n[Java](https://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html#DebuggingOptions)).\\nWant to know how hard your program is stressing the TLB? Use `perf record` and look for\\n`dtlb_load_misses.miss_causes_a_walk`.\\n\\nTwo final guiding questions, then: first, before attempting to apply some of the technology above to\\nyour own systems, can you first identify\\n[where/when you care](http://wiki.c2.com/?PrematureOptimization) about \\"high-performance\\"? As an\\nexample, if parts of a system rely on humans pushing buttons, CPU pinning won\'t have any measurable\\neffect. Humans are already far too slow to react in time. Second, if you\'re using benchmarks, are\\nthey being designed in a way that\'s actually helpful? Tools like\\n[Criterion](http://www.serpentine.com/criterion/) (also in\\n[Rust](https://github.com/bheisler/criterion.rs)) and Google\'s\\n[Benchmark](https://github.com/google/benchmark) output not only average run time, but variance as\\nwell; your benchmarking environment is subject to the same concerns your production environment is.\\n\\nFinally, I believe high-performance systems are a matter of philosophy, not necessarily technique.\\nRigorous focus on variance is the first step, and there are plenty of ways to measure and mitigate\\nit; once that\'s at an acceptable level, then optimize for speed."},{"id":"2019/05/making-bread","metadata":{"permalink":"/2019/05/making-bread","source":"@site/blog/2019-05-03-making-bread/index.mdx","title":"Making bread","description":"Having recently started my \\"gardening leave\\" between positions, I have some more personal time","date":"2019-05-03T12:00:00.000Z","tags":[],"readingTime":1.61,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/05/making-bread","title":"Making bread","date":"2019-05-03T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731207625000,"prevItem":{"title":"On building high performance systems","permalink":"/2019/06/high-performance-systems"},"nextItem":{"title":"Allocations in Rust: Summary","permalink":"/2019/02/summary"}},"content":"Having recently started my \\"gardening leave\\" between positions, I have some more personal time\\navailable. I\'m planning to stay productive, contributing to some open-source projects, but it also\\noccurred to me that despite [talking about](https://speice.io/2018/05/hello.html) bread pics, this\\nblog has been purely technical. Maybe I\'ll change the site title from \\"The Old Speice Guy\\" to \\"Bites\\nand Bytes\\"?\\n\\n\x3c!-- truncate --\x3e\\n\\nEither way, I\'m baking a little bit again, and figured it was worth taking a quick break to focus on\\nsome lighter material. I recently learned two critically important lessons: first, the temperature\\nof the dough when you put the yeast in makes a huge difference.\\n\\nPreviously, when I wasn\'t paying attention to dough temperature:\\n\\n![Whole weat dough](./whole-wheat-not-rising.jpg)\\n\\nCompared with what happens when I put the dough in the microwave for a defrost cycle because the\\nwater I used wasn\'t warm enough:\\n\\n![White dough](./white-dough-rising-before-fold.jpg)\\n\\nI mean, just look at the bubbles!\\n\\n![White dough with bubbles](./white-dough-rising-after-fold.jpg)\\n\\nAfter shaping the dough, I\'ve got two loaves ready:\\n\\n![Shaped loaves](./shaped-loaves.jpg)\\n\\nNow, the recipe normally calls for a Dutch Oven to bake the bread because it keeps the dough from\\ndrying out in the oven. Because I don\'t own a Dutch Oven, I typically put a casserole dish on the\\nbottom rack and fill it with water so there\'s still some moisture in the oven. This time, I forgot\\nto add the water and learned my second lesson: never add room-temperature water to a glass dish\\nthat\'s currently at 500 degrees.\\n\\n![Shattered glass dish](./shattered-glass.jpg)\\n\\nNeedless to say, trying to pull out sharp glass from an incredibly hot oven is not what I expected\\nto be doing during my garden leave.\\n\\nIn the end, the bread crust wasn\'t great, but the bread itself turned out pretty alright:\\n\\n![Baked bread](./final-product.jpg)\\n\\nI\'ve been writing a lot more during this break, so I\'m looking forward to sharing that in the\\nfuture. In the mean-time, I\'m planning on making a sandwich."},{"id":"2019/02/summary","metadata":{"permalink":"/2019/02/summary","source":"@site/blog/2019-02-09-summary/index.mdx","title":"Allocations in Rust: Summary","description":"While there\'s a lot of interesting detail captured in this series, it\'s often helpful to have a","date":"2019-02-09T12:00:00.000Z","tags":[],"readingTime":1.095,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/02/summary","title":"Allocations in Rust: Summary","date":"2019-02-09T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Making bread","permalink":"/2019/05/making-bread"},"nextItem":{"title":"Allocations in Rust: Compiler optimizations","permalink":"/2019/02/08/compiler-optimizations"}},"content":"While there\'s a lot of interesting detail captured in this series, it\'s often helpful to have a\\ndocument that answers some \\"yes/no\\" questions. You may not care about what an `Iterator` looks like\\nin assembly, you just need to know whether it allocates an object on the heap or not. And while Rust\\nwill prioritize the fastest behavior it can, here are the rules for each memory type:\\n\\n\x3c!-- truncate --\x3e\\n\\n**Global Allocation**:\\n\\n- `const` is a fixed value; the compiler is allowed to copy it wherever useful.\\n- `static` is a fixed reference; the compiler will guarantee it is unique.\\n\\n**Stack Allocation**:\\n\\n- Everything not using a smart pointer will be allocated on the stack.\\n- Structs, enums, iterators, arrays, and closures are all stack allocated.\\n- Cell types (`RefCell`) behave like smart pointers, but are stack-allocated.\\n- Inlining (`#[inline]`) will not affect allocation behavior for better or worse.\\n- Types that are marked `Copy` are guaranteed to have their contents stack-allocated.\\n\\n\\n**Heap Allocation**:\\n\\n- Smart pointers (`Box`, `Rc`, `Mutex`, etc.) allocate their contents in heap memory.\\n- Collections (`HashMap`, `Vec`, `String`, etc.) allocate their contents in heap memory.\\n- Some smart pointers in the standard library have counterparts in other crates that don\'t need heap\\n  memory. If possible, use those.\\n\\n![Container Sizes in Rust](./container-size.svg)\\n\\n-- [Raph Levien](https://docs.google.com/presentation/d/1q-c7UAyrUlM-eZyTo1pd8SZ0qwA_wYxmPZVOQkoDmH4/edit?usp=sharing)"},{"id":"/2019/02/08/compiler-optimizations","metadata":{"permalink":"/2019/02/08/compiler-optimizations","source":"@site/blog/2019-02-08-compiler-optimizations/index.mdx","title":"Allocations in Rust: Compiler optimizations","description":"A lot. The answer is a lot.","date":"2019-02-08T12:00:00.000Z","tags":[],"readingTime":3.695,"hasTruncateMarker":true,"authors":[],"frontMatter":{"title":"Allocations in Rust: Compiler optimizations","description":"A lot. The answer is a lot.","date":"2019-02-08T12:00:00.000Z","last_updated":{"date":"2019-02-10T12:00:00.000Z"},"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Allocations in Rust: Summary","permalink":"/2019/02/summary"},"nextItem":{"title":"Allocations in Rust: Dynamic memory","permalink":"/2019/02/a-heaping-helping"}},"content":"Up to this point, we\'ve been discussing memory usage in the Rust language by focusing on simple\\nrules that are mostly right for small chunks of code. We\'ve spent time showing how those rules work\\nthemselves out in practice, and become familiar with reading the assembly code needed to see each\\nmemory type (global, stack, heap) in action.\\n\\nThroughout the series so far, we\'ve put a handicap on the code. In the name of consistent and\\nunderstandable results, we\'ve asked the compiler to pretty please leave the training wheels on. Now\\nis the time where we throw out all the rules and take off the kid gloves. As it turns out, both the\\nRust compiler and the LLVM optimizers are incredibly sophisticated, and we\'ll step back and let them\\ndo their job.\\n\\n\x3c!-- truncate --\x3e\\n\\nSimilar to\\n[\\"What Has My Compiler Done For Me Lately?\\"](https://www.youtube.com/watch?v=bSkpMdDe4g4), we\'re\\nfocusing on interesting things the Rust language (and LLVM!) can do with memory management. We\'ll\\nstill be looking at assembly code to understand what\'s going on, but it\'s important to mention\\nagain: **please use automated tools like [alloc-counter](https://crates.io/crates/alloc_counter) to\\ndouble-check memory behavior if it\'s something you care about**. It\'s far too easy to mis-read\\nassembly in large code sections, you should always verify behavior if you care about memory usage.\\n\\nThe guiding principal as we move forward is this: _optimizing compilers won\'t produce worse programs\\nthan we started with._ There won\'t be any situations where stack allocations get moved to heap\\nallocations. There will, however, be an opera of optimization.\\n\\n**Update 2019-02-10**: When debugging a\\n[related issue](https://gitlab.com/sio4/code/alloc-counter/issues/1), it was discovered that the\\noriginal code worked because LLVM optimized out the entire function, rather than just the allocation\\nsegments. The code has been updated with proper use of\\n[`read_volatile`](https://doc.rust-lang.org/std/ptr/fn.read_volatile.html), and a previous section\\non vector capacity has been removed.\\n\\n## The Case of the Disappearing Box\\n\\nOur first optimization comes when LLVM can reason that the lifetime of an object is sufficiently\\nshort that heap allocations aren\'t necessary. In these cases, LLVM will move the allocation to the\\nstack instead! The way this interacts with `#[inline]` attributes is a bit opaque, but the important\\npart is that LLVM can sometimes do better than the baseline Rust language:\\n\\n```rust\\nuse std::alloc::{GlobalAlloc, Layout, System};\\nuse std::sync::atomic::{AtomicBool, Ordering};\\n\\npub fn cmp(x: u32) {\\n    // Turn on panicking if we allocate on the heap\\n    DO_PANIC.store(true, Ordering::SeqCst);\\n\\n    // The compiler is able to see through the constant `Box`\\n    // and directly compare `x` to 24 - assembly line 73\\n    let y = Box::new(24);\\n    let equals = x == *y;\\n\\n    // This call to drop is eliminated\\n    drop(y);\\n\\n    // Need to mark the comparison result as volatile so that\\n    // LLVM doesn\'t strip out all the code. If `y` is marked\\n    // volatile instead, allocation will be forced.\\n    unsafe { std::ptr::read_volatile(&equals) };\\n\\n    // Turn off panicking, as there are some deallocations\\n    // when we exit main.\\n    DO_PANIC.store(false, Ordering::SeqCst);\\n}\\n\\nfn main() {\\n    cmp(12)\\n}\\n\\n#[global_allocator]\\nstatic A: PanicAllocator = PanicAllocator;\\nstatic DO_PANIC: AtomicBool = AtomicBool::new(false);\\nstruct PanicAllocator;\\n\\nunsafe impl GlobalAlloc for PanicAllocator {\\n    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {\\n        if DO_PANIC.load(Ordering::SeqCst) {\\n            panic!(\\"Unexpected allocation.\\");\\n        }\\n        System.alloc(layout)\\n    }\\n\\n    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {\\n        if DO_PANIC.load(Ordering::SeqCst) {\\n            panic!(\\"Unexpected deallocation.\\");\\n        }\\n        System.dealloc(ptr, layout);\\n    }\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/BZ_Yp3)\\n\\n-- [Rust Playground](https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=4a765f753183d5b919f62c71d2109d5d)\\n\\n## Dr. Array or: how I learned to love the optimizer\\n\\nFinally, this isn\'t so much about LLVM figuring out different memory behavior, but LLVM stripping\\nout code that doesn\'t do anything. Optimizations of this type have a lot of nuance to them; if\\nyou\'re not careful, they can make your benchmarks look\\n[impossibly good](https://www.youtube.com/watch?v=nXaxk27zwlk&feature=youtu.be&t=1199). In Rust, the\\n`black_box` function (implemented in both\\n[`libtest`](https://doc.rust-lang.org/1.1.0/test/fn.black_box.html) and\\n[`criterion`](https://docs.rs/criterion/0.2.10/criterion/fn.black_box.html)) will tell the compiler\\nto disable this kind of optimization. But if you let LLVM remove unnecessary code, you can end up\\nrunning programs that previously caused errors:\\n\\n```rust\\n#[derive(Default)]\\nstruct TwoFiftySix {\\n    _a: [u64; 32]\\n}\\n\\n#[derive(Default)]\\nstruct EightK {\\n    _a: [TwoFiftySix; 32]\\n}\\n\\n#[derive(Default)]\\nstruct TwoFiftySixK {\\n    _a: [EightK; 32]\\n}\\n\\n#[derive(Default)]\\nstruct EightM {\\n    _a: [TwoFiftySixK; 32]\\n}\\n\\npub fn main() {\\n    // Normally this blows up because we can\'t reserve size on stack\\n    // for the `EightM` struct. But because the compiler notices we\\n    // never do anything with `_x`, it optimizes out the stack storage\\n    // and the program completes successfully.\\n    let _x = EightM::default();\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/daHn7P)\\n\\n-- [Rust Playground](https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=4c253bf26072119896ab93c6ef064dc0)"},{"id":"2019/02/a-heaping-helping","metadata":{"permalink":"/2019/02/a-heaping-helping","source":"@site/blog/2019-02-07-a-heaping-helping/index.mdx","title":"Allocations in Rust: Dynamic memory","description":"Managing dynamic memory is hard. Some languages assume users will do it themselves (C, C++), and","date":"2019-02-07T12:00:00.000Z","tags":[],"readingTime":5.86,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/02/a-heaping-helping","title":"Allocations in Rust: Dynamic memory","date":"2019-02-07T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Allocations in Rust: Compiler optimizations","permalink":"/2019/02/08/compiler-optimizations"},"nextItem":{"title":"Allocations in Rust: Fixed memory","permalink":"/2019/02/stacking-up"}},"content":"Managing dynamic memory is hard. Some languages assume users will do it themselves (C, C++), and\\nsome languages go to extreme lengths to protect users from themselves (Java, Python). In Rust, how\\nthe language uses dynamic memory (also referred to as the **heap**) is a system called _ownership_.\\nAnd as the docs mention, ownership\\n[is Rust\'s most unique feature](https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html).\\n\\nThe heap is used in two situations; when the compiler is unable to predict either the _total size of\\nmemory needed_, or _how long the memory is needed for_, it allocates space in the heap.\\n\\n\x3c!-- truncate --\x3e\\n\\nThis happens\\npretty frequently; if you want to download the Google home page, you won\'t know how large it is\\nuntil your program runs. And when you\'re finished with Google, we deallocate the memory so it can be\\nused to store other webpages. If you\'re interested in a slightly longer explanation of the heap,\\ncheck out\\n[The Stack and the Heap](https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html#the-stack-and-the-heap)\\nin Rust\'s documentation.\\n\\nWe won\'t go into detail on how the heap is managed; the\\n[ownership documentation](https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html) does a\\nphenomenal job explaining both the \\"why\\" and \\"how\\" of memory management. Instead, we\'re going to\\nfocus on understanding \\"when\\" heap allocations occur in Rust.\\n\\nTo start off, take a guess for how many allocations happen in the program below:\\n\\n```rust\\nfn main() {}\\n```\\n\\nIt\'s obviously a trick question; while no heap allocations occur as a result of that code, the setup\\nneeded to call `main` does allocate on the heap. Here\'s a way to show it:\\n\\n```rust\\n#![feature(integer_atomics)]\\nuse std::alloc::{GlobalAlloc, Layout, System};\\nuse std::sync::atomic::{AtomicU64, Ordering};\\n\\nstatic ALLOCATION_COUNT: AtomicU64 = AtomicU64::new(0);\\n\\nstruct CountingAllocator;\\n\\nunsafe impl GlobalAlloc for CountingAllocator {\\n    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {\\n        ALLOCATION_COUNT.fetch_add(1, Ordering::SeqCst);\\n        System.alloc(layout)\\n    }\\n\\n    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {\\n        System.dealloc(ptr, layout);\\n    }\\n}\\n\\n#[global_allocator]\\nstatic A: CountingAllocator = CountingAllocator;\\n\\nfn main() {\\n    let x = ALLOCATION_COUNT.fetch_add(0, Ordering::SeqCst);\\n    println!(\\"There were {} allocations before calling main!\\", x);\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=fb5060025ba79fc0f906b65a4ef8eb8e)\\n\\nAs of the time of writing, there are five allocations that happen before `main` is ever called.\\n\\nBut when we want to understand more practically where heap allocation happens, we\'ll follow this\\nguide:\\n\\n- Smart pointers hold their contents in the heap\\n- Collections are smart pointers for many objects at a time, and reallocate when they need to grow\\n\\nFinally, there are two \\"addendum\\" issues that are important to address when discussing Rust and the\\nheap:\\n\\n- Non-heap alternatives to many standard library types are available.\\n- Special allocators to track memory behavior should be used to benchmark code.\\n\\n## Smart pointers\\n\\nThe first thing to note are the \\"smart pointer\\" types. When you have data that must outlive the\\nscope in which it is declared, or your data is of unknown or dynamic size, you\'ll make use of these\\ntypes.\\n\\nThe term [smart pointer](https://en.wikipedia.org/wiki/Smart_pointer) comes from C++, and while it\'s\\nclosely linked to a general design pattern of\\n[\\"Resource Acquisition Is Initialization\\"](https://en.cppreference.com/w/cpp/language/raii), we\'ll\\nuse it here specifically to describe objects that are responsible for managing ownership of data\\nallocated on the heap. The smart pointers available in the `alloc` crate should look mostly\\nfamiliar:\\n\\n- [`Box`](https://doc.rust-lang.org/alloc/boxed/struct.Box.html)\\n- [`Rc`](https://doc.rust-lang.org/alloc/rc/struct.Rc.html)\\n- [`Arc`](https://doc.rust-lang.org/alloc/sync/struct.Arc.html)\\n- [`Cow`](https://doc.rust-lang.org/alloc/borrow/enum.Cow.html)\\n\\nThe [standard library](https://doc.rust-lang.org/std/) also defines some smart pointers to manage\\nheap objects, though more than can be covered here. Some examples are:\\n\\n- [`RwLock`](https://doc.rust-lang.org/std/sync/struct.RwLock.html)\\n- [`Mutex`](https://doc.rust-lang.org/std/sync/struct.Mutex.html)\\n\\nFinally, there is one [\\"gotcha\\"](https://www.merriam-webster.com/dictionary/gotcha): **cell types**\\n(like [`RefCell`](https://doc.rust-lang.org/stable/core/cell/struct.RefCell.html)) look and behave\\nsimilarly, but **don\'t involve heap allocation**. The\\n[`core::cell` docs](https://doc.rust-lang.org/stable/core/cell/index.html) have more information.\\n\\nWhen a smart pointer is created, the data it is given is placed in heap memory and the location of\\nthat data is recorded in the smart pointer. Once the smart pointer has determined it\'s safe to\\ndeallocate that memory (when a `Box` has\\n[gone out of scope](https://doc.rust-lang.org/stable/std/boxed/index.html) or a reference count\\n[goes to zero](https://doc.rust-lang.org/alloc/rc/index.html)), the heap space is reclaimed. We can\\nprove these types use heap memory by looking at code:\\n\\n```rust\\nuse std::rc::Rc;\\nuse std::sync::Arc;\\nuse std::borrow::Cow;\\n\\npub fn my_box() {\\n    // Drop at assembly line 1640\\n    Box::new(0);\\n}\\n\\npub fn my_rc() {\\n    // Drop at assembly line 1650\\n    Rc::new(0);\\n}\\n\\npub fn my_arc() {\\n    // Drop at assembly line 1660\\n    Arc::new(0);\\n}\\n\\npub fn my_cow() {\\n    // Drop at assembly line 1672\\n    Cow::from(\\"drop\\");\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/4AMQug)\\n\\n## Collections\\n\\nCollection types use heap memory because their contents have dynamic size; they will request more\\nmemory [when needed](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.reserve), and can\\n[release memory](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.shrink_to_fit) when it\'s\\nno longer necessary. This dynamic property forces Rust to heap allocate everything they contain. In\\na way, **collections are smart pointers for many objects at a time**. Common types that fall under\\nthis umbrella are [`Vec`](https://doc.rust-lang.org/stable/alloc/vec/struct.Vec.html),\\n[`HashMap`](https://doc.rust-lang.org/stable/std/collections/struct.HashMap.html), and\\n[`String`](https://doc.rust-lang.org/stable/alloc/string/struct.String.html) (not\\n[`str`](https://doc.rust-lang.org/std/primitive.str.html)).\\n\\nWhile collections store the objects they own in heap memory, _creating new collections will not\\nallocate on the heap_. This is a bit weird; if we call `Vec::new()`, the assembly shows a\\ncorresponding call to `real_drop_in_place`:\\n\\n```rust\\npub fn my_vec() {\\n    // Drop in place at line 481\\n    Vec::<u8>::new();\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/1WkNtC)\\n\\nBut because the vector has no elements to manage, no calls to the allocator will ever be dispatched:\\n\\n```rust\\nuse std::alloc::{GlobalAlloc, Layout, System};\\nuse std::sync::atomic::{AtomicBool, Ordering};\\n\\nfn main() {\\n    // Turn on panicking if we allocate on the heap\\n    DO_PANIC.store(true, Ordering::SeqCst);\\n\\n    // Interesting bit happens here\\n    let x: Vec<u8> = Vec::new();\\n    drop(x);\\n\\n    // Turn panicking back off, some deallocations occur\\n    // after main as well.\\n    DO_PANIC.store(false, Ordering::SeqCst);\\n}\\n\\n#[global_allocator]\\nstatic A: PanicAllocator = PanicAllocator;\\nstatic DO_PANIC: AtomicBool = AtomicBool::new(false);\\nstruct PanicAllocator;\\n\\nunsafe impl GlobalAlloc for PanicAllocator {\\n    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {\\n        if DO_PANIC.load(Ordering::SeqCst) {\\n            panic!(\\"Unexpected allocation.\\");\\n        }\\n        System.alloc(layout)\\n    }\\n\\n    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {\\n        if DO_PANIC.load(Ordering::SeqCst) {\\n            panic!(\\"Unexpected deallocation.\\");\\n        }\\n        System.dealloc(ptr, layout);\\n    }\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=831a297d176d015b1f9ace01ae416cc6)\\n\\nOther standard library types follow the same behavior; make sure to check out\\n[`HashMap::new()`](https://doc.rust-lang.org/std/collections/hash_map/struct.HashMap.html#method.new),\\nand [`String::new()`](https://doc.rust-lang.org/std/string/struct.String.html#method.new).\\n\\n## Heap Alternatives\\n\\nWhile it is a bit strange to speak of the stack after spending time with the heap, it\'s worth\\npointing out that some heap-allocated objects in Rust have stack-based counterparts provided by\\nother crates. If you have need of the functionality, but want to avoid allocating, there are\\ntypically alternatives available.\\n\\nWhen it comes to some standard library smart pointers\\n([`RwLock`](https://doc.rust-lang.org/std/sync/struct.RwLock.html) and\\n[`Mutex`](https://doc.rust-lang.org/std/sync/struct.Mutex.html)), stack-based alternatives are\\nprovided in crates like [parking_lot](https://crates.io/crates/parking_lot) and\\n[spin](https://crates.io/crates/spin). You can check out\\n[`lock_api::RwLock`](https://docs.rs/lock_api/0.1.5/lock_api/struct.RwLock.html),\\n[`lock_api::Mutex`](https://docs.rs/lock_api/0.1.5/lock_api/struct.Mutex.html), and\\n[`spin::Once`](https://mvdnes.github.io/rust-docs/spin-rs/spin/struct.Once.html) if you\'re in need\\nof synchronization primitives.\\n\\n[thread_id](https://crates.io/crates/thread-id) may be necessary if you\'re implementing an allocator\\nbecause [`thread::current().id()`](https://doc.rust-lang.org/std/thread/struct.ThreadId.html) uses a\\n[`thread_local!` structure](https://doc.rust-lang.org/stable/src/std/sys_common/thread_info.rs.html#17-36)\\nthat needs heap allocation.\\n\\n## Tracing Allocators\\n\\nWhen writing performance-sensitive code, there\'s no alternative to measuring your code. If you\\ndidn\'t write a benchmark,\\n[you don\'t care about it\'s performance](https://www.youtube.com/watch?v=2EWejmkKlxs&feature=youtu.be&t=263)\\nYou should never rely on your instincts when\\n[a microsecond is an eternity](https://www.youtube.com/watch?v=NH1Tta7purM).\\n\\nSimilarly, there\'s great work going on in Rust with allocators that keep track of what they\'re doing\\n(like [`alloc_counter`](https://crates.io/crates/alloc_counter)). When it comes to tracking heap\\nbehavior, it\'s easy to make mistakes; please write tests and make sure you have tools to guard\\nagainst future issues."},{"id":"2019/02/stacking-up","metadata":{"permalink":"/2019/02/stacking-up","source":"@site/blog/2019-02-06-stacking-up/index.mdx","title":"Allocations in Rust: Fixed memory","description":"const and static are perfectly fine, but it\'s relatively rare that we know at compile-time about","date":"2019-02-06T12:00:00.000Z","tags":[],"readingTime":15.165,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/02/stacking-up","title":"Allocations in Rust: Fixed memory","date":"2019-02-06T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Allocations in Rust: Dynamic memory","permalink":"/2019/02/a-heaping-helping"},"nextItem":{"title":"Allocations in Rust: Global memory","permalink":"/2019/02/the-whole-world"}},"content":"`const` and `static` are perfectly fine, but it\'s relatively rare that we know at compile-time about\\neither values or references that will be the same for the duration of our program. Put another way,\\nit\'s not often the case that either you or your compiler knows how much memory your entire program\\nwill ever need.\\n\\nHowever, there are still some optimizations the compiler can do if it knows how much memory\\nindividual functions will need. Specifically, the compiler can make use of \\"stack\\" memory (as\\nopposed to \\"heap\\" memory) which can be managed far faster in both the short- and long-term.\\n\\n\x3c!-- truncate --\x3e\\n\\nWhen requesting memory, the [`push` instruction](http://www.cs.virginia.edu/~evans/cs216/guides/x86.html)\\ncan typically complete in [1 or 2 cycles](https://agner.org/optimize/instruction_tables.ods) (&lt;1ns\\non modern CPUs). Contrast that to heap memory which requires an allocator (specialized\\nsoftware to track what memory is in use) to reserve space. When you\'re finished with stack memory,\\nthe `pop` instruction runs in 1-3 cycles, as opposed to an allocator needing to worry about memory\\nfragmentation and other issues with the heap. All sorts of incredibly sophisticated techniques have\\nbeen used to design allocators:\\n\\n- [Garbage Collection](<https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)>)\\n  strategies like [Tracing](https://en.wikipedia.org/wiki/Tracing_garbage_collection) (used in\\n  [Java](https://www.oracle.com/technetwork/java/javase/tech/g1-intro-jsp-135488.html)) and\\n  [Reference counting](https://en.wikipedia.org/wiki/Reference_counting) (used in\\n  [Python](https://docs.python.org/3/extending/extending.html#reference-counts))\\n- Thread-local structures to prevent locking the allocator in\\n  [tcmalloc](https://jamesgolick.com/2013/5/19/how-tcmalloc-works.html)\\n- Arena structures used in [jemalloc](http://jemalloc.net/), which\\n  [until recently](https://blog.rust-lang.org/2019/01/17/Rust-1.32.0.html#jemalloc-is-removed-by-default)\\n  was the primary allocator for Rust programs!\\n\\nBut no matter how fast your allocator is, the principle remains: the fastest allocator is the one\\nyou never use. As such, we\'re not going to discuss how exactly the\\n[`push` and `pop` instructions work](http://www.cs.virginia.edu/~evans/cs216/guides/x86.html), but\\nwe\'ll focus instead on the conditions that enable the Rust compiler to use faster stack-based\\nallocation for variables.\\n\\nSo, **how do we know when Rust will or will not use stack allocation for objects we create?**\\nLooking at other languages, it\'s often easy to delineate between stack and heap. Managed memory\\nlanguages (Python, Java,\\n[C#](https://blogs.msdn.microsoft.com/ericlippert/2010/09/30/the-truth-about-value-types/)) place\\neverything on the heap. JIT compilers ([PyPy](https://www.pypy.org/),\\n[HotSpot](https://www.oracle.com/technetwork/java/javase/tech/index-jsp-136373.html)) may optimize\\nsome heap allocations away, but you should never assume it will happen. C makes things clear with\\ncalls to special functions (like [malloc(3)](https://linux.die.net/man/3/malloc)) needed to access\\nheap memory. Old C++ has the [`new`](https://stackoverflow.com/a/655086/1454178) keyword, though\\nmodern C++/C++11 is more complicated with [RAII](https://en.cppreference.com/w/cpp/language/raii).\\n\\nFor Rust, we can summarize as follows: **stack allocation will be used for everything that doesn\'t\\ninvolve \\"smart pointers\\" and collections**. We\'ll skip over a precise definition of the term \\"smart\\npointer\\" for now, and instead discuss what we should watch for to understand when stack and heap\\nmemory regions are used:\\n\\n1. Stack manipulation instructions (`push`, `pop`, and `add`/`sub` of the `rsp` register) indicate\\n   allocation of stack memory:\\n\\n   ```rust\\n   pub fn stack_alloc(x: u32) -> u32 {\\n       // Space for `y` is allocated by subtracting from `rsp`,\\n       // and then populated\\n       let y = [1u8, 2, 3, 4];\\n       // Space for `y` is deallocated by adding back to `rsp`\\n       x\\n   }\\n   ```\\n\\n   -- [Compiler Explorer](https://godbolt.org/z/5WSgc9)\\n\\n2. Tracking when exactly heap allocation calls occur is difficult. It\'s typically easier to watch\\n   for `call core::ptr::real_drop_in_place`, and infer that a heap allocation happened in the recent\\n   past:\\n\\n   ```rust\\n   pub fn heap_alloc(x: usize) -> usize {\\n       // Space for elements in a vector has to be allocated\\n       // on the heap, and is then de-allocated once the\\n       // vector goes out of scope\\n       let y: Vec<u8> = Vec::with_capacity(x);\\n       x\\n   }\\n   ```\\n\\n   -- [Compiler Explorer](https://godbolt.org/z/epfgoQ) (`real_drop_in_place` happens on line 1317)\\n   <small>Note: While the\\n   [`Drop` trait](https://doc.rust-lang.org/std/ops/trait.Drop.html) is\\n   [called for stack-allocated objects](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=87edf374d8983816eb3d8cfeac657b46),\\n   the Rust standard library only defines `Drop` implementations for types that involve heap\\n   allocation.</small>\\n\\n3. If you don\'t want to inspect the assembly, use a custom allocator that\'s able to track and alert\\n   when heap allocations occur. Crates like\\n   [`alloc_counter`](https://crates.io/crates/alloc_counter) are designed for exactly this purpose.\\n\\nWith all that in mind, let\'s talk about situations in which we\'re guaranteed to use stack memory:\\n\\n- Structs are created on the stack.\\n- Function arguments are passed on the stack, meaning the\\n  [`#[inline]` attribute](https://doc.rust-lang.org/reference/attributes.html#inline-attribute) will\\n  not change the memory region used.\\n- Enums and unions are stack-allocated.\\n- [Arrays](https://doc.rust-lang.org/std/primitive.array.html) are always stack-allocated.\\n- Closures capture their arguments on the stack.\\n- Generics will use stack allocation, even with dynamic dispatch.\\n- [`Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html) types are guaranteed to be\\n  stack-allocated, and copying them will be done in stack memory.\\n- [`Iterator`s](https://doc.rust-lang.org/std/iter/trait.Iterator.html) in the standard library are\\n  stack-allocated even when iterating over heap-based collections.\\n\\n## Structs\\n\\nThe simplest case comes first. When creating vanilla `struct` objects, we use stack memory to hold\\ntheir contents:\\n\\n```rust\\nstruct Point {\\n    x: u64,\\n    y: u64,\\n}\\n\\nstruct Line {\\n    a: Point,\\n    b: Point,\\n}\\n\\npub fn make_line() {\\n    // `origin` is stored in the first 16 bytes of memory\\n    // starting at location `rsp`\\n    let origin = Point { x: 0, y: 0 };\\n    // `point` makes up the next 16 bytes of memory\\n    let point = Point { x: 1, y: 2 };\\n\\n    // When creating `ray`, we just move the content out of\\n    // `origin` and `point` into the next 32 bytes of memory\\n    let ray = Line { a: origin, b: point };\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/vri9BE)\\n\\nNote that while some extra-fancy instructions are used for memory manipulation in the assembly, the\\n`sub rsp, 64` instruction indicates we\'re still working with the stack.\\n\\n## Function arguments\\n\\nHave you ever wondered how functions communicate with each other? Like, once the variables are given\\nto you, everything\'s fine. But how do you \\"give\\" those variables to another function? How do you get\\nthe results back afterward? The answer: the compiler arranges memory and assembly instructions using\\na pre-determined [calling convention](http://llvm.org/docs/LangRef.html#calling-conventions). This\\nconvention governs the rules around where arguments needed by a function will be located (either in\\nmemory offsets relative to the stack pointer `rsp`, or in other registers), and where the results\\ncan be found once the function has finished. And when multiple languages agree on what the calling\\nconventions are, you can do things like having [Go call Rust code](https://blog.filippo.io/rustgo/)!\\n\\nPut simply: it\'s the compiler\'s job to figure out how to call other functions, and you can assume\\nthat the compiler is good at its job.\\n\\nWe can see this in action using a simple example:\\n\\n```rust\\nstruct Point {\\n    x: i64,\\n    y: i64,\\n}\\n\\n// We use integer division operations to keep\\n// the assembly clean, understanding the result\\n// isn\'t accurate.\\nfn distance(a: &Point, b: &Point) -> i64 {\\n    // Immediately subtract from `rsp` the bytes needed\\n    // to hold all the intermediate results - this is\\n    // the stack allocation step\\n\\n    // The compiler used the `rdi` and `rsi` registers\\n    // to pass our arguments, so read them in\\n    let x1 = a.x;\\n    let x2 = b.x;\\n    let y1 = a.y;\\n    let y2 = b.y;\\n\\n    // Do the actual math work\\n    let x_pow = (x1 - x2) * (x1 - x2);\\n    let y_pow = (y1 - y2) * (y1 - y2);\\n    let squared = x_pow + y_pow;\\n    squared / squared\\n\\n    // Our final result will be stored in the `rax` register\\n    // so that our caller knows where to retrieve it.\\n    // Finally, add back to `rsp` the stack memory that is\\n    // now ready to be used by other functions.\\n}\\n\\npub fn total_distance() {\\n    let start = Point { x: 1, y: 2 };\\n    let middle = Point { x: 3, y: 4 };\\n    let end = Point { x: 5, y: 6 };\\n\\n    let _dist_1 = distance(&start, &middle);\\n    let _dist_2 = distance(&middle, &end);\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/Qmx4ST)\\n\\nAs a consequence of function arguments never using heap memory, we can also infer that functions\\nusing the `#[inline]` attributes also do not heap allocate. But better than inferring, we can look\\nat the assembly to prove it:\\n\\n```rust\\nstruct Point {\\n    x: i64,\\n    y: i64,\\n}\\n\\n// Note that there is no `distance` function in the assembly output,\\n// and the total line count goes from 229 with inlining off\\n// to 306 with inline on. Even still, no heap allocations occur.\\n#[inline(always)]\\nfn distance(a: &Point, b: &Point) -> i64 {\\n    let x1 = a.x;\\n    let x2 = b.x;\\n    let y1 = a.y;\\n    let y2 = b.y;\\n\\n    let x_pow = (a.x - b.x) * (a.x - b.x);\\n    let y_pow = (a.y - b.y) * (a.y - b.y);\\n    let squared = x_pow + y_pow;\\n    squared / squared\\n}\\n\\npub fn total_distance() {\\n    let start = Point { x: 1, y: 2 };\\n    let middle = Point { x: 3, y: 4 };\\n    let end = Point { x: 5, y: 6 };\\n\\n    let _dist_1 = distance(&start, &middle);\\n    let _dist_2 = distance(&middle, &end);\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/30Sh66)\\n\\nFinally, passing by value (arguments with type\\n[`Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html)) and passing by reference (either\\nmoving ownership or passing a pointer) may have slightly different layouts in assembly, but will\\nstill use either stack memory or CPU registers:\\n\\n```rust\\npub struct Point {\\n    x: i64,\\n    y: i64,\\n}\\n\\n// Moving values\\npub fn distance_moved(a: Point, b: Point) -> i64 {\\n    let x1 = a.x;\\n    let x2 = b.x;\\n    let y1 = a.y;\\n    let y2 = b.y;\\n\\n    let x_pow = (x1 - x2) * (x1 - x2);\\n    let y_pow = (y1 - y2) * (y1 - y2);\\n    let squared = x_pow + y_pow;\\n    squared / squared\\n}\\n\\n// Borrowing values has two extra `mov` instructions on lines 21 and 22\\npub fn distance_borrowed(a: &Point, b: &Point) -> i64 {\\n    let x1 = a.x;\\n    let x2 = b.x;\\n    let y1 = a.y;\\n    let y2 = b.y;\\n\\n    let x_pow = (x1 - x2) * (x1 - x2);\\n    let y_pow = (y1 - y2) * (y1 - y2);\\n    let squared = x_pow + y_pow;\\n    squared / squared\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/06hGiv)\\n\\n## Enums\\n\\nIf you\'ve ever worried that wrapping your types in\\n[`Option`](https://doc.rust-lang.org/stable/core/option/enum.Option.html) or\\n[`Result`](https://doc.rust-lang.org/stable/core/result/enum.Result.html) would finally make them\\nlarge enough that Rust decides to use heap allocation instead, fear no longer: `enum` and union\\ntypes don\'t use heap allocation:\\n\\n```rust\\nenum MyEnum {\\n    Small(u8),\\n    Large(u64)\\n}\\n\\nstruct MyStruct {\\n    x: MyEnum,\\n    y: MyEnum,\\n}\\n\\npub fn enum_compare() {\\n    let x = MyEnum::Small(0);\\n    let y = MyEnum::Large(0);\\n\\n    let z = MyStruct { x, y };\\n\\n    let opt = Option::Some(z);\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/HK7zBx)\\n\\nBecause the size of an `enum` is the size of its largest element plus a flag, the compiler can\\npredict how much memory is used no matter which variant of an enum is currently stored in a\\nvariable. Thus, enums and unions have no need of heap allocation. There\'s unfortunately not a great\\nway to show this in assembly, so I\'ll instead point you to the\\n[`core::mem::size_of`](https://doc.rust-lang.org/stable/core/mem/fn.size_of.html#size-of-enums)\\ndocumentation.\\n\\n## Arrays\\n\\nThe array type is guaranteed to be stack allocated, which is why the array size must be declared.\\nInterestingly enough, this can be used to cause safe Rust programs to crash:\\n\\n```rust\\n// 256 bytes\\n#[derive(Default)]\\nstruct TwoFiftySix {\\n    _a: [u64; 32]\\n}\\n\\n// 8 kilobytes\\n#[derive(Default)]\\nstruct EightK {\\n    _a: [TwoFiftySix; 32]\\n}\\n\\n// 256 kilobytes\\n#[derive(Default)]\\nstruct TwoFiftySixK {\\n    _a: [EightK; 32]\\n}\\n\\n// 8 megabytes - exceeds space typically provided for the stack,\\n// though the kernel can be instructed to allocate more.\\n// On Linux, you can check stack size using `ulimit -s`\\n#[derive(Default)]\\nstruct EightM {\\n    _a: [TwoFiftySixK; 32]\\n}\\n\\nfn main() {\\n    // Because we already have things in stack memory\\n    // (like the current function call stack), allocating another\\n    // eight megabytes of stack memory crashes the program\\n    let _x = EightM::default();\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=587a6380a4914bcbcef4192c90c01dc4)\\n\\nThere aren\'t any security implications of this (no memory corruption occurs), but it\'s good to note\\nthat the Rust compiler won\'t move arrays into heap memory even if they can be reasonably expected to\\noverflow the stack.\\n\\n## Closures\\n\\nRules for how anonymous functions capture their arguments are typically language-specific. In Java,\\n[Lambda Expressions](https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html) are\\nactually objects created on the heap that capture local primitives by copying, and capture local\\nnon-primitives as (`final`) references.\\n[Python](https://docs.python.org/3.7/reference/expressions.html#lambda) and\\n[JavaScript](https://javascriptweblog.wordpress.com/2010/10/25/understanding-javascript-closures/)\\nboth bind _everything_ by reference normally, but Python can also\\n[capture values](https://stackoverflow.com/a/235764/1454178) and JavaScript has\\n[Arrow functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions).\\n\\nIn Rust, arguments to closures are the same as arguments to other functions; closures are simply\\nfunctions that don\'t have a declared name. Some weird ordering of the stack may be required to\\nhandle them, but it\'s the compiler\'s responsiblity to figure that out.\\n\\nEach example below has the same effect, but a different assembly implementation. In the simplest\\ncase, we immediately run a closure returned by another function. Because we don\'t store a reference\\nto the closure, the stack memory needed to store the captured values is contiguous:\\n\\n```rust\\nfn my_func() -> impl FnOnce() {\\n    let x = 24;\\n    // Note that this closure in assembly looks exactly like\\n    // any other function; you even use the `call` instruction\\n    // to start running it.\\n    move || { x; }\\n}\\n\\npub fn immediate() {\\n    my_func()();\\n    my_func()();\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/mgJ2zl), 25 total assembly instructions\\n\\nIf we store a reference to the closure, the Rust compiler keeps values it needs in the stack memory\\nof the original function. Getting the details right is a bit harder, so the instruction count goes\\nup even though this code is functionally equivalent to our original example:\\n\\n```rust\\npub fn simple_reference() {\\n    let x = my_func();\\n    let y = my_func();\\n    y();\\n    x();\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/K_dj5n), 55 total assembly instructions\\n\\nEven things like variable order can make a difference in instruction count:\\n\\n```rust\\npub fn complex() {\\n    let x = my_func();\\n    let y = my_func();\\n    x();\\n    y();\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/p37qFl), 70 total assembly instructions\\n\\nIn every circumstance though, the compiler ensured that no heap allocations were necessary.\\n\\n## Generics\\n\\nTraits in Rust come in two broad forms: static dispatch (monomorphization, `impl Trait`) and dynamic\\ndispatch (trait objects, `dyn Trait`). While dynamic dispatch is often _associated_ with trait\\nobjects being stored in the heap, dynamic dispatch can be used with stack allocated objects as well:\\n\\n```rust\\ntrait GetInt {\\n    fn get_int(&self) -> u64;\\n}\\n\\n// vtable stored at section L__unnamed_1\\nstruct WhyNotU8 {\\n    x: u8\\n}\\nimpl GetInt for WhyNotU8 {\\n    fn get_int(&self) -> u64 {\\n        self.x as u64\\n    }\\n}\\n\\n// vtable stored at section L__unnamed_2\\nstruct ActualU64 {\\n    x: u64\\n}\\nimpl GetInt for ActualU64 {\\n    fn get_int(&self) -> u64 {\\n        self.x\\n    }\\n}\\n\\n// `&dyn` declares that we want to use dynamic dispatch\\n// rather than monomorphization, so there is only one\\n// `retrieve_int` function that shows up in the final assembly.\\n// If we used generics, there would be one implementation of\\n// `retrieve_int` for each type that implements `GetInt`.\\npub fn retrieve_int(u: &dyn GetInt) {\\n    // In the assembly, we just call an address given to us\\n    // in the `rsi` register and hope that it was set up\\n    // correctly when this function was invoked.\\n    let x = u.get_int();\\n}\\n\\npub fn do_call() {\\n    // Note that even though the vtable for `WhyNotU8` and\\n    // `ActualU64` includes a pointer to\\n    // `core::ptr::real_drop_in_place`, it is never invoked.\\n    let a = WhyNotU8 { x: 0 };\\n    let b = ActualU64 { x: 0 };\\n\\n    retrieve_int(&a);\\n    retrieve_int(&b);\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/u_yguS)\\n\\nIt\'s hard to imagine practical situations where dynamic dispatch would be used for objects that\\naren\'t heap allocated, but it technically can be done.\\n\\n## Copy types\\n\\nUnderstanding move semantics and copy semantics in Rust is weird at first. The Rust docs\\n[go into detail](https://doc.rust-lang.org/stable/core/marker/trait.Copy.html) far better than can\\nbe addressed here, so I\'ll leave them to do the job. From a memory perspective though, their\\nguideline is reasonable:\\n[if your type can implemement `Copy`, it should](https://doc.rust-lang.org/stable/core/marker/trait.Copy.html#when-should-my-type-be-copy).\\nWhile there are potential speed tradeoffs to _benchmark_ when discussing `Copy` (move semantics for\\nstack objects vs. copying stack pointers vs. copying stack `struct`s), _it\'s impossible for `Copy`\\nto introduce a heap allocation_.\\n\\nBut why is this the case? Fundamentally, it\'s because the language controls what `Copy` means -\\n[\\"the behavior of `Copy` is not overloadable\\"](https://doc.rust-lang.org/std/marker/trait.Copy.html#whats-the-difference-between-copy-and-clone)\\nbecause it\'s a marker trait. From there we\'ll note that a type\\n[can implement `Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html#when-can-my-type-be-copy)\\nif (and only if) its components implement `Copy`, and that\\n[no heap-allocated types implement `Copy`](https://doc.rust-lang.org/std/marker/trait.Copy.html#implementors).\\nThus, assignments involving heap types are always move semantics, and new heap allocations won\'t\\noccur because of implicit operator behavior.\\n\\n```rust\\n#[derive(Clone)]\\nstruct Cloneable {\\n    x: Box<u64>\\n}\\n\\n// error[E0204]: the trait `Copy` may not be implemented for this type\\n#[derive(Copy, Clone)]\\nstruct NotCopyable {\\n    x: Box<u64>\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/VToRuK)\\n\\n## Iterators\\n\\nIn managed memory languages (like\\n[Java](https://www.youtube.com/watch?v=bSkpMdDe4g4&feature=youtu.be&t=357)), there\'s a subtle\\ndifference between these two code samples:\\n\\n```java\\npublic static int sum_for(List<Long> vals) {\\n    long sum = 0;\\n    // Regular for loop\\n    for (int i = 0; i < vals.length; i++) {\\n        sum += vals[i];\\n    }\\n    return sum;\\n}\\n\\npublic static int sum_foreach(List<Long> vals) {\\n    long sum = 0;\\n    // \\"Foreach\\" loop - uses iteration\\n    for (Long l : vals) {\\n        sum += l;\\n    }\\n    return sum;\\n}\\n```\\n\\nIn the `sum_for` function, nothing terribly interesting happens. In `sum_foreach`, an object of type\\n[`Iterator`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/Iterator.html)\\nis allocated on the heap, and will eventually be garbage-collected. This isn\'t a great design;\\niterators are often transient objects that you need during a function and can discard once the\\nfunction ends. Sounds exactly like the issue stack-allocated objects address, no?\\n\\nIn Rust, iterators are allocated on the stack. The objects to iterate over are almost certainly in\\nheap memory, but the iterator itself\\n([`Iter`](https://doc.rust-lang.org/std/slice/struct.Iter.html)) doesn\'t need to use the heap. In\\neach of the examples below we iterate over a collection, but never use heap allocation:\\n\\n```rust\\nuse std::collections::HashMap;\\n// There\'s a lot of assembly generated, but if you search in the text,\\n// there are no references to `real_drop_in_place` anywhere.\\n\\npub fn sum_vec(x: &Vec<u32>) {\\n    let mut s = 0;\\n    // Basic iteration over vectors doesn\'t need allocation\\n    for y in x {\\n        s += y;\\n    }\\n}\\n\\npub fn sum_enumerate(x: &Vec<u32>) {\\n    let mut s = 0;\\n    // More complex iterators are just fine too\\n    for (_i, y) in x.iter().enumerate() {\\n        s += y;\\n    }\\n}\\n\\npub fn sum_hm(x: &HashMap<u32, u32>) {\\n    let mut s = 0;\\n    // And it\'s not just Vec, all types will allocate the iterator\\n    // on stack memory\\n    for y in x.values() {\\n        s += y;\\n    }\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/FTT3CT)"},{"id":"2019/02/the-whole-world","metadata":{"permalink":"/2019/02/the-whole-world","source":"@site/blog/2019-02-05-the-whole-world/index.mdx","title":"Allocations in Rust: Global memory","description":"The first memory type we\'ll look at is pretty special: when Rust can prove that a value is fixed","date":"2019-02-05T12:00:00.000Z","tags":[],"readingTime":7.485,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/02/the-whole-world","title":"Allocations in Rust: Global memory","date":"2019-02-05T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Allocations in Rust: Fixed memory","permalink":"/2019/02/stacking-up"},"nextItem":{"title":"Allocations in Rust: Foreword","permalink":"/2019/02/understanding-allocations-in-rust"}},"content":"The first memory type we\'ll look at is pretty special: when Rust can prove that a _value_ is fixed\\nfor the life of a program (`const`), and when a _reference_ is unique for the life of a program\\n(`static` as a declaration, not\\n[`\'static`](https://doc.rust-lang.org/book/ch10-03-lifetime-syntax.html#the-static-lifetime) as a\\nlifetime), we can make use of global memory. This special section of data is embedded directly in\\nthe program binary so that variables are ready to go once the program loads; no additional\\ncomputation is necessary.\\n\\nUnderstanding the value/reference distinction is important for reasons we\'ll go into below, and\\nwhile the\\n[full specification](https://github.com/rust-lang/rfcs/blob/master/text/0246-const-vs-static.md) for\\nthese two keywords is available, we\'ll take a hands-on approach to the topic.\\n\\n\x3c!-- truncate --\x3e\\n\\n## `const` values\\n\\nWhen a _value_ is guaranteed to be unchanging in your program (where \\"value\\" may be scalars,\\n`struct`s, etc.), you can declare it `const`. This tells the compiler that it\'s safe to treat the\\nvalue as never changing, and enables some interesting optimizations; not only is there no\\ninitialization cost to creating the value (it is loaded at the same time as the executable parts of\\nyour program), but the compiler can also copy the value around if it speeds up the code.\\n\\nThe points we need to address when talking about `const` are:\\n\\n- `Const` values are stored in read-only memory - it\'s impossible to modify.\\n- Values resulting from calling a `const fn` are materialized at compile-time.\\n- The compiler may (or may not) copy `const` values wherever it chooses.\\n\\n### Read-Only\\n\\nThe first point is a bit strange - \\"read-only memory.\\"\\n[The Rust book](https://doc.rust-lang.org/book/ch03-01-variables-and-mutability.html#differences-between-variables-and-constants)\\nmentions in a couple places that using `mut` with constants is illegal, but it\'s also important to\\ndemonstrate just how immutable they are. _Typically_ in Rust you can use\\n[interior mutability](https://doc.rust-lang.org/book/ch15-05-interior-mutability.html) to modify\\nthings that aren\'t declared `mut`.\\n[`RefCell`](https://doc.rust-lang.org/std/cell/struct.RefCell.html) provides an example of this\\npattern in action:\\n\\n```rust\\nuse std::cell::RefCell;\\n\\nfn my_mutator(cell: &RefCell<u8>) {\\n    // Even though we\'re given an immutable reference,\\n    // the `replace` method allows us to modify the inner value.\\n    cell.replace(14);\\n}\\n\\nfn main() {\\n    let cell = RefCell::new(25);\\n    // Prints out 25\\n    println!(\\"Cell: {:?}\\", cell);\\n    my_mutator(&cell);\\n    // Prints out 14\\n    println!(\\"Cell: {:?}\\", cell);\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=8e4bea1a718edaff4507944e825a54b2)\\n\\nWhen `const` is involved though, interior mutability is impossible:\\n\\n```rust\\nuse std::cell::RefCell;\\n\\nconst CELL: RefCell<u8> = RefCell::new(25);\\n\\nfn my_mutator(cell: &RefCell<u8>) {\\n    cell.replace(14);\\n}\\n\\nfn main() {\\n    // First line prints 25 as expected\\n    println!(\\"Cell: {:?}\\", &CELL);\\n    my_mutator(&CELL);\\n    // Second line *still* prints 25\\n    println!(\\"Cell: {:?}\\", &CELL);\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=88fe98110c33c1b3a51e341f48b8ae00)\\n\\nAnd a second example using [`Once`](https://doc.rust-lang.org/std/sync/struct.Once.html):\\n\\n```rust\\nuse std::sync::Once;\\n\\nconst SURPRISE: Once = Once::new();\\n\\nfn main() {\\n    // This is how `Once` is supposed to be used\\n    SURPRISE.call_once(|| println!(\\"Initializing...\\"));\\n    // Because `Once` is a `const` value, we never record it\\n    // having been initialized the first time, and this closure\\n    // will also execute.\\n    SURPRISE.call_once(|| println!(\\"Initializing again???\\"));\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=c3cc5979b5e5434eca0f9ec4a06ee0ed)\\n\\nWhen the\\n[`const` specification](https://github.com/rust-lang/rfcs/blob/26197104b7bb9a5a35db243d639aee6e46d35d75/text/0246-const-vs-static.md)\\nrefers to [\\"rvalues\\"](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3055.pdf), this\\nbehavior is what they refer to. [Clippy](https://github.com/rust-lang/rust-clippy) will treat this\\nas an error, but it\'s still something to be aware of.\\n\\n### Initialization\\n\\nThe next thing to mention is that `const` values are loaded into memory _as part of your program\\nbinary_. Because of this, any `const` values declared in your program will be \\"realized\\" at\\ncompile-time; accessing them may trigger a main-memory lookup (with a fixed address, so your CPU may\\nbe able to prefetch the value), but that\'s it.\\n\\n```rust\\nuse std::cell::RefCell;\\n\\nconst CELL: RefCell<u32> = RefCell::new(24);\\n\\npub fn multiply(value: u32) -> u32 {\\n    // CELL is stored at `.L__unnamed_1`\\n    value * (*CELL.get_mut())\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/Th8boO)\\n\\nThe compiler creates one `RefCell`, uses it everywhere, and never needs to call the `RefCell::new`\\nfunction.\\n\\n### Copying\\n\\nIf it\'s helpful though, the compiler can choose to copy `const` values.\\n\\n```rust\\nconst FACTOR: u32 = 1000;\\n\\npub fn multiply(value: u32) -> u32 {\\n    // See assembly line 4 for the `mov edi, 1000` instruction\\n    value * FACTOR\\n}\\n\\npub fn multiply_twice(value: u32) -> u32 {\\n    // See assembly lines 22 and 29 for `mov edi, 1000` instructions\\n    value * FACTOR * FACTOR\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/ZtS54X)\\n\\nIn this example, the `FACTOR` value is turned into the `mov edi, 1000` instruction in both the\\n`multiply` and `multiply_twice` functions; the \\"1000\\" value is never \\"stored\\" anywhere, as it\'s\\nsmall enough to inline into the assembly instructions.\\n\\nFinally, getting the address of a `const` value is possible, but not guaranteed to be unique\\n(because the compiler can choose to copy values). I was unable to get non-unique pointers in my\\ntesting (even using different crates), but the specifications are clear enough: _don\'t rely on\\npointers to `const` values being consistent_. To be frank, caring about locations for `const` values\\nis almost certainly a code smell.\\n\\n## `static` values\\n\\nStatic variables are related to `const` variables, but take a slightly different approach. When we\\ndeclare that a _reference_ is unique for the life of a program, you have a `static` variable\\n(unrelated to the `\'static` lifetime). Because of the reference/value distinction with\\n`const`/`static`, static variables behave much more like typical \\"global\\" variables.\\n\\nBut to understand `static`, here\'s what we\'ll look at:\\n\\n- `static` variables are globally unique locations in memory.\\n- Like `const`, `static` variables are loaded at the same time as your program being read into\\n  memory.\\n- All `static` variables must implement the\\n  [`Sync`](https://doc.rust-lang.org/std/marker/trait.Sync.html) marker trait.\\n- Interior mutability is safe and acceptable when using `static` variables.\\n\\n### Memory Uniqueness\\n\\nThe single biggest difference between `const` and `static` is the guarantees provided about\\nuniqueness. Where `const` variables may or may not be copied in code, `static` variables are\\nguarantee to be unique. If we take a previous `const` example and change it to `static`, the\\ndifference should be clear:\\n\\n```rust\\nstatic FACTOR: u32 = 1000;\\n\\npub fn multiply(value: u32) -> u32 {\\n    // The assembly to `mul dword ptr [rip + example::FACTOR]` is how FACTOR gets used\\n    value * FACTOR\\n}\\n\\npub fn multiply_twice(value: u32) -> u32 {\\n    // The assembly to `mul dword ptr [rip + example::FACTOR]` is how FACTOR gets used\\n    value * FACTOR * FACTOR\\n}\\n```\\n\\n-- [Compiler Explorer](https://godbolt.org/z/uxmiRQ)\\n\\nWhere [previously](#copying) there were plenty of references to multiplying by 1000, the new\\nassembly refers to `FACTOR` as a named memory location instead. No initialization work needs to be\\ndone, but the compiler can no longer prove the value never changes during execution.\\n\\n### Initialization\\n\\nNext, let\'s talk about initialization. The simplest case is initializing static variables with\\neither scalar or struct notation:\\n\\n```rust\\n#[derive(Debug)]\\nstruct MyStruct {\\n    x: u32\\n}\\n\\nstatic MY_STRUCT: MyStruct = MyStruct {\\n    // You can even reference other statics\\n    // declared later\\n    x: MY_VAL\\n};\\n\\nstatic MY_VAL: u32 = 24;\\n\\nfn main() {\\n    println!(\\"Static MyStruct: {:?}\\", MY_STRUCT);\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=b538dbc46076f12db047af4f4403ee6e)\\n\\nThings can get a bit weirder when using `const fn` though. In most cases, it just works:\\n\\n```rust\\n#[derive(Debug)]\\nstruct MyStruct {\\n    x: u32\\n}\\n\\nimpl MyStruct {\\n    const fn new() -> MyStruct {\\n        MyStruct { x: 24 }\\n    }\\n}\\n\\nstatic MY_STRUCT: MyStruct = MyStruct::new();\\n\\nfn main() {\\n    println!(\\"const fn Static MyStruct: {:?}\\", MY_STRUCT);\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=8c796a6e7fc273c12115091b707b0255)\\n\\nHowever, there\'s a caveat: you\'re currently not allowed to use `const fn` to initialize static\\nvariables of types that aren\'t marked `Sync`. For example,\\n[`RefCell::new()`](https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.new) is a\\n`const fn`, but because\\n[`RefCell` isn\'t `Sync`](https://doc.rust-lang.org/std/cell/struct.RefCell.html#impl-Sync), you\'ll\\nget an error at compile time:\\n\\n```rust\\nuse std::cell::RefCell;\\n\\n// error[E0277]: `std::cell::RefCell<u8>` cannot be shared between threads safely\\nstatic MY_LOCK: RefCell<u8> = RefCell::new(0);\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=c76ef86e473d07117a1700e21fd45560)\\n\\nIt\'s likely that this will\\n[change in the future](https://github.com/rust-lang/rfcs/blob/master/text/0911-const-fn.md) though.\\n\\n### The `Sync` marker\\n\\nWhich leads well to the next point: static variable types must implement the\\n[`Sync` marker](https://doc.rust-lang.org/std/marker/trait.Sync.html). Because they\'re globally\\nunique, it must be safe for you to access static variables from any thread at any time. Most\\n`struct` definitions automatically implement the `Sync` trait because they contain only elements\\nwhich themselves implement `Sync` (read more in the\\n[Nomicon](https://doc.rust-lang.org/nomicon/send-and-sync.html)). This is why earlier examples could\\nget away with initializing statics, even though we never included an `impl Sync for MyStruct` in the\\ncode. To demonstrate this property, Rust refuses to compile our earlier example if we add a\\nnon-`Sync` element to the `struct` definition:\\n\\n```rust\\nuse std::cell::RefCell;\\n\\nstruct MyStruct {\\n    x: u32,\\n    y: RefCell<u8>,\\n}\\n\\n// error[E0277]: `std::cell::RefCell<u8>` cannot be shared between threads safely\\nstatic MY_STRUCT: MyStruct = MyStruct {\\n    x: 8,\\n    y: RefCell::new(8)\\n};\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=40074d0248f056c296b662dbbff97cfc)\\n\\n### Interior mutability\\n\\nFinally, while `static mut` variables are allowed, mutating them is an `unsafe` operation. If we\\nwant to stay in `safe` Rust, we can use interior mutability to accomplish similar goals:\\n\\n```rust\\nuse std::sync::Once;\\n\\n// This example adapted from https://doc.rust-lang.org/std/sync/struct.Once.html#method.call_once\\nstatic INIT: Once = Once::new();\\n\\nfn main() {\\n    // Note that while `INIT` is declared immutable, we\'re still allowed\\n    // to mutate its interior\\n    INIT.call_once(|| println!(\\"Initializing...\\"));\\n    // This code won\'t panic, as the interior of INIT was modified\\n    // as part of the previous `call_once`\\n    INIT.call_once(|| panic!(\\"INIT was called twice!\\"));\\n}\\n```\\n\\n--\\n[Rust Playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=3ba003a981a7ed7400240caadd384d59)"},{"id":"2019/02/understanding-allocations-in-rust","metadata":{"permalink":"/2019/02/understanding-allocations-in-rust","source":"@site/blog/2019-02-04-understanding-allocations-in-rust/index.mdx","title":"Allocations in Rust: Foreword","description":"There\'s an alchemy of distilling complex technical topics into articles and videos that change the","date":"2019-02-04T12:00:00.000Z","tags":[],"readingTime":3.785,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2019/02/understanding-allocations-in-rust","title":"Allocations in Rust: Foreword","date":"2019-02-04T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Allocations in Rust: Global memory","permalink":"/2019/02/the-whole-world"},"nextItem":{"title":"QADAPT - debug_assert! for allocations","permalink":"/2018/12/allocation-safety"}},"content":"There\'s an alchemy of distilling complex technical topics into articles and videos that change the\\nway programmers see the tools they interact with on a regular basis. I knew what a linker was, but\\nthere\'s a staggering amount of complexity in between\\n[the OS and `main()`](https://www.youtube.com/watch?v=dOfucXtyEsU). Rust programmers use the\\n[`Box`](https://doc.rust-lang.org/stable/std/boxed/struct.Box.html) type all the time, but there\'s a\\nrich history of the Rust language itself wrapped up in\\n[how special it is](https://manishearth.github.io/blog/2017/01/10/rust-tidbits-box-is-special/).\\n\\nIn a similar vein, this series attempts to look at code and understand how memory is used; the\\ncomplex choreography of operating system, compiler, and program that frees you to focus on\\nfunctionality far-flung from frivolous book-keeping. The Rust compiler relieves a great deal of the\\ncognitive burden associated with memory management, but we\'re going to step into its world for a\\nwhile.\\n\\nLet\'s learn a bit about memory in Rust.\\n\\n\x3c!-- truncate --\x3e\\n\\n---\\n\\nRust\'s three defining features of\\n[Performance, Reliability, and Productivity](https://www.rust-lang.org/) are all driven to a great\\ndegree by the how the Rust compiler understands memory usage. Unlike managed memory languages (Java,\\nPython), Rust\\n[doesn\'t really](https://words.steveklabnik.com/borrow-checking-escape-analysis-and-the-generational-hypothesis)\\ngarbage collect; instead, it uses an\\n[ownership](https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html) system to reason about\\nhow long objects will last in your program. In some cases, if the life of an object is fairly\\ntransient, Rust can make use of a very fast region called the \\"stack.\\" When that\'s not possible,\\nRust uses\\n[dynamic (heap) memory](https://en.wikipedia.org/wiki/Memory_management#Dynamic_memory_allocation)\\nand the ownership system to ensure you can\'t accidentally corrupt memory. It\'s not as fast, but it\\nis important to have available.\\n\\nThat said, there are specific situations in Rust where you\'d never need to worry about the\\nstack/heap distinction! If you:\\n\\n1. Never use `unsafe`\\n2. Never use `#![feature(alloc)]` or the [`alloc` crate](https://doc.rust-lang.org/alloc/index.html)\\n\\n...then it\'s not possible for you to use dynamic memory!\\n\\nFor some uses of Rust, typically embedded devices, these constraints are OK. They have very limited\\nmemory, and the program binary size itself may significantly affect what\'s available! There\'s no\\noperating system able to manage this\\n[\\"virtual memory\\"](https://en.wikipedia.org/wiki/Virtual_memory) thing, but that\'s not an issue\\nbecause there\'s only one running application. The\\n[embedonomicon](https://docs.rust-embedded.org/embedonomicon/preface.html) is ever in mind, and\\ninteracting with the \\"real world\\" through extra peripherals is accomplished by reading and writing\\nto [specific memory addresses](https://bob.cs.sonoma.edu/IntroCompOrg-RPi/sec-gpio-mem.html).\\n\\nMost Rust programs find these requirements overly burdensome though. C++ developers would struggle\\nwithout access to [`std::vector`](https://en.cppreference.com/w/cpp/container/vector) (except those\\nhardcore no-STL people), and Rust developers would struggle without\\n[`std::vec`](https://doc.rust-lang.org/std/vec/struct.Vec.html). But with the constraints above,\\n`std::vec` is actually a part of the\\n[`alloc` crate](https://doc.rust-lang.org/alloc/vec/struct.Vec.html), and thus off-limits. `Box`,\\n`Rc`, etc., are also unusable for the same reason.\\n\\nWhether writing code for embedded devices or not, the important thing in both situations is how much\\nyou know _before your application starts_ about what its memory usage will look like. In embedded\\ndevices, there\'s a small, fixed amount of memory to use. In a browser, you have no idea how large\\n[google.com](https://www.google.com)\'s home page is until you start trying to download it. The\\ncompiler uses this knowledge (or lack thereof) to optimize how memory is used; put simply, your code\\nruns faster when the compiler can guarantee exactly how much memory your program needs while it\'s\\nrunning. This series is all about understanding how the compiler reasons about your program, with an\\nemphasis on the implications for performance.\\n\\nNow let\'s address some conditions and caveats before going much further:\\n\\n- We\'ll focus on \\"safe\\" Rust only; `unsafe` lets you use platform-specific allocation API\'s\\n  ([`malloc`](https://www.tutorialspoint.com/c_standard_library/c_function_malloc.htm)) that we\'ll\\n  ignore.\\n- We\'ll assume a \\"debug\\" build of Rust code (what you get with `cargo run` and `cargo test`) and\\n  address (pun intended) release mode at the end (`cargo run --release` and `cargo test --release`).\\n- All content will be run using Rust 1.32, as that\'s the highest currently supported in the\\n  [Compiler Exporer](https://godbolt.org/). As such, we\'ll avoid upcoming innovations like\\n  [compile-time evaluation of `static`](https://github.com/rust-lang/rfcs/blob/master/text/0911-const-fn.md)\\n  that are available in nightly.\\n- Because of the nature of the content, being able to read assembly is helpful. We\'ll keep it\\n  simple, but I [found](https://stackoverflow.com/a/4584131/1454178) a\\n  [refresher](https://stackoverflow.com/a/26026278/1454178) on the `push` and `pop`\\n  [instructions](http://www.cs.virginia.edu/~evans/cs216/guides/x86.html) was helpful while writing\\n  this.\\n- I\'ve tried to be precise in saying only what I can prove using the tools (ASM, docs) that are\\n  available, but if there\'s something said in error it will be corrected expeditiously. Please let\\n  me know at [bradlee@speice.io](mailto:bradlee@speice.io)\\n\\nFinally, I\'ll do what I can to flag potential future changes but the Rust docs have a notice worth\\nrepeating:\\n\\n> Rust does not currently have a rigorously and formally defined memory model.\\n>\\n> -- [the docs](https://doc.rust-lang.org/std/ptr/fn.read_volatile.html)"},{"id":"2018/12/allocation-safety","metadata":{"permalink":"/2018/12/allocation-safety","source":"@site/blog/2018-12-15-allocation-safety/index.mdx","title":"QADAPT - debug_assert! for allocations","description":"I think it\'s part of the human condition to ignore perfectly good advice when it comes our way. A","date":"2018-12-15T12:00:00.000Z","tags":[],"readingTime":4.775,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/12/allocation-safety","title":"QADAPT - debug_assert! for allocations","date":"2018-12-15T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731204300000,"prevItem":{"title":"Allocations in Rust: Foreword","permalink":"/2019/02/understanding-allocations-in-rust"},"nextItem":{"title":"More \\"what companies really mean\\"","permalink":"/2018/12/what-small-business-really-means"}},"content":"I think it\'s part of the human condition to ignore perfectly good advice when it comes our way. A\\nbit over a month ago, I was dispensing sage wisdom for the ages:\\n\\n> I had a really great idea: build a custom allocator that allows you to track your own allocations.\\n> I gave it a shot, but learned very quickly: **never write your own allocator.**\\n>\\n> -- [me](/2018/10/case-study-optimization)\\n\\nI proceeded to ignore it, because we never really learn from our mistakes.\\n\\n\x3c!-- truncate --\x3e\\n\\nThere\'s another part of the human condition that derives joy from seeing things explode.\\n\\n<center>\\n![Explosions](./watch-the-world-burn.webp)\\n</center>\\n\\nAnd _that\'s_ the part I\'m going to focus on.\\n\\n## Why an Allocator?\\n\\nSo why, after complaining about allocators, would I still want to write one? There are three reasons\\nfor that:\\n\\n1. Allocation/dropping is slow\\n2. It\'s difficult to know exactly when Rust will allocate or drop, especially when using code that\\n   you did not write\\n3. I want automated tools to verify behavior, instead of inspecting by hand\\n\\nWhen I say \\"slow,\\" it\'s important to define the terms. If you\'re writing web applications, you\'ll\\nspend orders of magnitude more time waiting for the database than you will the allocator. However,\\nthere\'s still plenty of code where micro- or nano-seconds matter; think\\n[finance](https://www.youtube.com/watch?v=NH1Tta7purM),\\n[real-time audio](https://www.reddit.com/r/rust/comments/9hg7yj/synthesizer_progress_update/e6c291f),\\n[self-driving cars](https://polysync.io/blog/session-types-for-hearty-codecs/), and\\n[networking](https://carllerche.github.io/bytes/bytes/index.html). In these situations it\'s simply\\nunacceptable for you to spend time doing things that are not your program, and waiting on the\\nallocator is not cool.\\n\\nAs I continue to learn Rust, it\'s difficult for me to predict where exactly allocations will happen.\\nSo, I propose we play a quick trivia game: **Does this code invoke the allocator?**\\n\\n### Example 1\\n\\n```rust\\nfn my_function() {\\n    let v: Vec<u8> = Vec::new();\\n}\\n```\\n\\n**No**: Rust [knows how big](https://doc.rust-lang.org/std/mem/fn.size_of.html) the `Vec` type is,\\nand reserves a fixed amount of memory on the stack for the `v` vector. However, if we wanted to\\nreserve extra space (using `Vec::with_capacity`) the allocator would get invoked.\\n\\n### Example 2\\n\\n```rust\\nfn my_function() {\\n    let v: Box<Vec<u8>> = Box::new(Vec::new());\\n}\\n```\\n\\n**Yes**: Because Boxes allow us to work with things that are of unknown size, it has to allocate on\\nthe heap. While the `Box` is unnecessary in this snippet (release builds will optimize out the\\nallocation), reserving heap space more generally is needed to pass a dynamically sized type to\\nanother function.\\n\\n### Example 3\\n\\n```rust\\nfn my_function(v: Vec<u8>) {\\n    v.push(5);\\n}\\n```\\n\\n**Maybe**: Depending on whether the Vector we were given has space available, we may or may not\\nallocate. Especially when dealing with code that you did not author, it\'s difficult to verify that\\nthings behave as you expect them to.\\n\\n## Blowing Things Up\\n\\nSo, how exactly does QADAPT solve these problems? **Whenever an allocation or drop occurs in code\\nmarked allocation-safe, QADAPT triggers a thread panic.** We don\'t want to let the program continue\\nas if nothing strange happened, _we want things to explode_.\\n\\nHowever, you don\'t want code to panic in production because of circumstances you didn\'t predict.\\nJust like [`debug_assert!`](https://doc.rust-lang.org/std/macro.debug_assert.html), **QADAPT will\\nstrip out its own code when building in release mode to guarantee no panics and no performance\\nimpact.**\\n\\nFinally, there are three ways to have QADAPT check that your code will not invoke the allocator:\\n\\n### Using a procedural macro\\n\\nThe easiest method, watch an entire function for allocator invocation:\\n\\n```rust\\nuse qadapt::no_alloc;\\nuse qadapt::QADAPT;\\n\\n#[global_allocator]\\nstatic Q: QADAPT = QADAPT;\\n\\n#[no_alloc]\\nfn push_vec(v: &mut Vec<u8>) {\\n    // This triggers a panic if v.len() == v.capacity()\\n    v.push(5);\\n}\\n\\nfn main() {\\n    let v = Vec::with_capacity(1);\\n\\n    // This will *not* trigger a panic\\n    push_vec(&v);\\n\\n    // This *will* trigger a panic\\n    push_vec(&v);\\n}\\n```\\n\\n### Using a regular macro\\n\\nFor times when you need more precision:\\n\\n```rust\\nuse qadapt::assert_no_alloc;\\nuse qadapt::QADAPT;\\n\\n#[global_allocator]\\nstatic Q: QADAPT = QADAPT;\\n\\nfn main() {\\n    let v = Vec::with_capacity(1);\\n\\n    // No allocations here, we already have space reserved\\n    assert_no_alloc!(v.push(5));\\n\\n    // Even though we remove an item, it doesn\'t trigger a drop\\n    // because it\'s a scalar. If it were a `Box<_>` type,\\n    // a drop would trigger.\\n    assert_no_alloc!({\\n        v.pop().unwrap();\\n    });\\n}\\n```\\n\\n### Using function calls\\n\\nBoth the most precise and most tedious:\\n\\n```rust\\nuse qadapt::enter_protected;\\nuse qadapt::exit_protected;\\nuse qadapt::QADAPT;\\n\\n#[global_allocator]\\nstatic Q: QADAPT = QADAPT;\\n\\nfn main() {\\n    // This triggers an allocation (on non-release builds)\\n    let v = Vec::with_capacity(1);\\n\\n    enter_protected();\\n    // This does not trigger an allocation because we\'ve reserved size\\n    v.push(0);\\n    exit_protected();\\n\\n    // This triggers an allocation because we ran out of size,\\n    // but doesn\'t panic because we\'re no longer protected.\\n    v.push(1);\\n}\\n```\\n\\n### Caveats\\n\\nIt\'s important to point out that QADAPT code is synchronous, so please be careful when mixing in\\nasynchronous functions:\\n\\n```rust\\nuse futures::future::Future;\\nuse futures::future::ok;\\n\\n#[no_alloc]\\nfn async_capacity() -> impl Future<Item=Vec<u8>, Error=()> {\\n    ok(12).and_then(|e| Ok(Vec::with_capacity(e)))\\n}\\n\\nfn main() {\\n    // This doesn\'t trigger a panic because the `and_then` closure\\n    // wasn\'t run during the function call.\\n    async_capacity();\\n\\n    // Still no panic\\n    assert_no_alloc!(async_capacity());\\n\\n    // This will panic because the allocation happens during `unwrap`\\n    // in the `assert_no_alloc!` macro\\n    assert_no_alloc!(async_capacity().poll().unwrap());\\n}\\n```\\n\\n## Conclusion\\n\\nWhile there\'s a lot more to writing high-performance code than managing your usage of the allocator,\\nit\'s critical that you do use the allocator correctly. QADAPT will verify that your code is doing\\nwhat you expect. It\'s usable even on stable Rust from version 1.31 onward, which isn\'t the case for\\nmost allocators. Version 1.0 was released today, and you can check it out over at\\n[crates.io](https://crates.io/crates/qadapt) or on [github](https://github.com/bspeice/qadapt).\\n\\nI\'m hoping to write more about high-performance Rust in the future, and I expect that QADAPT will\\nhelp guide that. If there are topics you\'re interested in, let me know in the comments below!\\n\\n[qadapt]: https://crates.io/crates/qadapt"},{"id":"2018/12/what-small-business-really-means","metadata":{"permalink":"/2018/12/what-small-business-really-means","source":"@site/blog/2018-12-04-what-small-business-really-means/index.mdx","title":"More \\"what companies really mean\\"","description":"I recently stumbled across a phenomenal small article entitled","date":"2018-12-04T12:00:00.000Z","tags":[],"readingTime":1.205,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/12/what-small-business-really-means","title":"More \\"what companies really mean\\"","date":"2018-12-04T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731190109000,"prevItem":{"title":"QADAPT - debug_assert! for allocations","permalink":"/2018/12/allocation-safety"},"nextItem":{"title":"A case study in heaptrack","permalink":"/2018/10/case-study-optimization"}},"content":"I recently stumbled across a phenomenal small article entitled\\n[What Startups Really Mean By \\"Why Should We Hire You?\\"](https://angel.co/blog/what-startups-really-mean-by-why-should-we-hire-you).\\nHaving been interviewed by smaller companies (though not exactly startups), the questions and\\nsubtexts are the same. There\'s often a question behind the question that you\'re actually trying to\\nanswer, and I wish I spotted the nuance earlier in my career.\\n\\nLet me also make note of one more question/euphemism I\'ve come across:\\n\\n\x3c!-- truncate --\x3e\\n\\n## How do you feel about production support?\\n\\n**Translation**: _We\'re a fairly small team, and when things break on an evening/weekend/Christmas\\nDay, can we call on you to be there?_\\n\\nI\'ve met decidedly few people in my life who truly enjoy the \\"ops\\" side of \\"devops\\". They\'re\\nincredibly good at taking an impossible problem, pre-existing knowledge of arcane arts, and turning\\nthat into a functioning system at the end. And if they all left for lunch, we probably wouldn\'t make\\nit out the door before the zombie apocalypse.\\n\\nLarger organizations (in my experience, 500+ person organizations) have the luxury of hiring people\\nwho either enjoy that, or play along nicely enough that our systems keep working.\\n\\nSmall teams have no such luck. If you\'re interviewing at a small company, especially as a \\"data\\nscientist\\" or other somesuch position, be aware that systems can and do spontaneously combust at the\\nmost inopportune moments.\\n\\n**Terrible-but-popular answers include**: _It\'s a part of the job, and I\'m happy to contribute._"},{"id":"2018/10/case-study-optimization","metadata":{"permalink":"/2018/10/case-study-optimization","source":"@site/blog/2018-10-08-case-study-optimization/index.mdx","title":"A case study in heaptrack","description":"I remember early in my career someone joking that:","date":"2018-10-08T12:00:00.000Z","tags":[],"readingTime":4.26,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/10/case-study-optimization","title":"A case study in heaptrack","date":"2018-10-08T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731189722000,"prevItem":{"title":"More \\"what companies really mean\\"","permalink":"/2018/12/what-small-business-really-means"},"nextItem":{"title":"Isomorphic desktop apps with Rust","permalink":"/2018/09/isomorphic-apps"}},"content":"I remember early in my career someone joking that:\\n\\n> Programmers have it too easy these days. They should learn to develop in low memory environments\\n> and be more efficient.\\n\\n...though it\'s not like the first code I wrote was for a\\n[graphing calculator](https://web.archive.org/web/20180924060530/https://education.ti.com/en/products/calculators/graphing-calculators/ti-84-plus-se)\\npacking a whole 24KB of RAM.\\n\\nBut the principle remains: be efficient with the resources you have, because\\n[what Intel giveth, Microsoft taketh away](http://exo-blog.blogspot.com/2007/09/what-intel-giveth-microsoft-taketh-away.html).\\n\\n\x3c!-- truncate --\x3e\\n\\nMy professional work is focused on this kind of efficiency; low-latency financial markets demand\\nthat you understand at a deep level _exactly_ what your code is doing. As I continue experimenting\\nwith Rust for personal projects, it\'s exciting to bring a utilitarian mindset with me: there\'s\\nflexibility for the times I pretend to have a garbage collector, and flexibility for the times that\\nI really care about how memory is used.\\n\\nThis post is a (small) case study in how I went from the former to the latter. And ultimately, it\'s\\nintended to be a starting toolkit to empower analysis of your own code.\\n\\n## Curiosity\\n\\nWhen I first started building the [dtparse] crate, my intention was to mirror as closely as possible\\nthe equivalent [Python library][dateutil]. Python, as you may know, is garbage collected. Very\\nrarely is memory usage considered in Python, and I likewise wasn\'t paying too much attention when\\n`dtparse` was first being built.\\n\\nThis lackadaisical approach to memory works well enough, and I\'m not planning on making `dtparse`\\nhyper-efficient. But every so often, I\'ve wondered: \\"what exactly is going on in memory?\\" With the\\nadvent of Rust 1.28 and the\\n[Global Allocator trait](https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html), I had a really\\ngreat idea: _build a custom allocator that allows you to track your own allocations._ That way, you\\ncan do things like writing tests for both correct results and correct memory usage. I gave it a\\n[shot][qadapt], but learned very quickly: **never write your own allocator**. It went from \\"fun\\nweekend project\\" to \\"I have literally no idea what my computer is doing\\" at breakneck speed.\\n\\nInstead, I\'ll highlight a separate path I took to make sense of my memory usage: [heaptrack].\\n\\n## Turning on the System Allocator\\n\\nThis is the hardest part of the post. Because Rust uses\\n[its own allocator](https://github.com/rust-lang/rust/pull/27400#issue-41256384) by default,\\n`heaptrack` is unable to properly record unmodified Rust code. To remedy this, we\'ll make use of the\\n`#[global_allocator]` attribute.\\n\\nSpecifically, in `lib.rs` or `main.rs`, add this:\\n\\n```rust\\nuse std::alloc::System;\\n\\n#[global_allocator]\\nstatic GLOBAL: System = System;\\n```\\n\\n...and that\'s it. Everything else comes essentially for free.\\n\\n## Running heaptrack\\n\\nAssuming you\'ve installed heaptrack <small>(Homebrew in Mac, package manager\\nin Linux, ??? in Windows)</small>, all that\'s left is to fire up your application:\\n\\n```\\nheaptrack my_application\\n```\\n\\nIt\'s that easy. After the program finishes, you\'ll see a file in your local directory with a name\\nlike `heaptrack.my_appplication.XXXX.gz`. If you load that up in `heaptrack_gui`, you\'ll see\\nsomething like this:\\n\\n![heaptrack](./heaptrack-before.png)\\n\\n---\\n\\nAnd even these pretty colors:\\n\\n![pretty colors](./heaptrack-flamegraph.png)\\n\\n## Reading Flamegraphs\\n\\nTo make sense of our memory usage, we\'re going to focus on that last picture - it\'s called a\\n[\\"flamegraph\\"](http://www.brendangregg.com/flamegraphs.html). These charts are typically used to\\nshow how much time your program spends executing each function, but they\'re used here to show how\\nmuch memory was allocated during those functions instead.\\n\\nFor example, we can see that all executions happened during the `main` function:\\n\\n![allocations in main](./heaptrack-main-colorized.png)\\n\\n...and within that, all allocations happened during `dtparse::parse`:\\n\\n![allocations in dtparse](./heaptrack-dtparse-colorized.png)\\n\\n...and within _that_, allocations happened in two different places:\\n\\n![allocations in parseinfo](./heaptrack-parseinfo-colorized.png)\\n\\nNow I apologize that it\'s hard to see, but there\'s one area specifically that stuck out as an issue:\\n**what the heck is the `Default` thing doing?**\\n\\n![pretty colors](./heaptrack-flamegraph-default.png)\\n\\n## Optimizing dtparse\\n\\nSee, I knew that there were some allocations during calls to `dtparse::parse`, but I was totally\\nwrong about where the bulk of allocations occurred in my program. Let me post the code and see if\\nyou can spot the mistake:\\n\\n```rust\\n/// Main entry point for using `dtparse`.\\npub fn parse(timestr: &str) -> ParseResult<(NaiveDateTime, Option<FixedOffset>)> {\\n    let res = Parser::default().parse(\\n        timestr, None, None, false, false,\\n        None, false,\\n        &HashMap::new(),\\n    )?;\\n\\n    Ok((res.0, res.1))\\n}\\n```\\n\\n> [dtparse](https://github.com/bspeice/dtparse/blob/4d7c5dd99572823fa4a390b483c38ab020a2172f/src/lib.rs#L1286)\\n\\n---\\n\\nBecause `Parser::parse` requires a mutable reference to itself, I have to create a new\\n`Parser::default` every time it receives a string. This is excessive! We\'d rather have an immutable\\nparser that can be re-used, and avoid allocating memory in the first place.\\n\\nArmed with that information, I put some time in to\\n[make the parser immutable](https://github.com/bspeice/dtparse/commit/741afa34517d6bc1155713bbc5d66905fea13fad#diff-b4aea3e418ccdb71239b96952d9cddb6).\\nNow that I can re-use the same parser over and over, the allocations disappear:\\n\\n![allocations cleaned up](./heaptrack-flamegraph-after.png)\\n\\nIn total, we went from requiring 2 MB of memory in\\n[version 1.0.2](https://crates.io/crates/dtparse/1.0.2):\\n\\n![memory before](./heaptrack-closeup.png)\\n\\nAll the way down to 300KB in [version 1.0.3](https://crates.io/crates/dtparse/1.0.3):\\n\\n![memory after](./heaptrack-closeup-after.png)\\n\\n## Conclusion\\n\\nIn the end, you don\'t need to write a custom allocator to be efficient with memory, great tools\\nalready exist to help you understand what your program is doing.\\n\\n**Use them.**\\n\\nGiven that [Moore\'s Law](https://en.wikipedia.org/wiki/Moore%27s_law) is\\n[dead](https://www.technologyreview.com/s/601441/moores-law-is-dead-now-what/), we\'ve all got to do\\nour part to take back what Microsoft stole.\\n\\n[dtparse]: https://crates.io/crates/dtparse\\n[dateutil]: https://github.com/dateutil/dateutil\\n[heaptrack]: https://github.com/KDE/heaptrack\\n[qadapt]: https://crates.io/crates/qadapt"},{"id":"2018/09/isomorphic-apps","metadata":{"permalink":"/2018/09/isomorphic-apps","source":"@site/blog/2018-09-15-isomorphic-apps/index.mdx","title":"Isomorphic desktop apps with Rust","description":"I both despise Javascript and am stunned by its success doing some really cool things. It\'s","date":"2018-09-15T12:00:00.000Z","tags":[],"readingTime":9.905,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/09/isomorphic-apps","title":"Isomorphic desktop apps with Rust","date":"2018-09-15T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731188450000,"prevItem":{"title":"A case study in heaptrack","permalink":"/2018/10/case-study-optimization"},"nextItem":{"title":"Primitives in Rust are weird (and cool)","permalink":"/2018/09/primitives-in-rust-are-weird"}},"content":"I both despise Javascript and am stunned by its success doing some really cool things. It\'s\\n[this duality](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript) that\'s\\nled me to a couple of (very) late nights over the past weeks trying to reconcile myself as I\\nbootstrap a simple desktop application.\\n\\n\x3c!-- truncate --\x3e\\n\\nSee, as much as\\n[Webassembly isn\'t trying to replace Javascript](https://webassembly.org/docs/faq/#is-webassembly-trying-to-replace-javascript),\\n**I want Javascript gone**. There are plenty of people who don\'t share my views, and they are\\nprobably nicer and more fun at parties. But I cringe every time \\"Webpack\\" is mentioned, and I think\\nit\'s hilarious that the\\n[language specification](https://ecma-international.org/publications/standards/Ecma-402.htm)\\ndramatically outpaces anyone\'s\\n[actual implementation](https://kangax.github.io/compat-table/es2016plus/). The answer to this\\nconundrum is of course to recompile code from newer versions of the language to older versions _of\\nthe same language_ before running. At least [Babel] is a nice tongue-in-cheek reference.\\n\\nYet for as much hate as [Electron] receives, it does a stunningly good job at solving a really hard\\nproblem: _how the hell do I put a button on the screen and react when the user clicks it_? GUI\\nprogramming is hard, straight up. But if browsers are already able to run everywhere, why don\'t we\\ntake advantage of someone else solving the hard problems for us? I don\'t like that I have to use\\nJavascript for it, but I really don\'t feel inclined to whip out good ol\' [wxWidgets].\\n\\nNow there are other native solutions ([libui-rs], [conrod], [oh hey wxWdidgets again!][wxrust]), but\\nthose also have their own issues with distribution, styling, etc. With Electron, I can\\n`yarn create electron-app my-app` and just get going, knowing that packaging/upgrades/etc. are built\\nin.\\n\\nMy question is: given recent innovations with WASM, _are we Electron yet_?\\n\\nNo, not really.\\n\\nInstead, **what would it take to get to a point where we can skip Javascript in Electron apps?**\\n\\n# Setting the Stage\\n\\nTruth is, WASM/Webassembly is a pretty new technology and I\'m a total beginner in this area. There\\nmay already be solutions to the issues I discuss, but I\'m totally unaware of them, so I\'m going to\\ntry and organize what I did manage to discover.\\n\\nI should also mention that the content and things I\'m talking about here are not intended to be\\nprescriptive, but more \\"if someone else is interested, what do we already know doesn\'t work?\\" _I\\nexpect everything in this post to be obsolete within two months._ Even over the course of writing\\nthis, [a separate blog post](https://mnt.io/2018/08/28/from-rust-to-beyond-the-asm-js-galaxy/) had\\nto be modified because [upstream changes](https://github.com/WebAssembly/binaryen/pull/1642) broke a\\n[Rust tool](https://github.com/rustwasm/wasm-bindgen/pull/787) the post tried to use. The post\\nultimately\\n[got updated](https://mnt.io/2018/08/28/from-rust-to-beyond-the-asm-js-galaxy/#comment-477), **but\\nall this happened within the span of a week.** Things are moving quickly.\\n\\nI\'ll also note that we\'re going to skip [asm.js] and [emscripten]. Truth be told, I couldn\'t get\\neither of these to output anything, and so I\'m just going to say\\n[here be dragons.](https://en.wikipedia.org/wiki/Here_be_dragons) Everything I\'m discussing here\\nuses the `wasm32-unknown-unknown` target.\\n\\nThe code that I _did_ get running is available\\n[over here](https://github.com/speice-io/isomorphic-rust). Feel free to use it as a starting point,\\nbut I\'m mostly including the link as a reference for the things that were attempted.\\n\\n# An Example Running Application\\n\\nSo, I did _technically_ get a running application:\\n\\n![Electron app using WASM](./electron-percy-wasm.png)\\n\\n...which you can also try out if you want:\\n\\n```sh\\ngit clone https://github.com/speice-io/isomorphic-rust.git\\ncd isomorphic_rust/percy\\nyarn install && yarn start\\n```\\n\\n...but I wouldn\'t really call it a \\"high quality\\" starting point to base future work on. It\'s mostly\\nthere to prove this is possible in the first place. And that\'s something to be proud of! There\'s a\\nhuge amount of engineering that went into showing a window with the text \\"It\'s alive!\\".\\n\\nThere\'s also a lot of usability issues that prevent me from recommending anyone try Electron and\\nWASM apps at the moment, and I think that\'s the more important thing to discuss.\\n\\n# Issue the First: Complicated Toolchains\\n\\nI quickly established that [wasm-bindgen] was necessary to \\"link\\" my Rust code to Javascript. At\\nthat point you\'ve got an Electron app that starts an HTML page which ultimately fetches your WASM\\nblob. To keep things simple, the goal was to package everything using [webpack] so that I could just\\nload a `bundle.js` file on the page. That decision was to be the last thing that kinda worked in\\nthis process.\\n\\nThe first issue\\n[I ran into](https://www.reddit.com/r/rust/comments/98lpun/unable_to_load_wasm_for_electron_application/)\\nwhile attempting to bundle everything via `webpack` is a detail in the WASM spec:\\n\\n> This function accepts a Response object, or a promise for one, and ... **[if > it] does not match\\n> the `application/wasm` MIME type**, the returned promise will be rejected with a TypeError;\\n>\\n> [WebAssembly - Additional Web Embedding API](https://webassembly.org/docs/web/#additional-web-embedding-api)\\n\\nSpecifically, if you try and load a WASM blob without the MIME type set, you\'ll get an error. On the\\nweb this isn\'t a huge issue, as the server can set MIME types when delivering the blob. With\\nElectron, you\'re resolving things with a `file://` URL and thus can\'t control the MIME type:\\n\\n![TypeError: Incorrect response MIME type. Expected \'application/wasm\'.](./incorrect-MIME-type.png)\\n\\nThere are a couple of solutions depending on how far into the deep end you care to venture:\\n\\n- Embed a static file server in your Electron application\\n- Use a [custom protocol](https://electronjs.org/docs/api/protocol) and custom protocol handler\\n- Host your WASM blob on a website that you resolve at runtime\\n\\nBut all these are pretty bad solutions and defeat the purpose of using WASM in the first place.\\nInstead, my workaround was to\\n[open a PR with `webpack`](https://github.com/webpack/webpack/issues/7918) and use regex to remove\\ncalls to `instantiateStreaming` in the\\n[build script](https://github.com/speice-io/isomorphic-rust/blob/master/percy/build.sh#L21-L25):\\n\\n```sh\\ncargo +nightly build --target=wasm32-unknown-unknown && \\\\\\n    wasm-bindgen \\"$WASM_DIR/debug/$WASM_NAME.wasm\\" --out-dir \\"$APP_DIR\\" --no-typescript && \\\\\\n    # Have to use --mode=development so we can patch out the call to instantiateStreaming\\n    \\"$DIR/node_modules/webpack-cli/bin/cli.js\\" --mode=development \\"$APP_DIR/app_loader.js\\" -o \\"$APP_DIR/bundle.js\\" && \\\\\\n    sed -i \'s/.*instantiateStreaming.*//g\' \\"$APP_DIR/bundle.js\\"\\n```\\n\\nOnce that lands, the\\n[build process](https://github.com/speice-io/isomorphic-rust/blob/master/percy_patched_webpack/build.sh#L24-L27)\\nbecomes much simpler:\\n\\n```sh\\n\\ncargo +nightly build --target=wasm32-unknown-unknown && \\\\\\n    wasm-bindgen \\"$WASM_DIR/debug/$WASM_NAME.wasm\\" --out-dir \\"$APP_DIR\\" --no-typescript && \\\\\\n    \\"$DIR/node_modules/webpack-cli/bin/cli.js\\" --mode=production \\"$APP_DIR/app_loader.js\\" -o \\"$APP_DIR/bundle.js\\"\\n```\\n\\nBut we\'re not done yet! After we compile Rust into WASM and link WASM to Javascript (via\\n`wasm-bindgen` and `webpack`), we still have to make an Electron app. For this purpose I used a\\nstarter app from [Electron Forge], and then a\\n[`prestart` script](https://github.com/speice-io/isomorphic-rust/blob/master/percy/package.json#L8)\\nto actually handle starting the application.\\n\\nThe\\n[final toolchain](https://github.com/speice-io/isomorphic-rust/blob/master/percy/package.json#L8)\\nlooks something like this:\\n\\n- `yarn start` triggers the `prestart` script\\n- `prestart` checks for missing tools (`wasm-bindgen-cli`, etc.) and then:\\n  - Uses `cargo` to compile the Rust code into WASM\\n  - Uses `wasm-bindgen` to link the WASM blob into a Javascript file with exported symbols\\n  - Uses `webpack` to bundle the page start script with the Javascript we just generated\\n    - Uses `babel` under the hood to compile the `wasm-bindgen` code down from ES6 into something\\n      browser-compatible\\n- The `start` script runs an Electron Forge handler to do some sanity checks\\n- Electron actually starts\\n\\n...which is complicated. I think more work needs to be done to either build a high-quality starter\\napp that can manage these steps, or another tool that \\"just handles\\" the complexity of linking a\\ncompiled WASM file into something the Electron browser can run.\\n\\n# Issue the Second: WASM tools in Rust\\n\\nFor as much as I didn\'t enjoy the Javascript tooling needed to interface with Rust, the Rust-only\\nbits aren\'t any better at the moment. I get it, a lot of projects are just starting off, and that\\nleads to a fragmented ecosystem. Here\'s what I can recommend as a starting point:\\n\\nDon\'t check in your `Cargo.lock` files to version control. If there\'s a disagreement between the\\nversion of `wasm-bindgen-cli` you have installed and the `wasm-bindgen` you\'re compiling with in\\n`Cargo.lock`, you get a nasty error:\\n\\n```\\nit looks like the Rust project used to create this wasm file was linked against\\na different version of wasm-bindgen than this binary:\\n\\nrust wasm file: 0.2.21\\n    this binary: 0.2.17\\n\\nCurrently the bindgen format is unstable enough that these two version must\\nexactly match, so it\'s required that these two version are kept in sync by\\neither updating the wasm-bindgen dependency or this binary.\\n```\\n\\nNot that I ever managed to run into this myself (_coughs nervously_).\\n\\nThere are two projects attempting to be \\"application frameworks\\": [percy] and [yew]. Between those,\\nI managed to get [two](https://github.com/speice-io/isomorphic-rust/tree/master/percy)\\n[examples](https://github.com/speice-io/isomorphic-rust/tree/master/percy_patched_webpack) running\\nusing `percy`, but was unable to get an\\n[example](https://github.com/speice-io/isomorphic-rust/tree/master/yew) running with `yew` because\\nof issues with \\"missing modules\\" during the `webpack` step:\\n\\n```sh\\nERROR in ./dist/electron_yew_wasm_bg.wasm\\nModule not found: Error: Can\'t resolve \'env\' in \'/home/bspeice/Development/isomorphic_rust/yew/dist\'\\n @ ./dist/electron_yew_wasm_bg.wasm\\n @ ./dist/electron_yew_wasm.js\\n @ ./dist/app.js\\n @ ./dist/app_loader.js\\n```\\n\\nIf you want to work with the browser APIs directly, your choices are [percy-webapis] or [stdweb] (or\\neventually [web-sys]). See above for my `percy` examples, but when I tried\\n[an example with `stdweb`](https://github.com/speice-io/isomorphic-rust/tree/master/stdweb), I was\\nunable to get it running:\\n\\n```sh\\nERROR in ./dist/stdweb_electron_bg.wasm\\nModule not found: Error: Can\'t resolve \'env\' in \'/home/bspeice/Development/isomorphic_rust/stdweb/dist\'\\n @ ./dist/stdweb_electron_bg.wasm\\n @ ./dist/stdweb_electron.js\\n @ ./dist/app_loader.js\\n```\\n\\nAt this point I\'m pretty convinced that `stdweb` is causing issues for `yew` as well, but can\'t\\nprove it.\\n\\nI did also get a [minimal example](https://github.com/speice-io/isomorphic-rust/tree/master/minimal)\\nrunning that doesn\'t depend on any tools besides `wasm-bindgen`. However, it requires manually\\nwriting \\"`extern C`\\" blocks for everything you need from the browser. Es no bueno.\\n\\nFinally, from a tools and platform view, there are two up-and-coming packages that should be\\nmentioned: [js-sys] and [web-sys]. Their purpose is to be fundamental building blocks that exposes\\nthe browser\'s APIs to Rust. If you\'re interested in building an app framework from scratch, these\\nshould give you the most flexibility. I didn\'t touch either in my research, though I expect them to\\nbe essential long-term.\\n\\nSo there\'s a lot in play from the Rust side of things, and it\'s just going to take some time to\\nfigure out what works and what doesn\'t.\\n\\n# Issue the Third: Known Unknowns\\n\\nAlright, so after I managed to get an application started, I stopped there. It was a good deal of\\neffort to chain together even a proof of concept, and at this point I\'d rather learn [Typescript]\\nthan keep trying to maintain an incredibly brittle pipeline. Blasphemy, I know...\\n\\nThe important point I want to make is that there\'s a lot unknown about how any of this holds up\\noutside proofs of concept. Things I didn\'t attempt:\\n\\n- Testing\\n- Packaging\\n- Updates\\n- Literally anything related to why I wanted to use Electron in the first place\\n\\n# What it Would Take\\n\\nMuch as I don\'t like Javascript, the tools are too shaky for me to recommend mixing Electron and\\nWASM at the moment. There\'s a lot of innovation happening, so who knows? Someone might have an\\napplication in production a couple months from now. But at the moment, I\'m personally going to stay\\naway.\\n\\nLet\'s finish with a wishlist then - here are the things that I think need to happen before\\nElectron/WASM/Rust can become a thing:\\n\\n- Webpack still needs some updates. The necessary work is in progress, but hasn\'t landed yet\\n  ([#7983](https://github.com/webpack/webpack/pull/7983))\\n- Browser API libraries (`web-sys` and `stdweb`) need to make sure they can support running in\\n  Electron (see module error above)\\n- Projects need to stabilize. There\'s talk of `stdweb` being turned into a Rust API\\n  [on top of web-sys](https://github.com/rustwasm/team/issues/226#issuecomment-418475778), and percy\\n  [moving to web-sys](https://github.com/chinedufn/percy/issues/24), both of which are big changes\\n- `wasm-bindgen` is great, but still in the \\"move fast and break things\\" phase\\n- A good \\"boilerplate\\" app would dramatically simplify the start-up costs;\\n  [electron-react-boilerplate](https://github.com/chentsulin/electron-react-boilerplate) comes to\\n  mind as a good project to imitate\\n- More blog posts/contributors! I think Electron + Rust could be cool, but I have no idea what I\'m\\n  doing\\n\\n[wxwidgets]: https://wxwidgets.org/\\n[libui-rs]: https://github.com/LeoTindall/libui-rs/\\n[electron]: https://electronjs.org/\\n[babel]: https://babeljs.io/\\n[wxrust]: https://github.com/kenz-gelsoft/wxRust\\n[wasm-bindgen]: https://github.com/rustwasm/wasm-bindgen\\n[js-sys]: https://crates.io/crates/js-sys\\n[percy-webapis]: https://crates.io/crates/percy-webapis\\n[stdweb]: https://crates.io/crates/stdweb\\n[web-sys]: https://crates.io/crates/web-sys\\n[percy]: https://chinedufn.github.io/percy/\\n[virtual-dom-rs]: https://crates.io/crates/virtual-dom-rs\\n[yew]: https://github.com/DenisKolodin/yew\\n[react]: https://reactjs.org/\\n[elm]: http://elm-lang.org/\\n[asm.js]: http://asmjs.org/\\n[emscripten]: https://kripken.github.io/emscripten-site/\\n[typescript]: https://www.typescriptlang.org/\\n[electron forge]: https://electronforge.io/\\n[conrod]: https://github.com/PistonDevelopers/conrod\\n[webpack]: https://webpack.js.org/"},{"id":"2018/09/primitives-in-rust-are-weird","metadata":{"permalink":"/2018/09/primitives-in-rust-are-weird","source":"@site/blog/2018-09-01-primitives-in-rust-are-weird/index.mdx","title":"Primitives in Rust are weird (and cool)","description":"I wrote a really small Rust program a while back because I was curious. I was 100% convinced it","date":"2018-09-01T12:00:00.000Z","tags":[],"readingTime":6.945,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/09/primitives-in-rust-are-weird","title":"Primitives in Rust are weird (and cool)","date":"2018-09-01T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731187596000,"prevItem":{"title":"Isomorphic desktop apps with Rust","permalink":"/2018/09/isomorphic-apps"},"nextItem":{"title":"What I learned porting dateutil to Rust","permalink":"/2018/06/dateutil-parser-to-rust"}},"content":"I wrote a really small Rust program a while back because I was curious. I was 100% convinced it\\ncouldn\'t possibly run:\\n\\n```rust\\nfn main() {\\n    println!(\\"{}\\", 8.to_string())\\n}\\n```\\n\\nAnd to my complete befuddlement, it compiled, ran, and produced a completely sensible output.\\n\\n\x3c!-- truncate --\x3e\\n\\nThe reason I was so surprised has to do with how Rust treats a special category of things I\'m going to\\ncall _primitives_. In the current version of the Rust book, you\'ll see them referred to as\\n[scalars][rust_scalar], and in older versions they\'ll be called [primitives][rust_primitive], but\\nwe\'re going to stick with the name _primitive_ for the time being. Explaining why this program is so\\ncool requires talking about a number of other programming languages, and keeping a consistent\\nterminology makes things easier.\\n\\n**You\'ve been warned:** this is going to be a tedious post about a relatively minor issue that\\ninvolves Java, Python, C, and x86 Assembly. And also me pretending like I know what I\'m talking\\nabout with assembly.\\n\\n## Defining primitives (Java)\\n\\nThe reason I\'m using the name _primitive_ comes from how much of my life is Java right now. For the most part I like Java, but I digress. In Java, there\'s a special\\nname for some specific types of values:\\n\\n> ```\\n> bool    char    byte\\n> short   int     long\\n> float   double\\n> ```\\n\\nThey are referred to as [primitives][java_primitive]. And relative to the other bits of Java,\\nthey have two unique features. First, they don\'t have to worry about the\\n[billion-dollar mistake](https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions);\\nprimitives in Java can never be `null`. Second: *they can\'t have instance methods*.\\nRemember that Rust program from earlier? Java has no idea what to do with it:\\n\\n```java\\nclass Main {\\n    public static void main(String[] args) {\\n        int x = 8;\\n        System.out.println(x.toString()); // Triggers a compiler error\\n    }\\n}\\n````\\n\\nThe error is:\\n\\n```\\nMain.java:5: error: int cannot be dereferenced\\n        System.out.println(x.toString());\\n                            ^\\n1 error\\n```\\n\\nSpecifically, Java\'s [`Object`](https://docs.oracle.com/javase/10/docs/api/java/lang/Object.html)\\nand things that inherit from it are pointers under the hood, and we have to dereference them before\\nthe fields and methods they define can be used. In contrast, _primitive types are just values_ -\\nthere\'s nothing to be dereferenced. In memory, they\'re just a sequence of bits.\\n\\nIf we really want, we can turn the `int` into an\\n[`Integer`](https://docs.oracle.com/javase/10/docs/api/java/lang/Integer.html) and then dereference\\nit, but it\'s a bit wasteful:\\n\\n```java\\nclass Main {\\n    public static void main(String[] args) {\\n        int x = 8;\\n        Integer y = Integer.valueOf(x);\\n        System.out.println(y.toString());\\n    }\\n}\\n```\\n\\nThis creates the variable `y` of type `Integer` (which inherits `Object`), and at run time we\\ndereference `y` to locate the `toString()` function and call it. Rust obviously handles things a bit\\ndifferently, but we have to dig into the low-level details to see it in action.\\n\\n## Low Level Handling of Primitives (C)\\n\\nWe first need to build a foundation for reading and understanding the assembly code the final answer\\nrequires. Let\'s begin with showing how the `C` language (and your computer) thinks about \\"primitive\\"\\nvalues in memory:\\n\\n```c\\nvoid my_function(int num) {}\\n\\nint main() {\\n    int x = 8;\\n    my_function(x);\\n}\\n```\\n\\nThe [compiler explorer](https://godbolt.org/z/lgNYcc) gives us an easy way of showing off the\\nassembly-level code that\'s generated: <small>whose output has been lightly\\nedited</small>\\n\\n```nasm\\nmain:\\n        push    rbp\\n        mov     rbp, rsp\\n        sub     rsp, 16\\n\\n        ; We assign the value `8` to `x` here\\n        mov     DWORD PTR [rbp-4], 8\\n\\n        ; And copy the bits making up `x` to a location\\n        ; `my_function` can access (`edi`)\\n        mov     eax, DWORD PTR [rbp-4]\\n        mov     edi, eax\\n\\n        ; Call `my_function` and give it control\\n        call    my_function\\n\\n        mov     eax, 0\\n        leave\\n        ret\\n\\nmy_function:\\n        push    rbp\\n        mov     rbp, rsp\\n\\n        ; Copy the bits out of the pre-determined location (`edi`)\\n        ; to somewhere we can use\\n        mov     DWORD PTR [rbp-4], edi\\n        nop\\n\\n        pop     rbp\\n        ret\\n```\\n\\nAt a really low level of memory, we\'re copying bits around using the [`mov`][x86_guide] instruction;\\nnothing crazy. But to show how similar Rust is, let\'s take a look at our program translated from C\\nto Rust:\\n\\n```rust\\nfn my_function(x: i32) {}\\n\\nfn main() {\\n    let x = 8;\\n    my_function(x)\\n}\\n```\\n\\nAnd the assembly generated when we stick it in the\\n[compiler explorer](https://godbolt.org/z/cAlmk0): <small>again, lightly\\nedited</small>\\n\\n```nasm\\nexample::main:\\n  push rax\\n\\n  ; Look familiar? We\'re copying bits to a location for `my_function`\\n  ; The compiler just optimizes out holding `x` in memory\\n  mov edi, 8\\n\\n  ; Call `my_function` and give it control\\n  call example::my_function\\n\\n  pop rax\\n  ret\\n\\nexample::my_function:\\n  sub rsp, 4\\n\\n  ; And copying those bits again, just like in C\\n  mov dword ptr [rsp], edi\\n\\n  add rsp, 4\\n  ret\\n```\\n\\nThe generated Rust assembly is functionally pretty close to the C assembly: _When working with\\nprimitives, we\'re just dealing with bits in memory_.\\n\\nIn Java we have to dereference a pointer to call its functions; in Rust, there\'s no pointer to\\ndereference. So what exactly is going on with this `.to_string()` function call?\\n\\n## impl primitive (and Python)\\n\\nNow it\'s time to <strike>reveal my trap card</strike> show the revelation that tied all this\\ntogether: _Rust has implementations for its primitive types._ That\'s right, `impl` blocks aren\'t\\nonly for `structs` and `traits`, primitives get them too. Don\'t believe me? Check out\\n[u32](https://doc.rust-lang.org/std/primitive.u32.html),\\n[f64](https://doc.rust-lang.org/std/primitive.f64.html) and\\n[char](https://doc.rust-lang.org/std/primitive.char.html) as examples.\\n\\nBut the really interesting bit is how Rust turns those `impl` blocks into assembly. Let\'s break out\\nthe [compiler explorer](https://godbolt.org/z/6LBEwq) once again:\\n\\n```rust\\npub fn main() {\\n    8.to_string()\\n}\\n```\\n\\nAnd the interesting bits in the assembly: <small>heavily trimmed down</small>\\n\\n```nasm\\nexample::main:\\n  sub rsp, 24\\n  mov rdi, rsp\\n  lea rax, [rip + .Lbyte_str.u]\\n  mov rsi, rax\\n\\n  ; Cool stuff right here\\n  call <T as alloc::string::ToString>::to_string@PLT\\n\\n  mov rdi, rsp\\n  call core::ptr::drop_in_place\\n  add rsp, 24\\n  ret\\n```\\n\\nNow, this assembly is a bit more complicated, but here\'s the big revelation: **we\'re calling\\n`to_string()` as a function that exists all on its own, and giving it the instance of `8`**. Instead\\nof thinking of the value 8 as an instance of `u32` and then peeking in to find the location of the\\nfunction we want to call (like Java), we have a function that exists outside of the instance and\\njust give that function the value `8`.\\n\\nThis is an incredibly technical detail, but the interesting idea I had was this: _if `to_string()`\\nis a static function, can I refer to the unbound function and give it an instance?_\\n\\nBetter explained in code (and a [compiler explorer](https://godbolt.org/z/fJY-gA) link because I\\nseriously love this thing):\\n\\n```rust\\nstruct MyVal {\\n    x: u32\\n}\\n\\nimpl MyVal {\\n    fn to_string(&self) -> String {\\n        self.x.to_string()\\n    }\\n}\\n\\npub fn main() {\\n    let my_val = MyVal { x: 8 };\\n\\n    // THESE ARE THE SAME\\n    my_val.to_string();\\n    MyVal::to_string(&my_val);\\n}\\n```\\n\\nRust is totally fine \\"binding\\" the function call to the instance, and also as a static.\\n\\nMIND == BLOWN.\\n\\nPython does the same thing where I can both call functions bound to their instances and also call as\\nan unbound function where I give it the instance:\\n\\n```python\\nclass MyClass():\\n    x = 24\\n\\n    def my_function(self):\\n        print(self.x)\\n\\nm = MyClass()\\n\\nm.my_function()\\nMyClass.my_function(m)\\n```\\n\\nAnd Python tries to make you _think_ that primitives can have instance methods...\\n\\n```python\\n>>> dir(8)\\n[\'__abs__\', \'__add__\', \'__and__\', \'__class__\', \'__cmp__\', \'__coerce__\',\\n\'__delattr__\', \'__div__\', \'__divmod__\', \'__doc__\', \'__float__\', \'__floordiv__\',\\n...\\n\'__setattr__\', \'__sizeof__\', \'__str__\', \'__sub__\', \'__subclasshook__\', \'__truediv__\',\\n...]\\n\\n>>> # Theoretically `8.__str__()` should exist, but:\\n\\n>>> 8.__str__()\\n  File \\"<stdin>\\", line 1\\n    8.__str__()\\n             ^\\nSyntaxError: invalid syntax\\n\\n>>> # It will run if we assign it first though:\\n>>> x = 8\\n>>> x.__str__()\\n\'8\'\\n```\\n\\n...but in practice it\'s a bit complicated.\\n\\nSo while Python handles binding instance methods in a way similar to Rust, it\'s still not able to\\nrun the example we started with.\\n\\n## Conclusion\\n\\nThis was a super-roundabout way of demonstrating it, but the way Rust handles incredibly minor\\ndetails like primitives leads to really cool effects. Primitives are optimized like C in how they\\nhave a space-efficient memory layout, yet the language still has a lot of features I enjoy in Python\\n(like both instance and late binding).\\n\\nAnd when you put it together, there are areas where Rust does cool things nobody else can; as a\\nquirky feature of Rust\'s type system, `8.to_string()` is actually valid code.\\n\\nNow go forth and fool your friends into thinking you know assembly. This is all I\'ve got.\\n\\n[x86_guide]: http://www.cs.virginia.edu/~evans/cs216/guides/x86.html\\n[java_primitive]: https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html\\n[rust_scalar]: https://doc.rust-lang.org/book/second-edition/ch03-02-data-types.html#scalar-types\\n[rust_primitive]: https://doc.rust-lang.org/book/first-edition/primitive-types.html"},{"id":"2018/06/dateutil-parser-to-rust","metadata":{"permalink":"/2018/06/dateutil-parser-to-rust","source":"@site/blog/2018-06-25-dateutil-parser-to-rust/index.mdx","title":"What I learned porting dateutil to Rust","description":"I\'ve mostly been a lurker in Rust for a while, making a couple small contributions here and there.","date":"2018-06-25T12:00:00.000Z","tags":[],"readingTime":6.99,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/06/dateutil-parser-to-rust","title":"What I learned porting dateutil to Rust","date":"2018-06-25T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731201811000,"prevItem":{"title":"Primitives in Rust are weird (and cool)","permalink":"/2018/09/primitives-in-rust-are-weird"},"nextItem":{"title":"Hello!","permalink":"/2018/05/hello"}},"content":"I\'ve mostly been a lurker in Rust for a while, making a couple small contributions here and there.\\nSo launching [dtparse](https://github.com/bspeice/dtparse) feels like nice step towards becoming a\\nfunctioning member of society. But not too much, because then you know people start asking you to\\npay bills, and ain\'t nobody got time for that.\\n\\n\x3c!-- truncate --\x3e\\n\\nBut I built dtparse, and you can read about my thoughts on the process. Or don\'t. I won\'t tell you\\nwhat to do with your life (but you should totally keep reading).\\n\\n## Slow down, what?\\n\\nOK, fine, I guess I should start with _why_ someone would do this.\\n\\n[Dateutil](https://github.com/dateutil/dateutil) is a Python library for handling dates. The\\nstandard library support for time in Python is kinda dope, but there are a lot of extras that go\\ninto making it useful beyond just the [datetime](https://docs.python.org/3.6/library/datetime.html)\\nmodule. `dateutil.parser` specifically is code to take all the super-weird time formats people come\\nup with and turn them into something actually useful.\\n\\nDate/time parsing, it turns out, is just like everything else involving\\n[computers](https://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time) and\\n[time](https://infiniteundo.com/post/25509354022/more-falsehoods-programmers-believe-about-time): it\\nfeels like it shouldn\'t be that difficult to do, until you try to do it, and you realize that people\\nsuck and this is why\\n[we can\'t we have nice things](https://zachholman.com/talk/utc-is-enough-for-everyone-right). But\\nalas, we\'ll try and make contemporary art out of the rubble and give it a pretentious name like\\n_Time_.\\n\\n![A gravel mound](./gravel-mound.jpg)\\n\\n> [Time](https://www.goodfreephotos.com/united-states/montana/elkhorn/remains-of-the-mining-operation-elkhorn.jpg.php)\\n\\nWhat makes `dateutil.parser` great is that there\'s single function with a single argument that\\ndrives what programmers interact with:\\n[`parse(timestr)`](https://github.com/dateutil/dateutil/blob/6dde5d6298cfb81a4c594a38439462799ed2aef2/dateutil/parser/_parser.py#L1258).\\nIt takes in the time as a string, and gives you back a reasonable \\"look, this is the best anyone can\\npossibly do to make sense of your input\\" value. It doesn\'t expect much of you.\\n\\n[And now it\'s in Rust.](https://github.com/bspeice/dtparse/blob/7d565d3a78876dbebd9711c9720364fe9eba7915/src/lib.rs#L1332)\\n\\n## Lost in Translation\\n\\nHaving worked at a bulge-bracket bank watching Java programmers try to be Python programmers, I\'m\\nadmittedly hesitant to publish Python code that\'s trying to be Rust. Interestingly, Rust code can\\nactually do a great job of mimicking Python. It\'s certainly not idiomatic Rust, but I\'ve had better\\nexperiences than\\n[this guy](https://webcache.googleusercontent.com/search?q=cache:wkYMpktJtnUJ:https://jackstouffer.com/blog/porting_dateutil.html+&cd=3&hl=en&ct=clnk&gl=us)\\nwho attempted the same thing for D. These are the actual take-aways:\\n\\nWhen transcribing code, **stay as close to the original library as possible**. I\'m talking about\\nusing the same variable names, same access patterns, the whole shebang. It\'s way too easy to make a\\ncouple of typos, and all of a sudden your code blows up in new and exciting ways. Having a reference\\nmanual for verbatim what your code should be means that you don\'t spend that long debugging\\ncomplicated logic, you\'re more looking for typos.\\n\\nAlso, **don\'t use nice Rust things like enums**. While\\n[one time it worked out OK for me](https://github.com/bspeice/dtparse/blob/7d565d3a78876dbebd9711c9720364fe9eba7915/src/lib.rs#L88-L94),\\nI also managed to shoot myself in the foot a couple times because `dateutil` stores AM/PM as a\\nboolean and I mixed up which was true, and which was false (side note: AM is false, PM is true). In\\ngeneral, writing nice code _should not be a first-pass priority_ when you\'re just trying to recreate\\nthe same functionality.\\n\\n**Exceptions are a pain.** Make peace with it. Python code is just allowed to skip stack frames. So\\nwhen a co-worker told me \\"Rust is getting try-catch syntax\\" I properly freaked out. Turns out\\n[he\'s not quite right](https://github.com/rust-lang/rfcs/pull/243), and I\'m OK with that. And while\\n`dateutil` is pretty well-behaved about not skipping multiple stack frames,\\n[130-line try-catch blocks](https://github.com/dateutil/dateutil/blob/16561fc99361979e88cccbd135393b06b1af7e90/dateutil/parser/_parser.py#L730-L865)\\ntake a while to verify.\\n\\nAs another Python quirk, **be very careful about\\n[long nested if-elif-else blocks](https://github.com/dateutil/dateutil/blob/16561fc99361979e88cccbd135393b06b1af7e90/dateutil/parser/_parser.py#L494-L568)**.\\nI used to think that Python\'s whitespace was just there to get you to format your code correctly. I\\nthink that no longer. It\'s way too easy to close a block too early and have incredibly weird issues\\nin the logic. Make sure you use an editor that displays indentation levels so you can keep things\\nstraight.\\n\\n**Rust macros are not free.** I originally had the\\n[main test body](https://github.com/bspeice/dtparse/blob/b0e737f088eca8e83ab4244c6621a2797d247697/tests/compat.rs#L63-L217)\\nwrapped up in a macro using [pyo3](https://github.com/PyO3/PyO3). It took two minutes to compile.\\nAfter\\n[moving things to a function](https://github.com/bspeice/dtparse/blob/e017018295c670e4b6c6ee1cfff00dbb233db47d/tests/compat.rs#L76-L205)\\ncompile times dropped down to ~5 seconds. Turns out 150 lines \\\\* 100 tests = a lot of redundant code\\nto be compiled. My new rule of thumb is that any macros longer than 10-15 lines are actually\\nfunctions that need to be liberated, man.\\n\\nFinally, **I really miss list comprehensions and dictionary comprehensions.** As a quick comparison,\\nsee\\n[this dateutil code](https://github.com/dateutil/dateutil/blob/16561fc99361979e88cccbd135393b06b1af7e90/dateutil/parser/_parser.py#L476)\\nand\\n[the implementation in Rust](https://github.com/bspeice/dtparse/blob/7d565d3a78876dbebd9711c9720364fe9eba7915/src/lib.rs#L619-L629).\\nI probably wrote it wrong, and I\'m sorry. Ultimately though, I hope that these comprehensions can be\\nadded through macros or syntax extensions. Either way, they\'re expressive, save typing, and are\\nsuper-readable. Let\'s get more of that.\\n\\n## Using a young language\\n\\nNow, Rust is exciting and new, which means that there\'s opportunity to make a substantive impact. On\\nmore than one occasion though, I\'ve had issues navigating the Rust ecosystem.\\n\\nWhat I\'ll call the \\"canonical library\\" is still being built. In Python, if you need datetime\\nparsing, you use `dateutil`. If you want `decimal` types, it\'s already in the\\n[standard library](https://docs.python.org/3.6/library/decimal.html). While I might\'ve gotten away\\nwith `f64`, `dateutil` uses decimals, and I wanted to follow the principle of **staying as close to\\nthe original library as possible**. Thus began my quest to find a decimal library in Rust. What I\\nquickly found was summarized in a comment:\\n\\n> Writing a BigDecimal is easy. Writing a _good_ BigDecimal is hard.\\n>\\n> [-cmr](https://github.com/rust-lang/rust/issues/8937#issuecomment-34582794)\\n\\nIn practice, this means that there are at least [4](https://crates.io/crates/bigdecimal)\\n[different](https://crates.io/crates/rust_decimal)\\n[implementations](https://crates.io/crates/decimal) [available](https://crates.io/crates/decimate).\\nAnd that\'s a lot of decisions to worry about when all I\'m thinking is \\"why can\'t\\n[calendar reform](https://en.wikipedia.org/wiki/Calendar_reform) be a thing\\" and I\'m forced to dig\\nthrough a [couple](https://github.com/rust-lang/rust/issues/8937#issuecomment-31661916)\\n[different](https://github.com/rust-lang/rfcs/issues/334)\\n[threads](https://github.com/rust-num/num/issues/8) to figure out if the library I\'m look at is dead\\nor just stable.\\n\\nAnd even when the \\"canonical library\\" exists, there\'s no guarantees that it will be well-maintained.\\n[Chrono](https://github.com/chronotope/chrono) is the _de facto_ date/time library in Rust, and just\\nreleased version 0.4.4 like two days ago. Meanwhile,\\n[chrono-tz](https://github.com/chronotope/chrono-tz) appears to be dead in the water even though\\n[there are people happy to help maintain it](https://github.com/chronotope/chrono-tz/issues/19). I\\nknow relatively little about it, but it appears that most of the release process is automated;\\nkeeping that up to date should be a no-brainer.\\n\\n## Trial Maintenance Policy\\n\\nSpecifically given \\"maintenance\\" being an\\n[oft-discussed](https://www.reddit.com/r/rust/comments/48540g/thoughts_on_initiators_vs_maintainers/)\\nissue, I\'m going to try out the following policy to keep things moving on `dtparse`:\\n\\n1. Issues/PRs needing _maintainer_ feedback will be updated at least weekly. I want to make sure\\n   nobody\'s blocking on me.\\n\\n2. To keep issues/PRs needing _contributor_ feedback moving, I\'m going to (kindly) ask the\\n   contributor to check in after two weeks, and close the issue without resolution if I hear nothing\\n   back after a month.\\n\\nThe second point I think has the potential to be a bit controversial, so I\'m happy to receive\\nfeedback on that. And if a contributor responds with \\"hey, still working on it, had a kid and I\'m\\nrunning on 30 seconds of sleep a night,\\" then first: congratulations on sustaining human life. And\\nsecond: I don\'t mind keeping those requests going indefinitely. I just want to try and balance\\nkeeping things moving with giving people the necessary time they need.\\n\\nI should also note that I\'m still getting some best practices in place - CONTRIBUTING and\\nCONTRIBUTORS files need to be added, as well as issue/PR templates. In progress. None of us are\\nperfect.\\n\\n## Roadmap and Conclusion\\n\\nSo if I\'ve now built a `dateutil`-compatible parser, we\'re done, right? Of course not! That\'s not\\nnearly ambitious enough.\\n\\nUltimately, I\'d love to have a library that\'s capable of parsing everything the Linux `date` command\\ncan do (and not `date` on OSX, because seriously, BSD coreutils are the worst). I know Rust has a\\ncoreutils rewrite going on, and `dtparse` would potentially be an interesting candidate since it\\ndoesn\'t bring in a lot of extra dependencies. [`humantime`](https://crates.io/crates/humantime)\\ncould help pick up some of the (current) slack in dtparse, so maybe we can share and care with each\\nother?\\n\\nAll in all, I\'m mostly hoping that nobody\'s already done this and I haven\'t spent a bit over a month\\non redundant code. So if it exists, tell me. I need to know, but be nice about it, because I\'m going\\nto take it hard.\\n\\nAnd in the mean time, I\'m looking forward to building more. Onwards."},{"id":"2018/05/hello","metadata":{"permalink":"/2018/05/hello","source":"@site/blog/2018-05-28-hello/index.mdx","title":"Hello!","description":"I\'ll do what I can to keep this short, there\'s plenty of other things we both should be doing right","date":"2018-05-28T12:00:00.000Z","tags":[],"readingTime":0.375,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/05/hello","title":"Hello!","date":"2018-05-28T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731187596000,"prevItem":{"title":"What I learned porting dateutil to Rust","permalink":"/2018/06/dateutil-parser-to-rust"},"nextItem":{"title":"Captain\'s Cookbook: Practical usage","permalink":"/2018/01/captains-cookbook-part-2"}},"content":"I\'ll do what I can to keep this short, there\'s plenty of other things we both should be doing right\\nnow.\\n\\n\x3c!-- truncate --\x3e\\n\\nIf you\'re here for the bread pics, and to marvel in some other culinary side projects, I\'ve got you\\ncovered:\\n\\n![Saturday Bread](./bread.jpg)\\n\\nAnd no, I\'m not posting pictures of earlier attempts that ended up turning into rocks in the oven.\\n\\nOkay, just one:\\n\\n![Bread as rock](./rocks.jpg)\\n\\nThanks, and keep it amazing."},{"id":"2018/01/captains-cookbook-part-2","metadata":{"permalink":"/2018/01/captains-cookbook-part-2","source":"@site/blog/2018-01-16-captains-cookbook-part-2/index.mdx","title":"Captain\'s Cookbook: Practical usage","description":"A look at more practical usages of Cap\'N Proto","date":"2018-01-16T13:00:00.000Z","tags":[],"readingTime":6.51,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/01/captains-cookbook-part-2","title":"Captain\'s Cookbook: Practical usage","date":"2018-01-16T13:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731201811000,"prevItem":{"title":"Hello!","permalink":"/2018/05/hello"},"nextItem":{"title":"Captain\'s Cookbook: Project setup","permalink":"/2018/01/captains-cookbook-part-1"}},"content":"A look at more practical usages of Cap\'N Proto\\n\\n\x3c!-- truncate --\x3e\\n\\n[Part 1](/2018/01/captains-cookbook-part-1) of this series took a look at a basic starting project\\nwith Cap\'N Proto. In this section, we\'re going to take the (admittedly basic) schema and look at how we can add a pretty\\nbasic feature - sending Cap\'N Proto messages between threads. It\'s nothing complex, but I want to make sure that there\'s\\nsome documentation surrounding practical usage of the library.\\n\\nAs a quick refresher, we build a Cap\'N Proto message and go through the serialization/deserialization steps\\n[here](https://github.com/bspeice/capnp_cookbook_1/blob/master/src/main.rs). Our current example is going to build on\\nthe code we wrote there; after the deserialization step, we\'ll try and send the `point_reader` to a separate thread\\nfor verification.\\n\\nI\'m going to walk through the attempts as I made them and my thinking throughout.\\nIf you want to skip to the final project, check out the code available [here](https://github.com/bspeice/capnp_cookbook_2)\\n\\n## Attempt 1: Move the reference\\n\\nAs a first attempt, we\'re going to try and let Rust move the reference. Our code will look something like:\\n\\n```rust\\nfn main() {\\n\\n    // ...assume that we own a `buffer: Vec<u8>` containing the binary message content from\\n    // somewhere else\\n\\n    let deserialized = capnp::serialize::read_message(\\n        &mut buffer.as_slice(),\\n        capnp::message::ReaderOptions::new()\\n    ).unwrap();\\n\\n    let point_reader = deserialized.get_root::<point_capnp::point::Reader>().unwrap();\\n\\n    // By using `point_reader` inside the new thread, we\'re hoping that Rust can\\n    // safely move the reference and invalidate the original thread\'s usage.\\n    // Since the original thread doesn\'t use `point_reader` again, this should\\n    // be safe, right?\\n    let handle = std::thread:spawn(move || {\\n\\n        assert_eq!(point_reader.get_x(), 12);\\n\\n        assert_eq!(point_reader.get_y(), 14);\\n    });\\n\\n    handle.join().unwrap()\\n}\\n```\\n\\nWell, the Rust compiler doesn\'t really like this. We get four distinct errors back:\\n\\n```\\nerror[E0277]: the trait bound `*const u8: std::marker::Send` is not satisfied in `[closure@src/main.rs:31:37: 36:6 point_reader:point_capnp::point::Reader<\'_>]`                                                                                                                \\n  --\x3e src/main.rs:31:18                                             \\n   |                                                                \\n31 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `*const u8` cannot be sent between threads safely                                              \\n   |                                                                \\n\\nerror[E0277]: the trait bound `*const capnp::private::layout::WirePointer: std::marker::Send` is not satisfied in `[closure@src/main.rs:31:37: 36:6 point_reader:point_capnp::point::Reader<\'_>]`                                                                               \\n  --\x3e src/main.rs:31:18                                             \\n   |                                                                \\n31 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `*const capnp::private::layout::WirePointer` cannot be sent between threads safely             \\n   |                                                                \\n\\nerror[E0277]: the trait bound `capnp::private::arena::ReaderArena: std::marker::Sync` is not satisfied                                  \\n  --\x3e src/main.rs:31:18                                             \\n   |                                                                \\n31 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `capnp::private::arena::ReaderArena` cannot be shared between threads safely                   \\n   |                                                                \\n\\nerror[E0277]: the trait bound `*const std::vec::Vec<std::option::Option<std::boxed::Box<capnp::private::capability::ClientHook + \'static>>>: std::marker::Send` is not satisfied in `[closure@src/main.rs:31:37: 36:6 point_reader:point_capnp::point::Reader<\'_>]`             \\n  --\x3e src/main.rs:31:18                                             \\n   |                                                                \\n31 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `*const std::vec::Vec<std::option::Option<std::boxed::Box<capnp::private::capability::ClientHook + \'static>>>` cannot be sent between threads safely                                                                                   \\n   |                                                                \\n\\nerror: aborting due to 4 previous errors\\n```\\n\\nNote, I\'ve removed the help text for brevity, but suffice to say that these errors are intimidating.\\nPay attention to the text that keeps on getting repeated though: `XYZ cannot be sent between threads safely`.\\n\\nThis is a bit frustrating: we own the `buffer` from which all the content was derived, and we don\'t have any\\nunsafe accesses in our code. We guarantee that we wait for the child thread to stop first, so there\'s no possibility\\nof the pointer becoming invalid because the original thread exits before the child thread does. So why is Rust\\npreventing us from doing something that really should be legal?\\n\\nThis is what is known as [fighting the borrow checker](https://doc.rust-lang.org/1.8.0/book/references-and-borrowing.html).\\nLet our crusade begin.\\n\\n## Attempt 2: Put the `Reader` in a `Box`\\n\\nThe [`Box`](https://doc.rust-lang.org/std/boxed/struct.Box.html) type allows us to convert a pointer we have\\n(in our case the `point_reader`) into an \\"owned\\" value, which should be easier to send across threads.\\nOur next attempt looks something like this:\\n\\n```rust\\nfn main() {\\n\\n    // ...assume that we own a `buffer: Vec<u8>` containing the binary message content\\n    // from somewhere else\\n\\n    let deserialized = capnp::serialize::read_message(\\n        &mut buffer.as_slice(),\\n        capnp::message::ReaderOptions::new()\\n    ).unwrap();\\n\\n    let point_reader = deserialized.get_root::<point_capnp::point::Reader>().unwrap();\\n\\n    let boxed_reader = Box::new(point_reader);\\n\\n    // Now that the reader is `Box`ed, we\'ve proven ownership, and Rust can\\n    // move the ownership to the new thread, right?\\n    let handle = std::thread::spawn(move || {\\n\\n        assert_eq!(boxed_reader.get_x(), 12);\\n\\n        assert_eq!(boxed_reader.get_y(), 14);\\n    });\\n\\n    handle.join().unwrap();\\n}\\n```\\n\\nSpoiler alert: still doesn\'t work. Same errors still show up.\\n\\n```\\nerror[E0277]: the trait bound `*const u8: std::marker::Send` is not satisfied in `point_capnp::point::Reader<\'_>`                       \\n  --\x3e src/main.rs:33:18                                             \\n   |                                                                \\n33 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `*const u8` cannot be sent between threads safely                                              \\n   |                                                                \\n\\nerror[E0277]: the trait bound `*const capnp::private::layout::WirePointer: std::marker::Send` is not satisfied in `point_capnp::point::Reader<\'_>`                                                                                                                              \\n  --\x3e src/main.rs:33:18                                             \\n   |                                                                \\n33 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `*const capnp::private::layout::WirePointer` cannot be sent between threads safely             \\n   |                                                                \\n\\nerror[E0277]: the trait bound `capnp::private::arena::ReaderArena: std::marker::Sync` is not satisfied                                  \\n  --\x3e src/main.rs:33:18                                             \\n   |                                                                \\n33 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `capnp::private::arena::ReaderArena` cannot be shared between threads safely                   \\n   |                                                                \\n\\nerror[E0277]: the trait bound `*const std::vec::Vec<std::option::Option<std::boxed::Box<capnp::private::capability::ClientHook + \'static>>>: std::marker::Send` is not satisfied in `point_capnp::point::Reader<\'_>`                                                            \\n  --\x3e src/main.rs:33:18                                             \\n   |                                                                \\n33 |     let handle = std::thread::spawn(move || {                  \\n   |                  ^^^^^^^^^^^^^^^^^^ `*const std::vec::Vec<std::option::Option<std::boxed::Box<capnp::private::capability::ClientHook + \'static>>>` cannot be sent between threads safely                                                                                   \\n   |                                                                \\n\\nerror: aborting due to 4 previous errors\\n```\\n\\nLet\'s be a little bit smarter about the exceptions this time though. What is that\\n[`std::marker::Send`](https://doc.rust-lang.org/std/marker/trait.Send.html) thing the compiler keeps telling us about?\\n\\nThe documentation is pretty clear; `Send` is used to denote:\\n\\n> Types that can be transferred across thread boundaries.\\n\\nIn our case, we are seeing the error messages for two reasons:\\n\\n1.  Pointers (`*const u8`) are not safe to send across thread boundaries. While we\'re nice in our code\\nmaking sure that we wait on the child thread to finish before closing down, the Rust compiler can\'t make\\nthat assumption, and so complains that we\'re not using this in a safe manner.\\n\\n2.  The `point_capnp::point::Reader` type is itself not safe to send across threads because it doesn\'t\\nimplement the `Send` trait. Which is to say, the things that make up a `Reader` are themselves not thread-safe,\\nso the `Reader` is also not thread-safe.\\n\\nSo, how are we to actually transfer a parsed Cap\'N Proto message between threads?\\n\\n## Attempt 3: The `TypedReader`\\n\\nThe `TypedReader` is a new API implemented in the Cap\'N Proto [Rust code](https://crates.io/crates/capnp/0.8.14).\\nWe\'re interested in it here for two reasons:\\n\\n1.  It allows us to define an object where the _object_ owns the underlying data. In previous attempts,\\nthe current context owned the data, but the `Reader` itself had no such control.\\n\\n2.  We can compose the `TypedReader` using objects that are safe to `Send` across threads, guaranteeing\\nthat we can transfer parsed messages across threads.\\n\\nThe actual type info for the [`TypedReader`](https://github.com/capnproto/capnproto-rust/blob/f0efc35d7e9bd8f97ca4fdeb7c57fd7ea348e303/src/message.rs#L181)\\nis a bit complex. And to be honest, I\'m still really not sure what the whole point of the\\n[`PhantomData`](https://doc.rust-lang.org/std/marker/struct.PhantomData.html) thing is either.\\nMy impression is that it lets us enforce type safety when we know what the underlying Cap\'N Proto\\nmessage represents. That is, technically the only thing we\'re storing is the untyped binary message;\\n`PhantomData` just enforces the principle that the binary represents some specific object that has been parsed.\\n\\nEither way, we can carefully construct something which is safe to move between threads:\\n\\n```rust\\nfn main() {\\n\\n    // ...assume that we own a `buffer: Vec<u8>` containing the binary message content from somewhere else\\n\\n    let deserialized = capnp::serialize::read_message(\\n        &mut buffer.as_slice(),\\n        capnp::message::ReaderOptions::new()\\n    ).unwrap();\\n\\n    let point_reader: capnp::message::TypedReader<capnp::serialize::OwnedSegments, point_capnp::point::Owned> =\\n        capnp::message::TypedReader::new(deserialized);\\n\\n    // Because the point_reader is now working with OwnedSegments (which are owned vectors) and an Owned message\\n    // (which is \'static lifetime), this is now safe\\n    let handle = std::thread::spawn(move || {\\n\\n        // The point_reader owns its data, and we use .get() to retrieve the actual point_capnp::point::Reader\\n        // object from it\\n        let point_root = point_reader.get().unwrap();\\n\\n        assert_eq!(point_root.get_x(), 12);\\n\\n        assert_eq!(point_root.get_y(), 14);\\n    });\\n\\n    handle.join().unwrap();\\n}\\n```\\n\\nAnd while we\'ve left Rust to do the dirty work of actually moving the `point_reader` into the new thread,\\nwe could also use things like [`mpsc` channels](https://doc.rust-lang.org/std/sync/mpsc/index.html) to achieve a similar effect.\\n\\nSo now we\'re able to define basic Cap\'N Proto messages, and send them all around our programs."},{"id":"2018/01/captains-cookbook-part-1","metadata":{"permalink":"/2018/01/captains-cookbook-part-1","source":"@site/blog/2018-01-16-captains-cookbok-part-1/index.mdx","title":"Captain\'s Cookbook: Project setup","description":"A basic introduction to getting started with Cap\'N Proto.","date":"2018-01-16T12:00:00.000Z","tags":[],"readingTime":7.555,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2018/01/captains-cookbook-part-1","title":"Captain\'s Cookbook: Project setup","date":"2018-01-16T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1731201811000,"prevItem":{"title":"Captain\'s Cookbook: Practical usage","permalink":"/2018/01/captains-cookbook-part-2"},"nextItem":{"title":"PCA audio compression","permalink":"/2016/11/pca-audio-compression"}},"content":"A basic introduction to getting started with Cap\'N Proto.\\n\\n\x3c!-- truncate --\x3e\\n\\nI\'ve been working a lot with [Cap\'N Proto](https://capnproto.org/) recently with Rust, but there\'s a real dearth of information\\non how to set up and get going quickly. In the interest of trying to get more people using this (because I think it\'s\\nfantastic), I\'m going to work through a couple of examples detailing what exactly should be done to get going.\\n\\nSo, what is Cap\'N Proto? It\'s a data serialization library. It has contemporaries with [Protobuf](https://developers.google.com/protocol-buffers/)\\nand [FlatBuffers](https://google.github.io/flatbuffers/), but is better compared with FlatBuffers. The whole point behind it\\nis to define a schema language and serialization format such that:\\n\\n1. Applications that do not share the same base programming language can communicate\\n2. The data and schema you use can naturally evolve over time as your needs change\\n\\nAccompanying this are typically code generators that take the schemas you define for your application and give you back\\ncode for different languages to get data to and from that schema.\\n\\nNow, what makes Cap\'N Proto different from, say, Protobuf, is that there is no serialization/deserialization step the same way\\nas is implemented with Protobuf. Instead, the idea is that the message itself can be loaded in memory and used directly there.\\n\\nWe\'re going to take a look at a series of progressively more complex projects that use Cap\'N Proto in an effort to provide some\\nexamples of what idiomatic usage looks like, and shorten the startup time needed to make use of this library in Rust projects.\\nIf you want to follow along, feel free. If not, I\'ve posted [the final result](https://github.com/bspeice/capnp_cookbook_1)\\nfor reference.\\n\\n## Step 1: Installing `capnp`\\n\\nThe `capnp` binary itself is needed for taking the schema files you write and turning them into a format that can be used by the\\ncode generation libraries. Don\'t ask me what that actually means, I just know that you need to make sure this is installed.\\n\\nI\'ll refer you to [Cap\'N Proto\'s installation instructions](https://capnproto.org/install.html) here. As a quick TLDR though:\\n\\n- Linux users will likely have a binary shipped by their package manager - On Ubuntu, `apt install capnproto` is enough\\n- OS X users can use [Homebrew](https://brew.sh/) as an easy install path. Just `brew install capnp`\\n- Windows users are a bit more complicated. If you\'re using [Chocolatey](https://chocolatey.org/), there\'s [a package](https://chocolatey.org/packages/capnproto/) available. If that doesn\'t work however, you need to download [a release zip](https://capnproto.org/capnproto-c++-win32-0.6.1.zip) and make sure that the `capnp.exe` binary is in your `%PATH%` environment variable\\n\\nThe way you know you\'re done with this step is if the following command works in your shell:\\n\\n```bash\\ncapnp id\\n```\\n\\n## Step 2: Starting a Cap\'N Proto Rust project\\n\\nAfter the `capnp` binary is set up, it\'s time to actually create our Rust project. Nothing terribly complex here, just a simple\\n\\n```bash\\nmkdir capnp_cookbook_1\\ncd capnp_cookbook_1\\ncargo init --bin\\n```\\n\\nWe\'ll put the following content into `Cargo.toml`:\\n\\n```\\n[package]\\nname = \\"capnp_cookbook_1\\"\\nversion = \\"0.1.0\\"\\nauthors = [\\"Bradlee Speice <bspeice@kcg.com>\\"]\\n\\n[build-dependencies]\\ncapnpc = \\"0.8\\"  # 1\\n\\n[dependencies]\\ncapnp = \\"0.8\\"  # 2\\n```\\n\\nThis sets up: \\n\\n1. The Rust code generator (CAPNProto Compiler)\\n2. The Cap\'N Proto runtime library (CAPNProto runtime)\\n\\nWe\'ve now got everything prepared that we need for writing a Cap\'N Proto project.\\n\\n## Step 3: Writing a basic schema\\n\\nWe\'re going to start with writing a pretty trivial data schema that we can extend later. This is just intended to make sure\\nyou get familiar with how to start from a basic project.\\n\\nFirst, we\'re going to create a top-level directory for storing the schema files in:\\n\\n```bash\\n# Assuming we\'re starting from the `capnp_cookbook_1` directory created earlier\\n\\nmkdir schema\\ncd schema\\n```\\n\\nNow, we\'re going to put the following content in `point.capnp`:\\n\\n```\\n@0xab555145c708dad2;\\n\\nstruct Point {\\n    x @0 :Int32;\\n    y @1 :Int32;\\n}\\n```\\n\\nPretty easy, we\'ve now got structure for an object we\'ll be able to quickly encode in a binary format.\\n\\n## Step 4: Setting up the build process\\n\\nNow it\'s time to actually set up the build process to make sure that Cap\'N Proto generates the Rust code we\'ll eventually be using.\\nThis is typically done through a `build.rs` file to invoke the schema compiler.\\n\\nIn the same folder as your `Cargo.toml` file, please put the following content in `build.rs`:\\n\\n```rust\\nextern crate capnpc;\\n\\nfn main() {\\n    ::capnpc::CompilerCommand::new()\\n        .src_prefix(\\"schema\\")  // 1\\n        .file(\\"schema/point.capnp\\")  // 2\\n        .run().expect(\\"compiling schema\\");\\n}\\n```\\n\\nThis sets up the protocol compiler (`capnpc` from earlier) to compile the schema we\'ve built so far.\\n\\n1. Because Cap\'N Proto schema files can re-use types specified in other files, the `src_prefix()` tells the compiler\\nwhere to look for those extra files at.\\n2. We specify the schema file we\'re including by hand. In a much larger project, you could presumably build the `CompilerCommand`\\ndynamically, but we won\'t worry too much about that one for now.\\n\\n## Step 5: Running the build\\n\\nIf you\'ve done everything correctly so far, you should be able to actually build the project and see the auto-generated code.\\nRun a `cargo build` command, and if you don\'t see `cargo` complaining, you\'re doing just fine!\\n\\nSo where exactly does the generated code go to? I think it\'s critically important for people to be able to see what the generated\\ncode looks like, because you need to understand what you\'re actually programming against. The short answer is: the generated code lives\\nsomewhere in the `target/` directory.\\n\\nThe long answer is that you\'re best off running a `find` command to get the actual file path:\\n\\n```bash\\n# Assuming we\'re running from the capnp_cookbook_1 project folder\\nfind . -name point_capnp.rs\\n```\\n\\nAlternately, if the `find` command isn\'t available, the path will look something like:\\n\\n```\\n./target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs\\n```\\n\\nSee if there are any paths in your target directory that look similar.\\n\\nNow, the file content looks pretty nasty. I\'ve included an example [here](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs)\\nif you aren\'t following along at home. There are a couple things I\'ll try and point out though so you can get an idea of how\\nthe schema we wrote for the \\"Point\\" message is tied to the generated code.\\n\\nFirst, the Cap\'N Proto library splits things up into `Builder` and `Reader` structs. These are best thought of the same way\\nRust separates `mut` from non-`mut` code. `Builder`s are `mut` versions of your message, and `Reader`s are immutable versions.\\n\\nFor example, the [`Builder` impl](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L90) for `point` defines [`get_x()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L105), [`set_x()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L109), [`get_y()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L113), and [`set_y()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L117) methods.\\nIn comparison, the [`Reader` impl](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L38) only defines [`get_x()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L47) and [`get_y()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L51) methods.\\n\\nSo now we know that there are some `get` and `set` methods available for our `x` and `y` coordinates;\\nbut what do we actually do with those?\\n\\n## Step 6: Making a point\\n\\nSo we\'ve install Cap\'N Proto, gotten a project set up, and can generate schema code now. It\'s time to actually start building\\nCap\'N Proto messages! I\'m going to put the code you need here because it\'s small, and put some extra long comments inline. This code\\nshould go in [`src/main.rs`](https://github.com/bspeice/capnp_cookbook_1/blob/master/src/main.rs):\\n\\n```rust\\n// Note that we use `capnp` here, NOT `capnpc`\\nextern crate capnp;\\n\\n// We create a module here to define how we are to access the code\\n// being included.\\npub mod point_capnp {\\n    // The environment variable OUT_DIR is set by Cargo, and\\n    // is the location of all the code that was built as part\\n    // of the codegen step.\\n    // point_capnp.rs is the actual file to include\\n    include!(concat!(env!(\\"OUT_DIR\\"), \\"/point_capnp.rs\\"));\\n}\\n\\nfn main() {\\n\\n    // The process of building a Cap\'N Proto message is a bit tedious.\\n    // We start by creating a generic Builder; it acts as the message\\n    // container that we\'ll later be filling with content of our `Point`\\n    let mut builder = capnp::message::Builder::new_default();\\n\\n    // Because we need a mutable reference to the `builder` later,\\n    // we fence off this part of the code to allow sequential mutable\\n    // borrows. As I understand it, non-lexical lifetimes:\\n    // https://github.com/rust-lang/rust-roadmap/issues/16\\n    // will make this no longer necessary\\n    {\\n        // And now we can set up the actual message we\'re trying to create\\n        let mut point_msg = builder.init_root::<point_capnp::point::Builder>();\\n\\n        // Stuff our message with some content\\n        point_msg.set_x(12);\\n\\n        point_msg.set_y(14);\\n    }\\n\\n    // It\'s now time to serialize our message to binary. Let\'s set up a buffer for that:\\n    let mut buffer = Vec::new();\\n\\n    // And actually fill that buffer with our data\\n    capnp::serialize::write_message(&mut buffer, &builder).unwrap();\\n\\n    // Finally, let\'s deserialize the data\\n    let deserialized = capnp::serialize::read_message(\\n        &mut buffer.as_slice(),\\n        capnp::message::ReaderOptions::new()\\n    ).unwrap();\\n\\n    // `deserialized` is currently a generic reader; it understands\\n    // the content of the message we gave it (i.e. that there are two\\n    // int32 values) but doesn\'t really know what they represent (the Point).\\n    // This is where we map the generic data back into our schema.\\n    let point_reader = deserialized.get_root::<point_capnp::point::Reader>().unwrap();\\n\\n    // We can now get our x and y values back, and make sure they match\\n    assert_eq!(point_reader.get_x(), 12);\\n    assert_eq!(point_reader.get_y(), 14);\\n}\\n```\\n\\nAnd with that, we\'ve now got a functioning project. Here\'s the content I\'m planning to go over next as we build up\\nsome practical examples of Cap\'N Proto in action:"},{"id":"2016/11/pca-audio-compression","metadata":{"permalink":"/2016/11/pca-audio-compression","source":"@site/blog/2016-11-01-PCA-audio-compression/index.mdx","title":"PCA audio compression","description":"In which I apply Machine Learning techniques to Digital Signal Processing to astounding failure.","date":"2016-11-01T12:00:00.000Z","tags":[],"readingTime":10.39,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/11/pca-audio-compression","title":"PCA audio compression","date":"2016-11-01T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730863976000,"prevItem":{"title":"Captain\'s Cookbook: Project setup","permalink":"/2018/01/captains-cookbook-part-1"},"nextItem":{"title":"A Rustic re-podcasting server","permalink":"/2016/10/rustic-repodcasting"}},"content":"In which I apply Machine Learning techniques to Digital Signal Processing to astounding failure.\\n\\n\x3c!-- truncate --\x3e\\n\\nTowards a new (and pretty poor) compression scheme\\n--------------------------------------------------\\n\\nI\'m going to be working with some audio data for a while as I get prepared for a term project this semester. I\'ll be working (with a partner) to design a system for separating voices from music. Given my total lack of experience with [Digital Signal Processing][1] I figured that now was as good a time as ever to work on a couple of fun projects that would get me back up to speed.\\n\\nThe first project I want to work on: Designing a new compression scheme for audio data.\\n\\nA Brief Introduction to Audio Compression\\n-----------------------------------------\\n\\nAudio files when uncompressed (files ending with `.wav`) are huge. Like, 10.5 Megabytes per minute huge. Storage is cheap these days, but that\'s still an incredible amount of data that we don\'t really need. Instead, we\'d like to compress that data so that it\'s not taking up so much space. There are broadly two ways to accomplish this:\\n\\n1. Lossless compression - Formats like [FLAC][2], [ALAC][3], and [Monkey\'s Audio (.ape)][4] all go down this route. The idea is that when you compress and uncompress a file, you get exactly the same as what you started with.\\n\\n2. Lossy compression - Formats like [MP3][5], [Ogg][6], and [AAC (`.m4a`)][7] are far more popular, but make a crucial tradeoff: We can reduce the file size even more during compression, but the decompressed file won\'t be the same.\\n\\nThere is a fundamental tradeoff at stake: Using lossy compression sacrifices some of the integrity of the resulting file to save on storage space. Most people (I personally believe it\'s everybody) can\'t hear the difference, so this is an acceptable tradeoff. You have files that take up a 10<sup>th</sup> of the space, and nobody can tell there\'s a difference in audio quality.\\n\\nA PCA-based Compression Scheme\\n------------------------------\\n\\nWhat I want to try out is a [PCA][8] approach to encoding audio. The PCA technique comes from Machine Learning, where it is used for a process called [Dimensionality Reduction][9]. Put simply, the idea is the same as lossy compression: if we can find a way that represents the data well enough, we can save on space. There are a lot of theoretical concerns that lead me to believe this compression style will not end well, but I\'m interested to try it nonetheless.\\n\\nPCA works as follows: Given a dataset with a number of features, I find a way to approximate those original features using some \\"new features\\" that are statistically as close as possible to the original ones. This is comparable to a scheme like MP3: Given an original signal, I want to find a way of representing it that gets approximately close to what the original was. The difference is that PCA is designed for statistical data, and not signal data. But we won\'t let that stop us.\\n\\nThe idea is as follows: Given a signal, reshape it into 1024 columns by however many rows are needed (zero-padded if necessary). Run the PCA algorithm, and do dimensionality reduction with a couple different settings. The number of components I choose determines the quality: If I use 1024 components, I will essentially be using the original signal. If I use a smaller number of components, I start losing some of the data that was in the original file. This will give me an idea of whether it\'s possible to actually build an encoding scheme off of this, or whether I\'m wasting my time.\\n\\nRunning the Algorithm\\n---------------------\\n\\nThe audio I will be using comes from the song [Tabulasa][10], by [Broke for Free][11]. I\'ll be loading in the audio signal to Python and using [Scikit-Learn][12] to actually run the PCA algorithm.\\n\\nWe first need to convert the FLAC file I have to a WAV:\\n\\n[1]: https://en.wikipedia.org/wiki/Digital_signal_processing\\n[2]: https://en.wikipedia.org/wiki/FLAC\\n[3]: https://en.wikipedia.org/wiki/Apple_Lossless\\n[4]: https://en.wikipedia.org/wiki/Monkey%27s_Audio\\n[5]: https://en.wikipedia.org/wiki/MP3\\n[6]: https://en.wikipedia.org/wiki/Vorbis\\n[7]: https://en.wikipedia.org/wiki/Advanced_Audio_Coding\\n[8]: https://en.wikipedia.org/wiki/Principal_component_analysis\\n[9]: https://en.wikipedia.org/wiki/Dimensionality_reduction\\n[10]: https://brokeforfree.bandcamp.com/track/tabulasa\\n[11]: https://brokeforfree.bandcamp.com/album/xxvii\\n[12]: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA\\n\\n\\n```python\\n!ffmpeg -hide_banner -loglevel panic -i \\"Broke For Free/XXVII/01 Tabulasa.flac\\" \\"Tabulasa.wav\\" -c wav\\n```\\n\\nThen, let\'s go ahead and load a small sample so you can hear what is going on.\\n\\n\\n```python\\nfrom IPython.display import Audio\\nfrom scipy.io import wavfile\\n\\nsamplerate, tabulasa = wavfile.read(\'Tabulasa.wav\')\\n\\nstart = samplerate * 14 # 10 seconds in\\nend = start + samplerate * 10 # 5 second duration\\nAudio(data=tabulasa[start:end, 0], rate=samplerate)\\n```\\n\\nimport wav1 from \\"./1.wav\\";\\n\\n<audio controls src={wav1}/>\\n\\nNext, we\'ll define the code we will be using to do PCA. It\'s very short, as the PCA algorithm is very simple.\\n\\n\\n```python\\nfrom sklearn.decomposition import PCA\\nimport numpy as np\\n\\ndef pca_reduce(signal, n_components, block_size=1024):\\n    \\n    # First, zero-pad the signal so that it is divisible by the block_size\\n    samples = len(signal)\\n    hanging = block_size - np.mod(samples, block_size)\\n    padded = np.lib.pad(signal, (0, hanging), \'constant\', constant_values=0)\\n    \\n    # Reshape the signal to have 1024 dimensions\\n    reshaped = padded.reshape((len(padded) // block_size, block_size))\\n    \\n    # Second, do the actual PCA process\\n    pca = PCA(n_components=n_components)\\n    pca.fit(reshaped)\\n    \\n    transformed = pca.transform(reshaped)\\n    reconstructed = pca.inverse_transform(transformed).reshape((len(padded)))\\n    return pca, transformed, reconstructed\\n```\\n\\nNow that we\'ve got our functions set up, let\'s try actually running something. First, we\'ll use `n_components == block_size`, which implies that we should end up with the same signal we started with.\\n\\n\\n```python\\ntabulasa_left = tabulasa[:,0]\\n\\n_, _, reconstructed = pca_reduce(tabulasa_left, 1024, 1024)\\n\\nAudio(data=reconstructed[start:end], rate=samplerate)\\n```\\n\\nimport wav2 from \\"./2.wav\\";\\n\\n<audio controls src={wav2}/>\\n\\nOK, that does indeed sound like what we originally had. Let\'s drastically cut down the number of components we\'re doing this with as a sanity check: the audio quality should become incredibly poor.\\n\\n\\n```python\\n_, _, reconstructed = pca_reduce(tabulasa_left, 32, 1024)\\n\\nAudio(data=reconstructed[start:end], rate=samplerate)\\n```\\n\\nimport wav3 from \\"./3.wav\\";\\n\\n<audio controls src={wav3}/>\\n\\nAs expected, our reconstructed audio does sound incredibly poor! But there\'s something else very interesting going on here under the hood. Did you notice that the bassline comes across very well, but that there\'s no midrange or treble? The drums are almost entirely gone.\\n\\n[Drop the (Treble)][13]\\n-----------------------\\n\\nIt will help to understand PCA more fully when trying to read this part, but I\'ll do my best to break it down. PCA tries to find a way to best represent the dataset using \\"components.\\" Think of each \\"component\\" as containing some of the information you need in order to reconstruct the full audio. For example, you might have a \\"low frequency\\" component that contains all the information you need in order to hear the bassline. There might be other components that explain the high frequency things like singers, or melodies, that you also need.\\n\\nWhat makes PCA interesting is that it attempts to find the \\"most important\\" components in explaining the signal. In a signal processing world, this means that PCA is trying to find the signal amongst the noise in your data. In our case, this means that PCA, when forced to work with small numbers of components, will chuck out the noisy components first. It\'s doing it\'s best job to reconstruct the signal, but it has to make sacrifices somewhere.\\n\\nSo I\'ve mentioned that PCA identifies the \\"noisy\\" components in our dataset. This is equivalent to saying that PCA removes the \\"high frequency\\" components in this case: it\'s very easy to represent a low-frequency signal like a bassline. It\'s far more difficult to represent a high-frequency signal because it\'s changing all the time. When you force PCA to make a tradeoff by using a small number of components, the best it can hope to do is replicate the low-frequency sections and skip the high-frequency things.\\n\\nThis is a very interesting insight, and it also has echos (pardon the pun) of how humans understand music in general. Other encoding schemes (like MP3, etc.) typically chop off a lot of the high-frequency range as well. There is typically a lot of high-frequency noise in audio that is nearly impossible to hear, so it\'s easy to remove it without anyone noticing. PCA ends up doing something similar, and while that certainly wasn\'t the intention, it is an interesting effect.\\n\\n## A More Realistic Example\\n\\nSo we\'ve seen the edge cases so far: Using a large number of components results in audio very close to the original, and using a small number of components acts as a low-pass filter. How about we develop something that sounds \\"good enough\\" in practice, that we can use as a benchmark for size? We\'ll use ourselves as judges of audio quality, and build another function to help us estimate how much space we need to store everything in.\\n\\n[13]: https://youtu.be/Ua0KpfJsxKo?t=1m17s\\n\\n\\n```python\\nfrom bz2 import compress\\nimport pandas as pd\\n\\ndef raw_estimate(transformed, pca):\\n    # We assume that we\'ll be storing things as 16-bit WAV,\\n    # meaning two bytes per sample\\n    signal_bytes = transformed.tobytes()\\n    # PCA stores the components as floating point, we\'ll assume\\n    # that means 32-bit floats, so 4 bytes per element\\n    component_bytes = transformed.tobytes()\\n    \\n    # Return a result in megabytes\\n    return (len(signal_bytes) + len(component_bytes)) / (2**20)\\n\\n# Do an estimate for lossless compression applied on top of our\\n# PCA reduction\\ndef bz2_estimate(transformed, pca):\\n    bytestring = transformed.tobytes() + b\';\' + pca.components_.tobytes()\\n    compressed = compress(bytestring)\\n    return len(compressed) / (2**20)\\n\\ncompression_attempts = [\\n    (1, 1),\\n    (1, 2),\\n    (1, 4),\\n    (4, 32),\\n    (16, 256),\\n    (32, 256),\\n    (64, 256),\\n    (128, 1024),\\n    (256, 1024),\\n    (512, 1024),\\n    (128, 2048),\\n    (256, 2048),\\n    (512, 2048),\\n    (1024, 2048)\\n]\\n\\ndef build_estimates(signal, n_components, block_size):\\n    pca, transformed, recon = pca_reduce(tabulasa_left, n_components, block_size)\\n    raw_pca_estimate = raw_estimate(transformed, pca)\\n    bz2_pca_estimate = bz2_estimate(transformed, pca)\\n    raw_size = len(recon.tobytes()) / (2**20)\\n    return raw_size, raw_pca_estimate, bz2_pca_estimate\\n\\npca_compression_results = pd.DataFrame([\\n        build_estimates(tabulasa_left, n, bs)\\n        for n, bs in compression_attempts\\n    ])\\n\\npca_compression_results.columns = [\\"Raw\\", \\"PCA\\", \\"PCA w/ BZ2\\"]\\npca_compression_results.index = compression_attempts\\npca_compression_results\\n```\\n\\n<div>\\n<table>\\n  <thead>\\n    <tr>\\n      <th></th>\\n      <th>Raw</th>\\n      <th>PCA</th>\\n      <th>PCA w/ BZ2</th>\\n    </tr>\\n  </thead>\\n  <tbody>\\n    <tr>\\n      <th>(1, 1)</th>\\n      <td>69.054298</td>\\n      <td>138.108597</td>\\n      <td>16.431797</td>\\n    </tr>\\n    <tr>\\n      <th>(1, 2)</th>\\n      <td>69.054306</td>\\n      <td>69.054306</td>\\n      <td>32.981380</td>\\n    </tr>\\n    <tr>\\n      <th>(1, 4)</th>\\n      <td>69.054321</td>\\n      <td>34.527161</td>\\n      <td>16.715032</td>\\n    </tr>\\n    <tr>\\n      <th>(4, 32)</th>\\n      <td>69.054443</td>\\n      <td>17.263611</td>\\n      <td>8.481735</td>\\n    </tr>\\n    <tr>\\n      <th>(16, 256)</th>\\n      <td>69.054688</td>\\n      <td>8.631836</td>\\n      <td>4.274846</td>\\n    </tr>\\n    <tr>\\n      <th>(32, 256)</th>\\n      <td>69.054688</td>\\n      <td>17.263672</td>\\n      <td>8.542909</td>\\n    </tr>\\n    <tr>\\n      <th>(64, 256)</th>\\n      <td>69.054688</td>\\n      <td>34.527344</td>\\n      <td>17.097543</td>\\n    </tr>\\n    <tr>\\n      <th>(128, 1024)</th>\\n      <td>69.054688</td>\\n      <td>17.263672</td>\\n      <td>9.430644</td>\\n    </tr>\\n    <tr>\\n      <th>(256, 1024)</th>\\n      <td>69.054688</td>\\n      <td>34.527344</td>\\n      <td>18.870387</td>\\n    </tr>\\n    <tr>\\n      <th>(512, 1024)</th>\\n      <td>69.054688</td>\\n      <td>69.054688</td>\\n      <td>37.800940</td>\\n    </tr>\\n    <tr>\\n      <th>(128, 2048)</th>\\n      <td>69.062500</td>\\n      <td>8.632812</td>\\n      <td>6.185015</td>\\n    </tr>\\n    <tr>\\n      <th>(256, 2048)</th>\\n      <td>69.062500</td>\\n      <td>17.265625</td>\\n      <td>12.366942</td>\\n    </tr>\\n    <tr>\\n      <th>(512, 2048)</th>\\n      <td>69.062500</td>\\n      <td>34.531250</td>\\n      <td>24.736506</td>\\n    </tr>\\n    <tr>\\n      <th>(1024, 2048)</th>\\n      <td>69.062500</td>\\n      <td>69.062500</td>\\n      <td>49.517493</td>\\n    </tr>\\n  </tbody>\\n</table>\\n</div>\\n\\n\\n\\nAs we can see, there are a couple of instances where we do nearly 20 times better on storage space than the uncompressed file. Let\'s here what that sounds like:\\n\\n\\n```python\\n_, _, reconstructed = pca_reduce(tabulasa_left, 16, 256)\\nAudio(data=reconstructed[start:end], rate=samplerate)\\n```\\n\\nimport wav4 from \\"./4.wav\\";\\n\\n<audio controls src={wav4}/>\\n\\nIt sounds incredibly poor though. Let\'s try something that\'s a bit more realistic:\\n\\n\\n```python\\n_, _, reconstructed = pca_reduce(tabulasa_left, 1, 4)\\nAudio(data=reconstructed[start:end], rate=samplerate)\\n```\\n\\nimport wav5 from \\"./5.wav\\";\\n\\n<audio controls src={wav5}/>\\n\\nAnd just out of curiosity, we can try something that has the same ratio of components to block size. This should be close to an apples-to-apples comparison.\\n\\n\\n```python\\n_, _, reconstructed = pca_reduce(tabulasa_left, 64, 256)\\nAudio(data=reconstructed[start:end], rate=samplerate)\\n```\\n\\nimport wav6 from \\"./6.wav\\"\\n\\n<audio controls src={wav6}/>\\n\\nThe smaller block size definitely has better high-end response, but I personally think the larger block size sounds better overall.\\n\\n## Conclusions\\n\\nSo, what do I think about audio compression using PCA?\\n\\nStrangely enough, it actually works pretty well relative to what I expected. That said, it\'s a terrible idea in general.\\n\\nFirst off, you don\'t really save any space. The component matrix needed to actually run the PCA algorithm takes up a lot of space on its own, so it\'s very difficult to save space without sacrificing a huge amount of audio quality. And even then, codecs like AAC sound very nice even at bitrates that this PCA method could only dream of.\\n\\nSecond, there\'s the issue of audio streaming. PCA relies on two components: the datastream, and a matrix used to reconstruct the original signal. While it is easy to stream the data, you can\'t stream that matrix. And even if you divided the stream up into small blocks to give you a small matrix, you must guarantee that the matrix arrives; if you don\'t have that matrix, the data stream will make no sense whatsoever.\\n\\nAll said, this was an interesting experiment. It\'s really cool seeing PCA used for signal analysis where I haven\'t seen it applied before, but I don\'t think it will lead to any practical results. Look forward to more signal processing stuff in the future!"},{"id":"2016/10/rustic-repodcasting","metadata":{"permalink":"/2016/10/rustic-repodcasting","source":"@site/blog/2016-10-22-rustic-repodcasting/index.mdx","title":"A Rustic re-podcasting server","description":"Learning Rust by fire (it sounds better than learning by corrosion)","date":"2016-10-22T12:00:00.000Z","tags":[],"readingTime":10.405,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/10/rustic-repodcasting","title":"A Rustic re-podcasting server","date":"2016-10-22T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730863976000,"prevItem":{"title":"PCA audio compression","permalink":"/2016/11/pca-audio-compression"},"nextItem":{"title":"Event studies and earnings releases","permalink":"/2016/06/event-studies-and-earnings-releases"}},"content":"Learning Rust by fire (it sounds better than learning by corrosion)\\n\\n\x3c!-- truncate --\x3e\\n\\nI listen to a lot of Drum and Bass music, because it\'s beautiful music. And\\nthere\'s a particular site, [Bassdrive.com](http://bassdrive.com/) that hosts\\na lot of great content. Specifically, the\\n[archives](http://archives.bassdrivearchive.com/) section of the site has a\\nlist of the past shows that you can download and listen to. The issue is, it\'s\\njust a [giant list of links to download](http://archives.bassdrivearchive.com/6%20-%20Saturday/Electronic%20Warfare%20-%20The%20Overfiend/). I\'d really like\\nthis in a podcast format to take with me on the road, etc.\\n\\nSo I wrote the [elektricity](https://github.com/bspeice/elektricity) web\\napplication to actually accomplish all that. Whenever you request a feed, it\\ngoes out to Bassdrive, processes all the links on a page, and serves up some\\nfresh, tasty RSS to satisfy your ears. I hosted it on Heroku using the free\\ntier because it\'s really not resource-intensive at all.\\n\\n**The issue so far** is that I keep running out of free tier hours during a\\nmonth because my podcasting application likes to have a server scan for new\\nepisodes constantly. Not sure why it\'s doing that, but I don\'t have a whole\\nlot of control over it. It\'s a phenomenal application otherwise.\\n\\n**My (over-engineered) solution**: Re-write the application using the\\n[Rust](https://www.rust-lang.org/en-US/) programming language. I\'d like to run\\nthis on a small hacker board I own, and doing this in Rust would allow me to\\neasily cross-compile it. Plus, I\'ve been very interested in the Rust language\\nfor a while and this would be a great opportunity to really learn it well.\\nThe code is available [here](https://github.com/bspeice/nutone) as development\\nprogresses.\\n\\n## The Setup\\n\\nWe\'ll be using the [iron](http://ironframework.io/) library to handle the\\nserver, and [hyper](http://hyper.rs/) to fetch the data we need from elsewhere\\non the interwebs. [HTML5Ever](http://doc.servo.org/html5ever/index.html) allows\\nus to ingest the content that will be coming from Bassdrive, and finally,\\noutput is done with [handlebars-rust](http://sunng87.github.io/handlebars-rust/handlebars/index.html).\\n\\nIt will ultimately be interesting to see how much more work must be done to\\nactually get this working over another language like Python. Coming from a\\ndynamic state of mind it\'s super easy to just chain stuff together, ship it out,\\nand call it a day. I think I\'m going to end up getting much dirtier trying to\\nwrite all of this out.\\n\\n## Issue 1: Strings\\n\\nStrings in Rust are hard. I acknowledge Python can get away with some things\\nthat make strings super easy (and Python 3 has gotten better at cracking down\\non some bad cases, `str <-> bytes` specifically), but Rust is hard.\\n\\nLet\'s take for example the `404` error handler I\'m trying to write. The result\\nshould be incredibly simple: All I want is to echo back\\n`Didn\'t find URL: <url>`. Shouldn\'t be that hard right? In Python I\'d just do\\nsomething like:\\n\\n```python\\ndef echo_handler(request):\\n    return \\"You\'re visiting: {}\\".format(request.uri)\\n```\\n\\nAnd we\'d call it a day. Rust isn\'t so simple. Let\'s start with the trivial\\nexamples people post online:\\n\\n```rust\\nfn hello_world(req: &mut Request) -> IronResult<Response> {\\n    Ok(Response::with((status::Ok, \\"You found the server!\\")))\\n}\\n```\\n\\nDoesn\'t look too bad right? In fact, it\'s essentially the same as the Python\\nversion! All we need to do is just send back a string of some form. So, we\\nlook up the documentation for [`Request`](http://ironframework.io/doc/iron/request/struct.Request.html) and see a `url` field that will contain\\nwhat we want. Let\'s try the first iteration:\\n\\n```rust\\nfn hello_world(req: &mut Request) -> IronResult<Response> {\\n    Ok(Response::with((status::Ok, \\"You found the URL: \\" + req.url)))\\n}\\n```\\n\\nWhich yields the error:\\n\\n```\\n    error[E0369]: binary operation `+` cannot be applied to type `&\'static str`\\n```\\n\\nOK, what\'s going on here? Time to start Googling for [\\"concatenate strings in Rust\\"](https://www.google.com/#q=concatenate+strings+in+rust). That\'s what we\\nwant to do right? Concatenate a static string and the URL.\\n\\nAfter Googling, we come across a helpful [`concat!`](https://doc.rust-lang.org/std/macro.concat!.html) macro that looks really nice! Let\'s try that one:\\n\\n```rust\\nfn hello_world(req: &mut Request) -> IronResult<Response> {\\n    Ok(Response::with((status::Ok, concat!(\\"You found the URL: \\", req.url))))\\n}\\n```\\n\\nAnd the error:\\n\\n```\\n    error: expected a literal\\n```\\n\\nTurns out Rust actually blows up because the `concat!` macro expects us to know\\nat compile time what `req.url` is. Which, in my outsider opinion, is a bit\\nstrange. `println!` and `format!`, etc., all handle values they don\'t know at\\ncompile time. Why can\'t `concat!`? By any means, we need a new plan of attack.\\nHow about we try formatting strings?\\n\\n```rust\\nfn hello_world(req: &mut Request) -> IronResult<Response> {\\n    Ok(Response::with((status::Ok, format!(\\"You found the URL: {}\\", req.url))))\\n}\\n```\\n\\nAnd at long last, it works. Onwards!\\n\\n## Issue 2: Fighting with the borrow checker\\n\\nRust\'s single coolest feature is how the compiler can guarantee safety in your\\nprogram. As long as you don\'t use `unsafe` pointers in Rust, you\'re guaranteed\\nsafety. And not having truly manual memory management is really cool; I\'m\\ntotally OK with never having to write `malloc()` again.\\n\\nThat said, even [the Rust documentation](https://doc.rust-lang.org/book/ownership.html) makes a specific note:\\n\\n> Many new users to Rust experience something we like to call\\n> \u2018fighting with the borrow checker\u2019, where the Rust compiler refuses to\\n> compile a program that the author thinks is valid.\\n\\nIf you have to put it in the documentation, it\'s not a helpful note:\\nit\'s hazing.\\n\\nSo now that we have a handler which works with information from the request, we\\nwant to start making something that looks like an actual web application.\\nThe router provided by `iron` isn\'t terribly difficult so I won\'t cover it.\\nInstead, the thing that had me stumped for a couple hours was trying to\\ndynamically create routes.\\n\\nThe unfortunate thing with Rust (in my limited experience at the moment) is that\\nthere is a severe lack of non-trivial examples. Using the router is easy when\\nyou want to give an example of a static function. But how do you you start\\nworking on things that are a bit more complex?\\n\\nWe\'re going to cover that here. Our first try: creating a function which returns\\nother functions. This is a principle called [currying](http://stackoverflow.com/a/36321/1454178). We set up a function that allows us to keep some data in scope\\nfor another function to come later.\\n\\n```rust\\nfn build_handler(message: String) -> Fn(&mut Request) -> IronResult<Response> {\\n    move |_: &mut Request| {\\n        Ok(Response::with((status::Ok, message)))\\n    }\\n}\\n```\\n\\nWe\'ve simply set up a function that returns another anonymous function with the\\n`message` parameter scoped in. If you compile this, you get not 1, not 2, but 5\\nnew errors. 4 of them are the same though:\\n\\n```\\n    error[E0277]: the trait bound `for<\'r, \'r, \'r> std::ops::Fn(&\'r mut iron::Request<\'r, \'r>) -> std::result::Result<iron::Response, iron::IronError> + \'static: std::marker::Sized` is not satisfied\\n```\\n\\n...oookay. I for one, am not going to spend time trying to figure out what\'s\\ngoing on there.\\n\\nAnd it is here that I will save the audience many hours of frustrated effort.\\nAt this point, I decided to switch from `iron` to pure `hyper` since using\\n`hyper` would give me a much simpler API. All I would have to do is build a\\nfunction that took two parameters as input, and we\'re done. That said, it\\nultimately posed many more issues because I started getting into a weird fight\\nwith the `\'static` [lifetime](https://doc.rust-lang.org/book/lifetimes.html)\\nand being a Rust newbie I just gave up on trying to understand it.\\n\\nInstead, we will abandon (mostly) the curried function attempt, and instead\\ntake advantage of something Rust actually intends us to use: `struct` and\\n`trait`.\\n\\nRemember when I talked about a lack of non-trivial examples on the Internet?\\nThis is what I was talking about. I could only find *one* example of this\\navailable online, and it was incredibly complex and contained code we honestly\\ndon\'t need or care about. There was no documentation of how to build routes that\\ndidn\'t use static functions, etc. But, I\'m assuming you don\'t really care about\\nmy whining, so let\'s get to it.\\n\\nThe `iron` documentation mentions the [`Handler`](http://ironframework.io/doc/iron/middleware/trait.Handler.html) trait as being something we can implement.\\nDoes the function signature for that `handle()` method look familiar? It\'s what\\nwe\'ve been working with so far.\\n\\nThe principle is that we need to define a new `struct` to hold our data, then\\nimplement that `handle()` method to return the result. Something that looks\\nlike this might do:\\n\\n```rust\\nstruct EchoHandler {\\n    message: String\\n}\\n\\nimpl Handler for EchoHandler {\\n    fn handle(&self, _: &mut Request) -> IronResult<Response> {\\n        Ok(Response::with((status::Ok, self.message)))\\n    }\\n}\\n\\n// Later in the code when we set up the router...\\nlet echo = EchoHandler {\\n    message: \\"Is it working yet?\\"\\n}\\nrouter.get(\\"/\\", echo.handle, \\"index\\");\\n```\\n\\nWe attempt to build a struct, and give its `handle` method off to the router\\nso the router knows what to do.\\n\\nYou guessed it, more errors:\\n\\n```\\n    error: attempted to take value of method `handle` on type `EchoHandler`\\n```\\n\\nNow, the Rust compiler is actually a really nice fellow, and offers us help:\\n\\n```\\n    help: maybe a `()` to call it is missing? If not, try an anonymous function\\n```\\n\\nWe definitely don\'t want to call that function, so maybe try an anonymous\\nfunction as it recommends?\\n\\n```rust\\nrouter.get(\\"/\\", |req: &mut Request| echo.handle(req), \\"index\\");\\n```\\n\\nAnother error:\\n\\n```\\n    error[E0373]: closure may outlive the current function, but it borrows `echo`, which is owned by the current function\\n```\\n\\nAnother helpful message:\\n\\n```\\n    help: to force the closure to take ownership of `echo` (and any other referenced variables), use the `move` keyword\\n```\\n\\nWe\'re getting closer though! Let\'s implement this change:\\n\\n```rust\\nrouter.get(\\"/\\", move |req: &mut Request| echo.handle(req), \\"index\\");\\n```\\n\\nAnd here\'s where things get strange:\\n\\n```\\n    error[E0507]: cannot move out of borrowed content\\n      --\x3e src/main.rs:18:40\\n       |\\n    18 |         Ok(Response::with((status::Ok, self.message)))\\n       |                                        ^^^^ cannot move out of borrowed content\\n```\\n\\nNow, this took me another couple hours to figure out. I\'m going to explain it,\\nbut **keep this in mind: Rust only allows one reference at a time** (exceptions\\napply of course).\\n\\nWhen we attempt to use `self.message` as it has been created in the earlier\\n`struct`, we essentially are trying to give it away to another piece of code.\\nRust\'s semantics then state that *we may no longer access it* unless it is\\nreturned to us (which `iron`\'s code does not do). There are two ways to fix\\nthis:\\n\\n1. Only give away references (i.e. `&self.message` instead of `self.message`)\\ninstead of transferring ownership\\n2. Make a copy of the underlying value which will be safe to give away\\n\\nI didn\'t know these were the two options originally, so I hope this helps the\\naudience out. Because `iron` won\'t accept a reference, we are forced into the\\nsecond option: making a copy. To do so, we just need to change the function\\nto look like this:\\n\\n```rust\\nOk(Response::with((status::Ok, self.message.clone())))\\n```\\n\\nNot so bad, huh? My only complaint is that it took so long to figure out exactly\\nwhat was going on.\\n\\nAnd now we have a small server that we can configure dynamically. At long last.\\n\\n> Final sidenote: You can actually do this without anonymous functions. Just\\n> change the router line to:\\n>     `router.get(\\"/\\", echo, \\"index\\");`\\n>\\n> Rust\'s type system seems to figure out that we want to use the `handle()` method.\\n\\n## Conclusion\\n\\nAfter a good long days\' work, we now have the routing functionality set up on\\nour application. We should be able to scale this pretty well in the future:\\nthe RSS content we need to deliver in the future can be treated as a string, so\\nthe building blocks are in place.\\n\\nThere are two important things I learned starting with Rust today:\\n\\n1. Rust is a new language, and while the code is high-quality, the mindshare is coming.\\n2. I\'m a terrible programmer.\\n\\nNumber 1 is pretty obvious and not surprising to anyone. Number two caught me\\noff guard. I\'ve gotten used to having either a garbage collector (Java, Python,\\netc.) or playing a little fast and loose with scoping rules (C, C++). You don\'t\\nhave to worry about object lifetime there. With Rust, it\'s forcing me to fully\\nunderstand and use well the memory in my applications. In the final mistake I\\nfixed (using `.clone()`) I would have been fine in C++ to just give away that\\nreference and never use it again. I wouldn\'t have run into a \\"use-after-free\\"\\nerror, but I would have potentially been leaking memory. Rust forced me to be\\nincredibly precise about how I use it.\\n\\nAll said I\'m excited for using Rust more. I think it\'s super cool, it\'s just\\ngoing to take me a lot longer to do this than I originally thought."},{"id":"2016/06/event-studies-and-earnings-releases","metadata":{"permalink":"/2016/06/event-studies-and-earnings-releases","source":"@site/blog/2016-06-08-event-studies-and-earnings-releases/index.mdx","title":"Event studies and earnings releases","description":"Or, being suspicious of market insiders.","date":"2016-06-08T12:00:00.000Z","tags":[],"readingTime":16.01,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/06/event-studies-and-earnings-releases","title":"Event studies and earnings releases","date":"2016-06-08T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730863976000,"prevItem":{"title":"A Rustic re-podcasting server","permalink":"/2016/10/rustic-repodcasting"},"nextItem":{"title":"The unfair casino","permalink":"/2016/05/the-unfair-casino"}},"content":"Or, being suspicious of market insiders.\\n\\n\x3c!-- truncate --\x3e\\n\\n## The Market Just Knew\\n\\nI recently saw two examples of stock charts that have kept me thinking for a while. And now that the semester is complete, I finally have enough time to really look at them and give them the treatment they deserve. The first is good old Apple:\\n\\n\\n<details>\\n<summary>Code</summary>\\n\\n```python\\nfrom secrets import QUANDL_KEY\\nimport matplotlib.pyplot as plt\\nfrom matplotlib.dates import date2num\\nfrom matplotlib.finance import candlestick_ohlc\\nfrom matplotlib.dates import DateFormatter, WeekdayLocator,\\\\\\n    DayLocator, MONDAY\\nimport quandl\\nfrom datetime import datetime\\nimport pandas as pd\\n%matplotlib inline\\n\\ndef fetch_ticker(ticker, start, end):\\n    # Quandl is currently giving me issues with returning\\n    # the entire dataset and not slicing server-side.\\n    # So instead, we\'ll do it client-side!\\n    q_format = \'%Y-%m-%d\'\\n    ticker_data = quandl.get(\'YAHOO/\' + ticker,\\n                             start_date=start.strftime(q_format),\\n                             end_date=end.strftime(q_format),\\n                             authtoken=QUANDL_KEY)\\n    return ticker_data\\n\\ndef ohlc_dataframe(data, ax=None):\\n    # Much of this code re-used from:\\n    # http://matplotlib.org/examples/pylab_examples/finance_demo.html\\n    if ax is None:\\n        f, ax = plt.subplots()\\n    \\n    vals = [(date2num(date), *(data.loc[date]))\\n            for date in data.index]\\n    candlestick_ohlc(ax, vals)\\n    \\n    mondays = WeekdayLocator(MONDAY)\\n    alldays = DayLocator()\\n    weekFormatter = DateFormatter(\'%b %d\')\\n    ax.xaxis.set_major_locator(mondays)\\n    ax.xaxis.set_minor_locator(alldays)\\n    ax.xaxis.set_major_formatter(weekFormatter)\\n    return ax\\n```\\n</details>\\n\\n```python\\nAAPL = fetch_ticker(\'AAPL\', datetime(2016, 3, 1), datetime(2016, 5, 1))\\nax = ohlc_dataframe(AAPL)\\nplt.vlines(date2num(datetime(2016, 4, 26, 12)),\\n           ax.get_ylim()[0], ax.get_ylim()[1],\\n           color=\'b\',\\n          label=\'Earnings Release\')\\nplt.legend(loc=3)\\nplt.title(\\"Apple Price 3/1/2016 - 5/1/2016\\");\\n```\\n\\n![png](_notebook_files/_notebook_3_0.png)\\n\\nThe second chart is from Facebook:\\n\\n```python\\nFB = fetch_ticker(\'FB\', datetime(2016, 3, 1), datetime(2016, 5, 5))\\nax = ohlc_dataframe(FB)\\nplt.vlines(date2num(datetime(2016, 4, 27, 12)),\\n           ax.get_ylim()[0], ax.get_ylim()[1],\\n           color=\'b\', label=\'Earnings Release\')\\nplt.title(\'Facebook Price 3/5/2016 - 5/5/2016\')\\nplt.legend(loc=2);\\n```\\n    \\n![png](_notebook_files/_notebook_5_0.png)\\n    \\nThese two charts demonstrate two very specific phonomena: how the market prepares for earnings releases. Let\'s look at those charts again, but with some extra information. As we\'re about the see, the market \\"knew\\" in advance that Apple was going to perform poorly. The market expected that Facebook was going to perform poorly, and instead shot the lights out. Let\'s see that trend in action:\\n\\n<details>\\n<summary>Code</summary>\\n\\n```python\\ndef plot_hilo(ax, start, end, data):\\n    ax.plot([date2num(start), date2num(end)],\\n            [data.loc[start][\'High\'], data.loc[end][\'High\']],\\n            color=\'b\')\\n    ax.plot([date2num(start), date2num(end)],\\n            [data.loc[start][\'Low\'], data.loc[end][\'Low\']],\\n            color=\'b\')\\n\\nf, axarr = plt.subplots(1, 2)\\n\\nax_aapl = axarr[0]\\nax_fb = axarr[1]\\n\\n# Plot the AAPL trend up and down\\nohlc_dataframe(AAPL, ax=ax_aapl)\\nplot_hilo(ax_aapl, datetime(2016, 3, 1), datetime(2016, 4, 15), AAPL)\\nplot_hilo(ax_aapl, datetime(2016, 4, 18), datetime(2016, 4, 26), AAPL)\\nax_aapl.vlines(date2num(datetime(2016, 4, 26, 12)),\\n               ax_aapl.get_ylim()[0], ax_aapl.get_ylim()[1],\\n               color=\'g\', label=\'Earnings Release\')\\nax_aapl.legend(loc=2)\\nax_aapl.set_title(\'AAPL Price History\')\\n\\n# Plot the FB trend down and up\\nohlc_dataframe(FB, ax=ax_fb)\\nplot_hilo(ax_fb, datetime(2016, 3, 30), datetime(2016, 4, 27), FB)\\nplot_hilo(ax_fb, datetime(2016, 4, 28), datetime(2016, 5, 5), FB)\\nax_fb.vlines(date2num(datetime(2016, 4, 27, 12)),\\n             ax_fb.get_ylim()[0], ax_fb.get_ylim()[1],\\n             color=\'g\', label=\'Earnings Release\')\\nax_fb.legend(loc=2)\\nax_fb.set_title(\'FB Price History\')\\n\\nf.set_size_inches(18, 6)\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_7_0.png)\\n\\nAs we can see above, the market broke a prevailing trend on Apple in order to go down, and ultimately predict the earnings release. For Facebook, the opposite happened. While the trend was down, the earnings were fantastic and the market corrected itself much higher.\\n\\n## Formulating the Question\\n\\nWhile these are two specific examples, there are plenty of other examples you could cite one way or another. Even if the preponderance of evidence shows that the market correctly predicts earnings releases, we need not accuse people of collusion; for a company like Apple with many suppliers we can generally forecast how Apple has done based on those same suppliers.\\n\\nThe question then, is this: **how well does the market predict the earnings releases?** It\'s an incredibly broad question that I want to disect in a couple of different ways:\\n\\n1. Given a stock that has been trending down over the past N days before an earnings release, how likely does it continue downward after the release?\\n2. Given a stock trending up, how likely does it continue up?\\n3. Is there a difference in accuracy between large- and small-cap stocks?\\n4. How often, and for how long, do markets trend before an earnings release?\\n\\n**I want to especially thank Alejandro Saltiel for helping me retrieve the data.** He\'s great. And now for all of the interesting bits.\\n\\n## Event Studies\\n\\nBefore we go too much further, I want to introduce the actual event study. Each chart intends to capture a lot of information and present an easy-to-understand pattern:\\n\\n<details>\\n<summary>Code</summary>\\n\\n```python\\nimport numpy as np\\nimport pandas as pd\\nfrom pandas.tseries.holiday import USFederalHolidayCalendar\\nfrom pandas.tseries.offsets import CustomBusinessDay\\nfrom datetime import datetime, timedelta\\n\\n# If you remove rules, it removes them from *all* calendars\\n# To ensure we don\'t pop rules we don\'t want to, first make\\n# sure to fully copy the object\\ntrade_calendar = USFederalHolidayCalendar()\\ntrade_calendar.rules.pop(6) # Remove Columbus day\\ntrade_calendar.rules.pop(7) # Remove Veteran\'s day\\nTradeDay = lambda days: CustomBusinessDay(days, calendar=trade_calendar)\\n\\ndef plot_study(array):\\n    # Given a 2-d array, we assume the event happens at index `lookback`,\\n    # and create all of our summary statistics from there.\\n    lookback = int((array.shape[1] - 1) / 2)\\n    norm_factor = np.repeat(array[:,lookback].reshape(-1, 1), array.shape[1], axis=1)\\n    centered_data = array / norm_factor - 1\\n    lookforward = centered_data.shape[1] - lookback\\n    means = centered_data.mean(axis=0)\\n    lookforward_data = centered_data[:,lookforward:]\\n    std_dev = np.hstack([0, lookforward_data.std(axis=0)])\\n    maxes = lookforward_data.max(axis=0)\\n    mins = lookforward_data.min(axis=0)\\n    \\n    f, axarr = plt.subplots(1, 2)\\n    range_begin = -lookback\\n    range_end = lookforward\\n    axarr[0].plot(range(range_begin, range_end), means)\\n    axarr[1].plot(range(range_begin, range_end), means)\\n    axarr[0].fill_between(range(0, range_end),\\n                     means[-lookforward:] + std_dev,\\n                     means[-lookforward:] - std_dev,\\n                    alpha=.5, label=\\"$\\\\pm$ 1 s.d.\\")\\n    axarr[1].fill_between(range(0, range_end),\\n                     means[-lookforward:] + std_dev,\\n                     means[-lookforward:] - std_dev,\\n                    alpha=.5, label=\\"$\\\\pm$ 1 s.d.\\")\\n    \\n    max_err = maxes - means[-lookforward+1:]\\n    min_err = means[-lookforward+1:] - mins\\n    axarr[0].errorbar(range(1, range_end),\\n                  means[-lookforward+1:],\\n                  yerr=[min_err, max_err], label=\'Max & Min\')\\n    axarr[0].legend(loc=2)\\n    axarr[1].legend(loc=2)\\n    \\n    axarr[0].set_xlim((-lookback-1, lookback+1))\\n    axarr[1].set_xlim((-lookback-1, lookback+1))\\n    \\ndef plot_study_small(array):\\n    # Given a 2-d array, we assume the event happens at index `lookback`,\\n    # and create all of our summary statistics from there.\\n    lookback = int((array.shape[1] - 1) / 2)\\n    norm_factor = np.repeat(array[:,lookback].reshape(-1, 1), array.shape[1], axis=1)\\n    centered_data = array / norm_factor - 1\\n    lookforward = centered_data.shape[1] - lookback\\n    means = centered_data.mean(axis=0)\\n    lookforward_data = centered_data[:,lookforward:]\\n    std_dev = np.hstack([0, lookforward_data.std(axis=0)])\\n    maxes = lookforward_data.max(axis=0)\\n    mins = lookforward_data.min(axis=0)\\n    \\n    range_begin = -lookback\\n    range_end = lookforward\\n    plt.plot(range(range_begin, range_end), means)\\n    plt.fill_between(range(0, range_end),\\n                     means[-lookforward:] + std_dev,\\n                     means[-lookforward:] - std_dev,\\n                    alpha=.5, label=\\"$\\\\pm$ 1 s.d.\\")\\n    \\n    max_err = maxes - means[-lookforward+1:]\\n    min_err = means[-lookforward+1:] - mins\\n    plt.errorbar(range(1, range_end),\\n                  means[-lookforward+1:],\\n                  yerr=[min_err, max_err], label=\'Max & Min\')\\n    plt.legend(loc=2)\\n    plt.xlim((-lookback-1, lookback+1))\\n    \\ndef fetch_event_data(ticker, events, horizon=5):\\n    # Use horizon+1 to account for including the day of the event,\\n    # and half-open interval - that is, for a horizon of 5,\\n    # we should be including 11 events. Additionally, using the\\n    # CustomBusinessDay means we automatically handle issues if\\n    # for example a company reports Friday afternoon - the date\\n    # calculator will turn this into a \\"Saturday\\" release, but\\n    # we effectively shift that to Monday with the logic below.\\n    td_back = TradeDay(horizon+1)\\n    td_forward = TradeDay(horizon+1)\\n    \\n    start_date = min(events) - td_back\\n    end_date = max(events) + td_forward\\n    total_data = fetch_ticker(ticker, start_date, end_date)\\n    event_data = [total_data.ix[event-td_back:event+td_forward]\\\\\\n                      [0:horizon*2+1]\\\\\\n                      [\'Adjusted Close\']\\n                  for event in events]\\n    return np.array(event_data)\\n```\\n</details>\\n\\n```python\\n# Generate a couple of random events\\n\\nevent_dates = [datetime(2016, 5, 27) - timedelta(days=1) - TradeDay(x*20) for x in range(1, 40)]\\ndata = fetch_event_data(\'CELG\', event_dates)\\nplot_study_small(data)\\nplt.legend(loc=3)\\nplt.gcf().set_size_inches(12, 6);\\n\\n\\nplt.annotate(\'Mean price for days leading up to each event\',\\n             (-5, -.01), (-4.5, .025),\\n             arrowprops=dict(facecolor=\'black\', shrink=0.05))\\nplt.annotate(\'\', (-.1, .005), (-.5, .02),\\n             arrowprops={\'facecolor\': \'black\', \'shrink\': .05})\\nplt.annotate(\'$\\\\pm$ 1 std. dev. each day\', (5, .055), (2.5, .085),\\n            arrowprops={\'facecolor\': \'black\', \'shrink\': .05})\\nplt.annotate(\'Min/Max each day\', (.9, -.07), (-1, -.1),\\n            arrowprops={\'facecolor\': \'black\', \'shrink\': .05});\\n```\\n\\n\\n    \\n![png](_notebook_files/_notebook_11_0.png)\\n    \\n\\n\\nAnd as a quick textual explanation as well:\\n\\n- The blue line represents the mean price for each day, represented as a percentage of the price on the \'0-day\'. For example, if we defined an \'event\' as whenever the stock price dropped for three days, we would see a decreasing blue line to the left of the 0-day.\\n- The blue shaded area represents one standard deviation above and below the mean price for each day following an event. This is intended to give us an idea of what the stock price does in general following an event.\\n- The green bars are the minimum and maximum price for each day following an event. This instructs us as to how much it\'s possible for the stock to move.\\n\\n## Event Type 1: Trending down over the past N days\\n\\nThe first type of event I want to study is how stocks perform when they\'ve been trending down over the past couple of days prior to a release. However, we need to clarify what exactly is meant by \\"trending down.\\" To do so, we\'ll use the following metric: **the midpoint between each day\'s opening and closing price goes down over a period of N days**.\\n\\nIt\'s probably helpful to have an example:\\n\\n<details>\\n<summary>Code</summary>\\n```python\\nf, axarr = plt.subplots(1, 2)\\nf.set_size_inches(18, 6)\\n\\nFB_plot = axarr[0]\\nohlc_dataframe(FB[datetime(2016, 4, 18):], FB_plot)\\n\\nFB_truncated = FB[datetime(2016, 4, 18):datetime(2016, 4, 27)]\\nmidpoint = FB_truncated[\'Open\']/2 + FB_truncated[\'Close\']/2\\nFB_plot.plot(FB_truncated.index, midpoint, label=\'Midpoint\')\\nFB_plot.vlines(date2num(datetime(2016, 4, 27, 12)),\\n               ax_fb.get_ylim()[0], ax_fb.get_ylim()[1],\\n               color=\'g\', label=\'Earnings Release\')\\nFB_plot.legend(loc=2)\\nFB_plot.set_title(\'FB Midpoint Plot\')\\n\\nAAPL_plot = axarr[1]\\nohlc_dataframe(AAPL[datetime(2016, 4, 10):], AAPL_plot)\\nAAPL_truncated = AAPL[datetime(2016, 4, 10):datetime(2016, 4, 26)]\\nmidpoint = AAPL_truncated[\'Open\']/2 + AAPL_truncated[\'Close\']/2\\nAAPL_plot.plot(AAPL_truncated.index, midpoint, label=\'Midpoint\')\\nAAPL_plot.vlines(date2num(datetime(2016, 4, 26, 12)),\\n                 ax_aapl.get_ylim()[0], ax_aapl.get_ylim()[1],\\n                 color=\'g\', label=\'Earnings Release\')\\nAAPL_plot.legend(loc=3)\\nAAPL_plot.set_title(\'AAPL Midpoint Plot\');\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_14_0.png)\\n\\nGiven these charts, we can see that FB was trending down for the four days preceding the earnings release, and AAPL was trending down for a whopping 8 days (we don\'t count the peak day). This will define the methodology that we will use for the study.\\n\\nSo what are the results? For a given horizon, how well does the market actually perform?\\n\\n<details>\\n<summary>Code</summary>\\n\\n```python\\n# Read in the events for each stock;\\n# The file was created using the first code block in the Appendix\\nimport yaml\\nfrom dateutil.parser import parse\\nfrom progressbar import ProgressBar\\n\\ndata_str = open(\'earnings_dates.yaml\', \'r\').read()\\n# Need to remove invalid lines\\nfiltered = filter(lambda x: \'{\' not in x, data_str.split(\'\\\\n\'))\\nearnings_data = yaml.load(\'\\\\n\'.join(filtered))\\n\\n# Convert our earnings data into a list of (ticker, date) pairs\\n# to make it easy to work with.\\n# This is horribly inefficient, but should get us what we need\\nticker_dates = []\\nfor ticker, date_list in earnings_data.items():\\n    for iso_str in date_list:\\n        ticker_dates.append((ticker, parse(iso_str)))\\n\\ndef does_trend_down(ticker, event, horizon):\\n    # Figure out if the `event` has a downtrend for\\n    # the `horizon` days preceding it\\n    # As an interpretation note: it is assumed that\\n    # the closing price of day `event` is the reference\\n    # point, and we want `horizon` days before that.\\n    # The price_data.hdf was created in the second appendix code block\\n    try:\\n        ticker_data = pd.read_hdf(\'price_data.hdf\', ticker)\\n        data = ticker_data[event-TradeDay(horizon):event]\\n        midpoints = data[\'Open\']/2 + data[\'Close\']/2\\n\\n        # Shift dates one forward into the future and subtract\\n        # Effectively: do we trend down over all days?\\n        elems = midpoints - midpoints.shift(1)\\n        return len(elems)-1 == len(elems.dropna()[elems <= 0])\\n    except KeyError:\\n        # If the stock doesn\'t exist, it doesn\'t qualify as trending down\\n        # Mostly this is here to make sure the entire analysis doesn\'t \\n        # blow up if there were issues in data retrieval\\n        return False\\n\\ndef study_trend(horizon, trend_function):\\n    five_day_events = np.zeros((1, horizon*2 + 1))\\n    invalid_events = []\\n    for ticker, event in ProgressBar()(ticker_dates):\\n        if trend_function(ticker, event, horizon):\\n            ticker_data = pd.read_hdf(\'price_data.hdf\', ticker)\\n            event_data = ticker_data[event-TradeDay(horizon):event+TradeDay(horizon)][\'Close\']\\n\\n            try:\\n                five_day_events = np.vstack([five_day_events, event_data])\\n            except ValueError:\\n                # Sometimes we don\'t get exactly the right number of values due to calendar\\n                # issues. I\'ve fixed most everything I can, and the few issues that are left\\n                # I assume don\'t systemically bias the results (i.e. data could be missing\\n                # because it doesn\'t exist, etc.). After running through, ~1% of events get\\n                # discarded this way\\n                invalid_events.append((ticker, event))\\n            \\n\\n    # Remove our initial zero row\\n    five_day_events = five_day_events[1:,:]\\n    plot_study(five_day_events)\\n    plt.gcf().suptitle(\'Action over {} days: {} events\'\\n                       .format(horizon,five_day_events.shape[0]))\\n    plt.gcf().set_size_inches(18, 6)\\n    \\n# Start with a 5 day study\\nstudy_trend(5, does_trend_down)\\n```\\n\\n```\\n    100% (47578 of 47578) |###########################################################| Elapsed Time: 0:21:38 Time: 0:21:38\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_16_1.png)\\n\\nWhen a stock has been trending down for 5 days, once the earnings are announced it really doesn\'t move on average. However, the variability is *incredible*. This implies two important things:\\n\\n1. The market is just as often wrong about an earnings announcement before it happens as it is correct\\n2. The incredible width of the min/max bars and standard deviation area tell us that the market reacts *violently* after the earnings are released.\\n\\nLet\'s repeat the same study, but over a time horizon of 8 days and 3 days. Presumably if a stock has been going down for 8 days at a time before the earnings, the market should be more accurate.\\n\\n<details>\\n<summary>Code</summary>\\n\\n```python\\n# 8 day study next\\nstudy_trend(8, does_trend_down)\\n```\\n\\n```\\n    100% (47578 of 47578) |###########################################################| Elapsed Time: 0:20:29 Time: 0:20:29\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_18_1.png)\\n\\nHowever, looking only at stocks that trended down for 8 days prior to a release, the same pattern emerges: on average, the stock doesn\'t move, but the market reaction is often incredibly violent.\\n\\n\\n<details>\\n<summary>Code</summary>\\n```python\\n# 3 day study after that\\nstudy_trend(3, does_trend_down)\\n```\\n\\n```\\n    100% (47578 of 47578) |###########################################################| Elapsed Time: 0:26:26 Time: 0:26:26\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_20_1.png)\\n\\nFinally, when we look at a 3-day horizon, we start getting some incredible outliers. Stocks have a potential to move over ~300% up, and the standard deviation width is again, incredible. The results for a 3-day horizon follow the same pattern we\'ve seen in the 5- and 8-day horizons.\\n\\n## Event Type 2: Trending up for N days\\n\\nWe\'re now going to repeat the analysis, but do it for uptrends instead. That is, instead of looking at stocks that have been trending down over the past number of days, we focus only on stocks that have been trending up.\\n\\n<details>\\n<summary>Code</summary>\\n```python\\ndef does_trend_up(ticker, event, horizon):\\n    # Figure out if the `event` has an uptrend for\\n    # the `horizon` days preceding it\\n    # As an interpretation note: it is assumed that\\n    # the closing price of day `event` is the reference\\n    # point, and we want `horizon` days before that.\\n    # The price_data.hdf was created in the second appendix code block\\n    try:\\n        ticker_data = pd.read_hdf(\'price_data.hdf\', ticker)\\n        data = ticker_data[event-TradeDay(horizon):event]\\n        midpoints = data[\'Open\']/2 + data[\'Close\']/2\\n\\n        # Shift dates one forward into the future and subtract\\n        # Effectively: do we trend down over all days?\\n        elems = midpoints - midpoints.shift(1)\\n        return len(elems)-1 == len(elems.dropna()[elems >= 0])\\n    except KeyError:\\n        # If the stock doesn\'t exist, it doesn\'t qualify as trending down\\n        # Mostly this is here to make sure the entire analysis doesn\'t \\n        # blow up if there were issues in data retrieval\\n        return False\\n\\nstudy_trend(5, does_trend_up)\\n```\\n\\n```\\n    100% (47578 of 47578) |###########################################################| Elapsed Time: 0:22:51 Time: 0:22:51\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_23_1.png)\\n\\nThe patterns here are very similar. With the exception of noting that stocks can go to nearly 400% after an earnings announcement (most likely this included a takeover announcement, etc.), we still see large min/max bars and wide standard deviation of returns.\\n\\nWe\'ll repeat the pattern for stocks going up for both 8 and 3 days straight, but at this point, the results should be very predictable:\\n\\n<details>\\n<summary>Code</summary>\\n```python\\nstudy_trend(8, does_trend_up)\\n```\\n\\n```\\n    100% (47578 of 47578) |###########################################################| Elapsed Time: 0:20:51 Time: 0:20:51\\n```\\n</details>\\n    \\n![png](_notebook_files/_notebook_25_1.png)\\n\\n<details>\\n<summary>Code</summary>\\n```python\\nstudy_trend(3, does_trend_up)\\n```\\n\\n```\\n    100% (47578 of 47578) |###########################################################| Elapsed Time: 0:26:56 Time: 0:26:56\\n```\\n</details>\\n\\n![png](_notebook_files/_notebook_26_1.png)\\n\\n## Conclusion and Summary\\n\\nI guess the most important thing to summarize with is this: **looking at the entire market, stock performance prior to an earnings release has no bearing on the stock\'s performance.** Honestly: given the huge variability of returns after an earnings release, even when the stock has been trending for a long time, you\'re best off divesting before an earnings release and letting the market sort itself out.\\n\\n*However*, there is a big caveat. These results are taken when we look at the entire market. So while we can say that the market as a whole knows nothing and just reacts violently, I want to take a closer look into this data. Does the market typically perform poorly on large-cap/high liquidity stocks? Do smaller companies have investors that know them better and can thus predict performance better? Are specific market sectors better at prediction? Presumably technology stocks are more volatile than the industrials.\\n\\nSo there are some more interesting questions I still want to ask with this data. Knowing that the hard work of data processing is largely already done, it should be fairly simple to continue this analysis and get much more refined with it. Until next time.\\n\\n# Appendix\\n\\nExport event data for Russell 3000 companies:\\n\\n<details>\\n<summary>Code</summary>\\n```python\\nimport pandas as pd\\nfrom html.parser import HTMLParser\\nfrom datetime import datetime, timedelta\\nimport requests\\nimport re\\nfrom dateutil import parser\\nimport progressbar\\nfrom concurrent import futures\\nimport yaml\\n\\nclass EarningsParser(HTMLParser):\\n    store_dates = False\\n    earnings_offset = None\\n    dates = []\\n    \\n    def __init__(self, *args, **kwargs):\\n        super().__init__(*args, **kwargs)\\n        self.dates = []\\n    \\n    def handle_starttag(self, tag, attrs):\\n        if tag == \'table\':\\n            self.store_dates = True\\n            \\n    def handle_data(self, data):\\n        if self.store_dates:\\n            match = re.match(r\'\\\\d+/\\\\d+/\\\\d+\', data)\\n            if match:\\n                self.dates.append(match.group(0))\\n        \\n        # If a company reports before the bell, record the earnings date\\n        # being at midnight the day before. Ex: WMT reports 5/19/2016,\\n        # but we want the reference point to be the closing price on 5/18/2016\\n        if \'After Close\' in data:\\n            self.earnings_offset = timedelta(days=0)\\n        elif \'Before Open\' in data:\\n            self.earnings_offset = timedelta(days=-1)\\n                \\n    def handle_endtag(self, tag):\\n        if tag == \'table\':\\n            self.store_dates = False\\n            \\ndef earnings_releases(ticker):\\n    #print(\\"Looking up ticker {}\\".format(ticker))\\n    user_agent = \'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) \'\\\\\\n        \'Gecko/20100101 Firefox/46.0\'\\n    headers = {\'user-agent\': user_agent}\\n    base_url = \'http://www.streetinsider.com/ec_earnings.php?q={}\'\\\\\\n        .format(ticker)\\n    e = EarningsParser()\\n    s = requests.Session()\\n    a = requests.adapters.HTTPAdapter(max_retries=0)\\n    s.mount(\'http://\', a)\\n    e.feed(str(s.get(base_url, headers=headers).content))\\n    \\n    if e.earnings_offset is not None:\\n        dates = map(lambda x: parser.parse(x) + e.earnings_offset, e.dates)\\n        past = filter(lambda x: x < datetime.now(), dates)\\n        return list(map(lambda d: d.isoformat(), past))\\n\\n# Use a Russell-3000 ETF tracker (ticker IWV) to get a list of holdings\\nr3000 = pd.read_csv(\'https://www.ishares.com/us/products/239714/\'\\n                    \'ishares-russell-3000-etf/1449138789749.ajax?\'\\n                    \'fileType=csv&fileName=IWV_holdings&dataType=fund\',\\n                    header=10)\\nr3000_equities = r3000[(r3000[\'Exchange\'] == \'NASDAQ\') |\\n                       (r3000[\'Exchange\'] == \'New York Stock Exchange Inc.\')]\\n\\ndates_file = open(\'earnings_dates.yaml\', \'w\')\\n\\nwith futures.ThreadPoolExecutor(max_workers=8) as pool:\\n    fs = {pool.submit(earnings_releases, r3000_equities.ix[t][\'Ticker\']): t\\n          for t in r3000_equities.index}\\n    pbar = progressbar.ProgressBar(term_width=80,\\n                                   max_value=r3000_equities.index.max())\\n    \\n    for future in futures.as_completed(fs):\\n        i = fs[future]\\n        pbar.update(i)\\n        dates_file.write(yaml.dump({r3000_equities.ix[i][\'Ticker\']:\\n            future.result()}))\\n```\\n</details>\\n\\nDownloading stock price data needed for the event studies:\\n\\n<details>\\n<summary>Code</summary>\\n```python\\nfrom secrets import QUANDL_KEY\\nimport pandas as pd\\nimport yaml\\nfrom dateutil.parser import parse\\nfrom datetime import timedelta\\nimport quandl\\nfrom progressbar import ProgressBar\\n\\ndef fetch_ticker(ticker, start, end):\\n    # Quandl is currently giving me issues with returning\\n    # the entire dataset and not slicing server-side.\\n    # So instead, we\'ll do it client-side!\\n    q_format = \'%Y-%m-%d\'\\n    ticker_data = quandl.get(\'YAHOO/\' + ticker,\\n                             start_date=start.strftime(q_format),\\n                             end_date=end.strftime(q_format),\\n                             authtoken=QUANDL_KEY)\\n    return ticker_data\\n    \\ndata_str = open(\'earnings_dates.yaml\', \'r\').read()\\n# Need to remove invalid lines\\nfiltered = filter(lambda x: \'{\' not in x, data_str.split(\'\\\\n\'))\\nearnings_data = yaml.load(\'\\\\n\'.join(filtered))\\n\\n# Get the first 1500 keys - split up into two statements\\n# because of Quandl rate limits\\ntickers = list(earnings_data.keys())\\n\\nprice_dict = {}\\ninvalid_tickers = []\\nfor ticker in ProgressBar()(tickers[0:1500]):\\n    try:\\n        # Replace \'.\' with \'-\' in name for some tickers\\n        fixed = ticker.replace(\'.\', \'-\')\\n        event_strs = earnings_data[ticker]\\n        events = [parse(event) for event in event_strs]\\n        td = timedelta(days=20)\\n        price_dict[ticker] = fetch_ticker(fixed,\\n            min(events)-td, max(events)+td)\\n    except quandl.NotFoundError:\\n        invalid_tickers.append(ticker)\\n        \\n# Execute this after 10 minutes have passed\\nfor ticker in ProgressBar()(tickers[1500:]):\\n    try:\\n        # Replace \'.\' with \'-\' in name for some tickers\\n        fixed = ticker.replace(\'.\', \'-\')\\n        event_strs = earnings_data[ticker]\\n        events = [parse(event) for event in event_strs]\\n        td = timedelta(days=20)\\n        price_dict[ticker] = fetch_ticker(fixed,\\n            min(events)-td, max(events)+td)\\n    except quandl.NotFoundError:\\n        invalid_tickers.append(ticker)\\n \\nprices_store = pd.HDFStore(\'price_data.hdf\')\\nfor ticker, prices in price_dict.items():\\n    prices_store[ticker] = prices\\n```\\n</details>"},{"id":"2016/05/the-unfair-casino","metadata":{"permalink":"/2016/05/the-unfair-casino","source":"@site/blog/2016-05-15-the-unfair-casino/index.mdx","title":"The unfair casino","description":"Trying to figure out how exactly two dice are loaded in a cheating casino.","date":"2016-05-15T12:00:00.000Z","tags":[],"readingTime":14.62,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/05/the-unfair-casino","title":"The unfair casino","date":"2016-05-15T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730679360000,"prevItem":{"title":"Event studies and earnings releases","permalink":"/2016/06/event-studies-and-earnings-releases"},"nextItem":{"title":"Tick tock...","permalink":"/2016/04/tick-tock"}},"content":"Trying to figure out how exactly two dice are loaded in a cheating casino.\\n\\n\x3c!-- truncate --\x3e\\n\\nIn the ongoing eternal cycle of mathematicians asking generally useless questions about probability, I dreamt up another one. The scenario is as follows:\\n\\n**You\'re playing a game with two die, and you do not get to see what the outcome of the die are on each roll. All you get to see is their sum. Given an arbitrarily long list of the sum of two rolls, can you determine if one or both die are loaded, and what those loadings are?**\\n\\n## Proving we can detect cheating\\n\\nMy first question is simply, is this possible? There\'s a lot of trivial cases that make it obvious that there\'s cheating going on. But there are some edge cases that might give us more difficulty. First though, let\'s get a picture of what the fair distribution looks like. In principle, we can only detect cheating if the distribution of the fair die differs from the distribution of the loaded die.\\n\\n\\n```python\\nimport numpy as np\\nimport pandas as pd\\nimport matplotlib.pyplot as plt\\n%matplotlib inline\\n\\nfair_1 = np.random.randint(1, 7, 10000)\\nfair_2 = np.random.randint(1, 7, 10000)\\n\\npd.Series(fair_1 + fair_2).plot(kind=\'hist\', bins=11);\\nplt.title(\'Fair Distribution\');\\n```\\n    \\n![png](_notebook_files/_notebook_1_0.png)\\n\\nThis distribution makes sense: there are many ways to make a 7 (the most frequent observed value) and very few ways to make a 12 or 2; an important symmetry. As a special note, you can notice that the sum of two fair dice is a discrete case of the [Triangle Distribution][1], which is itself a special case of the [Irwin-Hall Distribution][2].\\n\\n## The Edge Cases\\n\\nGiven that we understand how the results of two fair dice are distributed, let\'s see some of the interesting edge cases that come up. This will give us assurance that when a casino is cheating, it is detectable (given sufficient data). To make this as hard as possible, we will think of scenarios where the expected value of the sum of loaded dice is the same as the expected value of the sum of fair dice.\\n\\n### Edge Case 1\\nWhat happens when one die is biased low, and one die is biased high? That is, where:\\n\\n$$\\n\\\\begin{align*}\\n\\\\begin{array}{cc}\\nD_1 = \\\\left\\\\{\\n\\\\begin{array}{lr}\\n1 & w.p. 1/3\\\\\\\\\\n2 & w.p. 1/3\\\\\\\\\\n3 & w.p. 1/12\\\\\\\\\\n4 & w.p. 1/12\\\\\\\\\\n5 & w.p. 1/12\\\\\\\\\\n6 & w.p. 1/12\\n\\\\end{array}\\n\\\\right. &\\nD_2 = \\\\left\\\\{\\n\\\\begin{array}{lr}\\n1 & w.p. 1/12\\\\\\\\\\n2 & w.p. 1/12\\\\\\\\\\n3 & w.p. 1/12\\\\\\\\\\n4 & w.p. 1/12\\\\\\\\\\n5 & w.p. 1/3\\\\\\\\\\n6 & w.p. 1/3\\n\\\\end{array}\\n\\\\right. \\\\\\\\\\n\\\\mathbb{E}[D_1] = 2.5 & \\\\mathbb{E}[D_2] = 4.5\\n\\\\end{array}\\\\\\\\\\n\\\\mathbb{E}[D_1 + D_2] = 7 = \\\\mathbb{E}[D_{fair} + D_{fair}]\\n\\\\end{align*}\\n$$\\n\\n[1]: https://en.wikipedia.org/wiki/Triangular_distribution\\n[2]: https://en.wikipedia.org/wiki/Irwin%E2%80%93Hall_distribution\\n\\n\\n```python\\ndef unfair_die(p_vals, n):\\n    x = np.random.multinomial(1, p_vals, n)\\n    return x.nonzero()[1] + 1\\n\\nd1 = [1/3, 1/3, 1/12, 1/12, 1/12, 1/12]\\nd2 = [1/12, 1/12, 1/12, 1/12, 1/3, 1/3]\\n\\nx1 = unfair_die(d1, 10000)\\nx2 = unfair_die(d2, 10000)\\n\\npd.Series(x1 + x2).plot(kind=\'hist\', bins=11);\\nplt.title(\'$D_1$ biased low, $D_2$ biased high\');\\n```\\n\\n![png](_notebook_files/_notebook_3_0.png)\\n\\nWe can see that while the 7 value remains the most likely (as expected), the distribution is not so nicely shaped any more. \\n\\n### Edge Case 2\\n\\nWhen one die is loaded low, and one is loaded high, we\'ve seen how we can detect them. How about when two die are loaded both low and high? That is, we have the following distribution:\\n\\n$$\\n\\\\begin{align*}\\n\\\\begin{array}{cc}\\nD_1 = \\\\left\\\\{\\n\\\\begin{array}{lr}\\n1 & w.p. 1/3\\\\\\\\\\n2 & w.p. 1/12\\\\\\\\\\n3 & w.p. 1/12\\\\\\\\\\n4 & w.p. 1/12\\\\\\\\\\n5 & w.p. 1/12\\\\\\\\\\n6 & w.p. 1/3\\n\\\\end{array}\\n\\\\right. &\\nD_2 = \\\\left\\\\{\\n\\\\begin{array}{lr}\\n1 & w.p. 1/3\\\\\\\\\\n2 & w.p. 1/12\\\\\\\\\\n3 & w.p. 1/12\\\\\\\\\\n4 & w.p. 1/12\\\\\\\\\\n5 & w.p. 1/12\\\\\\\\\\n6 & w.p. 1/3\\n\\\\end{array}\\n\\\\right. \\\\\\\\\\n\\\\mathbb{E}[D_1] = 3.5 & \\\\mathbb{E}[D_2] = 3.5\\n\\\\end{array}\\\\\\\\\\n\\\\mathbb{E}[D_1 + D_2] = 7 = \\\\mathbb{E}[D_{fair} + D_{fair}]\\n\\\\end{align*}\\n$$\\n\\nWe can see even that the expected value of each individual die is the same as the fair die! However, the distribution (if we are doing this correctly) should still be skewed:\\n\\n```python\\nd1 = [1/3, 1/12, 1/12, 1/12, 1/12, 1/3]\\nd2 = d1\\n\\nx1 = unfair_die(d1, 10000)\\nx2 = unfair_die(d2, 10000)\\n\\npd.Series(x1 + x2).plot(kind=\'hist\', bins=11)\\nplt.title(\\"$D_1$ and $D_2$ biased to 1 and 6\\");\\n```\\n    \\n![png](_notebook_files/_notebook_5_0.png)\\n\\nIn a very un-subtle way, we have of course made the values 2 and 12 far more likely.\\n\\n## Detection Conclusion\\n\\nThere are some trivial examples of cheating that are easy to detect: whenever the expected value of the sum of two fair dice deviates from the expected value for the sum of two fair dice, we can immediately conclude that there is cheating at stake.\\n\\nThe interesting edge cases occur when the expected value of the sum of loaded dice matches the expected value of the sum of fair dice. Considering the above examples (and a couple more I ran through in developing this), we have seen that in every circumstance having two unfair dice leads to a distribution of results different from the fair results.\\n\\nWe can thus finally state: **just by looking at the distribution of results from this game, we can immediately conclude whether there is cheating.**\\n\\n## Simulated Annealing\\n\\nWhat we really would like to do though, is see if there is any way to determine how exactly the dice are loaded. This is significantly more complicated, but we can borrow some algorithms from Machine Learning to figure out exactly how to perform this process. I\'m using the Simulated Annealing algorithm, and I discuss why this works and why I chose it over some of the alternatives in the [justification](#Justification-of-Simulated-Annealing). If you don\'t care about how I set up the model and just want to see the code, check out [the actual code](#The-actual-code).\\n\\n[Simulated Annealing][3] is a variation of the [Metropolis-Hastings Algorithm][4], but the important thing for us is: Simulated Annealing allows us to quickly optimize high-dimensional problems. But what exactly are we trying to optimize? Ideally, we want a function that can tell us whether one distribution for the dice better explains the results than another distribution. This is known as the **likelihood** function.\\n\\n### Deriving the Likelihood function\\n\\nTo derive our likelihood function, we want to know: **what is the probability of seeing a specific result given those hidden parameters?** This is actually a surprisingly difficult problem. While we can do a lot of calculations by hand, we need a more general solution since we will be working with very some interesting die distributions.\\n\\nWe first note that the sum of two dice can take on 11 different values - 2 through 12. This implies that each individual sum follows a [Categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution). That is:\\n\\n$$\\n\\\\begin{align*}\\n\\\\mathcal{L(x)} = \\\\left\\\\{\\n\\\\begin{array}{lr}\\np_2 & x = 2\\\\\\\\\\np_3 & x = 3\\\\\\\\\\n\\\\ldots & \\\\\\\\\\np_{11} & x = 11\\\\\\\\\\np_{12} & x = 12\\n\\\\end{array}\\n\\\\right.\\n\\\\end{align*}\\n$$\\n\\nWhere each $p_i$ is the probability of seeing that specific result. However, we need to calculate what each probability is! I\'ll save you the details, but [this author](http://math.stackexchange.com/a/1646360/320784) explains how to do it.\\n\\nNow, we would like to know the likelihood of our entire data-set. This is trivial:\\n\\n$$\\n\\\\begin{align*}\\n\\\\mathcal{L(\\\\mathbf{X})} &= \\\\prod_{i=1}^n L(x)\\n\\\\end{align*}\\n$$\\n\\nHowever, it\'s typically much easier to work with the $\\\\log(\\\\mathcal{L})$ function instead. This is critically important from a computational perspective: when you multiply so many small numbers together (i.e. the product of $L(x)$ terms) the computer suffers from rounding error; if we don\'t control for this, we will find that no matter the distributions we choose for each die, the \\"likelihood\\" will be close to zero because the computer is not precise enough.\\n\\n$$\\n\\\\begin{align*}\\n\\\\log(\\\\mathcal{L}) &= \\\\sum_{i=1}^n \\\\log(L)\\n\\\\end{align*}\\n$$\\n\\n### The process of Simulated Annealing\\n\\nThe means by which we optimize our likelihood function is the simulated annealing algorithm. The way it works is as follows:\\n\\n1. Start with a random guess for the parameters we are trying to optimize. In our case we are trying to guess the distribution of two dice, and so we \\"optimize\\" until we have a distribution that matches the data.\\n\\n2. For each iteration of the algorithm:\\n\\n    1. Generate a new \\"proposed\\" set of parameters based on the current parameters -\\n    i.e. slightly modify the current parameters to get a new set of parameters.\\n    2. Calculate the value of $\\\\log(\\\\mathcal{L})$ for each set of parameters. If the function value for the\\n    proposed parameter set is higher than for the current, automatically switch to the new parameter set\\n    and continue the next iteration.\\n    3. Given the new parameter set performs worse, determine a probability of switching to the new parameter set anyways: $\\\\mathcal{P}(p_{current}, p_{proposed})$\\n    4. Switch to the new parameter set with probability $\\\\mathcal{P}$. If you fail to switch, begin the next iteration.\\n    \\n3. The algorithm is complete after we fail to make a transition $n$ times in a row.\\n\\nIf everything goes according to plan, we will have a value that is close to the true distribution of each die.\\n\\n## The actual code\\n\\nWe start by defining the score function. This will tell us how well the proposed die densities actually explain the results.\\n\\n[3]:https://en.wikipedia.org/wiki/Simulated_annealing\\n[4]:https://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm\\n\\n\\n```python\\nimport numpy as np\\nfrom numpy import polynomial\\n\\ndef density_coef(d1_density, d2_density):\\n    # Calculating the probabilities of each outcome was taken\\n    # from this author: http://math.stackexchange.com/a/1710392/320784\\n    d1_p = polynomial.Polynomial(d1_density)\\n    d2_p = polynomial.Polynomial(d2_density)\\n    coefs = (d1_p * d2_p).coef\\n    return coefs\\n\\ndef score(x, d1_density, d2_density):\\n    # We\'ve now got the probabilities of each event, but we need\\n    # to shift the array a bit so we can use the x values to actually\\n    # index into it. This will allow us to do all the calculations\\n    # incredibly quickly\\n    coefs = density_coef(d1_density, d2_density)\\n    coefs = np.hstack((0, 0, coefs))\\n    return np.log(coefs[x]).sum()\\n```\\n\\nAfterward, we need to write something to permute the proposal densities. We make random modifications, and eventually the best one survives.\\n\\n\\n```python\\ndef permute(d1_density, d2_density):\\n    # To ensure we have legitimate densities, we will randomly\\n    # increase one die face probability by `change`,\\n    # and decrease one by `change`.\\n    # This means there are something less than (1/`change`)^12 possibilities\\n    # we are trying to search over.\\n    change = .01\\n    \\n    d1_index1, d1_index2 = np.random.randint(0, 6, 2)\\n    d2_index1, d2_index2 = np.random.randint(0, 6, 2)\\n    \\n    # Also make sure to copy. I\'ve had some weird aliasing issues\\n    # in the past that made everything blow up.\\n    new_d1 = np.float64(np.copy(d1_density))\\n    new_d2 = np.float64(np.copy(d2_density))\\n    \\n    # While this doesn\'t account for the possibility that some\\n    # values go negative, in practice this never happens\\n    new_d1[d1_index1] += change\\n    new_d1[d1_index2] -= change\\n    new_d2[d2_index1] += change\\n    new_d2[d2_index2] -= change\\n    \\n    return new_d1, new_d2\\n```\\n\\nNow we\'ve got the main algorithm code to do. This is what brings all the pieces together.\\n\\n\\n```python\\ndef optimize(data, conv_count=10, max_iter=1e4):\\n    switch_failures = 0\\n    iter_count = 0\\n    \\n    # Start with guessing fair dice\\n    cur_d1 = np.repeat(1/6, 6)\\n    cur_d2 = np.repeat(1/6, 6)\\n    cur_score = score(data, cur_d1, cur_d2)\\n    \\n    # Keep track of our best guesses - may not be\\n    # what we end with\\n    max_score = cur_score\\n    max_d1 = cur_d1\\n    max_d2 = cur_d2\\n    \\n    # Optimization stops when we have failed to switch `conv_count`\\n    # times (presumably because we have a great guess), or we reach\\n    # the maximum number of iterations.\\n    while switch_failures < conv_count and iter_count < max_iter:\\n        iter_count += 1\\n        if iter_count % (max_iter / 10) == 0:\\n            print(\'Iteration: {}; Current score (higher is better): {}\'.format(\\n                    iter_count, cur_score))\\n                \\n        new_d1, new_d2 = permute(cur_d1, cur_d2)\\n        new_score = score(data, new_d1, new_d2)\\n        \\n        if new_score > max_score:\\n            max_score = new_score\\n            max_d1 = new_d1\\n            max_d2 = new_d2\\n            \\n        if new_score > cur_score:\\n            # If the new permutation beats the old one,\\n            # automatically select it.\\n            cur_score = new_score\\n            cur_d1 = new_d1\\n            cur_d2 = new_d2\\n            switch_failures = 0\\n        else:\\n            # We didn\'t beat the current score, but allow\\n            # for possibly switching anyways.\\n            accept_prob = np.exp(new_score - cur_score)\\n            coin_toss = np.random.rand()\\n            if coin_toss < accept_prob:\\n                # We randomly switch to the new distribution\\n                cur_score = new_score\\n                cur_d1 = new_d1\\n                cur_d2 = new_d2\\n                switch_failures = 0\\n            else:\\n                switch_failures += 1\\n            \\n    # Return both our best guess, and the ending guess\\n    return max_d1, max_d2, cur_d1, cur_d2\\n```\\n\\nAnd now we have finished the hard work!\\n\\n## Catching the Casino\\n\\nLet\'s go through a couple of scenarios and see if we can catch the casino cheating with some loaded dice. **In every scenario we start with an assumption of fair dice**, and then try our hand to figure out what the *actual* distribution was.\\n\\n### Attempt 1\\n\\n\\nThe casino is using two dice that are both biased low. How well can we recover the distribution?\\n\\n\\n```python\\nimport time\\ndef simulate_casino(d1_dist, d2_dist, n=10000):\\n    d1_vals = unfair_die(d1_dist, n)\\n    d2_vals = unfair_die(d2_dist, n)\\n    \\n    start = time.perf_counter()\\n    max_d1, max_d2, final_d1, final_d2 = optimize(d1_vals + d2_vals)\\n    end = time.perf_counter()\\n    print(\\"Simulated Annealing time: {:.02f}s\\".format(end - start))\\n    \\n    coef_range = np.arange(2, 13) - .5\\n    plt.subplot(221)\\n    plt.bar(coef_range, density_coef(d1_dist, d2_dist), width=1)\\n    plt.title(\'True Distribution\')\\n    \\n    plt.subplot(222)\\n    plt.hist(d1_vals + d2_vals, bins=11)\\n    plt.title(\'Empirical Distribution\')\\n    \\n    plt.subplot(223)\\n    plt.bar(coef_range, density_coef(max_d1, max_d2), width=1)\\n    plt.title(\'Recovered Distribution\')\\n    \\n    plt.gcf().set_size_inches(10, 10)\\n\\n\\nsimulate_casino([2/9, 2/9, 2/9, 1/9, 1/9, 1/9],\\n                [2/9, 2/9, 2/9, 1/9, 1/9, 1/9])\\n```\\n\\n```\\n    Iteration: 1000; Current score (higher is better): -22147.004400281654\\n    Simulated Annealing time: 0.30s\\n```\\n    \\n![png](_notebook_files/_notebook_14_1.png)\\n\\n### Attempt 2\\n\\nThe casino now uses dice that are both biased towards 1 and 6.\\n\\n```python\\nsimulate_casino([1/3, 1/12, 1/12, 1/12, 1/12, 1/3],\\n                [1/3, 1/12, 1/12, 1/12, 1/12, 1/3])\\n```\\n\\n```\\n    Simulated Annealing time: 0.08s\\n```\\n    \\n![png](_notebook_files/_notebook_16_1.png)\\n\\n### Attempt 3\\n\\nThe casino will now use one die biased towards 1 and 6, and one die towards 3 and 4.\\n\\n\\n```python\\nsimulate_casino([1/3, 1/12, 1/12, 1/12, 1/12, 1/3],\\n                [1/12, 1/12, 1/3, 1/3, 1/12, 1/12])\\n```\\n\\n```\\n    Simulated Annealing time: 0.09s\\n```\\n    \\n![png](_notebook_files/_notebook_18_1.png)\\n\\n### Attempt 4\\n\\nWe\'ll now finally go to a fair casino to make sure that we can still recognize a positive result.\\n\\n```python\\nsimulate_casino(np.repeat(1/6, 6), np.repeat(1/6, 6))\\n```\\n\\n```\\n    Simulated Annealing time: 0.02s\\n```\\n    \\n![png](_notebook_files/_notebook_20_1.png)\\n\\n### Attempt 5\\n\\nWe\'ve so far been working with a large amount of data - 10,000 data points. Can we now scale things back to only 250 throws? We\'ll start with two dice biased high.\\n\\n\\n```python\\nsimulate_casino([1/9, 1/9, 1/9, 2/9, 2/9, 2/9],\\n                [1/9, 1/9, 1/9, 2/9, 2/9, 2/9],\\n                n=250)\\n```\\n\\n```\\n    Iteration: 1000; Current score (higher is better): -551.6995384525453\\n    Iteration: 2000; Current score (higher is better): -547.7803673440676\\n    Iteration: 3000; Current score (higher is better): -547.9805613193807\\n    Iteration: 4000; Current score (higher is better): -546.7574874775273\\n    Iteration: 5000; Current score (higher is better): -549.5798007672656\\n    Iteration: 6000; Current score (higher is better): -545.0354060154496\\n    Iteration: 7000; Current score (higher is better): -550.1134504086606\\n    Iteration: 8000; Current score (higher is better): -549.9306537114975\\n    Iteration: 9000; Current score (higher is better): -550.7075182119111\\n    Iteration: 10000; Current score (higher is better): -549.400679551826\\n    Simulated Annealing time: 1.94s\\n```\\n    \\n![png](_notebook_files/_notebook_22_1.png)\\n\\nThe results are surprisingly good. While the actual optimization process took much longer to finish than in the other examples, we still have a very good guess. As a caveat though: the recovered distribution tends to overfit the data. That is, if the data doesn\'t fit the underlying distribution well, the model will also fail.\\n\\n## Conclusion\\n\\nGiven the results above, we can see that we have indeed come up with a very good algorithm to determine the distribution of two dice given their results. As a benefit, we have even seen that results come back very quickly; it\'s not uncommon for the optimization to converge within a tenth of a second.\\n\\nAdditionally, we have seen that the algorithm can intuit the distribution even when there is not much data. While the final example shows that we can \'overfit\' on the dataset, we can still get valuable information from a relatively small dataset.\\n\\nWe can declare at long last: **the mathematicians have again triumphed over the casino**.\\n\\n---\\n\\n## Justification of Simulated Annealing\\n\\n### Why Simulated Annealing?\\n\\nSo why even use an algorithm with a fancy title like Simulated Annealing? First of all, because the title is sexy. Second of all, because this is a reasonably complicated problem to try and solve. We have a parameter space where each value $p_{ij} \\\\in (0, 1); i, j \\\\in \\\\{1, \\\\ldots, 6\\\\}$, for a total of 12 different variables we are trying to optimize over. Additionally, given a 12-dimensional function we are trying to optimize, simulated annealing makes sure that we don\'t fall into a local minimum.\\n\\n### Why not something else?\\n\\nThis is a fair question. There are two classes of algorithms that can also be used to solve this problem: [Non-linear optimization](https://en.wikipedia.org/wiki/Nonlinear_programming) methods, and the [EM algorithm](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm).\\n\\n1. I chose not to use non-linear optimization simply because I\'m a bit concerned that it will trap me in a local maximum. Instead of running multiple different optimizations from different starting points, I can just use simulated annealing to take that into account. In addition, throughout the course of testing the simulated annealing code converged *incredibly* quickly - far more quickly than any non-linear solver would be able to accomplish. \\n\\n2. The EM Algorithm was originally what I intended to write this blog post with. Indeed, the post was inspired by the [crooked casino](http://web.stanford.edu/class/stats366/hmmR2.html) example which uses the EM algorithm to solve it. However, after modeling the likelihood function I realized that the algebra would very quickly get out of hand. Trying to compute all the polynomial terms would not be fun, which would be needed to actually optimize for each parameter. So while the EM algorithm would likely be much faster in raw speed terms, the amount of time needed to program and verify it meant that I was far better off using a different method for optimization."},{"id":"2016/04/tick-tock","metadata":{"permalink":"/2016/04/tick-tock","source":"@site/blog/2016-04-06-tick-tock/index.mdx","title":"Tick tock...","description":"If all we have is a finite number of heartbeats left, what about me?","date":"2016-04-06T12:00:00.000Z","tags":[],"readingTime":10.125,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/04/tick-tock","title":"Tick tock...","date":"2016-04-06T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730863976000,"prevItem":{"title":"The unfair casino","permalink":"/2016/05/the-unfair-casino"},"nextItem":{"title":"Tweet like me","permalink":"/2016/03/tweet-like-me"}},"content":"If all we have is a finite number of heartbeats left, what about me?\\n\\n\x3c!-- truncate --\x3e\\n\\nWarning: this one is a bit creepier. But that\'s what you get when you come up with data science ideas as you\'re drifting off to sleep.\\n\\n## 2.5 Billion\\n\\nIf [PBS][1] is right, that\'s the total number of heartbeats we get. Approximately once every second that number goes down, and down, and down again...\\n\\n[1]: http://www.pbs.org/wgbh/nova/heart/heartfacts.html\\n\\n```python\\ntotal_heartbeats = 2500000000\\n```\\n\\nI got a Fitbit this past Christmas season, mostly because I was interested in the data and trying to work on some data science projects with it. This is going to be the first project, but there will likely be more (and not nearly as morbid). My idea was: If this is the final number that I\'m running up against, how far have I come, and how far am I likely to go? I\'ve currently had about 3 months\' time to estimate what my data will look like, so let\'s go ahead and see: given a lifetime 2.5 billion heart beats, how much time do I have left?\\n\\n## Statistical Considerations\\n\\nSince I\'m starting to work with health data, there are a few considerations I think are important before I start digging through my data.\\n\\n1. The concept of 2.5 billion as an agreed-upon number is tenuous at best. I\'ve seen anywhere from [2.21 billion][2] to [3.4 billion][3] so even if I knew exactly how many times my heart had beaten so far, the ending result is suspect at best. I\'m using 2.5 billion because that seems to be about the midpoint of the estimates I\'ve seen so far.\\n2. Most of the numbers I\'ve seen so far are based on extrapolating number of heart beats from life expectancy. As life expectancy goes up, the number of expected heart beats goes up too.\\n3. My estimation of the number of heartbeats in my life so far is based on 3 months worth of data, and I\'m extrapolating an entire lifetime based on this.\\n\\nSo while the ending number is **not useful in any medical context**, it is still an interesting project to work with the data I have on hand.\\n\\n## Getting the data\\n\\n[Fitbit](https://www.fitbit.com/) has an [API available](https://dev.fitbit.com/) for people to pull their personal data off the system. It requires registering an application, authentication with OAuth, and some other complicated things. **If you\'re not interested in how I fetch the data, skip [here](#Wild-Extrapolations-from-Small-Data)**.\\n\\n## Registering an application\\n\\nI\'ve already [registered a personal application](https://dev.fitbit.com/apps/new) with Fitbit, so I can go ahead and retrieve things like the client secret from a file.\\n\\n[1]: http://www.pbs.org/wgbh/nova/heart/heartfacts.html\\n[2]: http://gizmodo.com/5982977/how-many-heartbeats-does-each-species-get-in-a-lifetime\\n[3]: http://wonderopolis.org/wonder/how-many-times-does-your-heart-beat-in-a-lifetime/\\n\\n\\n```python\\n# Import all the OAuth secret information from a local file\\nfrom secrets import CLIENT_SECRET, CLIENT_ID, CALLBACK_URL\\n```\\n\\n### Handling OAuth 2\\n\\nSo, all the people that know what OAuth 2 is know what\'s coming next. For those who don\'t: OAuth is how people allow applications to access other data without having to know your password. Essentially the dialog goes like this:\\n\\n```\\nApplication: I\'ve got a user here who wants to use my application, but I need their data.\\nFitbit: OK, what data do you need access to, and for how long?\\nApplication: I need all of these scopes, and for this amount of time.\\nFitbit: OK, let me check with the user to make sure they really want to do this.\\n\\nFitbit: User, do you really want to let this application have your data?\\nUser: I do! And to prove it, here\'s my password.\\nFitbit: OK, everything checks out. I\'ll let the application access your data.\\n\\nFitbit: Application, you can access the user\'s data. Use this special value whenever you need to request data from me.\\nApplication: Thank you, now give me all the data.\\n```\\n\\nEffectively, this allows an application to gain access to a user\'s data without ever needing to know the user\'s password. That way, even if the other application is hacked, the user\'s original data remains safe. Plus, the user can let the data service know to stop providing the application access any time they want. All in all, very secure.\\n\\nIt does make handling small requests a bit challenging, but I\'ll go through the steps here. We\'ll be using the [Implicit Grant](https://dev.fitbit.com/docs/oauth2/) workflow, as it requires fewer steps in processing.\\n\\nFirst, we need to set up the URL the user would visit to authenticate:\\n\\n\\n```python\\nimport urllib\\n\\nFITBIT_URI = \'https://www.fitbit.com/oauth2/authorize\'\\nparams = {\\n    # If we need more than one scope, must be a CSV string\\n    \'scope\': \'heartrate\',\\n    \'response_type\': \'token\',\\n    \'expires_in\': 86400, # 1 day\\n    \'redirect_uri\': CALLBACK_URL,\\n    \'client_id\': CLIENT_ID\\n}\\n\\nrequest_url = FITBIT_URI + \'?\' + urllib.parse.urlencode(params)\\n```\\n\\nNow, here you would print out the request URL, go visit it, and get the full URL that it sends you back to. Because that is very sensitive information (specifically containing my `CLIENT_ID` that I\'d really rather not share on the internet), I\'ve skipped that step in the code here, but it happens in the background.\\n\\n\\n```python\\n# The `response_url` variable contains the full URL that\\n# FitBit sent back to us, but most importantly,\\n# contains the token we need for authorization.\\naccess_token = dict(urllib.parse.parse_qsl(response_url))[\'access_token\']\\n```\\n\\n### Requesting the data\\n\\nNow that we\'ve actually set up our access via the `access_token`, it\'s time to get the actual [heart rate data](https://dev.fitbit.com/docs/heart-rate/). I\'ll be using data from January 1, 2016 through March 31, 2016, and extrapolating wildly from that.\\n\\nFitbit only lets us fetch intraday data one day at a time, so I\'ll create a date range using pandas and iterate through that to pull down all the data.\\n\\n\\n```python\\nfrom requests_oauthlib import OAuth2Session\\nimport pandas as pd\\nfrom datetime import datetime\\n\\nsession = OAuth2Session(token={\\n        \'access_token\': access_token,\\n        \'token_type\': \'Bearer\'\\n    })\\n\\nformat_str = \'%Y-%m-%d\'\\nstart_date = datetime(2016, 1, 1)\\nend_date = datetime(2016, 3, 31)\\ndr = pd.date_range(start_date, end_date)\\n\\nurl = \'https://api.fitbit.com/1/user/-/activities/heart/date/{0}/1d/1min.json\'\\nhr_responses = [session.get(url.format(d.strftime(format_str))) for d in dr]\\n\\ndef record_to_df(record):\\n    if \'activities-heart\' not in record:\\n        return None\\n    date_str = record[\'activities-heart\'][0][\'dateTime\']\\n    df = pd.DataFrame(record[\'activities-heart-intraday\'][\'dataset\'])\\n        \\n    df.index = df[\'time\'].apply(\\n        lambda x: datetime.strptime(date_str + \' \' + x, \'%Y-%m-%d %H:%M:%S\'))\\n    return df\\n\\nhr_dataframes = [record_to_df(record.json()) for record in hr_responses]\\nhr_df_concat = pd.concat(hr_dataframes)\\n\\n\\n# There are some minutes with missing data, so we need to correct that\\nfull_daterange = pd.date_range(hr_df_concat.index[0],\\n                              hr_df_concat.index[-1],\\n                              freq=\'min\')\\nhr_df_full = hr_df_concat.reindex(full_daterange, method=\'nearest\')\\n\\nprint(\\"Heartbeats from {} to {}: {}\\".format(hr_df_full.index[0],\\n                                            hr_df_full.index[-1],\\n                                            hr_df_full[\'value\'].sum()))\\n```\\n\\n```\\n    Heartbeats from 2016-01-01 00:00:00 to 2016-03-31 23:59:00: 8139060\\n```\\n\\nAnd now we\'ve retrieved all the available heart rate data for January 1<sup>st</sup> through March 31<sup>st</sup>! Let\'s get to the actual analysis.\\n\\n## Wild Extrapolations from Small Data\\n\\nA fundamental issue of this data is that it\'s pretty small. I\'m using 3 months of data to make predictions about my entire life. But, purely as an exercise, I\'ll move forward.\\n\\n### How many heartbeats so far?\\n\\nThe first step is figuring out how many of the 2.5 billion heartbeats I\'ve used so far. We\'re going to try and work backward from the present day to when I was born to get that number. The easy part comes first: going back to January 1<sup>st</sup>, 1992. That\'s because I can generalize how many 3-month increments there were between now and then, account for leap years, and call that section done.\\n\\nBetween January 1992 and January 2016 there were 96 quarters, and 6 leap days. The number we\'re looking for is:\\n\\n$$\\n\\\\begin{equation*}\\nhr_q \\\\cdot n - hr_d \\\\cdot (n-m)\\n\\\\end{equation*}\\n$$\\n\\n- $hr_q$: Number of heartbeats per quarter\\n- $hr_d$: Number of heartbeats on leap day\\n- $n$: Number of quarters, in this case 96\\n- $m$: Number of leap days, in this case 6\\n\\n\\n```python\\nquarterly_count = hr_df_full[\'value\'].sum()\\nleap_day_count = hr_df_full[(hr_df_full.index.month == 2) &\\n                            (hr_df_full.index.day == 29)][\'value\'].sum()\\nnum_quarters = 96\\nleap_days = 6\\n\\njan_92_jan_16 = quarterly_count * num_quarters - leap_day_count * (num_quarters - leap_days)\\njan_92_jan_16\\n```\\n\\n```\\n    773609400\\n```\\n\\nSo between January 1992 and January 2016 I\'ve used $\\\\approx$ 774 million heartbeats. Now, I need to go back to my exact birthday. I\'m going to first find on average how many heartbeats I use in a minute, and multiply that by the number of minutes between my birthday and January 1992.\\n\\nFor privacy purposes I\'ll put the code here that I\'m using, but without any identifying information:\\n\\n\\n```python\\nminute_mean = hr_df_full[\'value\'].mean()\\n# Don\'t you wish you knew?\\n# birthday_minutes = ???\\n\\nbirthday_heartbeats = birthday_minutes * minute_mean\\n\\nheartbeats_until_2016 = int(birthday_heartbeats + jan_92_jan_16)\\nremaining_2016 = total_heartbeats - heartbeats_until_2016\\n\\nprint(\\"Heartbeats so far: {}\\".format(heartbeats_until_2016))\\nprint(\\"Remaining heartbeats: {}\\".format(remaining_2016))\\n```\\n\\n```\\n    Heartbeats so far: 775804660\\n    Remaining heartbeats: 1724195340\\n```\\n\\nIt would appear that my heart has beaten 775,804,660 times between my moment of birth and January 1<sup>st</sup> 2016, and that I have 1.72 billion left.\\n\\n### How many heartbeats longer?\\n\\nNow comes the tricky bit. I know how many heart beats I\'ve used so far, and how many I have remaining, so I\'d like to come up with a (relatively) accurate estimate of when exactly my heart should give out. We\'ll do this in a few steps, increasing in granularity.\\n\\nFirst step, how many heartbeats do I use in a 4-year period? I have data for a single quarter including leap day, so I want to know:\\n\\n$$\\n\\\\begin{equation*}\\nhr_q \\\\cdot n - hr_d \\\\cdot (n - m)\\n\\\\end{equation*}\\n$$\\n\\n- $hr_q$: Heartbeats per quarter\\n- $hr_d$: Heartbeats per leap day\\n- $n$: Number of quarters = 16\\n- $m$: Number of leap days = 1\\n\\n\\n```python\\nheartbeats_4year = quarterly_count * 16 - leap_day_count * (16 - 1)\\nheartbeats_4year\\n```\\n\\n```\\n    128934900\\n```\\n\\nNow, I can fast forward from 2016 the number of periods of 4 years I have left.\\n\\n```python\\nfour_year_periods = remaining_2016 // heartbeats_4year\\nremaining_4y = remaining_2016 - four_year_periods * heartbeats_4year\\n\\nprint(\\"Four year periods remaining: {}\\".format(four_year_periods))\\nprint(\\"Remaining heartbeats after 4 year periods: {}\\".format(remaining_4y))\\n```\\n\\n```\\n    Four year periods remaining: 13\\n    Remaining heartbeats after 4 year periods: 48041640\\n```\\n\\nGiven that there are 13 four-year periods left, I can move from 2016 all the way to 2068, and find that I will have 48 million heart beats left. Let\'s drop down to figuring out how many quarters that is. I know that 2068 will have a leap day (unless someone finally decides to get rid of them), so I\'ll subtract that out first. Then, I\'m left to figure out how many quarters exactly are left.\\n\\n```python\\nremaining_leap = remaining_4y - leap_day_count\\n# Ignore leap day in the data set\\nheartbeats_quarter = hr_df_full[(hr_df_full.index.month != 2) &\\n                                (hr_df_full.index.day != 29)][\'value\'].sum()\\nquarters_left = remaining_leap // heartbeats_quarter\\nremaining_year = remaining_leap - quarters_left * heartbeats_quarter\\n\\nprint(\\"Quarters left starting 2068: {}\\".format(quarters_left))\\nprint(\\"Remaining heartbeats after that: {}\\".format(remaining_year))\\n```\\n\\n```\\n    Quarters left starting 2068: 8\\n    Remaining heartbeats after that: 4760716\\n```\\n\\nSo, that analysis gets me through until January 1<sup>st</sup> 2070. Final step, using that minute estimate to figure out how many minutes past that I\'m predicted to have:\\n\\n\\n```python\\nfrom datetime import timedelta\\n\\nbase = datetime(2070, 1, 1)\\nminutes_left = remaining_year // minute_mean\\n\\nkaput = timedelta(minutes=minutes_left)\\nbase + kaput\\n```\\n\\n```\\n    datetime.datetime(2070, 2, 23, 5, 28)\\n```\\n\\nAccording to this, I\'ve got until February 23<sup>rd</sup>, 2070 at 5:28 PM in the evening before my heart gives out.\\n\\n## Summary\\n\\nWell, that\'s kind of a creepy date to know. As I said at the top though, **this number is totally useless in any medical context**. It ignores the rate at which we continue to get better at making people live longer, and is extrapolating from 3 months\' worth of data the rest of my life. Additionally, throughout my time developing this post I made many minor mistakes. I think they\'re all fixed now, but it\'s easy to mix a number up here or there and the analysis gets thrown off by a couple years.\\n\\nEven still, I think philosophically humans have a desire to know how much time we have left in the world. [Man is but a breath](https://www.biblegateway.com/passage/?search=psalm+144&version=ESV), and it\'s scary to think just how quickly that date may be coming up. This analysis asks an important question though: what are you going to do with the time you have left?\\n\\nThanks for sticking with me on this one, I promise it will be much less depressing next time!"},{"id":"2016/03/tweet-like-me","metadata":{"permalink":"/2016/03/tweet-like-me","source":"@site/blog/2016-03-28-tweet-like-me/index.mdx","title":"Tweet like me","description":"In which I try to create a robot that will tweet like I tweet.","date":"2016-03-28T12:00:00.000Z","tags":[],"readingTime":8.23,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/03/tweet-like-me","title":"Tweet like me","date":"2016-03-28T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730678252000,"prevItem":{"title":"Tick tock...","permalink":"/2016/04/tick-tock"},"nextItem":{"title":"Predicting Santander customer happiness","permalink":"/2016/03/predicting-santander-customer-happiness"}},"content":"In which I try to create a robot that will tweet like I tweet.\\n\\n\x3c!-- truncate --\x3e\\n\\nSo, I\'m taking a Machine Learning course this semester in school, and one of the topics we keep coming back to is natural language processing and the \'bag of words\' data structure. That is, given a sentence:\\n\\n`How much wood would a woodchuck chuck if a woodchuck could chuck wood?`\\n\\nWe can represent that sentence as the following list:\\n\\n`{\\n    How: 1\\n    much: 1\\n    wood: 2\\n    would: 2\\n    a: 2\\n    woodchuck: 2\\n    chuck: 2\\n    if: 1\\n}`\\n\\nIgnoring *where* the words happened, we\'re just interested in how *often* the words occurred. That got me thinking: I wonder what would happen if I built a robot that just imitated how often I said things? It\'s dangerous territory when computer scientists ask \\"what if,\\" but I got curious enough I wanted to follow through.\\n\\n## The Objective\\n\\nGiven an input list of Tweets, build up the following things:\\n\\n1. The distribution of starting words; since there are no \\"prior\\" words to go from, we need to treat this as a special case.\\n2. The distribution of words given a previous word; for example, every time I use the word `woodchuck` in the example sentence, there is a 50% chance it is followed by `chuck` and a 50% chance it is followed by `could`. I need this distribution for all words.\\n3. The distribution of quantity of hashtags; Do I most often use just one? Two? Do they follow something like a Poisson distribution?\\n4. Distribution of hashtags; Given a number of hashtags, what is the actual content? I\'ll treat hashtags as separate from the content of a tweet.\\n\\n## The Data\\n\\nI\'m using as input my tweet history. I don\'t really use Twitter anymore, but it seems like a fun use of the dataset. I\'d like to eventually build this to a point where I can imitate anyone on Twitter using their last 100 tweets or so, but I\'ll start with this as example code.\\n\\n## The Algorithm\\n\\nI\'ll be using the [NLTK](http://www.nltk.org/) library for doing a lot of the heavy lifting. First, let\'s import the data:\\n\\n\\n```python\\nimport pandas as pd\\n\\ntweets = pd.read_csv(\'tweets.csv\')\\ntext = tweets.text\\n\\n# Don\'t include tweets in reply to or mentioning people\\nreplies = text.str.contains(\'@\')\\ntext_norep = text.loc[~replies]\\n```\\n\\nAnd now that we\'ve got data, let\'s start crunching. First, tokenize and build out the distribution of first word:\\n\\n\\n```python\\nfrom nltk.tokenize import TweetTokenizer\\ntknzr = TweetTokenizer()\\ntokens = text_norep.map(tknzr.tokenize)\\n\\nfirst_words = tokens.map(lambda x: x[0])\\nfirst_words_alpha = first_words[first_words.str.isalpha()]\\nfirst_word_dist = first_words_alpha.value_counts() / len(first_words_alpha)\\n```\\n\\nNext, we need to build out the conditional distributions. That is, what is the probability of the next word given the current word is $X$? This one is a bit more involved. First, find all unique words, and then find what words proceed them. This can probably be done in a more efficient manner than I\'m currently doing here, but we\'ll ignore that for the moment.\\n\\n\\n```python\\nfrom functools import reduce\\n\\n# Get all possible words\\nall_words = reduce(lambda x, y: x+y, tokens, [])\\nunique_words = set(all_words)\\nactual_words = set([x if x[0] != \'.\' else None for x in unique_words])\\n\\nword_dist = {}\\nfor word in iter(actual_words):\\n    indices = [i for i, j in enumerate(all_words) if j == word]\\n    proceeding = [all_words[i+1] for i in indices]\\n    word_dist[word] = proceeding\\n```\\n\\nNow that we\'ve got the tweet analysis done, it\'s time for the fun part: hashtags! Let\'s count how many hashtags are in each tweet, I want to get a sense of the distribution.\\n\\n\\n```python\\nimport matplotlib.pyplot as plt\\n%matplotlib inline\\n\\nhashtags = text_norep.str.count(\'#\')\\nbins = hashtags.unique().max()\\nhashtags.plot(kind=\'hist\', bins=bins)\\n```\\n\\n```\\n    <matplotlib.axes._subplots.AxesSubplot at 0x18e59dc28d0>\\n```\\n\\n    \\n![png](_notebook_files/_notebook_7_1.png)\\n    \\n\\nThat looks like a Poisson distribution, kind of as I expected. I\'m guessing my number of hashtags per tweet is $\\\\sim Poi(1)$, but let\'s actually find the [most likely estimator](https://en.wikipedia.org/wiki/Poisson_distribution#Maximum_likelihood) which in this case is just $\\\\bar{\\\\lambda}$:\\n\\n\\n```python\\nmle = hashtags.mean()\\nmle\\n```\\n\\n```\\n    0.870236869207003\\n```\\n\\n\\nPretty close! So we can now simulate how many hashtags are in a tweet. Let\'s also find what hashtags are actually used:\\n\\n\\n```python\\nhashtags = [x for x in all_words if x[0] == \'#\']\\nn_hashtags = len(hashtags)\\n\\nunique_hashtags = list(set([x for x in unique_words if x[0] == \'#\']))\\nhashtag_dist = pd.DataFrame({\'hashtags\': unique_hashtags,\\n                             \'prob\': [all_words.count(h) / n_hashtags\\n                                      for h in unique_hashtags]})\\nlen(hashtag_dist)\\n```\\n\\n```\\n    603\\n```\\n\\nTurns out I have used 603 different hashtags during my time on Twitter. That means I was using a unique hashtag for about every third tweet.\\n\\nIn better news though, we now have all the data we need to go about actually constructing tweets! The process will happen in a few steps:\\n\\n1. Randomly select what the first word will be.\\n2. Randomly select the number of hashtags for this tweet, and then select the actual hashtags.\\n3. Fill in the remaining space of 140 characters with random words taken from my tweets.\\n\\nAnd hopefully, we won\'t have anything too crazy come out the other end. The way we do the selection follows a [Multinomial Distribution](https://en.wikipedia.org/wiki/Multinomial_distribution): given a lot of different values with specific probability, pick one. Let\'s give a quick example:\\n\\n```\\nx: .33\\ny: .5\\nz: .17\\n```\\n\\nThat is, I pick `x` with probability 33%, `y` with probability 50%, and so on. In context of our sentence construction, I\'ve built out the probabilities of specific words already - now I just need to simulate that distribution. Time for the engine to actually be developed!\\n\\n\\n```python\\nimport numpy as np\\n\\ndef multinom_sim(n, vals, probs):\\n    occurrences = np.random.multinomial(n, probs)\\n    results = occurrences * vals\\n    return \' \'.join(results[results != \'\'])\\n\\ndef sim_n_hashtags(hashtag_freq):\\n    return np.random.poisson(hashtag_freq)\\n\\ndef sim_hashtags(n, hashtag_dist):\\n    return multinom_sim(n, hashtag_dist.hashtags, hashtag_dist.prob)\\n\\ndef sim_first_word(first_word_dist):\\n    probs = np.float64(first_word_dist.values)\\n    return multinom_sim(1, first_word_dist.reset_index()[\'index\'], probs)\\n\\ndef sim_next_word(current, word_dist):\\n    dist = pd.Series(word_dist[current])\\n    probs = np.ones(len(dist)) / len(dist)\\n    return multinom_sim(1, dist, probs)\\n```\\n\\n## Pulling it all together\\n\\nI\'ve now built out all the code I need to actually simulate a sentence written by me. Let\'s try doing an example with five words and a single hashtag:\\n\\n\\n```python\\nfirst = sim_first_word(first_word_dist)\\nsecond = sim_next_word(first, word_dist)\\nthird = sim_next_word(second, word_dist)\\nfourth = sim_next_word(third, word_dist)\\nfifth = sim_next_word(fourth, word_dist)\\nhashtag = sim_hashtags(1, hashtag_dist)\\n\\n\' \'.join((first, second, third, fourth, fifth, hashtag))\\n```\\n\\n```\\n    \'My first all-nighter of friends #oldschool\'\\n```\\n\\nLet\'s go ahead and put everything together! We\'re going to simulate a first word, simulate the hashtags, and then simulate to fill the gap until we\'ve either taken up all the space or reached a period.\\n\\n\\n```python\\ndef simulate_tweet():\\n    chars_remaining = 140\\n    first = sim_first_word(first_word_dist)\\n    n_hash = sim_n_hashtags(mle)\\n    hashtags = sim_hashtags(n_hash, hashtag_dist)\\n    \\n    chars_remaining -= len(first) + len(hashtags)\\n    \\n    tweet = first\\n    current = first\\n    while chars_remaining > len(tweet) + len(hashtags) and current[0] != \'.\' and current[0] != \'!\':\\n        current = sim_next_word(current, word_dist)\\n        tweet += \' \' + current\\n        \\n    tweet = tweet[:-2] + tweet[-1]\\n        \\n    return \' \'.join((tweet, hashtags)).strip()\\n```\\n\\n## The results\\n\\nAnd now for something completely different: twenty random tweets dreamed up by my computer and my Twitter data. Here you go:\\n\\n\\n```python\\nfor i in range(0, 20):\\n    print(simulate_tweet())\\n    print()\\n```\\n\\n```\\n    Also , I\'m at 8 this morning. #thursdaysgohard #ornot\\n    \\n    Turns out of us breathe the code will want to my undergraduate career is becoming more night trying ? Religion is now as a chane #HYPE\\n    \\n    You know what recursion is to review the UNCC. #ornot\\n    \\n    There are really sore 3 bonfires in my first writing the library ground floor if awesome. #realtalk #impressed\\n    \\n    So we can make it out there\'s nothing but I\'m not let us so hot I could think I may be good. #SwingDance\\n    \\n    Happy Christmas , at Harris Teeter to be be godly or Roman Catholic ). #4b392b#4b392b #Isaiah26\\n    \\n    For context , I in the most decisive factor of the same for homework. #accomplishment\\n    \\n    Freaking done. #loveyouall\\n    \\n    New blog post : Don\'t jump in a quiz in with a knife fight. #haskell #earlybirthday\\n    \\n    God shows me legitimately want to get some food and one day.\\n    \\n    Stormed the queen city. #mindblown\\n    \\n    The day of a cold at least outside right before the semester ..\\n    \\n    Finished with the way back. #winners\\n    \\n    Waking up , OJ , I feel like Nick Jonas today.\\n    \\n    First draft of so hard drive. #humansvszombies\\n    \\n    Eric Whitacre is the wise creation.\\n    \\n    Ethics paper first , music in close to everyone who just be posting up with my sin , and Jerry Springr #TheLittleThings\\n    \\n    Love that you know enough time I\'ve eaten at 8 PM. #deepthoughts #stillblownaway\\n    \\n    Lead. #ThinkingTooMuch #Christmas\\n    \\n    Aamazing conference when you married #DepartmentOfRedundancyDepartment Yep , but there\'s a legitimate challenge.\\n```\\n\\n\\n...Which all ended up being a whole lot more nonsensical than I had hoped for. There are some good ones, so I\'ll call that an accomplishment! I was banking on grammar not being an issue: since my tweets use impeccable grammar, the program modeled off them should have pretty good grammar as well. There are going to be some hilarious edge cases (I\'m looking at you, `Ethics paper first, music in close to everyone`) that make no sense, and some hilarious edge cases (`Waking up, OJ, I feel like Nick Jonas today`) that make me feel like I should have a Twitter rap career. On the whole though, the structure came out alright.\\n\\n## Moving on from here\\n\\nDuring class we also talked about an interesting idea: trying to analyze corporate documents and corporate speech. I\'d be interested to know what this analysis applied to something like a couple of bank press releases could do. By any means, the code needs some work to clean it up before I get that far.\\n\\n## For further reading\\n\\nI\'m pretty confident I re-invented a couple wheels along the way - what I\'m doing feels a lot like what [Markov Chain Monte Carlo](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo) is intended to do. But I\'ve never worked explicitly with that before, so more research is needed."},{"id":"2016/03/predicting-santander-customer-happiness","metadata":{"permalink":"/2016/03/predicting-santander-customer-happiness","source":"@site/blog/2016-03-05-predicting-santander-customer-happiness/index.mdx","title":"Predicting Santander customer happiness","description":"My first Kaggle competition.","date":"2016-03-05T12:00:00.000Z","tags":[],"readingTime":6.95,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/03/predicting-santander-customer-happiness","title":"Predicting Santander customer happiness","date":"2016-03-05T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730863976000,"prevItem":{"title":"Tweet like me","permalink":"/2016/03/tweet-like-me"},"nextItem":{"title":"Profitability using the investment formula","permalink":"/2016/02/profitability-using-the-investment-formula"}},"content":"My first Kaggle competition.\\n\\n\x3c!-- truncate --\x3e\\n\\nIt\'s time! After embarking on a Machine Learning class this semester, and with a Saturday in which I don\'t have much planned, I wanted to put this class and training to work. It\'s my first competition submission. I want to walk you guys through how I\'m approaching this problem, because I thought it would be really neat. The competition is Banco Santander\'s [Santander Customer Satisfaction][1] competition. It seemed like an easy enough problem I could actually make decent progress on it.\\n\\n## Data Exploration\\n\\nFirst up: we need to load our data and do some exploratory work. Because we\'re going to be using this data for model selection prior to testing, we need to make a further split. I\'ve already gone ahead and done this work, please see the code in the [appendix below](#appendix).\\n\\n[1]: https://www.kaggle.com/c/santander-customer-satisfaction\\n\\n\\n```python\\nimport pandas as pd\\nimport numpy as np\\nimport matplotlib.pyplot as plt\\n%matplotlib inline\\n\\n# Record how long it takes to run the notebook - I\'m curious.\\nfrom datetime import datetime\\nstart = datetime.now()\\n\\ndataset = pd.read_csv(\'split_train.csv\')\\ndataset.index = dataset.ID\\nX = dataset.drop([\'TARGET\', \'ID\', \'ID.1\'], 1)\\ny = dataset.TARGET\\n```\\n\\n\\n```python\\ny.unique()\\n```\\n\\n```\\n    array([0, 1], dtype=int64)\\n```\\n\\n```python\\nlen(X.columns)\\n```\\n\\n```\\n    369\\n```\\n\\nOkay, so there are only [two classes we\'re predicting][2]: 1 for unsatisfied customers, 0 for satisfied customers. I would have preferred this to be something more like a regression, or predicting multiple classes: maybe the customer isn\'t the most happy, but is nowhere near closing their accounts. For now though, that\'s just the data we\'re working with.\\n\\nNow, I\'d like to make a scatter matrix of everything going on. Unfortunately as noted above, we have 369 different features. There\'s no way I can graphically make sense of that much data to start with.\\n\\nWe\'re also not told what the data actually represents: Are these survey results? Average time between contact with a customer care person? Frequency of contacting a customer care person? The idea is that I need to reduce the number of dimensions we\'re predicting across.\\n\\n### Dimensionality Reduction pt. 1 - Binary Classifiers\\n\\nMy first attempt to reduce the data dimensionality is to find all the binary classifiers in the dataset \\\\(i.e. 0 or 1 values\\\\) and see if any of those are good \\\\(or anti-good\\\\) predictors of the final data.\\n\\n[2]: https://www.kaggle.com/c/santander-customer-satisfaction/data\\n\\n\\n```python\\ncols = X.columns\\nb_class = []\\nfor c in cols:\\n    if len(X[c].unique()) == 2:\\n        b_class.append(c)\\n        \\nlen(b_class)\\n```\\n\\n```\\n    111\\n```\\n\\nSo there are 111 features in the dataset that are a binary label. Let\'s see if any of them are good at predicting the users satisfaction!\\n\\n```python\\n# First we need to `binarize` the data to 0-1; some of the labels are {0, 1},\\n# some are {0, 3}, etc.\\nfrom sklearn.preprocessing import binarize\\nX_bin = binarize(X[b_class])\\n\\naccuracy = [np.mean(X_bin[:,i] == y) for i in range(0, len(b_class))]\\nacc_df = pd.DataFrame({\\"Accuracy\\": accuracy}, index=b_class)\\nacc_df.describe()\\n```\\n\\n<div>\\n<table>\\n  <thead>\\n    <tr>\\n      <th></th>\\n      <th>Accuracy</th>\\n    </tr>\\n  </thead>\\n  <tbody>\\n    <tr>\\n      <th>count</th>\\n      <td>111.000000</td>\\n    </tr>\\n    <tr>\\n      <th>mean</th>\\n      <td>0.905159</td>\\n    </tr>\\n    <tr>\\n      <th>std</th>\\n      <td>0.180602</td>\\n    </tr>\\n    <tr>\\n      <th>min</th>\\n      <td>0.043598</td>\\n    </tr>\\n    <tr>\\n      <th>25%</th>\\n      <td>0.937329</td>\\n    </tr>\\n    <tr>\\n      <th>50%</th>\\n      <td>0.959372</td>\\n    </tr>\\n    <tr>\\n      <th>75%</th>\\n      <td>0.960837</td>\\n    </tr>\\n    <tr>\\n      <th>max</th>\\n      <td>0.960837</td>\\n    </tr>\\n  </tbody>\\n</table>\\n</div>\\n\\nWow! Looks like we\'ve got some incredibly predictive features! So much so that we should be a bit concerned. My initial guess for what\'s happening is that we have a sparsity issue: so many of the values are 0, and these likely happen to line up with satisfied customers.\\n\\nSo the question we must now answer, which I likely should have asked long before now: What exactly is the distribution of un/satisfied customers?\\n\\n```python\\nunsat = y[y == 1].count()\\nprint(\\"Satisfied customers: {}; Unsatisfied customers: {}\\".format(len(y) - unsat, unsat))\\nnaive_guess = np.mean(y == np.zeros(len(y)))\\nprint(\\"Naive guess accuracy: {}\\".format(naive_guess))\\n```\\n\\n```\\n    Satisfied customers: 51131; Unsatisfied customers: 2083\\n    Naive guess accuracy: 0.9608561656706882\\n```\\n\\nThis is a bit discouraging. A naive guess of \\"always satisfied\\" performs as well as our best individual binary classifier. What this tells me then, is that these data columns aren\'t incredibly helpful in prediction. I\'d be interested in a polynomial expansion of this data-set, but for now, that\'s more computation than I want to take on.\\n\\n### Dimensionality Reduction pt. 2 - LDA\\n\\nKnowing that our naive guess performs so well is a blessing and a curse:\\n\\n- Curse: The threshold for performance is incredibly high: We can only \\"improve\\" over the naive guess by 4%\\n- Blessing: All the binary classification features we just discovered are worthless on their own. We can throw them out and reduce the data dimensionality from 369 to 111.\\n\\nNow, in removing these features from the dataset, I\'m not saying that there is no \\"information\\" contained within them. There might be. But the only way we\'d know is through a polynomial expansion, and I\'m not going to take that on within this post.\\n\\nMy initial thought for a \\"next guess\\" is to use the [LDA][3] model for dimensionality reduction. However, it can only reduce dimensions to $1 - p$, with $p$ being the number of classes. Since this is a binary classification, every LDA model that I try will have dimensionality one; when I actually try this, the predictor ends up being slightly less accurate than the naive guess.\\n\\nInstead, let\'s take a different approach to dimensionality reduction: [principle components analysis][4]. This allows us to perform the dimensionality reduction without worrying about the number of classes. Then, we\'ll use a [Gaussian Naive Bayes][5] model to actually do the prediction. This model is chosen simply because it doesn\'t take a long time to fit and compute; because PCA will take so long, I just want a prediction at the end of this. We can worry about using a more sophisticated LDA/QDA/SVM model later.\\n\\nNow into the actual process: We\'re going to test out PCA dimensionality reduction from 1 - 20 dimensions, and then predict using a Gaussian Naive Bayes model. The 20 dimensions upper limit was selected because the accuracy never improves after you get beyond that \\\\(I found out by running it myself\\\\). Hopefully, we\'ll find that we can create a model better than the naive guess.\\n\\n[3]:http://scikit-learn.org/stable/modules/lda_qda.html\\n[4]:http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html\\n[5]:http://scikit-learn.org/stable/modules/naive_bayes.html#gaussian-naive-bayes\\n\\n\\n```python\\nfrom sklearn.naive_bayes import GaussianNB\\nfrom sklearn.decomposition import PCA\\n\\nX_no_bin = X.drop(b_class, 1)\\n\\ndef evaluate_gnb(dims):\\n    pca = PCA(n_components=dims)\\n    X_xform = pca.fit_transform(X_no_bin)\\n    \\n    gnb = GaussianNB()\\n    gnb.fit(X_xform, y)\\n    return gnb.score(X_xform, y)\\n\\ndim_range = np.arange(1, 21)\\nplt.plot(dim_range, [evaluate_gnb(dim) for dim in dim_range], label=\\"Gaussian NB Accuracy\\")\\nplt.axhline(naive_guess, label=\\"Naive Guess\\", c=\'k\')\\nplt.axhline(1 - naive_guess, label=\\"Inverse Naive Guess\\", c=\'k\')\\nplt.gcf().set_size_inches(12, 6)\\nplt.legend();\\n```\\n\\n![png](_notebook_files/_notebook_11_0.png)\\n\\n**sigh...** After all the effort and computational power, we\'re still at square one: we have yet to beat out the naive guess threshold. With PCA in play we end up performing terribly, but not terribly enough that we can guess against ourselves.\\n\\nLet\'s try one last-ditch attempt using the entire data set:\\n\\n\\n```python\\ndef evaluate_gnb_full(dims):\\n    pca = PCA(n_components=dims)\\n    X_xform = pca.fit_transform(X)\\n    \\n    gnb = GaussianNB()\\n    gnb.fit(X_xform, y)\\n    return gnb.score(X_xform, y)\\n\\ndim_range = np.arange(1, 21)\\nplt.plot(dim_range, [evaluate_gnb(dim) for dim in dim_range], label=\\"Gaussian NB Accuracy\\")\\nplt.axhline(naive_guess, label=\\"Naive Guess\\", c=\'k\')\\nplt.axhline(1 - naive_guess, label=\\"Inverse Naive Guess\\", c=\'k\')\\nplt.gcf().set_size_inches(12, 6)\\nplt.legend();\\n```\\n\\n![png](_notebook_files/_notebook_13_0.png)\\n\\nNothing. It is interesting to note that the graphs are almost exactly the same: This would imply again that the variables we removed earlier (all the binary classifiers) indeed have almost no predictive power. It seems this problem is high-dimensional, but with almost no data that can actually inform our decisions.\\n\\n## Summary for Day 1\\n\\nAfter spending a couple hours with this dataset, there seems to be a fundamental issue in play: We have very high-dimensional data, and it has no bearing on our ability to actually predict customer satisfaction. This can be a huge issue: it implies that **no matter what model we use, we fundamentally can\'t perform well.** I\'m sure most of this is because I\'m not an experienced data scientist. Even so, we have yet to develop a strategy that can actually beat out the village idiot; **so far, the bank is best off just assuming all its customers are satisfied.** Hopefully more to come soon.\\n\\n\\n```python\\nend = datetime.now()\\nprint(\\"Running time: {}\\".format(end - start))\\n```\\n\\n```\\n    Running time: 0:00:58.715714\\n```\\n\\n## Appendix\\n\\nCode used to split the initial training data:\\n\\n```python\\nfrom sklearn.cross_validation import train_test_split\\ndata = pd.read_csv(\'train.csv\')\\ndata.index = data.ID\\n\\ndata_train, data_validate = train_test_split(\\n    data, train_size=.7)\\n\\ndata_train.to_csv(\'split_train.csv\')\\ndata_validate.to_csv(\'split_validate.csv\')\\n```"},{"id":"2016/02/profitability-using-the-investment-formula","metadata":{"permalink":"/2016/02/profitability-using-the-investment-formula","source":"@site/blog/2016-02-26-profitability-using-the-investment-formula/index.mdx","title":"Profitability using the investment formula","description":"After developing a formula to guide our investing, how do we actually evaluate its performance in the real world?","date":"2016-02-26T12:00:00.000Z","tags":[],"readingTime":7.675,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/02/profitability-using-the-investment-formula","title":"Profitability using the investment formula","date":"2016-02-26T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730678252000,"prevItem":{"title":"Predicting Santander customer happiness","permalink":"/2016/03/predicting-santander-customer-happiness"},"nextItem":{"title":"Guaranteed money maker","permalink":"/2016/02/guaranteed-money-maker"}},"content":"After developing a formula to guide our investing, how do we actually evaluate its performance in the real world?\\n\\n\x3c!-- truncate --\x3e\\n\\nI\'ve previously talked about crafting an [Investment Formula](../2016-02-03-guaranteed-money-maker/index.mdx) that would guarantee making money if you could predict which direction the stock market was going to go. This is going to be the first in a series of posts trying to flesh out what an actual investment strategy based on this formula would look like.\\n\\nBut first, the formula doesn\'t take into account two very important things: **leverage**, and the **number of days invested**. That\'s why I want to set up what I\'m going to call the **Profitability Score**.\\n\\nThe definition is going to be very simple:\\n\\n- $p$: Profit made once you exit the investment\\n- $i$: Initial investment into the asset\\n- $m$: Maximum investment in the asset\\n- $l = m / i$: The maximum leverage of an investment, as the ratio of maximum invested to initial investment\\n- $d$: The number of days it takes to turn a profit\\n\\n$s = \\\\frac{1000 p}{i(l + d)} = \\\\frac{1000 p}{m + i\\\\cdot d}$\\n\\nCrazy, right? The score is simply the (normalized) profit you made divided by the leverage plus days invested. The $\\\\cdot 1000$ is just to turn the number into something more reasonable - people don\'t like hearing something with a profitability score of .001 for example.\\n\\n## Theoretical Justification\\n\\nThe formula itself is designed to be simple in principle: I like making a profit, and I want to penalize the leverage you incur and days you have to invest. Ideally, we want to have a stock that goes up all the time. However, the investment formula takes advantage of a different case: trying to profit from highly volatile assets. If we can make money when the investment only has one day up, let\'s do it!\\n\\nEven so, there are two potential issues: First, stocks that trend upward will have a higher profitability score - both leverage and days invested will be 1. To protect against only investing in this trend, I can do things like taking $\\\\log(d)$. I don\'t want to start biasing the scoring function until I have a practical reason to do so, so right now I\'ll leave it standing.\\n\\nThe second issue is how to penalize leverage and days invested relative to each other. As it currently stands, a leverage of 6x with only 1 day invested is the same as leveraging 2x with 3 days invested. In the future, I\'d again want to look at making the impact of days invested smaller - I can get over an extra 3 days in the market if it means that I don\'t have to incur a highly leveraged position.\\n\\nSo there could be things about the scoring function we change in the future, but I want to run some actual tests before we start worrying about things like that!\\n\\n## Running a simulation\\n\\nThis won\'t be an incredibly rigorous backtest, I just want to see some results from the work so far. Let\'s set up the simulation code again, and start looking into some random stocks. **If you\'ve read the last blog post, you can skip over the code.** The only difference is that it\'s been ported to python to make the data-wrangling easier. Julia doesn\'t yet support some of the multi-index things I\'m trying to do.\\n\\n\\n```python\\nimport numpy as np\\nimport pandas as pd\\nimport matplotlib.pyplot as plt\\nfrom Quandl import get as qget\\n%matplotlib inline\\napi_key = \'\'\\n\\nprofitability = lambda p, i, m, d: 1000*p / (m + i*d)\\n\\ndef is_profitable(current_price, purchase_history, open_history):\\n    shares = (purchase_history / open_history).sum()\\n    return current_price * shares > sum(purchase_history)\\n\\ndef daily_investment(current_open, current_close, purchase_history, open_history):\\n    t1 = current_close / current_open - 1\\n    t2 = (purchase_history - purchase_history * current_close / open_history).sum()\\n    return t2 / t1\\n\\ndef simulate_day(open_vals, close_vals, init, expected, bias):\\n    invested = np.array([init])\\n    \\n    day = 1\\n    profitable = is_profitable(close_vals[day-1], invested, open_vals[0:len(invested)]) \\\\\\n        or is_profitable(open_vals[day], invested, open_vals[0:len(invested)])\\n\\n    while not profitable:\\n        expected_close = open_vals[day] * expected\\n        todays_purchase = daily_investment(open_vals[day], expected_close, invested, open_vals[0:day])\\n        invested = np.append(invested, todays_purchase + bias)\\n        # expected_profit = expected_close * (invested / open_vals[0:len(invested)]).sum() - invested.sum()\\n        day += 1\\n        profitable = is_profitable(close_vals[day-1], invested, open_vals[0:len(invested)]) \\\\\\n            or is_profitable(open_vals[day], invested, open_vals[0:len(invested)])\\n            \\n    shares = (invested / open_vals[0:len(invested)]).sum()\\n    \\n    # Make sure we can\'t see into the future - we know either today\'s close or tomorrow\'s open\\n    # will be profitable, but we need to check which one.\\n    if is_profitable(close_vals[day-1], invested, open_vals[0:len(invested)]):\\n        ending_price = close_vals[day-1]\\n    else:\\n        ending_price = open_vals[day]\\n        \\n    profit = shares * ending_price - sum(invested)\\n    return invested, profit\\n\\ndef simulate_ts(name, start, end, initial, expected, bias):\\n    ticker_info = qget(name, trim_start=start, api_key=api_key)\\n    evaluation_times = ticker_info[:end].index\\n    \\n    # Handle Google vs. YFinance data\\n    if \\"Adjusted Close\\" in ticker_info.columns:\\n        close_column = \\"Adjusted Close\\"\\n    else:\\n        close_column = \\"Close\\"\\n        \\n    sim = {d: simulate_day(ticker_info[d:][\\"Open\\"], ticker_info[d:][close_column],\\n                       100, 1.02, 10) for d in evaluation_times}\\n    sim_series = pd.Series(sim)\\n    result = pd.DataFrame()\\n    result[\\"profit\\"] = sim_series.apply(lambda x: x[1])\\n    result[\\"max\\"] = sim_series.apply(lambda x: max(x[0]))\\n    result[\\"days\\"] = sim_series.apply(lambda x: len(x[0]))\\n    result[\\"score\\"] = sim_series.apply(lambda x: profitability(x[1], x[0][0], max(x[0]), len(x[0])))\\n    result[\\"investments\\"] = sim_series.apply(lambda x: x[0])\\n    \\n    return result\\n\\ndef simulate_tickers(tickers):\\n    from datetime import datetime\\n    results = {}\\n    for ticker in tickers:\\n        start = datetime(2015, 1, 1)\\n        results_df = simulate_ts(ticker, start, datetime(2016, 1, 1), 100, 1.01, 10)\\n        results[ticker] = results_df\\n        \\n    return pd.concat(list(results.values()), keys=list(results.keys()), axis=1)\\n```\\n\\n## And now the interesting part\\n\\nLet\'s start looking into the data! FANG stocks have been big over the past year, let\'s see how they look:\\n\\n\\n```python\\nfang_df = simulate_tickers([\\"YAHOO/FB\\", \\"YAHOO/AAPL\\", \\"YAHOO/NFLX\\", \\"YAHOO/GOOG\\"])\\n```\\n\\n```python\\nfang_df.xs(\'days\', axis=1, level=1).hist()\\nplt.gcf().set_size_inches(18, 8);\\nplt.gcf().suptitle(\\"Distribution of Days Until Profitability\\", fontsize=18);\\n```\\n\\n![png](_notebook_files/_notebook_4_0.png)\\n\\n```python\\nfang_df.xs(\'score\', axis=1, level=1).plot()\\nplt.gcf().set_size_inches(18, 6)\\nplt.gcf().suptitle(\\"Profitability score over time\\", fontsize=18);\\n```\\n    \\n![png](_notebook_files/_notebook_5_0.png)\\n    \\nLet\'s think about these graphs. First, the histogram. What we like seeing is a lot of 1\'s - that means there were a lot of days that the stock went up and we didn\'t have to worry about actually implementing the strategy - we were able to close the trade at a profit.\\n\\nLooking at the profitability score over time though is a bit more interesting. First off, stocks that are more volatile will tend to have a higher profitability score, no two ways about that. However, Netflix consistently outperformed on this metric. We know that 2015 was a good year for Netflix, so that\'s a (small) sign the strategy is performing as expected.\\n\\nThe final interesting note happens around the end of August 2015. Around this period, the markets were selling off in a big way due to issues in China (not unlike what\'s happening now). Even so, all of the FANG stocks saw an uptick in profitability around this time. This is another sign that the strategy being developed performs better during periods of volatility, rather than from riding markets up or down.\\n\\nWhat about FANG vs. some cyclicals?\\n\\n```python\\ncyclic_df = simulate_tickers([\\"YAHOO/X\\", \\"YAHOO/CAT\\", \\"YAHOO/NFLX\\", \\"YAHOO/GOOG\\"])\\n```\\n\\n```python\\ncyclic_df.xs(\'days\', axis=1, level=1).hist()\\nplt.gcf().set_size_inches(18, 8);\\nplt.gcf().suptitle(\\"Distribution of Days Until Profitability\\", fontsize=18);\\n```\\n    \\n![png](_notebook_files/_notebook_8_0.png)\\n    \\n\\n```python\\ncyclic_df.xs(\'score\', axis=1, level=1).plot()\\nplt.gcf().set_size_inches(18, 6)\\nplt.gcf().suptitle(\\"Profitability score over time\\", fontsize=18);\\n```\\n\\n![png](_notebook_files/_notebook_9_0.png)\\n\\nSome more interesting results come from this as well. First off, US Steel (X) has a much smoother distribution of days until profitability - it doesn\'t have a huge number of values at 1 and then drop off. Intuitively, we\'re not terribly large fans of this, we want a stock to go up! However, on the profitability score it is the only serious contender to Netflix.\\n\\nSecond, we see the same trend around August - the algorithm performs well in volatile markets.\\n\\nFor a final test, let\'s try some biotech and ETFs!\\n\\n```python\\nbiotech_df = simulate_tickers([\'YAHOO/REGN\', \'YAHOO/CELG\', \'GOOG/NASDAQ_BIB\', \'GOOG/NASDAQ_IBB\'])\\n```\\n\\n```python\\nbiotech_df.xs(\'days\', axis=1, level=1).hist()\\nplt.gcf().set_size_inches(18, 8);\\nplt.gcf().suptitle(\\"Distribution of Days Until Profitability\\", fontsize=18);\\n```\\n    \\n![png](_notebook_files/_notebook_12_0.png)\\n    \\n```python\\nbiotech_df.xs(\'score\', axis=1, level=1).plot()\\nplt.gcf().set_size_inches(18, 6)\\nplt.gcf().suptitle(\\"Profitability score over time\\", fontsize=18);\\n```\\n    \\n![png](_notebook_files/_notebook_13_0.png)\\n\\nIn this example, we don\'t see a whole lot of interesting things: the scores are all fairly close together with notable exceptions in late August, and mid-October.\\n\\nWhat is interesting is that during the volatile period, the ETF\'s performed significantly better than the stocks did in terms of profitability. The leveraged ETF (BIB) performed far above anyone else, and it appears that indeed, it is most profitable during volatile periods. Even so, it was far more likely to take multiple days to give a return. Its count of 1-day investments trails the other ETF and both stocks by a decent margin.\\n\\nAnd consider me an OCD freak, but I just really like Celgene\'s distribution - it looks nice and smooth.\\n\\n## Summary and plans for the next post\\n\\nSo far I\'m really enjoying playing with this strategy - there\'s a lot of depth here to understand, though the preliminary results seem to indicate that it profits mostly from taking the other side of a volatile trade. I\'d be interested to run results later on data from January - It\'s been a particularly volatile start to the year so it would be neat to see whether this strategy would work then.\\n\\nFor the next post, I want to start playing with some of the parameters: How do the bias and expected close influence the process? The values have been fairly conservative so far, it will be interesting to see how the simulations respond afterward."},{"id":"2016/02/guaranteed-money-maker","metadata":{"permalink":"/2016/02/guaranteed-money-maker","source":"@site/blog/2016-02-03-guaranteed-money-maker/index.mdx","title":"Guaranteed money maker","description":"Developing an investment strategy based on the Martingale betting strategy","date":"2016-02-03T12:00:00.000Z","tags":[],"readingTime":7.695,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/02/guaranteed-money-maker","title":"Guaranteed money maker","date":"2016-02-03T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730678252000,"prevItem":{"title":"Profitability using the investment formula","permalink":"/2016/02/profitability-using-the-investment-formula"},"nextItem":{"title":"Cloudy in Seattle","permalink":"/2016/01/cloudy-in-seattle"}},"content":"Developing an investment strategy based on the Martingale betting strategy\\n\\nIf you can see into the future, that is.\\n\\n\x3c!-- truncate --\x3e\\n\\nMy previous class in Stochastic Calculus covered a lot of interesting topics, and the important one for today is the [Gambler\'s Ruin][1] problem. If you\'re interested in some of the theory behind it, also make sure to check out [random walks][2]. The important bit is that we studied the [Martingale Betting Strategy][3], which describes for us a **guaranteed way** to <small>eventually</small> make money.\\n\\nThe strategy goes like this: You are going to toss a fair coin with a friend. If you guess heads or tails correctly, you get back double the money you bet. If you guess incorrectly, you lose money. How should you bet?\\n\\nThe correct answer is that you should double your bet each time you lose. Then when you finally win, you\'ll be guaranteed to make back everything you lost and then &#36;1 extra! Consider the scenario:\\n\\n1. You bet &#36;1, and guess incorrectly. You\'re 1 dollar in the hole.\\n2. You bet &#36;2, and guess incorrectly. You\'re 3 dollars in the hole now.\\n3. You bet &#36;4, and guess incorrectly. You\'re 7 dollars in the hole.\\n4. You bet &#36;8, and guess correctly! You now get back those 8 dollars you bet, plus 8 extra for winning, for a **total profit of one dollar**!\\n\\nMathematically, we can prove that as long as you have unlimited money to bet, you are guaranteed to make money.\\n\\n## Applying the Martingale Strategy\\n\\nBut we\'re all realistic people, and once you start talking about \\"unlimited money\\" eyebrows should be raised. Even still, this is an interesting strategy to investigate, and I want to apply it to the stock market. As long as we can guarantee there\'s a single day in which the stock goes up, we should be able to make money right? The question is just how much we have to invest to guarantee this.\\n\\nNow it\'s time for the math. We\'ll use the following definitions:\\n\\n- $o_i$ = the share price at the opening of day $i$\\n- $c_i$ = the share price at the close of day $i$\\n- $d_i$ = the amount of money we want to invest at the beginning of day $i$\\n\\nWith those definitions in place, I\'d like to present the formula that is **guaranteed to make you money**. I call it *Bradlee\'s Investment Formula*:\\n\\n$c_n \\\\sum_{i=1}^n \\\\frac{d_i}{o_i} > \\\\sum_{i=1}^{n} d_i$\\n\\nIt might not look like much, but if you can manage to make it so that this formula holds true, you will be guaranteed to make money. The intuition behind the formula is this: The closing share price times the number of shares you have purchased ends up greater than the amount of money you invested.\\n\\nThat is, on day $n$, <small>if you know what the closing price will be</small> you can set up the amount of money you invest that day to **guarantee you make money**. I\'ll even teach you to figure out how much money that is! Take a look:\\n\\n$$\\n\\\\begin{align*}\\nc_n \\\\sum_{i=1}^{n-1} \\\\frac{d_i}{o_i} + \\\\frac{c_nd_n}{o_n} &> \\\\sum_{i=1}^{n-1}d_i + d_n\\\\\\\\\\n\\\\frac{c_nd_n}{o_n} - d_n &> \\\\sum_{i=1}^{n-1}(d_i - \\\\frac{c_nd_i}{o_i})\\\\\\\\\\nd_n (\\\\frac{c_n - o_n}{o_n}) &> \\\\sum_{i=1}^{n-1} d_i(1 - \\\\frac{c_n}{o_i})\\\\\\\\\\nd_n &> \\\\frac{o_n}{c_n - o_n} \\\\sum_{i=1}^{n-1} d_i(1 - \\\\frac{1}{o_i})\\n\\\\end{align*}\\n$$\\n\\nIf you invest exactly $d_n$ that day, you\'ll break even. But if you can make sure the money you invest is greater than that quantity on the right <small>(which requires that you have a crystal ball tell you the stock\'s closing price)</small> you are **guaranteed to make money!**\\n\\n## Interesting Implications\\n\\nOn a more serious note though, the formula above tells us a couple of interesting things:\\n\\n1. It\'s impossible to make money without the closing price at some point being greater than the opening price (or vice-versa if you are short selling) - there is no amount of money you can invest that will turn things in your favor.\\n2. Close prices of the past aren\'t important if you\'re concerned about the bottom line. While chart technicians use price history to make judgment calls, in the end, the closing price on anything other than the last day is irrelevant.\\n3. It\'s possible to make money as long as there is a single day where the closing price is greater than the opening price! You might have to invest a lot to do so, but it\'s possible.\\n4. You must make a prediction about where the stock will close at if you want to know how much to invest. That is, we can set up our investment for the day to make money if the stock goes up 1%, but if it only goes up .5% we\'ll still lose money.\\n5. It\'s possible the winning move is to scale back your position. Consider the scenario:\\n    - You invest money and the stock closes down the day .5%\\n    - You invest tomorrow expecting the stock to go up 1%\\n    - The winning investment to break even (assuming a 1% increase) is to scale back the position, since the shares you purchased at the beginning would then be profitable\\n\\n## Running the simulation\\n\\nSo now that we\'ve defined our investment formula,we need to tweak a couple things in order to make an investment strategy we can actually work with. There are two issues we need to address:\\n\\n1. The formula only tells us how much to invest if we want to break even ($d_n$). If we actually want to turn a profit, we need to invest more than that, which we will refer to as the **bias**.\\n2. The formula assumes we know what the closing price will be on any given day. If we don\'t know this, we can still invest assuming the stock price will close at a level we choose. If the price doesn\'t meet this objective, we try again tomorrow! This predetermined closing price will be referred to as the **expectation**.\\n\\nNow that we\'ve defined our *bias* and *expectation*, we can actually build a strategy we can simulate. Much like the martingale strategy told you to bet twice your previous bet in order to make money, we\'ve designed a system that tells us how much to bet in order to make money as well.\\n\\nNow, let\'s get to the code!\\n\\n[1]: https://en.wikipedia.org/wiki/Gambler\'s_ruin\\n[2]: https://en.wikipedia.org/wiki/Random_walk\\n[3]: https://en.wikipedia.org/wiki/Martingale_%28betting_system%29\\n\\n\\n```julia\\nusing Quandl\\napi_key = \\"\\"\\ndaily_investment = function(current_open, current_close, purchase_history, open_history)\\n    # We\'re not going to safeguard against divide by 0 - that\'s the user\'s responsibility\\n    t1 = current_close / current_open - 1\\n    t2 = sum(purchase_history - purchase_history*current_close ./ open_history)\\n    return t2 / t1\\nend;\\n```\\n\\nAnd let\'s code a way to run simulations quickly:\\n\\n\\n```julia\\nis_profitable = function(current_price, purchase_history, open_history)\\n    shares = sum(purchase_history ./ open_history)\\n    return current_price*shares > sum(purchase_history)\\nend\\n\\nsimulate = function(name, start, init, expected, bias)\\n    ticker_info = quandlget(name, from=start, api_key=api_key)\\n    open_vals = ticker_info[\\"Open\\"].values\\n    close_vals = ticker_info[\\"Close\\"].values\\n    invested = [init]\\n    \\n    # The simulation stops once we\'ve made a profit\\n    day = 1\\n    profitable = is_profitable(close_vals[day], invested, open_vals[1:length(invested)]) ||\\n        is_profitable(open_vals[day+1], invested, open_vals[1:length(invested)])\\n    while !profitable\\n        expected_close = open_vals[day+1] * expected\\n        todays_purchase = daily_investment(open_vals[day+1], expected_close, invested, open_vals[1:day])\\n        invested = [invested; todays_purchase + bias]\\n        # expected_profit = expected_close * sum(invested ./ open_vals[1:length(invested)]) - sum(invested)\\n        day += 1\\n        profitable = is_profitable(close_vals[day], invested, open_vals[1:length(invested)]) ||\\n            is_profitable(open_vals[day+1], invested, open_vals[1:length(invested)])\\n    end\\n    \\n    shares = sum(invested ./ open_vals[1:length(invested)])\\n    max_profit = max(close_vals[day], open_vals[day+1])\\n    profit = shares * max_profit - sum(invested)\\n    return (invested, profit)\\nend\\n\\nsim_summary = function(investments, profit)\\n    leverages = [sum(investments[1:i]) for i=1:length(investments)]\\n    max_leverage = maximum(leverages) / investments[1]\\n    println(\\"Max leverage: $(max_leverage)\\")\\n    println(\\"Days invested: $(length(investments))\\")\\n    println(\\"Profit: $profit\\")\\nend;\\n```\\n\\nNow, let\'s get some data and run a simulation! Our first test:\\n\\n- We\'ll invest 100 dollars in LMT, and expect that the stock will close up 1% every day. We\'ll invest $d_n$ + 10 dollars every day that we haven\'t turned a profit, and end the simulation once we\'ve made a profit.\\n\\n\\n```julia\\ninvestments, profit = simulate(\\"YAHOO/LMT\\", Date(2015, 11, 29), 100, 1.01, 10)\\nsim_summary(investments, profit)\\n```\\n\\n```\\n    Max leverage: 5.590373200042106\\n    Days invested: 5\\n    Profit: 0.6894803101560001\\n```\\n\\nThe result: We need to invest 5.6x our initial position over a period of 5 days to make approximately .69&#162;\\n\\n- Now let\'s try the same thing, but we\'ll assume the stock closes up 2% instead.\\n\\n\\n```julia\\ninvestments, profit = simulate(\\"YAHOO/LMT\\", Date(2015, 11, 29), 100, 1.02, 10)\\nsim_summary(investments, profit)\\n```\\n\\n```\\n    Max leverage: 1.854949900247809\\n    Days invested: 25\\n    Profit: 0.08304813163696423\\n```\\n\\nIn this example, we only get up to a 1.85x leveraged position, but it takes 25 days to turn a profit of 8&#162;\\n\\n## Summary\\n\\nWe\'ve defined an investment strategy that can tell us how much to invest when we know what the closing position of a stock will be. We can tweak the strategy to actually make money, but plenty of work needs to be done so that we can optimize the money invested.\\n\\nIn the next post I\'m going to post more information about some backtests and strategy tests on this strategy (unless of course this experiment actually produces a significant profit potential, and then I\'m keeping it for myself).\\n\\n### Side note and disclaimer\\n\\nThe claims made in this presentation about being able to guarantee making money are intended as a joke and do not constitute investment advice of any sort."},{"id":"2016/01/cloudy-in-seattle","metadata":{"permalink":"/2016/01/cloudy-in-seattle","source":"@site/blog/2016-01-23-cloudy-in-seattle/index.mdx","title":"Cloudy in Seattle","description":"Building on prior analysis, is Seattle\'s reputation as a depressing city actually well-earned?","date":"2016-01-23T12:00:00.000Z","tags":[],"readingTime":3.965,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/01/cloudy-in-seattle","title":"Cloudy in Seattle","date":"2016-01-23T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730683607000,"prevItem":{"title":"Guaranteed money maker","permalink":"/2016/02/guaranteed-money-maker"},"nextItem":{"title":"Complaining about the weather","permalink":"/2016/01/complaining-about-the-weather"}},"content":"Building on prior analysis, is Seattle\'s reputation as a depressing city actually well-earned?\\n\\n\x3c!-- truncate --\x3e\\n\\n```python\\nimport pickle\\nimport pandas as pd\\nimport numpy as np\\nfrom bokeh.plotting import output_notebook, figure, show\\nfrom bokeh.palettes import RdBu4 as Palette\\nfrom datetime import datetime\\nimport warnings\\n\\noutput_notebook()\\n```\\n\\n```\\nBokehJS successfully loaded.\\n```\\n\\n## Examining other cities\\n\\nAfter taking some time to explore how the weather in North Carolina stacked up over the past years, I was interested in doing the same analysis for other cities. Growing up with family from Binghamton, NY I was always told it was very cloudy there. And Seattle has a nasty reputation for being very depressing and cloudy. All said, the cities I want to examine are:\\n- Binghamton, NY\\n- Cary, NC\\n- Seattle, WA\\n- New York City, NY\\n\\nI\'d be interested to try this analysis worldwide at some point - comparing London and Seattle might be an interesting analysis. For now though, we\'ll stick with trying out the US data.\\n\\nThere will be plenty of charts. I want to know: **How has average cloud cover and precipitation chance changed over the years for each city mentioned?** This will hopefully tell us whether Seattle has actually earned its reputation for being a depressing city.\\n\\n\\n```python\\ncity_forecasts = pickle.load(open(\'city_forecasts.p\', \'rb\'))\\nforecasts_df = pd.DataFrame.from_dict(city_forecasts)\\n```\\n\\n\\n```python\\ncities = [\'binghamton\', \'cary\', \'nyc\', \'seattle\']\\ncity_colors = {cities[i]: Palette[i] for i in range(0, 4)}\\n\\ndef safe_cover(frame):\\n    if frame and \'cloudCover\' in frame:\\n        return frame[\'cloudCover\']\\n    else:\\n        return np.NaN\\n\\ndef monthly_avg_cloudcover(city, year, month):\\n    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),\\n                             end=datetime(year, month + 1, 1, 12),\\n                             freq=\'D\', closed=\'left\')\\n    cloud_cover_vals = list(map(lambda x: safe_cover(forecasts_df[city][x][\'currently\']), dates))\\n    cloud_cover_samples = len(list(filter(lambda x: x is not np.NaN, cloud_cover_vals)))\\n    # Ignore an issue with nanmean having all NaN values. We\'ll discuss the data issues below.\\n    with warnings.catch_warnings():\\n        warnings.simplefilter(\'ignore\')\\n        return np.nanmean(cloud_cover_vals), cloud_cover_samples\\n```\\n\\n\\n```python\\nyears = range(1990, 2016)\\ndef city_avg_cc(city, month):\\n    return [monthly_avg_cloudcover(city, y, month) for y in years]\\n\\nmonths = [\\n    (\'July\', 7),\\n    (\'August\', 8),\\n    (\'September\', 9),\\n    (\'October\', 10),\\n    (\'November\', 11)\\n]\\n\\nfor month, month_id in months:\\n    month_averages = {city: city_avg_cc(city, month_id) for city in cities}\\n    f = figure(title=\\"{} Average Cloud Cover\\".format(month),\\n               x_axis_label=\'Year\',\\n               y_axis_label=\'Cloud Cover Percentage\')\\n    for city in cities:\\n        f.line(years, [x[0] for x in month_averages[city]],\\n               legend=city, color=city_colors[city])\\n    show(f)\\n```\\n\\n![July average cloud cover chart](./1.png)\\n![August average cloud cover chart](./2.png)\\n![September average cloud cover chart](./3.png)\\n![October average cloud cover chart](./4.png)\\n![November average cloud cover chart](./5.png)\\n\\nWell, as it so happens it looks like there are some data issues. July\'s data is a bit sporadic, and 2013 seems to be missing from most months as well. I think really only two things can really be confirmed here:\\n- Seattle, specifically for the months of October and November, is in fact significantly more cloudy on average than are other cities\\n- All cities surveyed have seen average cloud cover decline over the months studied. There are data issues, but the trend seems clear.\\n\\nLet\'s now move from cloud cover data to looking at average rainfall chance.\\n\\n\\n```python\\ndef safe_precip(frame):\\n    if frame and \'precipProbability\' in frame:\\n        return frame[\'precipProbability\']\\n    else:\\n        return np.NaN\\n\\ndef monthly_avg_precip(city, year, month):\\n    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),\\n                             end=datetime(year, month + 1, 1, 12),\\n                             freq=\'D\', closed=\'left\')\\n    precip_vals = list(map(lambda x: safe_precip(forecasts_df[city][x][\'currently\']), dates))\\n    precip_samples = len(list(filter(lambda x: x is not np.NaN, precip_vals)))\\n    # Ignore an issue with nanmean having all NaN values. We\'ll discuss the data issues below.\\n    with warnings.catch_warnings():\\n        warnings.simplefilter(\'ignore\')\\n        return np.nanmean(precip_vals), precip_samples\\n\\ndef city_avg_precip(city, month):\\n    return [monthly_avg_precip(city, y, month) for y in years]\\n\\nfor month, month_id in months:\\n    month_averages = {city: city_avg_cc(city, month_id) for city in cities}\\n    f = figure(title=\\"{} Average Precipitation Chance\\".format(month),\\n               x_axis_label=\'Year\',\\n               y_axis_label=\'Precipitation Chance Percentage\')\\n    for city in cities:\\n        f.line(years, [x[0] for x in month_averages[city]],\\n               legend=city, color=city_colors[city])\\n    show(f)\\n```\\n\\n![July average precipitation chance chart](./6.png)\\n![August average precipitation chance chart](./7.png)\\n![September average precipitation chance chart](./8.png)\\n![October average precipitation chance chart](./9.png)\\n![November average precipitation chance chart](./10.png)\\n\\nThe same data issue caveats apply here: 2013 seems to be missing some data, and July has some issues as well. However, this seems to confirm the trends we saw with cloud cover:\\n- Seattle, specifically for the months of August, October, and November has had a consistently higher chance of rain than other cities surveyed.\\n- Average precipitation chance, just like average cloud cover, has been trending down over time.\\n\\n## Conclusion\\n\\nI have to admit I was a bit surprised after doing this analysis. Seattle showed a higher average cloud cover and average precipitation chance than did the other cities surveyed. Maybe Seattle is actually an objectively more depressing city to live in.\\n\\nWell that\'s all for weather data at the moment. It\'s been a great experiment, but I think this is about as far as I\'ll be able to get with weather data without some domain knowledge. Talk again soon!"},{"id":"2016/01/complaining-about-the-weather","metadata":{"permalink":"/2016/01/complaining-about-the-weather","source":"@site/blog/2016-01-01-complaining-about-the-weather/index.mdx","title":"Complaining about the weather","description":"Figuring out whether people should be complaining about the recent weather in North Carolina.","date":"2016-01-01T12:00:00.000Z","tags":[],"readingTime":7.475,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2016/01/complaining-about-the-weather","title":"Complaining about the weather","date":"2016-01-01T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730678252000,"prevItem":{"title":"Cloudy in Seattle","permalink":"/2016/01/cloudy-in-seattle"},"nextItem":{"title":"Testing Cramer","permalink":"/2015/12/testing-cramer"}},"content":"Figuring out whether people should be complaining about the recent weather in North Carolina.\\n\\n\x3c!-- truncate --\x3e\\n\\n```python\\nfrom bokeh.plotting import figure, output_notebook, show\\nfrom bokeh.palettes import PuBuGn9 as Palette\\nimport pandas as pd\\nimport numpy as np\\nfrom datetime import datetime\\nimport pickle\\n\\noutput_notebook()\\n```\\n\\n```\\nBokehJS successfully loaded.\\n```\\n\\nI\'m originally from North Carolina, and I\'ve been hearing a lot of people talking about how often it\'s been raining recently. They\'re excited for any day that has sun.\\n\\nSo I got a bit curious: Has North Carolina over the past few months actually had more cloudy and rainy days recently than in previous years? This shouldn\'t be a particularly challenging task, but I\'m interested to know if people\'s perceptions actually reflect reality.\\n\\nThe data we\'ll use comes from [forecast.io](https://forecast.io), since they can give us a cloud cover percentage. I\'ve gone ahead and retrieved the data to a pickle file, and included the [code that was used to generate it](#Generating-the-Forecast-file). First up: What was the average cloud cover in North Carolina during August - November, and how many days were cloudy? We\'re going to assume that a \\"cloudy\\" day is defined as any day in which the cloud cover is above 50%.\\n\\n```python\\ncity_forecasts = pickle.load(open(\'city_forecasts.p\', \'rb\'))\\nforecast_df = pd.DataFrame.from_dict(city_forecasts)\\n```\\n\\n```python\\ncary_forecast = forecast_df[\'cary\']\\nyears = range(1990, 2016)\\nmonths = range(7, 12)\\nmonths_str = [\'July\', \'August\', \'September\', \'October\', \'November\']\\n\\ndef safe_cover(frame):\\n    if frame and \'cloudCover\' in frame:\\n        return frame[\'cloudCover\']\\n    else:\\n        return np.NaN\\n\\ndef monthly_avg_cloudcover(year, month):\\n    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),\\n                             end=datetime(year, month + 1, 1, 12),\\n                             freq=\'D\', closed=\'left\')\\n    cloud_cover_vals = list(map(lambda x: safe_cover(cary_forecast[x][\'currently\']), dates))\\n    cloud_cover_samples = len(list(filter(lambda x: x is not np.NaN, cloud_cover_vals)))\\n    return np.nanmean(cloud_cover_vals), cloud_cover_samples\\n\\n\\nmonthly_cover_vals = [[monthly_avg_cloudcover(y, m)[0] for y in years] for m in months]\\n\\nf = figure(title=\'Monthly Average Cloud Cover\',\\n           x_range=(1990, 2015),\\n          x_axis_label=\'Year\')\\nfor x in range(0, len(months)):\\n    f.line(years, monthly_cover_vals[x], legend=months_str[x], color=Palette[x])\\nshow(f)\\n```\\n\\n![Monthly average cloud cover chart](./1.png)\\n\\nAs we can see from the chart above, on the whole the monthly average cloud cover has been generally trending down over time. The average cloud cover is also lower than it was last year - it seems people are mostly just complaining. There are some data issues that start in 2012 that we need to be aware of - the cloud cover percentage doesn\'t exist for all days. Even so, the data that we have seems to reflect the wider trend, so we\'ll assume for now that the missing data doesn\'t skew our results.\\n\\nThere\'s one more metric we want to check though - how many cloudy days were there? This is probably a better gauge of sentiment than the average monthly cover.\\n\\n\\n```python\\ndef monthly_cloudy_days(year, month):\\n    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),\\n                             end=datetime(year, month + 1, 1, 12),\\n                             freq=\'D\', closed=\'left\')\\n    cloud_cover_vals = list(map(lambda x: safe_cover(cary_forecast[x][\'currently\']), dates))\\n    cloud_cover_samples = len(list(filter(lambda x: x is not np.NaN, cloud_cover_vals)))\\n    cloudy_days = [cover > .5 for cover in cloud_cover_vals]\\n    return np.count_nonzero(cloudy_days), cloud_cover_samples\\n\\nmonthly_days_vals = [[monthly_cloudy_days(y, m)[0] for y in years] for m in months]\\nmonthly_cover_samples = [[monthly_cloudy_days(y, m)[1] for y in years] for m in months]\\n\\nf = figure(title=\'Monthly Cloudy Days\',\\n           x_range=(1990, 2015),\\n          x_axis_label=\'Year\')\\nfor x in range(0, len(months)):\\n    f.line(years, monthly_days_vals[x], legend=months_str[x], color=Palette[x])\\nshow(f)\\n\\nf = figure(title=\'Monthly Cloud Cover Samples\',\\n          x_range=(1990, 2015),\\n          x_axis_label=\'Year\',\\n          height=300)\\nfor x in range(0, len(months)):\\n    f.line(years, monthly_cover_samples[x], legend=months_str[x], color=Palette[x])\\nshow(f)\\n```\\n\\n![Monthly cloudy days chart](./2.png)\\n\\n![Monthly cloud cover samples chart](./3.png)\\n\\nOn the whole, the number of cloudy days seems to reflect the trend with average cloud cover - it\'s actually becoming more sunny as time progresses. That said, we need to be careful in how we view this number - because there weren\'t as many samples in 2015 as previous years, the number of days can get thrown off. In context though, even if most days not recorded were in fact cloudy, the overall count for 2015 would still be lower than previous years.\\n\\n# Tracking Precipitation Chances\\n\\nIn addition to checking cloud cover, I wanted to check precipitation data as well - what is the average precipitation chance over a month, and how many days during a month is rain likely? The thinking is that days with a high-precipitation chance will also be days in which it is cloudy or depressing.\\n\\n\\n```python\\ndef safe_precip(frame):\\n    if frame and \'precipProbability\' in frame:\\n        return frame[\'precipProbability\']\\n    else:\\n        return np.NaN\\n\\ndef monthly_avg_precip(year, month):\\n    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),\\n                             end=datetime(year, month + 1, 1, 12),\\n                             freq=\'D\', closed=\'left\')\\n    precip_vals = list(map(lambda x: safe_precip(cary_forecast[x][\'currently\']), dates))\\n    precip_samples = len(list(filter(lambda x: x is not np.NaN, precip_vals)))\\n    return np.nanmean(precip_vals), precip_samples\\n\\nmonthly_avg_precip_vals = [[monthly_avg_precip(y, m)[0] for y in years] for m in months]\\n\\nf = figure(title=\'Monthly Average Precipitation Chance\',\\n           x_range=(1990, 2015),\\n          x_axis_label=\'Year\')\\nfor x in range(0, len(months)):\\n    f.line(years, monthly_avg_precip_vals[x], legend=months_str[x], color=Palette[x])\\nshow(f)\\n```\\n\\n![Monthly average precipitation chance chart](./4.png)\\n\\nAs we can see from the chart, the average chance of precipitation over a month more or less stays within a band of 0 - .1 for all months over all years. This is further evidence that the past few months are no more cloudy or rainy than previous years. Like the cloud cover though, we still want to get a count of all the rainy days, in addition to the average chance. We\'ll define a \\"rainy day\\" as any day in which the chance of rain is greater than 25%.\\n\\n\\n```python\\ndef monthly_rainy_days(year, month):\\n    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),\\n                             end=datetime(year, month + 1, 1, 12),\\n                             freq=\'D\', closed=\'left\')\\n    precip_prob_vals = list(map(lambda x: safe_precip(cary_forecast[x][\'currently\']), dates))\\n    precip_prob_samples = len(list(filter(lambda x: x is not np.NaN, precip_prob_vals)))\\n    precip_days = [prob > .25 for prob in precip_prob_vals]\\n    return np.count_nonzero(precip_days), precip_prob_samples\\n\\nmonthly_precip_days_vals = [[monthly_rainy_days(y, m)[0] for y in years] for m in months]\\nmonthly_precip_samples = [[monthly_rainy_days(y, m)[1] for y in years] for m in months]\\n\\nf = figure(title=\'Monthly Rainy Days\',\\n           x_range=(1990, 2015),\\n          x_axis_label=\'Year\')\\nfor x in range(0, len(months)):\\n    f.line(years, monthly_precip_days_vals[x], legend=months_str[x], color=Palette[x])\\nshow(f)\\n\\nf = figure(title=\'Monthly Rainy Days Samples\',\\n          x_range=(1990, 2015),\\n          x_axis_label=\'Year\',\\n          height=300)\\nfor x in range(0, len(months)):\\n    f.line(years, monthly_precip_samples[x], legend=months_str[x], color=Palette[x])\\nshow(f)\\n```\\n\\n![Monthly rainy days chart](./5.png)\\n\\n![Monthly rainy days samples chart](./6.png)\\n\\nAfter trying to find the number of days that are rainy, we can see that November hit its max value for rainy days in 2015. However, that value is 6, as compared to a previous maximum of 5. While it is a new record, the value isn\'t actually all that different. And for other months, the values are mostly in-line with the averages.\\n\\n# Summary and Conclusions\\n\\nAfter having looked at forecast data for Cary, it appears that the months of July - November this year in terms of weather were at worst on par with prior years, if not slightly more sunny. This seems to be a case of confirmation bias: someone complains about a string of cloudy or rainy days, and suddenly you start noticing them more.\\n\\nWhile this analysis doesn\'t take into account other areas of North Carolina, my initial guess would be to assume that other areas also will show similar results: nothing interesting is happening. Maybe that will be for another blog post later!\\n\\nComing soon: I\'ll compare rain/cloud conditions in North Carolina to some other places in the U.S.!\\n\\n# Generating the Forecast file\\n\\nThe following code was generates the file that was used throughout the blog post. Please note that I\'m retrieving data for other cities to use in a future blog post, only Cary data was used for this post.\\n\\n```python\\nimport pandas as pd\\nfrom functools import reduce\\nimport requests\\nfrom datetime import datetime\\n\\n# Coordinate data from http://itouchmap.com/latlong.html\\ncary_loc = (35.79154,-78.781117)\\nnyc_loc = (40.78306,-73.971249)\\nseattle_loc = (47.60621,-122.332071)\\nbinghamton_loc = (42.098687,-75.917974)\\ncities = {\\n    \'cary\': cary_loc,\\n    \'nyc\': nyc_loc,\\n    \'seattle\': seattle_loc,\\n    \'binghamton\': binghamton_loc\\n}\\n\\napikey = \'\' # My super-secret API Key\\n\\ndef get_forecast(lat, long, date=None):\\n    forecast_base = \\"https://api.forecast.io/forecast/\\"\\n    if date is None:\\n        url = forecast_base + apikey + \'/{},{}\'.format(lat, long)\\n    else:\\n        epoch = int(date.timestamp())\\n        url = forecast_base + apikey + \'/{},{},{}\'.format(lat, long, epoch)\\n        \\n    return requests.get(url).json()\\n    \\nyears = range(1990,2016)\\n# For datetimes, the 12 is for getting the weather at noon.\\n# We\'re doing this over midnight because we\'re more concerned\\n# with what people see, and people don\'t typically see the weather\\n# at midnight.\\ndt_indices = [pd.date_range(start=datetime(year, 7, 1, 12),\\n                          end=datetime(year, 11, 30, 12))\\n              for year in years]\\ndt_merge = reduce(lambda x, y: x.union(y), dt_indices)\\n\\n# Because we have to pay a little bit to use the API, we use for loops here\\n# instead of a comprehension - if something breaks, we want to preserve the\\n# data already retrieved\\ncity_forecasts = {}\\nfor city, loc in cities.items():\\n    print(\\"Retrieving data for {} starting at {}\\".format(city,\\n                                                         datetime.now().strftime(\\"%I:%M:%S %p\\")))\\n    for dt in dt_merge:\\n        try:\\n            city_forecasts[(city, dt)] = get_forecast(*loc, dt)\\n        except Exception as e:\\n            print(e)\\n            city_forecasts[(city, dt)] = None\\nprint(\\"End forecast retrieval: {}\\".format(datetime.now().strftime(\\"%I:%M:%S %p\\")))\\n\\nimport pickle\\npickle.dump(city_forecasts, open(\'city_forecasts.p\', \'wb\'))\\n\\n### Output:\\n# Retrieving data for binghamton starting at 05:13:42 PM\\n# Retrieving data for seattle starting at 05:30:51 PM\\n# Retrieving data for nyc starting at 05:48:30 PM\\n# Retrieving data for cary starting at 06:08:32 PM\\n# End forecast retrieval: 06:25:21 PM\\n```"},{"id":"2015/12/testing-cramer","metadata":{"permalink":"/2015/12/testing-cramer","source":"@site/blog/2015-12-26-testing-cramer/index.mdx","title":"Testing Cramer","description":"Pursuant to attending a graduate school studying Financial Engineering, I\'ve been a fan of the Mad Money TV show featuring the bombastic Jim Cramer. One of the things that he\'s said is that you shouldn\'t use the futures to predict where the stock market is going to go. But he says it often enough, I\'ve begun to wonder - who is he trying to convince?","date":"2015-12-26T12:00:00.000Z","tags":[],"readingTime":8.23,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2015/12/testing-cramer","title":"Testing Cramer","date":"2015-12-26T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730678252000,"prevItem":{"title":"Complaining about the weather","permalink":"/2016/01/complaining-about-the-weather"},"nextItem":{"title":"Autocallable Bonds","permalink":"/2015/11/autocallable"}},"content":"Pursuant to attending a graduate school studying Financial Engineering, I\'ve been a fan of the [Mad Money][1] TV show featuring the bombastic Jim Cramer. One of the things that he\'s said is that you shouldn\'t use the futures to predict where the stock market is going to go. But he says it often enough, I\'ve begun to wonder - who is he trying to convince?\\n\\n\x3c!-- truncate --\x3e\\n\\nIt makes sense that because futures on things like the S&P 500 are traded continuously, they would price in market information before the stock market opens. So is Cramer right to be convinced that strategies based on the futures are a poor idea? I wanted to test it out.\\n\\nThe first question is where to get the future\'s data. I\'ve been part of [Seeking Alpha][2] for a bit, and they publish the [Wall Street Breakfast][3] newsletter which contains daily future\'s returns as of 6:20 AM EST. I\'d be interested in using that data to see if we can actually make some money.\\n\\nFirst though, let\'s get the data:\\n\\n## Downloading Futures data from Seeking Alpha\\n\\nWe\'re going to define two HTML parsing classes - one to get the article URL\'s from a page, and one to get the actual data from each article.\\n\\n[1]: http://www.cnbc.com/mad-money/\\n[2]: http://seekingalpha.com/\\n[3]: http://seekingalpha.com/author/wall-street-breakfast?s=wall-street-breakfast\\n\\n\\n```python\\nclass ArticleListParser(HTMLParser):\\n    \\"\\"\\"Given a web page with articles on it, parse out the article links\\"\\"\\"\\n    \\n    articles = []\\n    \\n    def handle_starttag(self, tag, attrs):\\n        #if tag == \'div\' and (\\"id\\", \\"author_articles_wrapper\\") in attrs:\\n        #    self.fetch_links = True\\n        if tag == \'a\' and (\'class\', \'dashboard_article_link\') in attrs:\\n            href = list(filter(lambda x: x[0] == \'href\', attrs))[0][1]\\n            self.articles.append(href)\\n            \\nbase_url = \\"http://seekingalpha.com/author/wall-street-breakfast/articles\\"\\narticle_page_urls = [base_url] + [base_url + \'/{}\'.format(i) for i in range(2, 20)]\\n\\nglobal_articles = []\\nfor page in article_page_urls:\\n    # We need to switch the user agent, as SA blocks the standard requests agent\\n    articles_html = requests.get(page,\\n                                headers={\\"User-Agent\\": \\"Wget/1.13.4\\"})\\n    parser = ArticleListParser()\\n    parser.feed(articles_html.text)\\n    global_articles += (parser.articles)\\n```\\n\\n\\n```python\\nclass ArticleReturnParser(HTMLParser):\\n    \\"Given an article, parse out the futures returns in it\\"\\n    \\n    record_font_tags = False\\n    in_font_tag = False\\n    counter = 0\\n    # data = {} # See __init__\\n    \\n    def __init__(self, *args, **kwargs):\\n        super().__init__(*args, **kwargs)\\n        self.data = {}\\n    \\n    def handle_starttag(self, tag, attrs):\\n        if tag == \'span\' and (\'itemprop\', \'datePublished\') in attrs:\\n            date_string = list(filter(lambda x: x[0] == \'content\', attrs))[0][1]\\n            date = dtparser.parse(date_string)\\n            self.data[\'date\'] = date\\n            \\n        self.in_font_tag = tag == \'font\'\\n        \\n    def safe_float(self, string):\\n        try:\\n            return float(string[:-1]) / 100\\n        except ValueError:\\n            return np.NaN\\n            \\n    def handle_data(self, content):\\n        if not self.record_font_tags and \\"Futures at 6\\" in content:\\n            self.record_font_tags = True\\n            \\n        if self.record_font_tags and self.in_font_tag:\\n            if self.counter == 0:\\n                self.data[\'DOW\'] = self.safe_float(content)\\n            elif self.counter == 1:\\n                self.data[\'S&P\'] = self.safe_float(content)\\n            elif self.counter == 2:\\n                self.data[\'NASDAQ\'] = self.safe_float(content)\\n            elif self.counter == 3:\\n                self.data[\'Crude\'] = self.safe_float(content)\\n            elif self.counter == 4:\\n                self.data[\'Gold\'] = self.safe_float(content)\\n            \\n            self.counter += 1\\n            \\n    def handle_endtag(self, tag):\\n        self.in_font_tag = False\\n\\ndef retrieve_data(url):\\n    sa = \\"http://seekingalpha.com\\"\\n    article_html = requests.get(sa + url,\\n                               headers={\\"User-Agent\\": \\"Wget/1.13.4\\"})\\n    parser = ArticleReturnParser()\\n    parser.feed(article_html.text)\\n    parser.data.update({\\"url\\": url})\\n    parser.data.update({\\"text\\": article_html.text})\\n    return parser.data\\n\\n# This copy **MUST** be in place. I\'m not sure why,\\n# as you\'d think that the data being returned would already\\n# represent a different memory location. Even so, it blows up\\n# if you don\'t do this.\\narticle_list = list(set(global_articles))\\narticle_data = [copy(retrieve_data(url)) for url in article_list]\\n# If there\'s an issue downloading the article, drop it.\\narticle_df = pd.DataFrame.from_dict(article_data).dropna()\\n```\\n\\n## Fetching the Returns data\\n\\nNow that we have the futures data, we\'re going to compare across 4 different indices - the S&P 500 index, Dow Jones Industrial, Russell 2000, and NASDAQ 100. Let\'s get the data off of Quandl to make things easier!\\n\\n\\n```python\\n# article_df is sorted by date, so we get the first row.\\nstart_date = article_df.sort_values(by=\'date\').iloc[0][\'date\'] - relativedelta(days=1)\\nSPY = Quandl.get(\\"GOOG/NYSE_SPY\\", trim_start=start_date)\\nDJIA = Quandl.get(\\"GOOG/AMS_DIA\\", trim_start=start_date)\\nRUSS = Quandl.get(\\"GOOG/AMEX_IWM\\", trim_start=start_date)\\nNASDAQ = Quandl.get(\\"GOOG/EPA_QQQ\\", trim_start=start_date)\\n```\\n\\n## Running the Comparison\\n\\nThere are two types of tests I want to determine: How accurate each futures category is at predicting the index\'s opening change over the close before, and predicting the index\'s daily return.\\n\\nLet\'s first calculate how good each future is at predicting the opening return over the previous day. I expect that the futures will be more than 50% accurate, since the information is recorded 3 hours before the markets open.\\n\\n\\n```python\\ndef calculate_opening_ret(frame):\\n    # I\'m not a huge fan of the appending for loop,\\n    # but it\'s a bit verbose for a comprehension\\n    data = {}\\n    for i in range(1, len(frame)):\\n        date = frame.iloc[i].name\\n        prior_close = frame.iloc[i-1][\'Close\']\\n        open_val = frame.iloc[i][\'Open\']\\n        data[date] = (open_val - prior_close) / prior_close\\n        \\n    return data\\n\\nSPY_open_ret = calculate_opening_ret(SPY)\\nDJIA_open_ret = calculate_opening_ret(DJIA)\\nRUSS_open_ret = calculate_opening_ret(RUSS)\\nNASDAQ_open_ret = calculate_opening_ret(NASDAQ)\\n\\ndef signs_match(list_1, list_2):\\n    # This is a surprisingly difficult task - we have to match\\n    # up the dates in order to check if opening returns actually match\\n    index_dict_dt = {key.to_datetime(): list_2[key] for key in list_2.keys()}\\n    \\n    matches = []\\n    for row in list_1.iterrows():\\n        row_dt = row[1][1]\\n        row_value = row[1][0]\\n        index_dt = datetime(row_dt.year, row_dt.month, row_dt.day)\\n        if index_dt in list_2:\\n            index_value = list_2[index_dt]\\n            if (row_value > 0 and index_value > 0) or \\\\\\n                (row_value < 0 and index_value < 0) or \\\\\\n                (row_value == 0 and index_value == 0):\\n                    matches += [1]\\n            else:\\n                matches += [0]\\n            #print(\\"{}\\".format(list_2[index_dt]))\\n    return matches\\n    \\n    \\nprediction_dict = {}\\nmatches_dict = {}\\ncount_dict = {}\\nindex_dict = {\\"SPY\\": SPY_open_ret, \\"DJIA\\": DJIA_open_ret, \\"RUSS\\": RUSS_open_ret, \\"NASDAQ\\": NASDAQ_open_ret}\\nindices = [\\"SPY\\", \\"DJIA\\", \\"RUSS\\", \\"NASDAQ\\"]\\nfutures = [\\"Crude\\", \\"Gold\\", \\"DOW\\", \\"NASDAQ\\", \\"S&P\\"]\\nfor index in indices:\\n    matches_dict[index] = {future: signs_match(article_df[[future, \'date\']],\\n                                               index_dict[index]) for future in futures}\\n    count_dict[index] = {future: len(matches_dict[index][future]) for future in futures}\\n    prediction_dict[index] = {future: np.mean(matches_dict[index][future])\\n                              for future in futures}\\nprint(\\"Articles Checked: \\")\\nprint(pd.DataFrame.from_dict(count_dict))\\nprint()\\nprint(\\"Prediction Accuracy:\\")\\nprint(pd.DataFrame.from_dict(prediction_dict))\\n```\\n\\n```\\n    Articles Checked: \\n            DJIA  NASDAQ  RUSS  SPY\\n    Crude    268     268   271  271\\n    DOW      268     268   271  271\\n    Gold     268     268   271  271\\n    NASDAQ   268     268   271  271\\n    S&P      268     268   271  271\\n    \\n    Prediction Accuracy:\\n                DJIA    NASDAQ      RUSS       SPY\\n    Crude   0.544776  0.522388  0.601476  0.590406\\n    DOW     0.611940  0.604478  0.804428  0.841328\\n    Gold    0.462687  0.455224  0.464945  0.476015\\n    NASDAQ  0.615672  0.608209  0.797048  0.830258\\n    S&P     0.604478  0.597015  0.811808  0.848708\\n```\\n\\nThis data is very interesting. Some insights:\\n\\n- Both DOW and NASDAQ futures are pretty bad at predicting their actual market openings\\n- NASDAQ and Dow are fairly unpredictable; Russell 2000 and S&P are very predictable\\n- Gold is a poor predictor in general - intuitively Gold should move inverse to the market, but it appears to be about as accurate as a coin flip.\\n\\nAll said though it appears that futures data is important for determining market direction for both the S&P 500 and Russell 2000. Cramer is half-right: futures data isn\'t very helpful for the Dow and NASDAQ indices, but is great for the S&P and Russell indices.\\n\\n## The next step - Predicting the close\\n\\nGiven the code we currently have, I\'d like to predict the close of the market as well. We can re-use most of the code, so let\'s see what happens:\\n\\n\\n```python\\ndef calculate_closing_ret(frame):\\n    # I\'m not a huge fan of the appending for loop,\\n    # but it\'s a bit verbose for a comprehension\\n    data = {}\\n    for i in range(0, len(frame)):\\n        date = frame.iloc[i].name\\n        open_val = frame.iloc[i][\'Open\']\\n        close_val = frame.iloc[i][\'Close\']\\n        data[date] = (close_val - open_val) / open_val\\n        \\n    return data\\n\\nSPY_close_ret = calculate_closing_ret(SPY)\\nDJIA_close_ret = calculate_closing_ret(DJIA)\\nRUSS_close_ret = calculate_closing_ret(RUSS)\\nNASDAQ_close_ret = calculate_closing_ret(NASDAQ)\\n\\ndef signs_match(list_1, list_2):\\n    # This is a surprisingly difficult task - we have to match\\n    # up the dates in order to check if opening returns actually match\\n    index_dict_dt = {key.to_datetime(): list_2[key] for key in list_2.keys()}\\n    \\n    matches = []\\n    for row in list_1.iterrows():\\n        row_dt = row[1][1]\\n        row_value = row[1][0]\\n        index_dt = datetime(row_dt.year, row_dt.month, row_dt.day)\\n        if index_dt in list_2:\\n            index_value = list_2[index_dt]\\n            if (row_value > 0 and index_value > 0) or \\\\\\n                (row_value < 0 and index_value < 0) or \\\\\\n                (row_value == 0 and index_value == 0):\\n                    matches += [1]\\n            else:\\n                matches += [0]\\n            #print(\\"{}\\".format(list_2[index_dt]))\\n    return matches\\n    \\n    \\nmatches_dict = {}\\ncount_dict = {}\\nprediction_dict = {}\\nindex_dict = {\\"SPY\\": SPY_close_ret, \\"DJIA\\": DJIA_close_ret,\\n              \\"RUSS\\": RUSS_close_ret, \\"NASDAQ\\": NASDAQ_close_ret}\\nindices = [\\"SPY\\", \\"DJIA\\", \\"RUSS\\", \\"NASDAQ\\"]\\nfutures = [\\"Crude\\", \\"Gold\\", \\"DOW\\", \\"NASDAQ\\", \\"S&P\\"]\\nfor index in indices:\\n    matches_dict[index] = {future: signs_match(article_df[[future, \'date\']],\\n                                               index_dict[index]) for future in futures}\\n    count_dict[index] = {future: len(matches_dict[index][future]) for future in futures}\\n    prediction_dict[index] = {future: np.mean(matches_dict[index][future])\\n                              for future in futures}\\n    \\nprint(\\"Articles Checked:\\")\\nprint(pd.DataFrame.from_dict(count_dict))\\nprint()\\nprint(\\"Prediction Accuracy:\\")\\nprint(pd.DataFrame.from_dict(prediction_dict))\\n```\\n\\n```\\n    Articles Checked:\\n            DJIA  NASDAQ  RUSS  SPY\\n    Crude    268     268   271  271\\n    DOW      268     268   271  271\\n    Gold     268     268   271  271\\n    NASDAQ   268     268   271  271\\n    S&P      268     268   271  271\\n    \\n    Prediction Accuracy:\\n                DJIA    NASDAQ      RUSS       SPY\\n    Crude   0.533582  0.529851  0.501845  0.542435\\n    DOW     0.589552  0.608209  0.535055  0.535055\\n    Gold    0.455224  0.451493  0.483395  0.512915\\n    NASDAQ  0.582090  0.626866  0.531365  0.538745\\n    S&P     0.585821  0.608209  0.535055  0.535055\\n```\\n\\nWell, it appears that the futures data is terrible at predicting market close. NASDAQ predicting NASDAQ is the most interesting data point, but 63% accuracy isn\'t accurate enough to make money consistently.\\n\\n## Final sentiments\\n\\nThe data bears out very close to what I expected would happen:\\n\\n- Futures data is more accurate than a coin flip for predicting openings, which makes sense since it is recorded only 3 hours before the actual opening\\n- Futures data is about as acccurate as a coin flip for predicting closings, which means there is no money to be made in trying to predict the market direction for the day given the futures data.\\n\\nIn summary:\\n\\n- Cramer is half right: Futures data is not good for predicting the market open of the Dow and NASDAQ indices. Contrary to Cramer though, it is very good for predicting the S&P and Russell indices - we can achieve an accuracy slightly over 80% for each. \\n- Making money in the market is hard. We can\'t just go to the futures and treat them as an oracle for where the market will close.\\n\\nI hope you\'ve enjoyed this, I quite enjoyed taking a deep dive in the analytics this way. I\'ll be posting more soon!"},{"id":"2015/11/autocallable","metadata":{"permalink":"/2015/11/autocallable","source":"@site/blog/2015-11-27-autocallable/index.mdx","title":"Autocallable Bonds","description":"For a final project, my group was tasked with understanding three exotic derivatives: The Athena, Phoenix without memory, and Phoenix with memory autocallable products.","date":"2015-11-27T12:00:00.000Z","tags":[],"readingTime":11.085,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2015/11/autocallable","title":"Autocallable Bonds","date":"2015-11-27T12:00:00.000Z","authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1730678252000,"prevItem":{"title":"Testing Cramer","permalink":"/2015/12/testing-cramer"},"nextItem":{"title":"Welcome, and an algorithm","permalink":"/2015/11/welcome"}},"content":"For a final project, my group was tasked with understanding three exotic derivatives: The Athena, Phoenix without memory, and Phoenix with memory autocallable products.\\n\\n\x3c!-- truncate --\x3e\\n\\nMy only non-core class this semester has been in Structure Products. We\'ve been surveying a wide variety of products, and the final project was to pick one to report on.\\nBecause these are all very similar, we decided to demonstrate all 3 products at once.\\n\\nWhat follows below is a notebook demonstrating the usage of [Julia](http://julialang.com) for Monte-Carlo simulation of some exotic products.\\n\\n---\\n\\n```julia\\nusing Gadfly\\n```\\n\\n# Athena/Phoenix Simulation\\n\\n## Underlying simulation\\n\\nIn order to price the autocallable bonds, we need to simulate the underlying assets. Let\'s go ahead and set up the simulation first, as this lays the foundation for what we\'re trying to do. We\'re going to use [JNJ](http://finance.yahoo.com/q?s=jnj) as the basis for our simulation. This implies the following parameters:\\n\\n- $S_0$ = \\\\$102.2 (as of time of writing)\\n- $q$ = 2.84%\\n- $r$ = [.49, .9, 1.21, 1.45, 1.69] (term structure as of time of writing, linear interpolation)\\n- $\\\\mu$ = $r - q$ (note that this implies a negative drift because of current low rates)\\n- $\\\\sigma$ = $\\\\sigma_{imp}$ = 15.62% (from VIX implied volatility)\\n\\nWe additionally define some parameters for simulation:\\n\\n- `T`: The number of years to simulate\\n- `m`: The number of paths to simulate\\n- `n`: The number of steps to simulate in a year\\n\\n\\n```julia\\nS0 = 102.2\\nnominal = 100\\nq = 2.84 / 100\\n\u03C3 = 15.37 / 100\\nterm = [0, .49, .9, 1.21, 1.45, 1.69] / 100 + 1\\n\\n###\\n# Potential: Based on PEP\\n# S0 = 100.6\\n# \u03C3 = 14.86\\n# q = 2.7\\n###\\n\\n# Simulation parameters\\nT = 5 # Using years as the unit of time\\nn = 250 # simulations per year\\nm = 100000 # paths\\nnum_simulations = 5; # simulation rounds per price\\n```\\n\\n### Defining the simulation\\nTo make things simpler, we simulate a single year at a time. This allows us to easily add in a dividend policy without too much difficulty, and update the simulation every year to match the term structure. The underlying uses GBM for simulation between years.\\n\\n\\n```julia\\nsimulate_gbm = function(S0, \u03BC, \u03C3, T, n)\\n    # Set the initial state\\n    m = length(S0)\\n    t = T / n\\n    motion = zeros(m, n)\\n    motion[:,1] = S0\\n    \\n    # Build out all states\\n    for i=1:(n-1)\\n        motion[:,i+1] = motion[:,i] .* exp((\u03BC - \u03C3^2/2)*t) .* exp(sqrt(t) * \u03C3 .* randn(m))\\n    end\\n    \\n    return motion\\nend\\n\\nfunction display_motion(motion, T)\\n    # Given a matrix of paths, display the motion\\n    n = length(motion[1,:])\\n    m = length(motion[:,1])\\n    x = repmat(1:n, m)\\n    \\n    # Calculate the ticks we\'re going to use. We\'d like to\\n    # have an xtick every month, so calculate where those\\n    # ticks will actually be at.\\n    if (T > 3)\\n        num_ticks = T\\n        xlabel = \\"Years\\"\\n    else\\n        num_ticks = T * 12\\n        xlabel = \\"Months\\"\\n    end\\n    tick_width = n / num_ticks\\n    x_ticks = []\\n    for i=1:round(num_ticks)\\n        x_ticks = vcat(x_ticks, i*tick_width)\\n    end\\n    \\n    # Use one color for each path. I\'m not sure if there\'s\\n    # a better way to do this without going through DataFrames\\n    colors = []\\n    for i = 1:m\\n        colors = vcat(colors, ones(n)*i)\\n    end\\n    \\n    plot(x=x, y=motion\', color=colors, Geom.line,\\n    Guide.xticks(ticks=x_ticks, label=false),\\n    Guide.xlabel(xlabel),\\n    Guide.ylabel(\\"Value\\"))\\nend;\\n```\\n\\n### Example simulation\\n\\nLet\'s go ahead and run a sample simulation to see what the functions got us!\\n\\n\\n```julia\\ninitial = ones(5) * S0\\n# Using \u03BC=0, T=.25 for now, we\'ll use the proper values later\\nmotion = simulate_gbm(initial, 0, \u03C3, .25, 200) \\n\\ndisplay_motion(motion, .25)\\n```\\n\\n![](./_notebook_files/_notebook_6_0.svg)\\n\\n### Computing the term structure\\n\\nNow that we\'ve got the basic motion set up, let\'s start making things a bit more sophisticated for the model. We\'re going to assume that the drift of the stock is the difference between the implied forward rate and the quarterly dividend rate.\\n\\nWe\'re given the yearly term structure, and need to calculate the quarterly forward rate to match this structure. The term structure is assumed to follow:\\n\\n$d(0, t) = d(0,t-1)\\\\cdot f_{i-1, i}$\\n\\nWhere $f_{i-1, i}$ is the quarterly forward rate.\\n\\n```julia\\nforward_term = function(yearly_term)\\n    # It is assumed that we have a yearly term structure passed in, and starts at year 0\\n    # This implies a nominal rate above 0 for the first year!\\n    years = length(term)-1 # because we start at 0\\n    structure = [(term[i+1] / term[i]) for i=1:years]\\nend;\\n```\\n\\n### Illustrating the term structure\\n\\nNow that we\'ve got our term structure, let\'s validate that we\'re getting the correct results! If we\'ve done this correctly, then:\\n\\n```\\nterm[2] == term[1] * structure[1]\\n```\\n\\n```julia\\n# Example term structure taken from:\\n# http://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/TextView.aspx?data=yield\\n# Linear interpolation used years in-between periods, assuming real-dollar\\n# interest rates\\nforward_yield = forward_term(term)\\ncalculated_term2 = term[1] * forward_yield[1]\\n\\nprintln(\\"Actual term[2]: $(term[2]); Calculated term[2]: $(calculated_term2)\\")\\n```\\n\\n```\\n    Actual term[2]: 1.0049; Calculated term[2]: 1.0049\\n```\\n\\n### The full underlying simulation\\n\\nNow that we have the term structure set up, we can actually start doing some real simulation! Let\'s construct some paths through the full 5-year time frame. In order to do this, we will simulate 1 year at a time, and use the forward rates at those times to compute the drift. Thus, there will be 5 total simulations batched together.\\n\\n\\n```julia\\nfull_motion = ones(5) * S0\\nfull_term = vcat(term[1], forward_yield)\\nfor i=1:T\\n    \u03BC = (full_term[i] - 1 - q)\\n    year_motion = simulate_gbm(full_motion[:,end], \u03BC, \u03C3, 1, n)\\n    full_motion = hcat(full_motion, year_motion)\\nend\\n\\ndisplay_motion(full_motion, T)\\n```\\n\\n![](./_notebook_files/_notebook_12_0.svg)\\n\\n### Final simulation\\n\\nWe\'re now going to actually build out the full motion that we\'ll use for computing the pricing of our autocallable products. It will be largely the same, but we will use far more sample paths for the simulation.\\n\\n\\n```julia\\nfull_simulation = function(S0, T, n, m, term)\\n    forward = vcat(term[1], forward_term(term))\\n\\n    # And an S0 to kick things off.\\n    final_motion = ones(m) * S0\\n    for i=1:T\\n        \u03BC = (forward[i] - 1 - q)\\n        year_motion = simulate_gbm(final_motion[:,end], \u03BC, \u03C3, 1, n)\\n        final_motion = hcat(final_motion, year_motion)\\n    end\\n    return final_motion\\nend\\n\\ntic()\\nfull_simulation(S0, T, n, m, term)\\ntime = toq()\\n@printf(\\"Time to run simulation: %.2fs\\", time)\\n```\\n\\n```\\n    Time to run simulation: 5.34s\\n```\\n\\n## Athena Simulation\\n\\nNow that we\'ve defined our underlying simulation, let\'s actually try and price an Athena note. Athena has the following characteristics:\\n\\n- Automatically called if the underlying is above the **call barrier** at observation\\n- Accelerated coupon paid if the underlying is above the **call barrier** at observation\\n    - The coupon paid is $c \\\\cdot i$ with $i$ as the current year, and $c$ the coupon rate\\n- Principle protection up until a **protection barrier** at observation; All principle at risk if this barrier not met\\n- Observed yearly\\n\\n```julia\\ncall_barrier = S0\\nstrike = S0\\nprotection_barrier = S0 * .6\\ncoupon = nominal * .07\\n\\nprice_athena = function(initial_price, year_prices, call_barrier,\\n        protection_barrier, coupon, forward_structure)\\n\\n    total_coupons = 0\\n    \\n    t = length(year_prices)\\n\\n    for i=1:t\\n        price = year_prices[i]\\n        if price \u2265 call_barrier\\n            return (nominal + coupon*i) * exp((prod(forward_structure[i:end])-1)*(t-i))\\n        end\\n    end\\n\\n    # We\'ve reached maturity, time to check capital protection\\n    if year_prices[end] > protection_barrier\\n        return nominal\\n    else\\n        put = (strike - year_prices[end]) / strike\\n        return nominal*(1-put)\\n    end\\nend\\n\\nforward_structure = forward_term(term)\\nprice_function = (year_prices) -> price_athena(S0, year_prices,\\n    call_barrier, protection_barrier, coupon, forward_structure)\\n\\nathena = function()\\n    year_indexes = [n*i for i=1:T]\\n    motion = full_simulation(S0, T, n, m, term)\\n    payoffs = [price_function(motion[i, year_indexes]) for i=1:m]\\n    return mean(payoffs)\\nend\\n\\nmean_payoffs = zeros(num_simulations)\\nfor i=1:num_simulations\\n    tic()\\n    mean_payoffs[i] = athena()\\n    time = toq()\\n    @printf(\\"Mean of simulation %i: \\\\$%.4f; Simulation time: %.2fs\\\\n\\", i, mean_payoffs[i], time)\\nend\\n\\nfinal_mean = mean(mean_payoffs)\\nprintln(\\"Mean over $num_simulations simulations: $(mean(mean_payoffs))\\")\\npv = final_mean * (exp(-(prod(forward_structure)-1)*T))\\n@printf(\\"Present value of Athena note: \\\\$%.2f, notional: \\\\$%.2f\\", pv, nominal)\\n```\\n\\n```\\n    Mean of simulation 1: $103.2805; Simulation time: 5.59s\\n    Mean of simulation 2: $103.3796; Simulation time: 5.05s\\n    Mean of simulation 3: $103.4752; Simulation time: 5.18s\\n    Mean of simulation 4: $103.4099; Simulation time: 5.37s\\n    Mean of simulation 5: $103.3260; Simulation time: 5.32s\\n    Mean over 5 simulations: 103.37421610015554\\n    Present value of Athena note: $95.00, notional: $100.00\\n```\\n\\n## Phoenix without Memory Simulation\\n\\nLet\'s move into pricing a Phoenix without memory. It\'s very similar to the Athena production, with the exception that we introduce a coupon barrier so coupons are paid even when the underlying is below the initial price.\\n\\nThe Phoenix product has the following characteristics (example [here](https://www.rbccm.com/usstructurednotes/file-780079.pdf)):\\n\\n- Automatically called if the underlying is above the **call barrier** at observation\\n- Coupon paid if the underlying is above a **coupon barrier** at observation\\n- Principle protection up until a **protection barrier** at observation; All principle at risk if this barrier not met\\n- Observed yearly\\n\\nSome example paths (all assume that a call barrier of the current price, and coupon barrier some level below that):\\n\\n- At the end of year 1, the stock is above the call barrier; the note is called and you receive the value of the stock plus the coupon being paid.\\n- At the end of year 1, the stock is above the coupon barrier, but not the call barrier; you receive the coupon. At the end of year 2, the stock is below the coupon barrier; you receive nothing. At the end of year 3, the stock is above the call barrier; the note is called and you receive the value of the stock plus a coupon for year 3.\\n\\nWe\'re going to re-use the same simulation, with the following parameters:\\n\\n- Call barrier: 100%\\n- Coupon barrier: 70%\\n- Coupon: 6%\\n- Capital protection until 70% (at maturity)\\n\\n\\n```julia\\ncall_barrier = S0\\ncoupon_barrier = S0 * .8\\nprotection_barrier = S0 * .6\\ncoupon = nominal * .06\\n\\nprice_phoenix_no_memory = function(initial_price, year_prices, call_barrier, coupon_barrier,\\n        protection_barrier, coupon, forward_structure)\\n\\n    total_coupons = 0\\n    t = length(year_prices)\\n\\n    for i=1:t\\n        price = year_prices[i]\\n        if price \u2265 call_barrier\\n            return (nominal + coupon + total_coupons)*exp((prod(forward_structure[i:end])-1)*(t-i))\\n        elseif price \u2265 coupon_barrier\\n            total_coupons = total_coupons * exp(forward_structure[i]-1) + coupon\\n        else\\n            total_coupons *= exp(forward_structure[i]-1)\\n        end\\n    end\\n\\n    # We\'ve reached maturity, time to check capital protection\\n    if year_prices[end] > protection_barrier\\n        return nominal + total_coupons\\n    else\\n        put = (strike - year_prices[end]) / strike\\n        return nominal*(1-put)\\n    end\\nend\\n\\nforward_structure = forward_term(term)\\nprice_function = (year_prices) -> price_phoenix_no_memory(S0, year_prices,\\n    call_barrier, coupon_barrier, protection_barrier, coupon, forward_structure)\\n\\nphoenix_no_memory = function()\\n    year_indexes = [n*i for i=1:T]\\n    motion = full_simulation(S0, T, n, m, term)\\n    payoffs = [price_function(motion[i, year_indexes]) for i=1:m]\\n    return mean(payoffs)\\nend\\n\\nmean_payoffs = zeros(num_simulations)\\nfor i=1:num_simulations\\n    tic()\\n    mean_payoffs[i] = phoenix_no_memory()\\n    time = toq()\\n    @printf(\\"Mean of simulation %i: \\\\$%.4f; Simulation time: %.2fs\\\\n\\", i, mean_payoffs[i], time)\\nend\\n\\nfinal_mean = mean(mean_payoffs)\\nprintln(\\"Mean over $num_simulations simulations: $(mean(mean_payoffs))\\")\\npv = final_mean * exp(-(prod(forward_structure)-1)*(T))\\n@printf(\\"Present value of Phoenix without memory note: \\\\$%.2f\\", pv)\\n```\\n\\n```\\n    Mean of simulation 1: $106.0562; Simulation time: 5.72s\\n    Mean of simulation 2: $106.0071; Simulation time: 5.85s\\n    Mean of simulation 3: $105.9959; Simulation time: 5.87s\\n    Mean of simulation 4: $106.0665; Simulation time: 5.93s\\n    Mean of simulation 5: $106.0168; Simulation time: 5.81s\\n    Mean over 5 simulations: 106.02850857209883\\n    Present value of Phoenix without memory note: $97.44\\n```\\n\\n## Phoenix with Memory Simulation\\n\\nThe Phoenix with Memory structure is very similar to the Phoenix, but as the name implies, has a special \\"memory\\" property: **It remembers any coupons that haven\'t been paid at prior observation times, and pays them all if the underlying crosses the coupon barrier**. For example:\\n- Note issued with 100% call barrier, 70% coupon barrier. At year 1, the underlying is at 50%, so no coupons are paid. At year 2, the underlying is at 80%, so coupons for both year 1 and 2 are paid, resulting in a double coupon.\\n\\nYou can also find an example [here](https://www.rbccm.com/usstructurednotes/file-781232.pdf).\\n\\nLet\'s go ahead and set up the simulation! The parameters will be the same, but we can expect that the value will go up because of the memory attribute\\n\\n\\n```julia\\ncall_barrier = S0\\ncoupon_barrier = S0 * .8\\nprotection_barrier = S0 * .6\\ncoupon = nominal * .07\\n\\nprice_phoenix_with_memory = function(initial_price, year_prices, call_barrier,\\n    coupon_barrier, protection_barrier, coupon, forward_structure)\\n\\n    last_coupon = 0\\n    total_coupons = 0\\n    \\n    t = length(year_prices)\\n\\n    for i=1:t\\n        price = year_prices[i]\\n        if price > call_barrier\\n            return (nominal + coupon + total_coupons)*exp((prod(forward_structure[i:end])-1)*(t-i))\\n        elseif price > coupon_barrier\\n            ####################################################################\\n            # The only difference between with/without memory is the below lines\\n            memory_coupons = (i - last_coupon) * coupon\\n            last_coupon = i\\n            total_coupons = total_coupons * exp(forward_structure[i]-1) + memory_coupons\\n            ####################################################################\\n        else\\n            total_coupons *= exp(forward_structure[i]-1)\\n        end\\n    end\\n\\n    # We\'ve reached maturity, time to check capital protection\\n    if year_prices[end] > protection_barrier\\n        return nominal + total_coupons\\n    else\\n        put = (strike - year_prices[end]) / strike\\n        return nominal*(1-put)\\n    end\\nend\\n\\nforward_structure = forward_term(term)\\nprice_function = (year_prices) -> price_phoenix_with_memory(S0, year_prices,\\n    call_barrier, coupon_barrier, protection_barrier, coupon, forward_structure)\\n\\nphoenix_with_memory = function()\\n    year_indexes = [n*i for i=1:T]\\n    motion = full_simulation(S0, T, n, m, term)\\n    payoffs = [price_function(motion[i, year_indexes]) for i=1:m]\\n    return mean(payoffs)\\nend\\n\\nmean_payoffs = zeros(num_simulations)\\nfor i=1:num_simulations\\n    tic()\\n    mean_payoffs[i] = phoenix_with_memory()\\n    time = toq()\\n    @printf(\\"Mean of simulation %i: \\\\$%.4f; Simulation time: %.2fs\\\\n\\",\\n        i, mean_payoffs[i], time)\\nend\\n\\nfinal_mean = mean(mean_payoffs)\\nprintln(\\"Mean over $num_simulations simulations: $(mean(mean_payoffs))\\")\\npv = final_mean * exp(-(prod(forward_structure)-1)*(T))\\n@printf(\\"Present value of Phoenix with memory note: \\\\$%.2f\\", pv)\\n```\\n\\n```\\n    Mean of simulation 1: $108.8612; Simulation time: 5.89s\\n    Mean of simulation 2: $109.0226; Simulation time: 5.90s\\n    Mean of simulation 3: $108.9175; Simulation time: 5.92s\\n    Mean of simulation 4: $108.9426; Simulation time: 5.94s\\n    Mean of simulation 5: $108.8087; Simulation time: 6.06s\\n    Mean over 5 simulations: 108.91052564051816\\n    Present value of Phoenix with memory note: $100.09\\n```"},{"id":"2015/11/welcome","metadata":{"permalink":"/2015/11/welcome","source":"@site/blog/2015-11-14-welcome/index.mdx","title":"Welcome, and an algorithm","description":"Hello! Glad to meet you. I\'m currently a student at Columbia University studying Financial Engineering, and want to give an overview of the projects I\'m working on!","date":"2015-11-19T12:00:00.000Z","tags":[],"readingTime":4.175,"hasTruncateMarker":true,"authors":[{"name":"Bradlee Speice","socials":{"github":"https://github.com/bspeice"},"key":"bspeice","page":null}],"frontMatter":{"slug":"2015/11/welcome","title":"Welcome, and an algorithm","date":"2015-11-19T12:00:00.000Z","last_update":{"date":"2015-12-05T12:00:00.000Z"},"authors":["bspeice"],"tags":[]},"unlisted":false,"lastUpdatedAt":1449316800000,"prevItem":{"title":"Autocallable Bonds","permalink":"/2015/11/autocallable"}},"content":"Hello! Glad to meet you. I\'m currently a student at Columbia University studying Financial Engineering, and want to give an overview of the projects I\'m working on!\\n\\n\x3c!-- truncate --\x3e\\n\\nTo start things off, Columbia has been hosting a trading competition that myself and another partner are competing in. I\'m including a notebook of the algorithm that we\'re using, just to give a simple overview of a miniature algorithm.\\n\\nThe competition is scored in 3 areas:\\n\\n- Total return\\n- [Sharpe ratio](https://en.wikipedia.org/wiki/Sharpe_ratio)\\n- Maximum drawdown\\n\\nOur algorithm uses a basic momentum strategy: in the given list of potential portfolios, pick the stocks that have been performing well in the past 30 days. Then, optimize for return subject to the drawdown being below a specific level. We didn\'t include the Sharpe ratio as a constraint, mostly because we were a bit late entering the competition.\\n\\nI\'ll be updating this post with the results of our algorithm as they come along!\\n\\n---\\n\\n**UPDATE 12/5/2015**: Now that the competition has ended, I wanted to update how the algorithm performed. Unfortunately, it didn\'t do very well. I\'m planning to make some tweaks over the coming weeks, and do another forward test in January.\\n\\n- After week 1: Down .1%\\n- After week 2: Down 1.4%\\n- After week 3: Flat\\n\\nAnd some statistics for all teams participating in the competition:\\n\\n| Statistic          | Value  |\\n|--------------------|--------|\\n| Max Return         | 74.1%  |\\n| Min Return         | -97.4% |\\n| Average Return     | -.1%   |\\n| Std Dev of Returns | 19.6%  |\\n\\n---\\n\\n## Trading Competition Optimization\\n\\n**Goal: Max return given maximum Sharpe and Drawdown**\\n\\n\\n```python\\nfrom IPython.display import display\\nimport Quandl\\nfrom datetime import datetime, timedelta\\n\\ntickers = [\'XOM\', \'CVX\', \'CLB\', \'OXY\', \'SLB\']\\nmarket_ticker = \'GOOG/NYSE_VOO\'\\nlookback = 30\\nd_col = \'Close\'\\n\\ndata = {tick: Quandl.get(\'YAHOO/{}\'.format(tick))[-lookback:] for tick in tickers}\\nmarket = Quandl.get(market_ticker)\\n```\\n\\n## Calculating the Return\\n\\nWe first want to know how much each ticker returned over the prior period.\\n\\n```python\\nreturns = {tick: data[tick][d_col].pct_change() for tick in tickers}\\n\\ndisplay({tick: returns[tick].mean() for tick in tickers})\\n```\\n\\n```\\n    {\'CLB\': -0.0016320202164526894,\\n     \'CVX\': 0.0010319531629488911,\\n     \'OXY\': 0.00093418904454400551,\\n     \'SLB\': 0.00098431254720448159,\\n     \'XOM\': 0.00044165797556096868}\\n```\\n\\n## Calculating the Sharpe ratio\\n\\nSharpe: ${R - R_M \\\\over \\\\sigma}$\\n\\nWe use the average return over the lookback period, minus the market average return, over the ticker standard deviation to calculate the Sharpe. Shorting a stock turns a negative Sharpe positive.\\n\\n\\n```python\\nmarket_returns = market.pct_change()\\n\\nsharpe = lambda ret: (ret.mean() - market_returns[d_col].mean()) / ret.std()\\nsharpes = {tick: sharpe(returns[tick]) for tick in tickers}\\n\\ndisplay(sharpes)\\n```\\n\\n```\\n    {\'CLB\': -0.10578734457846127,\\n     \'CVX\': 0.027303529817677398,\\n     \'OXY\': 0.022622210057414487,\\n     \'SLB\': 0.026950946344858676,\\n     \'XOM\': -0.0053519259698605499}\\n```\\n\\n## Calculating the drawdown\\n\\nThis one is easy - what is the maximum daily change over the lookback period? That is, because we will allow short positions, we are not concerned strictly with maximum downturn, but in general, what is the largest 1-day change?\\n\\n\\n```python\\ndrawdown = lambda ret: ret.abs().max()\\ndrawdowns = {tick: drawdown(returns[tick]) for tick in tickers}\\n\\ndisplay(drawdowns)\\n```\\n\\n```\\n    {\'CLB\': 0.043551495607375035,\\n     \'CVX\': 0.044894389686214398,\\n     \'OXY\': 0.051424517867144637,\\n     \'SLB\': 0.034774627850375328,\\n     \'XOM\': 0.035851524605672758}\\n```\\n\\n# Performing the optimization\\n\\n$$\\n\\\\begin{align*}\\nmax\\\\ \\\\ & \\\\mu \\\\cdot \\\\omega\\\\\\\\\\ns.t.\\\\ \\\\ & \\\\vec{1} \\\\omega = 1\\\\\\\\\\n& \\\\vec{S} \\\\omega \\\\ge s\\\\\\\\\\n& \\\\vec{D} \\\\cdot | \\\\omega | \\\\le d\\\\\\\\\\n& \\\\left|\\\\omega\\\\right| \\\\le l\\\\\\\\\\n\\\\end{align*}\\n$$\\n\\nWe want to maximize average return subject to having a full portfolio, Sharpe above a specific level, drawdown below a level, and leverage not too high - that is, don\'t have huge long/short positions.\\n\\n```python\\nimport numpy as np\\nfrom scipy.optimize import minimize\\n\\n#sharpe_limit = .1\\ndrawdown_limit = .05\\nleverage = 250\\n\\n# Use the map so we can guarantee we maintain the correct order\\n\\n# So we can write as upper-bound\\n# sharpe_a = np.array(list(map(lambda tick: sharpes[tick], tickers))) * -1 \\ndd_a = np.array(list(map(lambda tick: drawdowns[tick], tickers)))\\n\\n# Because minimizing\\nreturns_a = np.array(list(map(lambda tick: returns[tick].mean(), tickers))) \\n\\nmeets_sharpe = lambda x: sum(abs(x) * sharpe_a) - sharpe_limit\\ndef meets_dd(x):\\n    portfolio = sum(abs(x))\\n    if portfolio < .1:\\n        # If there are no stocks in the portfolio,\\n        # we can accidentally induce division by 0,\\n        # or division by something small enough to cause infinity\\n        return 0\\n    \\n    return drawdown_limit - sum(abs(x) * dd_a) / sum(abs(x))\\n\\nis_portfolio = lambda x: sum(x) - 1\\n\\ndef within_leverage(x):\\n    return leverage - sum(abs(x))\\n\\nobjective = lambda x: sum(x * returns_a) * -1 # Because we\'re minimizing\\nbounds = ((None, None),) * len(tickers)\\nx = np.zeros(len(tickers))\\n\\nconstraints = [\\n    {\\n        \'type\': \'eq\',\\n        \'fun\': is_portfolio\\n    }, {\\n        \'type\': \'ineq\',\\n        \'fun\': within_leverage\\n    #}, {\\n    #    \'type\': \'ineq\',\\n    #    \'fun\': meets_sharpe\\n    }, {\\n        \'type\': \'ineq\',\\n        \'fun\': meets_dd\\n    }\\n]\\n\\noptimal = minimize(objective, x, bounds=bounds, constraints=constraints,\\n                  options={\'maxiter\': 500})\\n\\n# Optimization time!\\ndisplay(optimal.message)\\n\\ndisplay(\\"Holdings: {}\\".format(list(zip(tickers, optimal.x))))\\n\\n# multiply by -100 to scale, and compensate for minimizing\\nexpected_return = optimal.fun * -100\\ndisplay(\\"Expected Return: {:.3f}%\\".format(expected_return))\\n\\nexpected_drawdown = sum(abs(optimal.x) * dd_a) / sum(abs(optimal.x)) * 100\\ndisplay(\\"Expected Max Drawdown: {0:.2f}%\\".format(expected_drawdown))\\n\\n# TODO: Calculate expected Sharpe\\n```\\n\\n```\\n    \'Optimization terminated successfully.\'\\n    \\"Holdings: [(\'XOM\', 5.8337945679814904),\\n                (\'CVX\', 42.935064321851307),\\n                (\'CLB\', -124.5),\\n                (\'OXY\', 36.790387773552119),\\n                (\'SLB\', 39.940753336615096)]\\"\\n    \'Expected Return: 32.375%\'\\n    \'Expected Max Drawdown: 4.34%\'\\n```"}]}}')}}]);