From f50e65204c04c6787dc63666228956c4a8098281 Mon Sep 17 00:00:00 2001
From: Bradlee Speice <bradlee@speice.io>
Date: Sat, 28 Sep 2019 00:18:20 -0400
Subject: [PATCH] Finish first draft

---
 _posts/2019-09-01-binary-format-shootout.md | 54 +++++++++++++++++++--
 1 file changed, 49 insertions(+), 5 deletions(-)

diff --git a/_posts/2019-09-01-binary-format-shootout.md b/_posts/2019-09-01-binary-format-shootout.md
index 2001966..64b3ef3 100644
--- a/_posts/2019-09-01-binary-format-shootout.md
+++ b/_posts/2019-09-01-binary-format-shootout.md
@@ -1,7 +1,7 @@
 ---
 layout: post
 title: "Binary Format Shootout"
-description: "Making sense of binary streams"
+description: "Cap'n Proto vs. Flatbuffers vs. SBE"
 category: 
 tags: [rust]
 ---
@@ -161,11 +161,40 @@ message to indicate the size. Not specifically a problem, but I would've rather
 
 Ultimately, I enjoyed using Flatbuffers, and had to do significantly less work to make it perform well.
 
-# Final Results
+# Part 3: Simple Binary Encoding
 
-NOTE: Need to expand on this, but numbers reported below are from the IEX's 2019-09-03 data, took average over 10 runs.
+Support for SBE was added by the author of one of my favorite
+[Rust blog posts](https://web.archive.org/web/20190427124806/https://polysync.io/blog/session-types-for-hearty-codecs/).
+I've [talked previously]({% post_url 2019-06-31-high-performance-systems %}) about how important variance is in
+high-performance systems, so it was encouraging to read about a format that
+[directly addressed](https://github.com/real-logic/simple-binary-encoding/wiki/Why-Low-Latency) my concerns. SBE has by far
+the simplest binary format, but it does make some tradeoffs.
 
-Serialization
+Both Cap'n Proto and Flatbuffers use [pointers in their messages](https://capnproto.org/encoding.html#structs) to handle
+variable-length data, [unions](https://capnproto.org/language.html#unions), and a couple other features. In contrast,
+messages in SBE are essentially [primitive structs](https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-samples/src/main/resources/example-schema.xml);
+variable-length data is supported, but there's no union type.
+
+As mentioned in the beginning, the Rust port of SBE is certainly usable, but is essentially unmaintained. However, if you
+don't need union types, and can accept that schemas are XML documents, it's still worth using.
+
+# Results
+
+After building a test harness [for](https://github.com/bspeice/speice.io-md_shootout/blob/master/src/capnp_runner.rs)
+[each](https://github.com/bspeice/speice.io-md_shootout/blob/master/src/flatbuffers_runner.rs)
+[protocol](https://github.com/bspeice/speice.io-md_shootout/blob/master/src/sbe_runner.rs),
+it was time to actually take them for a spin. I used
+[this script](https://github.com/bspeice/speice.io-md_shootout/blob/master/run_shootout.sh) to manage the test process,
+and the raw results are [here](https://github.com/bspeice/speice.io-md_shootout/blob/master/shootout.csv). All data
+reported below is the average of 10 runs over a single day of IEX data. Data checks were implemented to make sure
+that each format achieved the same results.
+
+## Serialization
+
+Serialization measures on a
+[per-message basis](https://github.com/bspeice/speice.io-md_shootout/blob/master/src/main.rs#L268-L272)
+how long it takes to convert the pre-parsed IEX message into the desired format
+and write to a pre-allocated buffer.
 
 | Schema               | Median | 99th Pctl | 99.9th Pctl | Total  |
 |:---------------------|:-------|:----------|:------------|:-------|
@@ -174,7 +203,13 @@ Serialization
 | Flatbuffers          | 355ns  | 2185ns    | 3497ns      | 14.31s |
 | SBE                  | 91ns   | 1535ns    | 2423ns      | 3.91s  |
 
-Deserialization
+## Deserialization
+
+Deserialization measures on a
+[per-message basis](https://github.com/bspeice/speice.io-md_shootout/blob/master/src/main.rs#L294-L298)
+how long it takes to read the message encoded during deserialization and
+perform some basic aggregation. The aggregation code is the same for each format,
+so any performance differences are due solely to the format implementation.
 
 | Schema               | Median | 99th Pctl | 99.9th Pctl | Total  |
 |:---------------------|:-------|:----------|:------------|:-------|
@@ -182,3 +217,12 @@ Deserialization
 | Cap'n Proto Unpacked | 366ns  | 737ns     | 1583ns      | 12.32s |
 | Flatbuffers          | 173ns  | 421ns     | 1007ns      | 6.00s  |
 | SBE                  | 116ns  | 286ns     | 659ns       | 4.05s  |
+
+# Conclusion
+
+Building a benchmark turned out to be incredibly helpful in making a decision; because a
+"union" type isn't important to me, I'll be using SBE for my personal projects.
+
+And while SBE was the fastest in terms of both median and worst-case performance, its worst case
+performance was proportionately far higher than any other format. Further research is necessary
+to figure out why this is the case. But that's for another time.