diff --git a/content/articles/2018-01-16-captains-cookbook-part-1.md b/content/articles/2018-01-16-captains-cookbook-part-1.md new file mode 100644 index 0000000..e0f394e --- /dev/null +++ b/content/articles/2018-01-16-captains-cookbook-part-1.md @@ -0,0 +1,244 @@ +Title: Captain's Cookbook - Part 1 +Date: 2018-01-16 +Category: Blog +Tags: capnproto rust +Authors: Bradlee Speice +Summary: A basic introduction to getting started with Cap'N Proto +[//]: <> "Modified: " + +# Captain's Cookbook - Part 1 + +I've been working a lot with [Cap'N Proto](https://capnproto.org/) recently with Rust, but there's a real dearth of information +on how to set up and get going quickly. In the interest of trying to get more people using this (because I think it's +fantastic), I'm going to work through a couple of examples detailing what exactly should be done to get going. + +So, what is Cap'N Proto? It's a data serialization library. It has contemporaries with [Protobuf](https://developers.google.com/protocol-buffers/) +and [FlatBuffers](https://google.github.io/flatbuffers/), but is better compared with FlatBuffers. The whole point behind it +is to define a schema language and serialization format such that: + +1. Applications that do not share the same base programming language can communicate +2. The data and schema you use can naturally evolve over time as your needs change + +Accompanying this are typically code generators that take the schemas you define for your application and give you back +code for different languages to get data to and from that schema. + +Now, what makes Cap'N Proto different from, say, Protobuf, is that there is no serialization/deserialization step the same way +as is implemented with Protobuf. Instead, the idea is that the message itself can be loaded in memory and used directly there. + +We're going to take a look at a series of progressively more complex projects that use Cap'N Proto in an effort to provide some +examples of what idiomatic usage looks like, and shorten the startup time needed to make use of this library in Rust projects. +If you want to follow along, feel free. If not, I've posted [the final result](https://github.com/bspeice/capnp_cookbook_1) +for reference. + +# Step 1: Installing `capnp` + +The `capnp` binary itself is needed for taking the schema files you write and turning them into a format that can be used by the +code generation libraries. Don't ask me what that actually means, I just know that you need to make sure this is installed. + +I'll refer you to [Cap'N Proto's installation instructions](https://capnproto.org/install.html) here. As a quick TLDR though: + +- Linux users will likely have a binary shipped by their package manager - On Ubuntu, `apt install capnproto` is enough +- OS X users can use [Homebrew](https://brew.sh/) as an easy install path. Just `brew install capnp` +- Windows users are a bit more complicated. If you're using [Chocolatey](https://chocolatey.org/), there's [a package](https://chocolatey.org/packages/capnproto/) available. If that doesn't work however, you need to download [a release zip](https://capnproto.org/capnproto-c++-win32-0.6.1.zip) and make sure that the `capnp.exe` binary is in your `%PATH%` environment variable + +The way you know you're done with this step is if the following command works in your shell: + +```bash +capnp id +``` + +# Step 2: Starting a Cap'N Proto Rust project + +After the `capnp` binary is set up, it's time to actually create our Rust project. Nothing terribly complex here, just a simple + +```bash +mkdir capnp_cookbook_1 +cd capnp_cookbook_1 +cargo init --bin +``` + +We'll put the following content into `Cargo.toml`: + +``` +[package] +name = "capnp_cookbook_1" +version = "0.1.0" +authors = ["Bradlee Speice "] + +[build-dependencies] +capnpc = "0.8" # 1 + +[dependencies] +capnp = "0.8" # 2 +``` + +This sets up: + +1. The Rust code generator (CAPNProto Compiler) +2. The Cap'N Proto runtime library (CAPNProto runtime) + +We've now got everything prepared that we need for writing a Cap'N Proto project. + +# Step 3: Writing a basic schema + +We're going to start with writing a pretty trivial data schema that we can extend later. This is just intended to make sure +you get familiar with how to start from a basic project. + +First, we're going to create a top-level directory for storing the schema files in: + +```bash +# Assuming we're starting from the `capnp_cookbook_1` directory created earlier + +mkdir schema +cd schema +``` + +Now, we're going to put the following content in `point.capnp`: + +``` +@0xab555145c708dad2; + +struct Point { + x @0 :Int32; + y @1 :Int32; +} +``` + +Pretty easy, we've now got structure for an object we'll be able to quickly encode in a binary format. + +# Step 4: Setting up the build process + +Now it's time to actually set up the build process to make sure that Cap'N Proto generates the Rust code we'll eventually be using. +This is typically done through a `build.rs` file to invoke the schema compiler. + +In the same folder as your `Cargo.toml` file, please put the following content in `build.rs`: + +```rust +extern crate capnpc; + +fn main() { + ::capnpc::CompilerCommand::new() + .src_prefix("schema") // 1 + .file("schema/point.capnp") // 2 + .run().expect("compiling schema"); +} +``` + +This sets up the protocol compiler (`capnpc` from earlier) to compile the schema we've built so far. + +1. Because Cap'N Proto schema files can re-use types specified in other files, the `src_prefix()` tells the compiler +where to look for those extra files at. +2. We specify the schema file we're including by hand. In a much larger project, you could presumably build the `CompilerCommand` +dynamically, but we won't worry too much about that one for now. + +# Step 5: Running the build + +If you've done everything correctly so far, you should be able to actually build the project and see the auto-generated code. +Run a `cargo build` command, and if you don't see `cargo` complaining, you're doing just fine! + +So where exactly does the generated code go to? I think it's critically important for people to be able to see what the generated +code looks like, because you need to understand what you're actually programming against. The short answer is: the generated code lives +somewhere in the `target/` directory. + +The long answer is that you're best off running a `find` command to get the actual file path: + +```bash +# Assuming we're running from the capnp_cookbook_1 project folder +find . -name point_capnp.rs +``` + +Alternately, if the `find` command isn't available, the path will look something like: + +``` +./target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs +``` + +See if there are any paths in your target directory that look similar. + +Now, the file content looks pretty nasty. I've included an example [here](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs) +if you aren't following along at home. There are a couple things I'll try and point out though so you can get an idea of how +the schema we wrote for the "Point" message is tied to the generated code. + +First, the Cap'N Proto library splits things up into `Builder` and `Reader` structs. These are best thought of the same way +Rust separates `mut` from non-`mut` code. `Builder`s are `mut` versions of your message, and `Reader`s are immutable versions. + +For example, the [`Builder` impl](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L90) for `point` defines [`get_x()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L105), [`set_x()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L109), [`get_y()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L113), and [`set_y()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L117) methods. +In comparison, the [`Reader` impl](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L38) only defines [`get_x()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L47) and [`get_y()`](https://github.com/bspeice/capnp_cookbook_1/blob/master/target/debug/build/capnp_cookbook_1-c6e2990393c32fe6/out/point_capnp.rs#L51) methods. + +So now we know that there are some `get` and `set` methods available for our `x` and `y` coordinates; +but what do we actually do with those? + +# Step 6: Making a point + +So we've install Cap'N Proto, gotten a project set up, and can generate schema code now. It's time to actually start building +Cap'N Proto messages! I'm going to put the code you need here because it's small, and put some extra long comments inline. This code +should go in [`src/main.rs`](https://github.com/bspeice/capnp_cookbook_1/blob/master/src/main.rs): + +```rust +// Note that we use `capnp` here, NOT `capnpc` +extern crate capnp; + +// We create a module here to define how we are to access the code +// being included. +pub mod point_capnp { + // The environment variable OUT_DIR is set by Cargo, and + // is the location of all the code that was built as part + // of the codegen step. + // point_capnp.rs is the actual file to include + include!(concat!(env!("OUT_DIR"), "/point_capnp.rs")); +} + +fn main() { + + // The process of building a Cap'N Proto message is a bit tedious. + // We start by creating a generic Builder; it acts as the message + // container that we'll later be filling with content of our `Point` + let mut builder = capnp::message::Builder::new_default(); + + // Because we need a mutable reference to the `builder` later, + // we fence off this part of the code to allow sequential mutable + // borrows. As I understand it, non-lexical lifetimes: + // https://github.com/rust-lang/rust-roadmap/issues/16 + // will make this no longer necessary + { + // And now we can set up the actual message we're trying to create + let mut point_msg = builder.init_root::(); + + // Stuff our message with some content + point_msg.set_x(12); + + point_msg.set_y(14); + } + + // It's now time to serialize our message to binary. Let's set up a buffer for that: + let mut buffer = Vec::new(); + + // And actually fill that buffer with our data + capnp::serialize::write_message(&mut buffer, &builder).unwrap(); + + // Finally, let's deserialize the data + let deserialized = capnp::serialize::read_message( + &mut buffer.as_slice(), + capnp::message::ReaderOptions::new() + ).unwrap(); + + // `deserialized` is currently a generic reader; it understands + // the content of the message we gave it (i.e. that there are two + // int32 values) but doesn't really know what they represent (the Point). + // This is where we map the generic data back into our schema. + let point_reader = deserialized.get_root::().unwrap(); + + // We can now get our x and y values back, and make sure they match + assert_eq!(point_reader.get_x(), 12); + assert_eq!(point_reader.get_y(), 14); +} +``` + +And with that, we've now got a functioning project. Here's the content I'm planning to go over next as we build up +some practical examples of Cap'N Proto in action: + +## Next steps: + +Part 2: Using [TypedReader](https://github.com/capnproto/capnproto-rust/blob/master/src/message.rs#L181) to send messages across thread boundaries + +Part 3: Serialization and Deserialization of multiple Cap'N Proto messages diff --git a/pelicanconf.py b/pelicanconf.py index 94f3544..bfe755d 100644 --- a/pelicanconf.py +++ b/pelicanconf.py @@ -39,6 +39,7 @@ PLUGINS = ['liquid_tags.notebook'] NOTEBOOK_DIR = 'notebooks' THEME='nest' +MD_EXTENSIONS = ['codehilite(css_class=highlight)', 'extra'] #NEST_INDEX_HEADER_TITLE="Bradlee Speice" NEST_INDEX_HEADER_SUBTITLE="Exploring the intersection of Computer Science and Financial Engineering" NEST_HEADER_LOGO="/images/logo.svg"