Flatbuffers — Why?
I started web development in 2010. Yeah, I know, I started when times were easy. The days of IE8+, jQuery was all the rage, and PHP was hotter than a pistol. During my time developing I questioned so many of the practices: “Why am I using PHP?”, “Who uses IE7?”, and “Why is jQuery so awesome?”. But one thing that never crossed my mind is “why JSON?” JSON is ubiquitous and human readable. In fact, I had my first and only white paper during my time in academia is about JSON. Yeah, the paper sucked, but none the less, JSON is awesome! So for the rest of my career, up until this year, I have never considered anything but the all mighty JSON. But then it happened… my friend, Guy* **, explained to me why he does not JSON (yes, the verb form of JSON). Of course my natural reaction was…
“Are you suggesting XML?” - Me
“XML?” - Guy
After a 15 second conversation about real life, you know, adult things, I quickly realized I am a fool!
This is the motivation for explaining what Flatbuffers are and why they are amazingly cool. So this will be a series of articles that rely as much as possible on data, and little as possible on feelings. You know, practical science shit.
Let us JSON
The following is from JSON.org.
This series of articles is me disagreeing with the above statement. Well, not the whole statement, just the last part.
These properties make JSON an ideal data-interchange language.
This article is meant to introduce flatbuffers, what they are, and the high level view of the algorithm used. From there I’ll be making a set of comparisons of their runtime. All experiments will be done in node.
video. To make this into valid json, the following function is applied
Intro to flatbuffers
To understand why flatbuffers are so awesome you first must understand what happens when working with JSON. So to recap…
- Some object structure has to be converted from a class or map into a string.
- When receiving json, that string needs to be converted back into an object.
- Yes, the entire structure, at once.
The above points are extremely simplified and hide even more about why flatbuffers are awesome (Zero Copy), but its not our concern for now.
How it works by understanding arrays
To understand flatbuffers lets take the previously mentioned
video object and transform it. Instead of listing the keys and values in a map, let us just put each one of the values into an array.
Now that we have this
array of data, we need a way to understand it. Lets hand generate some bindings.
The video array can now be understood through an interface. To get out the rating of a video I would do the following.
Let us expand our example from a single array representing a single video to a single array representing
N videos. How would this change our interface? Lets rewrite the interface to consider the array is filled with many videos.
The offset is required to know where in the array buffer to start from. From there it is simple addition to get to the values.
So now you have the idea. Arrays. But instead of arrays think
Uint8Array or a contiguous memory blocks. In the end, what makes flatbuffers so powerful is the ability to point to memory and say you are a
Video. There is no need to parse the entire block to read a single field. This becomes increasingly more powerful with larger payloads.
Where we are going next
This is just the introduction to Flatbuffers, and I have plenty more to say. The outline for this series of blogs will be the following:
- How to work with flatbuffers.
- Cost of JSON serialization vs Flatbuffers
- Server Side rendering and Flatbuffers.
In the end I hope to rate flatbuffers vs json on 3 points:
- Usability - The ease of use of construction and working of flat binary structs.
- Debuggability - How is debugging issues when it comes to flatbuffers?
- Performance - Both CPU and Memory performance.
Hopefully by the end I can make you a believer in FlatBuffers.
* - Guy: He is pretty neet and he has strong opinions.
** - A shout out to the Honey Badgers. Alex;