FlatBuffers

Why

• Access to serialized data without parsing/unpacking
• Memory Efficiency and Speed
• Backwards and Forwards Compatibility
• Small Footprint

How to Define FlatBuffer schema (.fbs)

namespace

FlatBuffers has support for namespaces to place the generated code into. There is mixed level of support for namespaces (some languages don’t have namespaces), but for the C family of languages, it is fully supported.

• place the generated code into

namespace MyGame.Sample;

key (on a field)

this field is meant to be used as a key when sorting a vector of the type of table it sits in.

Can be used for in-place binary search.

table Monster {
    name:string (key); // (key) 标记关键字段
    // ...
}

enum

Enums definitions can be defined with the backing numerical type. Implicit numbering is supported, so that Green would have a value of 1.

• numerical type
• Implicit numbering is supported

enum Color:byte { Red = 0, Green, Blue = 2 }

union

A union represents a single value from a set of possible values. Its effectively an enum (to represent the type actually store) and a value, combined into one. In this example, the union is not very useful, since it only has a single type.

• Similar to the union of C++

union Equipment { Weapon }

struct

A struct is a collection of scalar fields with names.

It is itself a scalar type, which uses less memory and has faster lookup.

However, once a struct is defined, it cannot be changed.

Use tables for data structures that can evolve over time.

• struct will be serialized inline in the table without any need for offset.

struct Vec3 { 
  x:float; 
  y:float;
  z:float;
}

scalar types

FlatBuffers has the standard set of scalar numerical types (int8, int16, int32, int64, uint8, uint16, uint32, uint64, float, double), as well as bool. Note, scalars are fixed width, varints are not supported.

table

Tables are the main data structure for grouping data together. It can evolve by adding and deprecating fields over time, while preserving forward and backwards compatibility.

table Monster {
    pos:Vec3; // A field that happens to be a struct. This means the data of the Vec3 struct will be serialized inline in the table without any need for offset.

    mana:short = 150; // Fields can be provided a default value. Default values can be configured to not be serialized at all while still providing the default value while deserializing. However, once set, a default value cannot be changed.

    hp:short = 100;
    
    name:string; // A string field which points to a serialized string external to the table.
    
    friendly:bool = false (deprecated); // A deprecated field that is no longer being used. This is used instead of removing the field outright.
    
    inventory:[ubyte]; // A vector field that points to a vector of bytes. Like strings, the vector data is serialized elsewhere and this field just stores an offset to the vector.
    
    color:Color = Blue;
    
    weapons:[Weapon];  // Vector of tables and structs are also possible.
    
    equipped:Equipment;  // A field to a union type.
    
    path:[Vec3];
    
}

root_type

The root of the flatbuffer is always a table. This indicates the type of table the “entry” point of the flatbuffer will point to.

root_type Monster; 

compiler to transform the schema into language-specific code

flatc compiler

Compiling Schema

flatc --python -o ./generated_py monster.fbs hero.fbs

flatc --csharp -o ./generated_cs monster.fbs hero.fbs

Serialized

使用 FlatBuffers 时，必须单独序列化每部分数据，并且要按照特定顺序（深度优先、前序遍历）进行序列化。

例如，任何引用类型（例如table ）必须先序列化vector，string然后才能被其他结构引用。

因此，典型的做法是从叶节点到根节点序列化数据

FlatBufferBuilder

monster.fbs

table Weapon {
  name:string;
  damage:short;
}

table Monster {
  pos:Vec3;
  mana:short = 150;
  hp:short = 100;
  name:string;
  friendly:bool = false (deprecated);
  inventory:[ubyte];
  color:Color = Blue;
  weapons:[Weapon];
  equipped:Equipment;
  path:[Vec3];
}

Steps

// Construct a Builder with 1024 byte backing array.
// The builder will automatically resize the backing buffer when necessary.
FlatBufferBuilder builder = new FlatBufferBuilder(1024);

// Step Strings
Offset<String> weaponOneName = builder.CreateString("Sword");
Offset<String> weaponTwoName = builder.CreateString("Axe");
short weaponOneDamage = 3;
short weaponTwoDamage = 5;

// Step Tables
// Use the `CreateWeapon()` helper function to create the weapons, since we set every field.
Offset<Weapon> sword = Weapon.CreateWeapon(builder, weaponOneName, weaponOneDamage);
Offset<Weapon> axe = Weapon.CreateWeapon(builder, weaponTwoName, weaponTwoDamage);


// Step Vectors
// Create an array of the two weapon offsets.
var weaps = new Offset<Weapon>[2];
weaps[0] = sword;
weaps[1] = axe;

// Pass the `weaps` array into the `CreateWeaponsVector()` method to create a FlatBuffer vector.
var weapons = Monster.CreateWeaponsVector(builder, weaps);

// Create a `vector` representing the inventory of the Orc. 
// Each number could correspond to an item that can be claimed after he is slain.
// Note: Since we prepend the bytes, this loop iterates in reverse order.
Monster.StartInventoryVector(builder, 10);
for (int i = 9; i >= 0; i--)
{
    builder.AddByte((byte)i);
}
Offset<Vector<byte>> inventory = builder.EndVector();

// Start building a path vector of length 2.
Monster.StartPathVector(fbb, 2);

// Serialize the individual Vec3 structs
Vec3.CreateVec3(builder, 1.0f, 2.0f, 3.0f);
Vec3.CreateVec3(builder, 4.0f, 5.0f, 6.0f);

// End the vector to get the offset
Offset<Vector<Vec3>> path = fbb.EndVector();

// Create the remaining data needed for the Monster.
var name = builder.CreateString("Orc");


// Setp Unions
// Create our monster using `StartMonster()` and `EndMonster()`.
Monster.StartMonster(builder);
Monster.AddPos(builder, Vec3.CreateVec3(builder, 1.0f, 2.0f, 3.0f));
Monster.AddHp(builder, (short)300);
Monster.AddName(builder, name);
Monster.AddInventory(builder, inv);
Monster.AddColor(builder, Color.Red);
Monster.AddWeapons(builder, weapons);
// For union fields, we explicitly add the auto-generated enum for the type of value stored in the union.
Monster.AddEquippedType(builder, Equipment.Weapon);
// And we just use the `.Value` property of the already serialized axe.
Monster.AddEquipped(builder, axe.Value); // Axe
Monster.AddPath(builder, path);
Offset<Monster> orc = Monster.EndMonster(builder);


// Setp Finishing
// Call `Finish()` to instruct the builder that this monster is complete.
// You could also call `Monster.FinishMonsterBuffer(builder, orc);`
builder.Finish(orc.Value);


// Setp Buffer Access
// This must be called after `Finish()`.
//
// The data in this ByteBuffer does NOT start at 0, but at buf.Position.
// The end of the data is marked by buf.Length, so the size is
// buf.Length - buf.Position.
FlatBuffers.ByteBuffer dataBuffer = builder.DataBuffer;

// Alternatively this copies the above data out of the ByteBuffer for you:
byte[] buf = builder.SizedByteArray();

// Now you can write the bytes to a file or send them over the network. The buffer stays valid until the Builder is cleared or destroyed.
// Make sure your file mode (or transfer protocol) is set to BINARY, and not TEXT. If you try to transfer a flatbuffer in TEXT mode, the buffer will be corrupted and be hard to diagnose.

####

Deserialization

反序列化这个术语有点用词不当，因为 FlatBuffers 在访问时不会反序列化整个缓冲区。

它只是“解码”所请求的数据，而其他所有数据保持不变。

应用程序可以决定是否复制或读取数据。

// Root Access
byte[] bytes = /* the data you just read */
// Get an view to the root object inside the buffer.
Monster monster = Monster.GetRootAsMonster(new ByteBuffer(bytes));


// Table Access
// For C#, unlike most other languages support by FlatBuffers, most values
// (except for vectors and unions) are available as properties instead of
// accessor methods.
var hp = monster.Hp; // 300
var mana = monster.Mana; // 150 - That is because the generated accessors return a hard-coded default value when it doesn't find the value in the buffer.
var name = monster.Name; // "Orc"


// Nested Object Access
var pos = monster.Pos.Value;
var x = pos.X;
var y = pos.Y;
var z = pos.Z;


// Vector Access
int invLength = monster.InventoryLength;
var thirdItem = monster.Inventory(2);

int weaponsLength = monster.WeaponsLength;
var secondWeaponName = monster.Weapons(1).Name;
var secondWeaponDamage = monster.Weapons(1).Damage;

// Union Access
var unionType = monster.EquippedType;

if (unionType == Equipment.Weapon) {
    var weapon = monster.Equipped<Weapon>().Value;

    var weaponName = weapon.Name;     // "Axe"
    var weaponDamage = weapon.Damage; // 5
}