MD
Uploaded byManfred Dreese
PDF, PPTX199 views

Data Serialization Frameworks for Java and the IoT

Data serialization frameworks allow for efficient encoding of messages for transmission between systems. They provide standardized interfaces and code generation to compose and deserialize messages. While providing compact binary encodings, they lack features for error correction and type identification within messages. Such frameworks can be used to transmit IoT data via message brokers or define APIs between microservices.

Embed presentation

Download as PDF, PPTX
Data Serialization Frameworks for Java and the IoT Manfred Dreese, codecentric AG 1
AGENDA 2 Shortcomings and no-frills What Data-Serialization Frameworks do Architectural considerations API, Speed and Size Battle Back to binary encoded messages?
Why size matters (again) 3
Standard Interface Definitions languages or hand-crafted binary Protocols 4 History of binary serialization frameworks 80s String-encoded Interfaces, String encoded verification String-encoded Protocols 90s Renaissance of binary protocols as backend- turbocharger. Just with more frills. 2000 Binary Serialization to Turbocharge embedded Devices 2010
Standard Interface Definitions languages or hand-crafted binary Protocols 5 History of binary serialization frameworks 80s String-encoded Interfaces, String encoded verification String-encoded Protocols 90s Renaissance of binary protocols as backend- turbocharger. Just with more frills. 2000 Binary Serialization to Turbocharge embedded Devices 2010> ASN-1 Thrift Protobuf JSON Avro CBOR lots_of {custom_protocol}
Message length considerations for IoT appliances 6 Cloud Application IoT Solution Powerful, scalable Servers Finite number of constrained devices High amount of influence on technology stack Mixed technology stacks, processor architectures and languages High connections speeds Serial links, Industrial bus connections, Cellular networks. No restrictions on data transfer Low bandwidth and duty cycle regulations Nodes available all the time Power Saving, Mobile Data Saving, possible bad signal level Data transfer is (almost) free of charge (Literally) expensive data transfer when mobile data is involved. Processing power for data transfer is neglectible Sophisticated String processing or data transfer in resource competition with actual functionality
What a serialization Framework does 7
Code generator 8 How data serialization frameworks work Schema-driven Frameworks Interface Definition Java code Golang code Javascript code {any} code
IDL/Schema Definition Code generation Message Composition and Serialization 9 Example 1 2 3 Message Transfer 4 Message Deserialization 5
IDL/Schema Definition Code generation Message Composition and Serialization 10 Example syntax = "proto3"; import "Common.proto"; import "google/protobuf/timestamp.proto"; package jcon.telemetry; option java_package = "de.m9d.telemetry.engine"; option go_package = "m9d.de/telemetry/engine“; message Telegram { string engineId = 1; google.protobuf.Timestamp timeFrom = 2; google.protobuf.Timestamp timeTo = 3; double totalWork = 4; Curve engineSpeed = 5; Curve temperature = 6; Curve oilPressure = 7; Curve fuelConsumption = 8; } 1 2 3 Define interface using a meta-language. Message Transfer 4 Message Deserialization 5
11 Example <plugin> <groupId>com.github.os72</groupId> <artifactId>protoc-jar-maven-plugin</artifactId> <version>3.5.0</version> <executions> <execution> <phase>generate-sources</phase> <goals> <goal>run</goal> </goals> <configuration> <protocVersion>3.5.0</protocVersion> <includeStdTypes>true</includeStdTypes> <includeDirectories> </includeDirectories> <inputDirectories> <include>../idl</include> </inputDirectories> </configuration> </execution> </executions> </plugin> <dependency>    <groupId>com.google.protobuf</groupId>    <artifactId>protobuf-java</artifactId>    <version>3.5.1</version> </dependency> IDL/Schema Definition Code generation Message Composition and Serialization 1 2 3 Use code-generator (CLI / build plugin) to build producer/consumer code for your platform Message Transfer 4 Message Deserialization 5 protoc -I=../idl —cpp_out=api/ —wrap=*.proto
12 Example Common.Curve.Builder sampleCurve = Common.Curve.newBuilder(); for (int x = 0 ; x < 32 ; x++) { sampleCurve.addPoints( Common.Point.newBuilder() .setX(x) .setY(x) .build()); } EngineTelemetry.Telegram message= EngineTelemetry.Telegram .newBuilder() .setTimeFrom(Timestamp.newBuilder() .setSeconds(fromTimestamp) .setTimeTo(Timestamp.newBuilder() .setSeconds(toTimestamp) .setTotalWork(42.23) .setEngineId(engineUid.toString()) .setFuelConsumption(sampleCurve) .setOilPressure(sampleCurve) .build(); byte[] proto = message.toByteArray(); IDL/Schema Definition Code generation Message Composition and Serialization 1 2 3 Message Transfer 4 Message Deserialization 5 Use generated code or generic API to compose a message.
13 0 |24 30 37 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4 10 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e 20 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc 30 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@: 40 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ? 50 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @ 60 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @ 70 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @ 80 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @ 90 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A
14 0 |22 65 6E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077 10 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4- 20 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f 30 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": { 40 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389 50 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo": 60 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153 70 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW 80 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil 90 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x": B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{" C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0 D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y": E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0,
15 Example EngineTelemetry.Telegram deserialized = EngineTelemetry.Telegram.parseFrom(wire); IDL/Schema Definition Code generation Message Composition and Serialization 1 2 3 Message Transfer 4 Message Deserialization 5 The same way back
Generated Builder API Builder API or types serializers/deserializers based on Interface definition Generic Builder API Field-By-Field API Abstracted-away Workhorse Serving as a Serializer Option for i.E. Jackson 16 API types
17 ..it is not A uint32 is a uint32… Message with low int values telegram. .setMaximumTorque(32) .setTotalWork(32) .build(); telegram. .setMaximumTorque(65535) .setTotalWork(65535) .build(); 2 8 20 21 0 0 0 0 0 0 40 40 (13 bytes) 4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40 (15 bytes) Message with high int values
18 40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a 50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479 60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c 70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d 80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A Random Access With fixed length fields -Predictable offset of any item -Decode just a subset of the message -Just point reading routines to raw data, no need to allocate additional memory
Shortcomings 19
20 No error correction Encode message Meet the bad guys EngineTelemetry.Telegram telemetry= EngineTelemetry.Telegram.newBuilder() .setTimeFrom(Timestamp.now()) .setTotalWork(65535) .build(); wire[10] = 0x40; Decoding works fine! EngineTelemetry.Telegram deserialized = EngineTelemetry.Telegram.parseFrom(wire); deserialized: { „timeFrom": {"seconds": 65535}, "totalWork": 65535.0078125 } Well, fine enough for another Diesel joke… 4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40 4 8 FF FF 3 21 0 0 0 0 40 FF EF 40
21 40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a 50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479 60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c 70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d 80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A // CAFE, BABE? could be anything! No frills, no type announcement -Type awareness has to be established on decoration or transport
22 Non-opiniated with regards to transport -Just an array of bytes -No encryption or compression by default -No transport or remote method methods by default -Except Add-Ons (gRPC, AVRO RPC) -Integrateable in existing transport infrastructure -Raw sockets -MQTT, AMQP, Kafka, … -Machine bus protocols -XMPP -Local memory
Serialization Frameworks and Microservices 23
24 Use-case: Binary encoded content Message Broker (MQTT/AMQP/…) IoT Device Consumers Submit
25 Use-case: API generation Code generator Service A API description V1.23.42 Service C API code Service B API code Checkout Generate Generate
26 Use-case: Backend Payload Backend Service B Backend Service CBackend Service A Message Broker Pub/Sub Pub/Sub Pub/Sub
27 Best Usecases -High message sizes and frequencies -Structure data -Lists, Arrays, Time Series -Cross-Platform and Languages
Java APIs comparism (and BATTLE!) 28
Serialize a engine telemetry data item as fast as possible With as little data transfer as possible Only no-brainer optimizations are allowed Given situation 29 A fictional telemetry use-case EngineID Uuid (String encoding if no native type) Observation Interval Unix timestamp from, to Total amount of work Double Fuel consumption curve 32 float tupels (0,0)..(32,32) Oil Pressure curve 32 float tupels (0,0)..(32,32) Serialization in Java
JSON 30 0 |22 65 6E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077 10 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4- 20 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f 30 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": { 40 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389 50 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo": 60 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153 70 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW 80 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil 90 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x": B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{" C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0 D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y": E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0, F0 |22 79 22 3A 20 34 2E 30 7D 2C 7B 22 78 22 3A 20 |"y": 4.0 },{"x": 100 |35 2E 30 2C 22 79 22 3A 20 35 2E 30 7D 2C 7B 22 |5.0,"y": 5.0},{" 110 |78 22 3A 20 36 2E 30 2C 22 79 22 3A 20 36 2E 30 |x": 6.0, "y": 6.0 120 |7D 2C 7B 22 78 22 3A 20 37 2E 30 2C 22 79 22 3A |},{"x": 7.0,"y": 130 |20 37 2E 30 7D 2C 7B 22 78 22 3A 20 38 2E 30 2C | 7.0},{" x": 8.0, 140 |22 79 22 3A 20 38 2E 30 7D 2C 7B 22 78 22 3A 20 |"y": 8.0 },{"x": 150 |39 2E 30 2C 22 79 22 3A 20 39 2E 30 7D 2C 7B 22 |9.0,"y": 9.0},{" 160 |78 22 3A 20 31 30 2E 30 2C 22 79 22 3A 20 31 30 |x": 10.0 ,"y": 10 170 |2E 30 7D 2C 7B 22 78 22 3A 20 31 31 2E 30 2C 22 |.0},{"x" : 11.0," 180 |79 22 3A 20 31 31 2E 30 7D 2C 7B 22 78 22 3A 20 |y": 11.0 },{"x": 190 |31 32 2E 30 2C 22 79 22 3A 20 31 32 2E 30 7D 2C |12.0,"y" : 12.0}, 1A0 |7B 22 78 22 3A 20 31 33 2E 30 2C 22 79 22 3A 20 |{"x": 13 .0,"y": 1B0 |31 33 2E 30 7D 2C 7B 22 78 22 3A 20 31 34 2E 30 |13.0},{" x": 14.0 1C0 |2C 22 79 22 3A 20 31 34 2E 30 7D 2C 7B 22 78 22 |,"y": 14 .0},{"x" 1D0 |3A 20 31 35 2E 30 2C 22 79 22 3A 20 31 35 2E 30 |: 15.0," y": 15.0 1E0 |7D 2C 7B 22 78 22 3A 20 31 36 2E 30 2C 22 79 22 |},{"x": 16.0,"y" 1F0 |3A 20 31 36 2E 30 7D 2C 7B 22 78 22 3A 20 31 37 |: 16.0}, {"x": 17 200 |2E 30 2C 22 79 22 3A 20 31 37 2E 30 7D 2C 7B 22 |.0,"y": 17.0},{" 210 |78 22 3A 20 31 38 2E 30 2C 22 79 22 3A 20 31 38 |x": 18.0 ,"y": 18 220 |2E 30 7D 2C 7B 22 78 22 3A 20 31 39 2E 30 2C 22 |.0},{"x" : 19.0," 230 |79 22 3A 20 31 39 2E 30 7D 2C 7B 22 78 22 3A 20 |y": 19.0 },{"x": 240 |32 30 2E 30 2C 22 79 22 3A 20 32 30 2E 30 7D 2C |20.0,"y" : 20.0}, 250 |7B 22 78 22 3A 20 32 31 2E 30 2C 22 79 22 3A 20 |{"x": 21 .0,"y": 260 |32 31 2E 30 7D 2C 7B 22 78 22 3A 20 32 32 2E 30 |21.0},{" x": 22.0 270 |2C 22 79 22 3A 20 32 32 2E 30 7D 2C 7B 22 78 22 |,"y": 22 .0},{"x" // (cut) 60000ns 1537 bytes
31 Google Protocol Buffers syntax = "proto3"; import "Common.proto"; import "google/protobuf/timestamp.proto"; package jcon.telemetry; option java_package = "de.m9d.telemetry.engine"; option go_package = "de.m9d/telemetry/engine"; message Telegram { string engineId = 1; google.protobuf.Timestamp timeFrom = 2; google.protobuf.Timestamp timeTo = 3; double totalWork = 4; Curve engineSpeed = 5; Curve temperature = 6; Curve oilPressure = 7; Curve fuelConsumption = 8; } syntax = "proto3"; package jcon.telemetry; option java_package = "de.m9d.telemetry.engine"; option go_package = "de.m9d/telemetry/engine"; message Point { float x = 1; float y = 2; } message Curve { repeated Point points = 1; }
32 public void serializeProtobuf() { Common.Curve.Builder sampleCurve = Common.Curve.newBuilder(); for (int x = 0 ; x < 32 ; x++) { sampleCurve.addPoints(Common.Point.newBuilder().setX(x).setY(x).build()); } EngineTelemetry.Telegram message= EngineTelemetry.Telegram.newBuilder() .setTimeFrom(Timestamp.newBuilder().setSeconds(Instant.now().getEpochSecond()).build()) .setTimeTo(Timestamp.newBuilder().setSeconds(Instant.now().getEpochSecond()-3600).build()) .setTotalWork(42.23) .setEngineId(UUID.randomUUID().toString()) .setFuelConsumption(sampleCurve) .setOilPressure(sampleCurve) .build(); byte[] proto = message.toByteArray(); } Google Protocol Buffers
Google Protocol Buffers 33 0 |24 30 37 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4 10 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e 20 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc 30 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@: 40 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ? 50 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @ 60 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @ 70 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @ 80 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @ 90 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A F0 |70 41 15 0 0 70 41 A A D 0 0 80 41 15 0 |pA pA A 100 | 0 80 41 A A D 0 0 88 41 15 0 0 88 41 A | A A A 110 | A D 0 0 90 41 15 0 0 90 41 A A D 0 0 | A A 120 |98 41 15 0 0 98 41 A A D 0 0 A0 41 15 0 | A A A 130 | 0 A0 41 A A D 0 0 A8 41 15 0 0 A8 41 A | A A A 140 | A D 0 0 B0 41 15 0 0 B0 41 A A D 0 0 | A A 150 |B8 41 15 0 0 B8 41 A A D 0 0 C0 41 15 0 | A A A 160 | 0 C0 41 A A D 0 0 C8 41 15 0 0 C8 41 A | A A A 170 | A D 0 0 D0 41 15 0 0 D0 41 A A D 0 0 | A A 180 |D8 41 15 0 0 D8 41 A A D 0 0 E0 41 15 0 | A A A 190 | 0 E0 41 A A D 0 0 E8 41 15 0 0 E8 41 A | A A A 1A0 | A D 0 0 F0 41 15 0 0 F0 41 A A D 0 0 | A A 1B0 |F8 41 15 0 0 F8 41 42 F6 2 A 0 A A D 0 | A AB 1C0 | 0 80 3F 15 0 0 80 3F A A D 0 0 0 40 15 | ? ? @ 1D0 | 0 0 0 40 A A D 0 0 40 40 15 0 0 40 40 | @ @@ @@ 1E0 | A A D 0 0 80 40 15 0 0 80 40 A A D 0 | @ @ 1F0 | 0 A0 40 15 0 0 A0 40 A A D 0 0 C0 40 15 | @ @ @ 200 | 0 0 C0 40 A A D 0 0 E0 40 15 0 0 E0 40 | @ @ @ 210 | A A D 0 0 0 41 15 0 0 0 41 A A D 0 | A A 220 | 0 10 41 15 0 0 10 41 A A D 0 0 20 41 15 | A A A 230 | 0 0 20 41 A A D 0 0 30 41 15 0 0 30 41 | A 0A 0A 240 | A A D 0 0 40 41 15 0 0 40 41 A A D 0 | @A @A 250 | 0 50 41 15 0 0 50 41 A A D 0 0 60 41 15 | PA PA `A 260 | 0 0 60 41 A A D 0 0 70 41 15 0 0 70 41 | `A pA pA 270 | A A D 0 0 80 41 15 0 0 80 41 A A D 0 | A A 280 | 0 88 41 15 0 0 88 41 A A D 0 0 90 41 15 | A A A 290 | 0 0 90 41 A A D 0 0 98 41 15 0 0 98 41 | A A A // (cut) 2700ns 817 bytes
34 Captain Proto @0xcfe6e6668c89c78f; using Java = import "/java.capnp"; $Java.package("de.m9d.telemetry.engine"); $Java.outerClassname("Telemetry"); struct Telegram { code @0 :Text; engineId @1 :Text; timeFrom @2 :UInt32; timeTo @3 :UInt32; totalWork @4 :Float64; engineSpeed @5: Curve; temperature @6: Curve; oilPressure @7: Curve; fuelConsumption @8: Curve; } struct Curve { points @0: List(Point); } struct Point { x @0: Float32; y @1: Float32; }
35 Captain Proto public void serializeCaptainProto() throws IOException { String engineId = UUID.randomUUID().toString(); org.capnproto.MessageBuilder message = new org.capnproto.MessageBuilder(); de.m9d.telemetry.engine.Telemetry.Telegram.Builder builder = message.initRoot(Telemetry.Telegram.factory); builder.setEngineId(engineId); builder.setTimeFrom((int) Instant.now().getEpochSecond()); builder.setTimeTo((int) Instant.now().getEpochSecond()); builder.setTotalWork((float)42.561); builder.initEngineSpeed(); de.m9d.telemetry.engine.Telemetry.Curve.Builder engineSpeedBuilder = builder.getEngineSpeed(); engineSpeedBuilder.initPoints(32); for (int idx = 0 ; idx < 32 ; idx ++) { engineSpeedBuilder.getPoints().get(idx).setX(idx); engineSpeedBuilder.getPoints().get(idx).setY(idx); } // Same for oil pressure ByteArrayOutputStream bos = new ByteArrayOutputStream(); WritableByteChannel wbc = Channels.newChannel(bos); org.capnproto.Serialize.write(wbc, message); byte[] proto = bos.toByteArray(); }
Captain Proto 36 0 | 0 0 0 52 0 0 0 0 0 0 0 2 0 6 0 DC | R 10 |BD BB 5B DC BD BB 5B 77 3E 2A 42 0 0 0 0 0 | [ [w >*B 20 | 0 0 0 0 0 0 0 11 0 0 0 2A 1 0 0 20 | * 30 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 A0 | 40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a 50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479 60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c 70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d 80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A E0 | 0 30 41 0 0 30 41 0 0 40 41 0 0 40 41 0 | 0A 0A @A @A F0 | 0 50 41 0 0 50 41 0 0 60 41 0 0 60 41 0 | PA PA `A `A 100 | 0 70 41 0 0 70 41 0 0 80 41 0 0 80 41 0 | pA pA A A 110 | 0 88 41 0 0 88 41 0 0 90 41 0 0 90 41 0 | A A A A 120 | 0 98 41 0 0 98 41 0 0 A0 41 0 0 A0 41 0 | A A A A 130 | 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 B0 41 0 | A A A A 140 | 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 C0 41 0 | A A A A 150 | 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 D0 41 0 | A A A A 160 | 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 E0 41 0 | A A A A 170 | 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 F0 41 0 | A A A A 180 | 0 F8 41 0 0 F8 41 1 0 0 0 7 1 0 0 80 | A A 190 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 1A0 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ 1B0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ 1C0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ 1D0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A 1E0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A 1F0 | 0 30 41 0 0 30 41 0 0 40 41 0 0 40 41 0 | 0A 0A @A @A 200 | 0 50 41 0 0 50 41 0 0 60 41 0 0 60 41 0 | PA PA `A `A 210 | 0 70 41 0 0 70 41 0 0 80 41 0 0 80 41 0 | pA pA A A 220 | 0 88 41 0 0 88 41 0 0 90 41 0 0 90 41 0 | A A A A 230 | 0 98 41 0 0 98 41 0 0 A0 41 0 0 A0 41 0 | A A A A 240 | 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 B0 41 0 | A A A A 250 | 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 C0 41 0 | A A A A 260 | 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 D0 41 0 | A A A A 270 | 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 E0 41 0 | A A A A 280 | 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 F0 41 0 | A A A A 290 | 0 F8 41 0 0 F8 41 5200ns 664 bytes
37 AVRO [ { "namespace": "jcon.telemetryserver.domain", "type": "record", "name": "Curve", "fields": [ {"name": "points", "type": {"type": "array", "items": { "name":"Point", "type":"record", "fields":[ {"name":"x", "type":"float"}, {"name":"y", "type":"float"} ] }}} ] }, { "namespace": "jcon.telemetryserver.domain", "type": "record", "name": "Telegram", "fields": [ {"name": "engineId", "type": "string"}, {"name": "timeFrom", "type": {"type": "long", "logicalType": "timestamp-millis"}}, {"name": "timeTo", "type": {"type": "long", "logicalType": "timestamp-millis"}}, {"name": "totalWork", "type": "double"}, {"name": "engineSpeed", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }, {"name": "temperature", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }, {"name": "oilPressure", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }, {"name": "fuelConsumption", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null } ] } ]
38 AVRO public void serializeWithAvroAndSchemas() throws IOException { ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); DatumWriter<Telegram> writer = new SpecificDatumWriter<>(Telegram.getClassSchema()); BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null); List<Point> points = new ArrayList<>(); for (float x = 0 ; x < 32 ; x++) { points.add(Point.newBuilder().setX(x).setY(x).build()); } Curve curve = Curve.newBuilder() .setPoints(points) .build(); Telegram telegram = Telegram.newBuilder() .setEngineId(UUID.randomUUID().toString()) .setTimeFrom(DateTime.now()) .setTimeTo(DateTime.now()) .setTotalWork(42.23) .setEngineSpeed(curve) .setFuelConsumption(curve) .build(); writer.write(telegram, encoder); encoder.flush(); byte[] wire = outputStream.toByteArray(); }
AVRO 39 0 |64 63 62 33 33 39 33 65 2D 35 34 63 61 2D 34 66 |dcb3393e -54ca-4f 10 |39 39 2D 38 31 35 64 2D 35 62 36 64 35 31 33 35 |99-815d- 5b6d5135 20 |37 30 31 34 84 F3 A9 85 CA 59 B0 F3 A9 85 CA 59 |7014 Y Y 30 |3D A D7 A3 70 1D 45 40 2 40 0 0 0 0 0 0 |= p E@ @ 40 | 0 0 0 0 80 3F 0 0 80 3F 0 0 0 40 0 0 | ? ? @ 50 | 0 40 0 0 40 40 0 0 40 40 0 0 80 40 0 0 | @ @@ @@ @ 60 |80 40 0 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 | @ @ @ @ 70 |C0 40 0 0 E0 40 0 0 E0 40 0 0 0 41 0 0 | @ @ @ A 80 | 0 41 0 0 10 41 0 0 10 41 0 0 20 41 0 0 | A A A A 90 |20 41 0 0 30 41 0 0 30 41 0 0 40 41 0 0 | A 0A 0A @A A0 |40 41 0 0 50 41 0 0 50 41 0 0 60 41 0 0 |@A PA PA `A B0 |60 41 0 0 70 41 0 0 70 41 0 0 80 41 0 0 |`A pA pA A C0 |80 41 0 0 88 41 0 0 88 41 0 0 90 41 0 0 | A A A A D0 |90 41 0 0 98 41 0 0 98 41 0 0 A0 41 0 0 | A A A A E0 |A0 41 0 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 | A A A A F0 |B0 41 0 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 | A A A A 100 |C0 41 0 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 | A A A A 110 |D0 41 0 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 | A A A A 120 |E0 41 0 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 | A A A A 130 |F0 41 0 0 F8 41 0 0 F8 41 0 0 0 2 40 0 | A A A @ 140 | 0 0 0 0 0 0 0 0 0 80 3F 0 0 80 3F 0 | ? ? 150 | 0 0 40 0 0 0 40 0 0 40 40 0 0 40 40 0 | @ @ @@ @@ 160 | 0 80 40 0 0 80 40 0 0 A0 40 0 0 A0 40 0 | @ @ @ @ 170 | 0 C0 40 0 0 C0 40 0 0 E0 40 0 0 E0 40 0 | @ @ @ @ 180 | 0 0 41 0 0 0 41 0 0 10 41 0 0 10 41 0 | A A A A 190 | 0 20 41 0 0 20 41 0 0 30 41 0 0 30 41 0 | A A 0A 0A 1A0 | 0 40 41 0 0 40 41 0 0 50 41 0 0 50 41 0 | @A @A PA PA 1B0 | 0 60 41 0 0 60 41 0 0 70 41 0 0 70 41 0 | `A `A pA pA 1C0 | 0 80 41 0 0 80 41 0 0 88 41 0 0 88 41 0 | A A A A 1D0 | 0 90 41 0 0 90 41 0 0 98 41 0 0 98 41 0 | A A A A 1E0 | 0 A0 41 0 0 A0 41 0 0 A8 41 0 0 A8 41 0 | A A A A 1F0 | 0 B0 41 0 0 B0 41 0 0 B8 41 0 0 B8 41 0 | A A A A 200 | 0 C0 41 0 0 C0 41 0 0 C8 41 0 0 C8 41 0 | A A A A 210 | 0 D0 41 0 0 D0 41 0 0 D8 41 0 0 D8 41 0 | A A A A 220 | 0 E0 41 0 0 E0 41 0 0 E8 41 0 0 E8 41 0 | A A A A 230 | 0 F0 41 0 0 F0 41 0 0 F8 41 0 0 F8 41 0 | A A A A 240 | 5400ns 577 bytes
40 CBOR public void serializeCbor() throws IOException, CborException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); CborBuilder di = new CborBuilder(); ArrayBuilder ab = di.addArray(); for (float x = 0 ; x < 32 ; x++) { ab.addArray() .add(x) .add(x) .end(); } List<DataItem> curves = di.build(); new CborEncoder(baos).encode(new CborBuilder() .add(UUID.randomUUID().toString()) .add(Instant.now().getEpochSecond()) .add(Instant.now().getEpochSecond()) .add(42.23) .add(curves.get(0)) .add(curves.get(0)) .build()); byte[] encodedBytes = baos.toByteArray(); }
CBOR 41 0 |24 35 39 38 32 31 33 63 36 2D 64 65 37 37 2D 34 |$598213c 6-de77-4 10 |33 63 63 2D 62 64 38 61 2D 30 31 37 30 30 64 35 |3cc-bd8a -01700d5 20 |30 62 62 35 39 1A 5B BB C4 CC 1A 5B BB C4 CC FB |0bb59 [ [ 30 |40 45 1D 70 A3 D7 A 3D 98 20 82 F9 0 0 F9 0 |@E p = 40 | 0 82 F9 3C 0 F9 3C 0 82 F9 40 0 F9 40 0 82 | < < @ @ 50 |F9 42 0 F9 42 0 82 F9 44 0 F9 44 0 82 F9 45 | B B D D E 60 | 0 F9 45 0 82 F9 46 0 F9 46 0 82 F9 47 0 F9 | E F F G 70 |47 0 82 F9 48 0 F9 48 0 82 F9 48 80 F9 48 80 |G H H H H 80 |82 F9 49 0 F9 49 0 82 F9 49 80 F9 49 80 82 F9 | I I I I 90 |4A 0 F9 4A 0 82 F9 4A 80 F9 4A 80 82 F9 4B 0 |J J J J K A0 |F9 4B 0 82 F9 4B 80 F9 4B 80 82 F9 4C 0 F9 4C | K K K L L B0 | 0 82 F9 4C 40 F9 4C 40 82 F9 4C 80 F9 4C 80 82 | L@ L@ L L C0 |F9 4C C0 F9 4C C0 82 F9 4D 0 F9 4D 0 82 F9 4D | L L M M M D0 |40 F9 4D 40 82 F9 4D 80 F9 4D 80 82 F9 4D C0 F9 |@ M@ M M M E0 |4D C0 82 F9 4E 0 F9 4E 0 82 F9 4E 40 F9 4E 40 |M N N N@ N@ F0 |82 F9 4E 80 F9 4E 80 82 F9 4E C0 F9 4E C0 82 F9 | N N N N 100 |4F 0 F9 4F 0 82 F9 4F 40 F9 4F 40 82 F9 4F 80 |O O O @ O@ O 110 |F9 4F 80 82 F9 4F C0 F9 4F C0 98 20 82 F9 0 0 | O O O 120 |F9 0 0 82 F9 3C 0 F9 3C 0 82 F9 40 0 F9 40 | < < @ @ 130 | 0 82 F9 42 0 F9 42 0 82 F9 44 0 F9 44 0 82 | B B D D 140 |F9 45 0 F9 45 0 82 F9 46 0 F9 46 0 82 F9 47 | E E F F G 150 | 0 F9 47 0 82 F9 48 0 F9 48 0 82 F9 48 80 F9 | G H H H 160 |48 80 82 F9 49 0 F9 49 0 82 F9 49 80 F9 49 80 |H I I I I 170 |82 F9 4A 0 F9 4A 0 82 F9 4A 80 F9 4A 80 82 F9 | J J J J 180 |4B 0 F9 4B 0 82 F9 4B 80 F9 4B 80 82 F9 4C 0 |K K K K L 190 |F9 4C 0 82 F9 4C 40 F9 4C 40 82 F9 4C 80 F9 4C | L L@ L@ L L 1A0 |80 82 F9 4C C0 F9 4C C0 82 F9 4D 0 F9 4D 0 82 | L L M M 1B0 |F9 4D 40 F9 4D 40 82 F9 4D 80 F9 4D 80 82 F9 4D | M@ M@ M M M 1C0 |C0 F9 4D C0 82 F9 4E 0 F9 4E 0 82 F9 4E 40 F9 | M N N N@ 1D0 |4E 40 82 F9 4E 80 F9 4E 80 82 F9 4E C0 F9 4E C0 |N@ N N N N 1E0 |82 F9 4F 0 F9 4F 0 82 F9 4F 40 F9 4F 40 82 F9 | O O O@ O@ 1F0 |4F 80 F9 4F 80 82 F9 4F C0 F9 4F C0 14000ns 509 bytes
Battle Results 42 Message size Time needed Cross- Platform JSON 1537 60000nsYes Java Serialization (not cross-language) 1022 5000nsNo Protocol Buffers 817 2100nsYes Captain Proto 664 5200nsYes CBOR 509 14000nsYes AVRO 577 5700nsYes
43 Thank you!
44

More Related Content

PPT
Unit 4 dica
byPavan Mukku
 
PDF
The IoT Academy IoT Training Arduino Part 3 programming
byThe IOT Academy
 
PDF
TC74AC244FT PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
PDF
TC74AC244F PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
PPT
Lec12 Intro to Computer Engineering by Hsien-Hsin Sean Lee Georgia Tech -- Ad...
byHsien-Hsin Sean Lee, Ph.D.
 
PPT
04 comb ex
byAravindharamanan S
 
PDF
TC74AC244FW PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
PPT
Arithmetic by aniket bhute
byAniket Bhute
 
Unit 4 dica
byPavan Mukku
 
The IoT Academy IoT Training Arduino Part 3 programming
byThe IOT Academy
 
TC74AC244FT PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
TC74AC244F PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
Lec12 Intro to Computer Engineering by Hsien-Hsin Sean Lee Georgia Tech -- Ad...
byHsien-Hsin Sean Lee, Ph.D.
 
04 comb ex
byAravindharamanan S
 
TC74AC244FW PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
Arithmetic by aniket bhute
byAniket Bhute
 

What's hot

PDF
HHVM on AArch64 - BUD17-400K1
byLinaro
 
PDF
TC74LCX244FW PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
PDF
Arduino based applications part 1
byJawaher Abdulwahab Fadhil
 
PDF
Seven segment display
byMurali Sahukari
 
DOCX
C++ file
bysimarsimmygrewal
 
PDF
Logic gates verification
byMotilal nehru national institute
 
DOCX
Computer Networks Lab File
byKandarp Tiwari
 
PDF
Logic gates verification using universal gates
byMotilal nehru national institute
 
DOC
Program for hamming code using c
bysnsanth
 
PDF
crack satellite
byTecnicoAInstrumentos
 
PDF
Ecad &amp;vlsi lab 18
byShekar Midde
 
DOCX
Direct analog
bysrikanthsailu
 
PPTX
Lec06
bysiddu kadiwal
 
PDF
TC74AC244P PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
PDF
第二回 冬のスイッチ大勉強会 - FullColorLED & MPU-6050編 -
byWataru Kani
 
PDF
TC74LCX244FT PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
PDF
TC74LCX244F PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
DOC
VLSI experiments II
byGouthaman V
 
PDF
Roberto Gallea: Workshop Arduino, giorno #2 Arduino + Processing
byDemetrio Siragusa
 
PDF
Playing with camera preview buffers on BlackBerry 10
byRaimon Ràfols
 
HHVM on AArch64 - BUD17-400K1
byLinaro
 
TC74LCX244FW PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
Arduino based applications part 1
byJawaher Abdulwahab Fadhil
 
Seven segment display
byMurali Sahukari
 
C++ file
bysimarsimmygrewal
 
Logic gates verification
byMotilal nehru national institute
 
Computer Networks Lab File
byKandarp Tiwari
 
Logic gates verification using universal gates
byMotilal nehru national institute
 
Program for hamming code using c
bysnsanth
 
crack satellite
byTecnicoAInstrumentos
 
Ecad &amp;vlsi lab 18
byShekar Midde
 
Direct analog
bysrikanthsailu
 
Lec06
bysiddu kadiwal
 
TC74AC244P PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
第二回 冬のスイッチ大勉強会 - FullColorLED & MPU-6050編 -
byWataru Kani
 
TC74LCX244FT PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
TC74LCX244F PSpice Model (Free SPICE Model)
byTsuyoshi Horigome
 
VLSI experiments II
byGouthaman V
 
Roberto Gallea: Workshop Arduino, giorno #2 Arduino + Processing
byDemetrio Siragusa
 
Playing with camera preview buffers on BlackBerry 10
byRaimon Ràfols
 

Similar to Data Serialization Frameworks for Java and the IoT

PDF
Scalable Service-Oriented Middleware over IP
byDai Yang
 
PDF
Open source Tools and Frameworks for M2M - Sierra Wireless Developer Days
byBenjamin Cabé
 
PDF
Apache Arrow Workshop at VLDB 2019 / BOSS Session
byWes McKinney
 
PDF
Pushing JavaEE outside of the enterprise: Home Automation & IoT - David Delab...
byCodemotion Tel Aviv
 
PPTX
Thrift vs Protocol Buffers vs Avro - Biased Comparison
byIgor Anishchenko
 
PDF
Apache Etch Introduction @ FOSDEM 2011
bygrandyho
 
PDF
Config interface
byRyan Boland
 
PPT
WebSocket JSON Hackday
bySomay Nakhal
 
PDF
Story Writing Byte Serializer in Golang
byHuy Do
 
PDF
GOTO Night with Todd Montgomery: Aeron: What, why and what next?
byAlexandra Masterson
 
PDF
Building a maintainable bi-directional cross platform protocol
byPavel Dovbush
 
PPTX
Data Encoding in Remote Procedure calls.
bykalojo7178
 
PDF
Teach your (micro)services talk Protocol Buffers with gRPC.
byMihai Iachimovschi
 
PDF
10 fn s03
byScott Foster
 
PDF
10 fn s03
byScott Foster
 
PPTX
Golang proto buff_ixxo
bywww.ixxo.io
 
PDF
Rest style web services (google protocol buffers) prasad nirantar
byIndicThreads
 
PPTX
Serialization and performance by Sergey Morenets
byAlex Tumanoff
 
PDF
3 f6 9a_corba
byop205
 
PDF
3 f6 9a_corba
byop205
 
Scalable Service-Oriented Middleware over IP
byDai Yang
 
Open source Tools and Frameworks for M2M - Sierra Wireless Developer Days
byBenjamin Cabé
 
Apache Arrow Workshop at VLDB 2019 / BOSS Session
byWes McKinney
 
Pushing JavaEE outside of the enterprise: Home Automation & IoT - David Delab...
byCodemotion Tel Aviv
 
Thrift vs Protocol Buffers vs Avro - Biased Comparison
byIgor Anishchenko
 
Apache Etch Introduction @ FOSDEM 2011
bygrandyho
 
Config interface
byRyan Boland
 
WebSocket JSON Hackday
bySomay Nakhal
 
Story Writing Byte Serializer in Golang
byHuy Do
 
GOTO Night with Todd Montgomery: Aeron: What, why and what next?
byAlexandra Masterson
 
Building a maintainable bi-directional cross platform protocol
byPavel Dovbush
 
Data Encoding in Remote Procedure calls.
bykalojo7178
 
Teach your (micro)services talk Protocol Buffers with gRPC.
byMihai Iachimovschi
 
10 fn s03
byScott Foster
 
10 fn s03
byScott Foster
 
Golang proto buff_ixxo
bywww.ixxo.io
 
Rest style web services (google protocol buffers) prasad nirantar
byIndicThreads
 
Serialization and performance by Sergey Morenets
byAlex Tumanoff
 
3 f6 9a_corba
byop205
 
3 f6 9a_corba
byop205
 

Recently uploaded

PDF
Hack&Verse Kick-off and infor session Event ppt
bysanidhyasahu1120
 
PDF
our environment ppt made by many people name rarang yadav
byrarang180
 
PDF
quantumcomputing-230309064424-9aa92847 (2).pdf (1).pdf
byganeshkonakanchi123
 
PDF
Healthcare Mobile App Development: Features & Benefits
byjoealwyn38
 
PDF
DSD-INT 2025 iMOD Parallel coupler development plan - Verkaik
byDeltares
 
PPTX
Orion Context Broker introduction 20260114
byFermin Galan
 
PDF
DSD-INT 2025 Large scale unsaturated zone modelling - Sequential Steady State...
byDeltares
 
PDF
DSD-INT 2025 Groundwater table lowering due to surface water lowering - Reusen
byDeltares
 
PDF
Latest Courier Management Software 2026 | Complete Courier & Logistics Solution
byCourier Softwares
 
PPTX
How MathCo Solved Enterprise AI Validation Challenges
byIT IDOL Technologies
 
PPTX
Architectural Styles in Software Engineering
byM Munim
 
PDF
Inside An AI Powered ERP System for Process Manufacturers
byBatchMaster Software Pvt. Ltd.
 
PDF
Building Strong IT Systems Using the Site Reliability Engineering
byAvekshaa Technologies
 
PDF
Transferring Ideas to Impact -Driving Innovation and Demand with OnePlan
byOnePlan Solutions
 
PPTX
Online Contractor Safety Induction Training Software
bySHEQ Network Limited
 
PDF
Driving Finance Growth with Business Central ERP
byNavision India
 
PDF
Navigating Product Evolution: When MVPs Mature Beyond Their Initial Scope
byCadabra Studio
 
PPTX
1-introduction-to-databases notes start .pptx
byBeatrice1625
 
PPTX
EMR Software for Gynaecology Clinics How EasyClinic Supports Continuity, Priv...
byEasy Clinic
 
PDF
Cybernetic Global Intelligence Securing Businesses in a Digital-First World.pdf
byCybernetic Global Intelligence
 
Hack&Verse Kick-off and infor session Event ppt
bysanidhyasahu1120
 
our environment ppt made by many people name rarang yadav
byrarang180
 
quantumcomputing-230309064424-9aa92847 (2).pdf (1).pdf
byganeshkonakanchi123
 
Healthcare Mobile App Development: Features & Benefits
byjoealwyn38
 
DSD-INT 2025 iMOD Parallel coupler development plan - Verkaik
byDeltares
 
Orion Context Broker introduction 20260114
byFermin Galan
 
DSD-INT 2025 Large scale unsaturated zone modelling - Sequential Steady State...
byDeltares
 
DSD-INT 2025 Groundwater table lowering due to surface water lowering - Reusen
byDeltares
 
Latest Courier Management Software 2026 | Complete Courier & Logistics Solution
byCourier Softwares
 
How MathCo Solved Enterprise AI Validation Challenges
byIT IDOL Technologies
 
Architectural Styles in Software Engineering
byM Munim
 
Inside An AI Powered ERP System for Process Manufacturers
byBatchMaster Software Pvt. Ltd.
 
Building Strong IT Systems Using the Site Reliability Engineering
byAvekshaa Technologies
 
Transferring Ideas to Impact -Driving Innovation and Demand with OnePlan
byOnePlan Solutions
 
Online Contractor Safety Induction Training Software
bySHEQ Network Limited
 
Driving Finance Growth with Business Central ERP
byNavision India
 
Navigating Product Evolution: When MVPs Mature Beyond Their Initial Scope
byCadabra Studio
 
1-introduction-to-databases notes start .pptx
byBeatrice1625
 
EMR Software for Gynaecology Clinics How EasyClinic Supports Continuity, Priv...
byEasy Clinic
 
Cybernetic Global Intelligence Securing Businesses in a Digital-First World.pdf
byCybernetic Global Intelligence
 

Data Serialization Frameworks for Java and the IoT

  • 1.
    Data Serialization Frameworksfor Java and the IoT Manfred Dreese, codecentric AG 1
  • 2.
    AGENDA 2 Shortcomings and no-frills WhatData-Serialization Frameworks do Architectural considerations API, Speed and Size Battle Back to binary encoded messages?
  • 3.
  • 4.
    Standard Interface Definitions languages orhand-crafted binary Protocols 4 History of binary serialization frameworks 80s String-encoded Interfaces, String encoded verification String-encoded Protocols 90s Renaissance of binary protocols as backend- turbocharger. Just with more frills. 2000 Binary Serialization to Turbocharge embedded Devices 2010
  • 5.
    Standard Interface Definitions languages orhand-crafted binary Protocols 5 History of binary serialization frameworks 80s String-encoded Interfaces, String encoded verification String-encoded Protocols 90s Renaissance of binary protocols as backend- turbocharger. Just with more frills. 2000 Binary Serialization to Turbocharge embedded Devices 2010> ASN-1 Thrift Protobuf JSON Avro CBOR lots_of {custom_protocol}
  • 6.
    Message length considerationsfor IoT appliances 6 Cloud Application IoT Solution Powerful, scalable Servers Finite number of constrained devices High amount of influence on technology stack Mixed technology stacks, processor architectures and languages High connections speeds Serial links, Industrial bus connections, Cellular networks. No restrictions on data transfer Low bandwidth and duty cycle regulations Nodes available all the time Power Saving, Mobile Data Saving, possible bad signal level Data transfer is (almost) free of charge (Literally) expensive data transfer when mobile data is involved. Processing power for data transfer is neglectible Sophisticated String processing or data transfer in resource competition with actual functionality
  • 7.
  • 8.
    Code generator 8 How dataserialization frameworks work Schema-driven Frameworks Interface Definition Java code Golang code Javascript code {any} code
  • 9.
    IDL/Schema Definition Code generation MessageComposition and Serialization 9 Example 1 2 3 Message Transfer 4 Message Deserialization 5
  • 10.
    IDL/Schema Definition Code generation MessageComposition and Serialization 10 Example syntax = "proto3"; import "Common.proto"; import "google/protobuf/timestamp.proto"; package jcon.telemetry; option java_package = "de.m9d.telemetry.engine"; option go_package = "m9d.de/telemetry/engine“; message Telegram { string engineId = 1; google.protobuf.Timestamp timeFrom = 2; google.protobuf.Timestamp timeTo = 3; double totalWork = 4; Curve engineSpeed = 5; Curve temperature = 6; Curve oilPressure = 7; Curve fuelConsumption = 8; } 1 2 3 Define interface using a meta-language. Message Transfer 4 Message Deserialization 5
  • 11.
    11 Example <plugin> <groupId>com.github.os72</groupId> <artifactId>protoc-jar-maven-plugin</artifactId> <version>3.5.0</version> <executions> <execution> <phase>generate-sources</phase> <goals> <goal>run</goal> </goals> <configuration> <protocVersion>3.5.0</protocVersion> <includeStdTypes>true</includeStdTypes> <includeDirectories> </includeDirectories> <inputDirectories> <include>../idl</include> </inputDirectories> </configuration> </execution> </executions> </plugin> <dependency>    <groupId>com.google.protobuf</groupId>    <artifactId>protobuf-java</artifactId>    <version>3.5.1</version> </dependency> IDL/Schema Definition Code generation MessageComposition and Serialization 1 2 3 Use code-generator (CLI / build plugin) to build producer/consumer code for your platform Message Transfer 4 Message Deserialization 5 protoc -I=../idl —cpp_out=api/ —wrap=*.proto
  • 12.
    12 Example Common.Curve.Builder sampleCurve =Common.Curve.newBuilder(); for (int x = 0 ; x < 32 ; x++) { sampleCurve.addPoints( Common.Point.newBuilder() .setX(x) .setY(x) .build()); } EngineTelemetry.Telegram message= EngineTelemetry.Telegram .newBuilder() .setTimeFrom(Timestamp.newBuilder() .setSeconds(fromTimestamp) .setTimeTo(Timestamp.newBuilder() .setSeconds(toTimestamp) .setTotalWork(42.23) .setEngineId(engineUid.toString()) .setFuelConsumption(sampleCurve) .setOilPressure(sampleCurve) .build(); byte[] proto = message.toByteArray(); IDL/Schema Definition Code generation Message Composition and Serialization 1 2 3 Message Transfer 4 Message Deserialization 5 Use generated code or generic API to compose a message.
  • 13.
    13 0 |24 3037 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4 10 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e 20 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc 30 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@: 40 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ? 50 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @ 60 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @ 70 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @ 80 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @ 90 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A
  • 14.
    14 0 |22 656E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077 10 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4- 20 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f 30 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": { 40 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389 50 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo": 60 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153 70 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW 80 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil 90 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x": B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{" C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0 D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y": E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0,
  • 15.
    15 Example EngineTelemetry.Telegram deserialized = EngineTelemetry.Telegram.parseFrom(wire); IDL/SchemaDefinition Code generation Message Composition and Serialization 1 2 3 Message Transfer 4 Message Deserialization 5 The same way back
  • 16.
    Generated Builder API Builder APIor types serializers/deserializers based on Interface definition Generic Builder API Field-By-Field API Abstracted-away Workhorse Serving as a Serializer Option for i.E. Jackson 16 API types
  • 17.
    17 ..it is not Auint32 is a uint32… Message with low int values telegram. .setMaximumTorque(32) .setTotalWork(32) .build(); telegram. .setMaximumTorque(65535) .setTotalWork(65535) .build(); 2 8 20 21 0 0 0 0 0 0 40 40 (13 bytes) 4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40 (15 bytes) Message with high int values
  • 18.
    18 40 | 00 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a 50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479 60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c 70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d 80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A Random Access With fixed length fields -Predictable offset of any item -Decode just a subset of the message -Just point reading routines to raw data, no need to allocate additional memory
  • 19.
  • 20.
    20 No error correction Encodemessage Meet the bad guys EngineTelemetry.Telegram telemetry= EngineTelemetry.Telegram.newBuilder() .setTimeFrom(Timestamp.now()) .setTotalWork(65535) .build(); wire[10] = 0x40; Decoding works fine! EngineTelemetry.Telegram deserialized = EngineTelemetry.Telegram.parseFrom(wire); deserialized: { „timeFrom": {"seconds": 65535}, "totalWork": 65535.0078125 } Well, fine enough for another Diesel joke… 4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40 4 8 FF FF 3 21 0 0 0 0 40 FF EF 40
  • 21.
    21 40 | 00 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a 50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479 60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c 70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d 80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A // CAFE, BABE? could be anything! No frills, no type announcement -Type awareness has to be established on decoration or transport
  • 22.
    22 Non-opiniated with regardsto transport -Just an array of bytes -No encryption or compression by default -No transport or remote method methods by default -Except Add-Ons (gRPC, AVRO RPC) -Integrateable in existing transport infrastructure -Raw sockets -MQTT, AMQP, Kafka, … -Machine bus protocols -XMPP -Local memory
  • 23.
  • 24.
    24 Use-case: Binary encodedcontent Message Broker (MQTT/AMQP/…) IoT Device Consumers Submit
  • 25.
    25 Use-case: API generation Codegenerator Service A API description V1.23.42 Service C API code Service B API code Checkout Generate Generate
  • 26.
    26 Use-case: Backend Payload BackendService B Backend Service CBackend Service A Message Broker Pub/Sub Pub/Sub Pub/Sub
  • 27.
    27 Best Usecases -High messagesizes and frequencies -Structure data -Lists, Arrays, Time Series -Cross-Platform and Languages
  • 28.
  • 29.
    Serialize a enginetelemetry data item as fast as possible With as little data transfer as possible Only no-brainer optimizations are allowed Given situation 29 A fictional telemetry use-case EngineID Uuid (String encoding if no native type) Observation Interval Unix timestamp from, to Total amount of work Double Fuel consumption curve 32 float tupels (0,0)..(32,32) Oil Pressure curve 32 float tupels (0,0)..(32,32) Serialization in Java
  • 30.
    JSON 30 0 |22 656E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077 10 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4- 20 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f 30 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": { 40 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389 50 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo": 60 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153 70 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW 80 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil 90 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x": B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{" C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0 D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y": E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0, F0 |22 79 22 3A 20 34 2E 30 7D 2C 7B 22 78 22 3A 20 |"y": 4.0 },{"x": 100 |35 2E 30 2C 22 79 22 3A 20 35 2E 30 7D 2C 7B 22 |5.0,"y": 5.0},{" 110 |78 22 3A 20 36 2E 30 2C 22 79 22 3A 20 36 2E 30 |x": 6.0, "y": 6.0 120 |7D 2C 7B 22 78 22 3A 20 37 2E 30 2C 22 79 22 3A |},{"x": 7.0,"y": 130 |20 37 2E 30 7D 2C 7B 22 78 22 3A 20 38 2E 30 2C | 7.0},{" x": 8.0, 140 |22 79 22 3A 20 38 2E 30 7D 2C 7B 22 78 22 3A 20 |"y": 8.0 },{"x": 150 |39 2E 30 2C 22 79 22 3A 20 39 2E 30 7D 2C 7B 22 |9.0,"y": 9.0},{" 160 |78 22 3A 20 31 30 2E 30 2C 22 79 22 3A 20 31 30 |x": 10.0 ,"y": 10 170 |2E 30 7D 2C 7B 22 78 22 3A 20 31 31 2E 30 2C 22 |.0},{"x" : 11.0," 180 |79 22 3A 20 31 31 2E 30 7D 2C 7B 22 78 22 3A 20 |y": 11.0 },{"x": 190 |31 32 2E 30 2C 22 79 22 3A 20 31 32 2E 30 7D 2C |12.0,"y" : 12.0}, 1A0 |7B 22 78 22 3A 20 31 33 2E 30 2C 22 79 22 3A 20 |{"x": 13 .0,"y": 1B0 |31 33 2E 30 7D 2C 7B 22 78 22 3A 20 31 34 2E 30 |13.0},{" x": 14.0 1C0 |2C 22 79 22 3A 20 31 34 2E 30 7D 2C 7B 22 78 22 |,"y": 14 .0},{"x" 1D0 |3A 20 31 35 2E 30 2C 22 79 22 3A 20 31 35 2E 30 |: 15.0," y": 15.0 1E0 |7D 2C 7B 22 78 22 3A 20 31 36 2E 30 2C 22 79 22 |},{"x": 16.0,"y" 1F0 |3A 20 31 36 2E 30 7D 2C 7B 22 78 22 3A 20 31 37 |: 16.0}, {"x": 17 200 |2E 30 2C 22 79 22 3A 20 31 37 2E 30 7D 2C 7B 22 |.0,"y": 17.0},{" 210 |78 22 3A 20 31 38 2E 30 2C 22 79 22 3A 20 31 38 |x": 18.0 ,"y": 18 220 |2E 30 7D 2C 7B 22 78 22 3A 20 31 39 2E 30 2C 22 |.0},{"x" : 19.0," 230 |79 22 3A 20 31 39 2E 30 7D 2C 7B 22 78 22 3A 20 |y": 19.0 },{"x": 240 |32 30 2E 30 2C 22 79 22 3A 20 32 30 2E 30 7D 2C |20.0,"y" : 20.0}, 250 |7B 22 78 22 3A 20 32 31 2E 30 2C 22 79 22 3A 20 |{"x": 21 .0,"y": 260 |32 31 2E 30 7D 2C 7B 22 78 22 3A 20 32 32 2E 30 |21.0},{" x": 22.0 270 |2C 22 79 22 3A 20 32 32 2E 30 7D 2C 7B 22 78 22 |,"y": 22 .0},{"x" // (cut) 60000ns 1537 bytes
  • 31.
    31 Google Protocol Buffers syntax= "proto3"; import "Common.proto"; import "google/protobuf/timestamp.proto"; package jcon.telemetry; option java_package = "de.m9d.telemetry.engine"; option go_package = "de.m9d/telemetry/engine"; message Telegram { string engineId = 1; google.protobuf.Timestamp timeFrom = 2; google.protobuf.Timestamp timeTo = 3; double totalWork = 4; Curve engineSpeed = 5; Curve temperature = 6; Curve oilPressure = 7; Curve fuelConsumption = 8; } syntax = "proto3"; package jcon.telemetry; option java_package = "de.m9d.telemetry.engine"; option go_package = "de.m9d/telemetry/engine"; message Point { float x = 1; float y = 2; } message Curve { repeated Point points = 1; }
  • 32.
    32 public void serializeProtobuf(){ Common.Curve.Builder sampleCurve = Common.Curve.newBuilder(); for (int x = 0 ; x < 32 ; x++) { sampleCurve.addPoints(Common.Point.newBuilder().setX(x).setY(x).build()); } EngineTelemetry.Telegram message= EngineTelemetry.Telegram.newBuilder() .setTimeFrom(Timestamp.newBuilder().setSeconds(Instant.now().getEpochSecond()).build()) .setTimeTo(Timestamp.newBuilder().setSeconds(Instant.now().getEpochSecond()-3600).build()) .setTotalWork(42.23) .setEngineId(UUID.randomUUID().toString()) .setFuelConsumption(sampleCurve) .setOilPressure(sampleCurve) .build(); byte[] proto = message.toByteArray(); } Google Protocol Buffers
  • 33.
    Google Protocol Buffers 33 0|24 30 37 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4 10 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e 20 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc 30 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@: 40 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ? 50 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @ 60 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @ 70 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @ 80 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @ 90 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A F0 |70 41 15 0 0 70 41 A A D 0 0 80 41 15 0 |pA pA A 100 | 0 80 41 A A D 0 0 88 41 15 0 0 88 41 A | A A A 110 | A D 0 0 90 41 15 0 0 90 41 A A D 0 0 | A A 120 |98 41 15 0 0 98 41 A A D 0 0 A0 41 15 0 | A A A 130 | 0 A0 41 A A D 0 0 A8 41 15 0 0 A8 41 A | A A A 140 | A D 0 0 B0 41 15 0 0 B0 41 A A D 0 0 | A A 150 |B8 41 15 0 0 B8 41 A A D 0 0 C0 41 15 0 | A A A 160 | 0 C0 41 A A D 0 0 C8 41 15 0 0 C8 41 A | A A A 170 | A D 0 0 D0 41 15 0 0 D0 41 A A D 0 0 | A A 180 |D8 41 15 0 0 D8 41 A A D 0 0 E0 41 15 0 | A A A 190 | 0 E0 41 A A D 0 0 E8 41 15 0 0 E8 41 A | A A A 1A0 | A D 0 0 F0 41 15 0 0 F0 41 A A D 0 0 | A A 1B0 |F8 41 15 0 0 F8 41 42 F6 2 A 0 A A D 0 | A AB 1C0 | 0 80 3F 15 0 0 80 3F A A D 0 0 0 40 15 | ? ? @ 1D0 | 0 0 0 40 A A D 0 0 40 40 15 0 0 40 40 | @ @@ @@ 1E0 | A A D 0 0 80 40 15 0 0 80 40 A A D 0 | @ @ 1F0 | 0 A0 40 15 0 0 A0 40 A A D 0 0 C0 40 15 | @ @ @ 200 | 0 0 C0 40 A A D 0 0 E0 40 15 0 0 E0 40 | @ @ @ 210 | A A D 0 0 0 41 15 0 0 0 41 A A D 0 | A A 220 | 0 10 41 15 0 0 10 41 A A D 0 0 20 41 15 | A A A 230 | 0 0 20 41 A A D 0 0 30 41 15 0 0 30 41 | A 0A 0A 240 | A A D 0 0 40 41 15 0 0 40 41 A A D 0 | @A @A 250 | 0 50 41 15 0 0 50 41 A A D 0 0 60 41 15 | PA PA `A 260 | 0 0 60 41 A A D 0 0 70 41 15 0 0 70 41 | `A pA pA 270 | A A D 0 0 80 41 15 0 0 80 41 A A D 0 | A A 280 | 0 88 41 15 0 0 88 41 A A D 0 0 90 41 15 | A A A 290 | 0 0 90 41 A A D 0 0 98 41 15 0 0 98 41 | A A A // (cut) 2700ns 817 bytes
  • 34.
    34 Captain Proto @0xcfe6e6668c89c78f; using Java= import "/java.capnp"; $Java.package("de.m9d.telemetry.engine"); $Java.outerClassname("Telemetry"); struct Telegram { code @0 :Text; engineId @1 :Text; timeFrom @2 :UInt32; timeTo @3 :UInt32; totalWork @4 :Float64; engineSpeed @5: Curve; temperature @6: Curve; oilPressure @7: Curve; fuelConsumption @8: Curve; } struct Curve { points @0: List(Point); } struct Point { x @0: Float32; y @1: Float32; }
  • 35.
    35 Captain Proto public voidserializeCaptainProto() throws IOException { String engineId = UUID.randomUUID().toString(); org.capnproto.MessageBuilder message = new org.capnproto.MessageBuilder(); de.m9d.telemetry.engine.Telemetry.Telegram.Builder builder = message.initRoot(Telemetry.Telegram.factory); builder.setEngineId(engineId); builder.setTimeFrom((int) Instant.now().getEpochSecond()); builder.setTimeTo((int) Instant.now().getEpochSecond()); builder.setTotalWork((float)42.561); builder.initEngineSpeed(); de.m9d.telemetry.engine.Telemetry.Curve.Builder engineSpeedBuilder = builder.getEngineSpeed(); engineSpeedBuilder.initPoints(32); for (int idx = 0 ; idx < 32 ; idx ++) { engineSpeedBuilder.getPoints().get(idx).setX(idx); engineSpeedBuilder.getPoints().get(idx).setY(idx); } // Same for oil pressure ByteArrayOutputStream bos = new ByteArrayOutputStream(); WritableByteChannel wbc = Channels.newChannel(bos); org.capnproto.Serialize.write(wbc, message); byte[] proto = bos.toByteArray(); }
  • 36.
    Captain Proto 36 0 |0 0 0 52 0 0 0 0 0 0 0 2 0 6 0 DC | R 10 |BD BB 5B DC BD BB 5B 77 3E 2A 42 0 0 0 0 0 | [ [w >*B 20 | 0 0 0 0 0 0 0 11 0 0 0 2A 1 0 0 20 | * 30 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 A0 | 40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a 50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479 60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c 70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d 80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A E0 | 0 30 41 0 0 30 41 0 0 40 41 0 0 40 41 0 | 0A 0A @A @A F0 | 0 50 41 0 0 50 41 0 0 60 41 0 0 60 41 0 | PA PA `A `A 100 | 0 70 41 0 0 70 41 0 0 80 41 0 0 80 41 0 | pA pA A A 110 | 0 88 41 0 0 88 41 0 0 90 41 0 0 90 41 0 | A A A A 120 | 0 98 41 0 0 98 41 0 0 A0 41 0 0 A0 41 0 | A A A A 130 | 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 B0 41 0 | A A A A 140 | 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 C0 41 0 | A A A A 150 | 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 D0 41 0 | A A A A 160 | 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 E0 41 0 | A A A A 170 | 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 F0 41 0 | A A A A 180 | 0 F8 41 0 0 F8 41 1 0 0 0 7 1 0 0 80 | A A 190 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 | 1A0 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @ 1B0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @ 1C0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @ 1D0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A 1E0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A 1F0 | 0 30 41 0 0 30 41 0 0 40 41 0 0 40 41 0 | 0A 0A @A @A 200 | 0 50 41 0 0 50 41 0 0 60 41 0 0 60 41 0 | PA PA `A `A 210 | 0 70 41 0 0 70 41 0 0 80 41 0 0 80 41 0 | pA pA A A 220 | 0 88 41 0 0 88 41 0 0 90 41 0 0 90 41 0 | A A A A 230 | 0 98 41 0 0 98 41 0 0 A0 41 0 0 A0 41 0 | A A A A 240 | 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 B0 41 0 | A A A A 250 | 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 C0 41 0 | A A A A 260 | 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 D0 41 0 | A A A A 270 | 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 E0 41 0 | A A A A 280 | 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 F0 41 0 | A A A A 290 | 0 F8 41 0 0 F8 41 5200ns 664 bytes
  • 37.
    37 AVRO [ { "namespace": "jcon.telemetryserver.domain", "type": "record", "name":"Curve", "fields": [ {"name": "points", "type": {"type": "array", "items": { "name":"Point", "type":"record", "fields":[ {"name":"x", "type":"float"}, {"name":"y", "type":"float"} ] }}} ] }, { "namespace": "jcon.telemetryserver.domain", "type": "record", "name": "Telegram", "fields": [ {"name": "engineId", "type": "string"}, {"name": "timeFrom", "type": {"type": "long", "logicalType": "timestamp-millis"}}, {"name": "timeTo", "type": {"type": "long", "logicalType": "timestamp-millis"}}, {"name": "totalWork", "type": "double"}, {"name": "engineSpeed", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }, {"name": "temperature", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }, {"name": "oilPressure", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }, {"name": "fuelConsumption", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null } ] } ]
  • 38.
    38 AVRO public void serializeWithAvroAndSchemas()throws IOException { ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); DatumWriter<Telegram> writer = new SpecificDatumWriter<>(Telegram.getClassSchema()); BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null); List<Point> points = new ArrayList<>(); for (float x = 0 ; x < 32 ; x++) { points.add(Point.newBuilder().setX(x).setY(x).build()); } Curve curve = Curve.newBuilder() .setPoints(points) .build(); Telegram telegram = Telegram.newBuilder() .setEngineId(UUID.randomUUID().toString()) .setTimeFrom(DateTime.now()) .setTimeTo(DateTime.now()) .setTotalWork(42.23) .setEngineSpeed(curve) .setFuelConsumption(curve) .build(); writer.write(telegram, encoder); encoder.flush(); byte[] wire = outputStream.toByteArray(); }
  • 39.
    AVRO 39 0 |64 6362 33 33 39 33 65 2D 35 34 63 61 2D 34 66 |dcb3393e -54ca-4f 10 |39 39 2D 38 31 35 64 2D 35 62 36 64 35 31 33 35 |99-815d- 5b6d5135 20 |37 30 31 34 84 F3 A9 85 CA 59 B0 F3 A9 85 CA 59 |7014 Y Y 30 |3D A D7 A3 70 1D 45 40 2 40 0 0 0 0 0 0 |= p E@ @ 40 | 0 0 0 0 80 3F 0 0 80 3F 0 0 0 40 0 0 | ? ? @ 50 | 0 40 0 0 40 40 0 0 40 40 0 0 80 40 0 0 | @ @@ @@ @ 60 |80 40 0 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 | @ @ @ @ 70 |C0 40 0 0 E0 40 0 0 E0 40 0 0 0 41 0 0 | @ @ @ A 80 | 0 41 0 0 10 41 0 0 10 41 0 0 20 41 0 0 | A A A A 90 |20 41 0 0 30 41 0 0 30 41 0 0 40 41 0 0 | A 0A 0A @A A0 |40 41 0 0 50 41 0 0 50 41 0 0 60 41 0 0 |@A PA PA `A B0 |60 41 0 0 70 41 0 0 70 41 0 0 80 41 0 0 |`A pA pA A C0 |80 41 0 0 88 41 0 0 88 41 0 0 90 41 0 0 | A A A A D0 |90 41 0 0 98 41 0 0 98 41 0 0 A0 41 0 0 | A A A A E0 |A0 41 0 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 | A A A A F0 |B0 41 0 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 | A A A A 100 |C0 41 0 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 | A A A A 110 |D0 41 0 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 | A A A A 120 |E0 41 0 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 | A A A A 130 |F0 41 0 0 F8 41 0 0 F8 41 0 0 0 2 40 0 | A A A @ 140 | 0 0 0 0 0 0 0 0 0 80 3F 0 0 80 3F 0 | ? ? 150 | 0 0 40 0 0 0 40 0 0 40 40 0 0 40 40 0 | @ @ @@ @@ 160 | 0 80 40 0 0 80 40 0 0 A0 40 0 0 A0 40 0 | @ @ @ @ 170 | 0 C0 40 0 0 C0 40 0 0 E0 40 0 0 E0 40 0 | @ @ @ @ 180 | 0 0 41 0 0 0 41 0 0 10 41 0 0 10 41 0 | A A A A 190 | 0 20 41 0 0 20 41 0 0 30 41 0 0 30 41 0 | A A 0A 0A 1A0 | 0 40 41 0 0 40 41 0 0 50 41 0 0 50 41 0 | @A @A PA PA 1B0 | 0 60 41 0 0 60 41 0 0 70 41 0 0 70 41 0 | `A `A pA pA 1C0 | 0 80 41 0 0 80 41 0 0 88 41 0 0 88 41 0 | A A A A 1D0 | 0 90 41 0 0 90 41 0 0 98 41 0 0 98 41 0 | A A A A 1E0 | 0 A0 41 0 0 A0 41 0 0 A8 41 0 0 A8 41 0 | A A A A 1F0 | 0 B0 41 0 0 B0 41 0 0 B8 41 0 0 B8 41 0 | A A A A 200 | 0 C0 41 0 0 C0 41 0 0 C8 41 0 0 C8 41 0 | A A A A 210 | 0 D0 41 0 0 D0 41 0 0 D8 41 0 0 D8 41 0 | A A A A 220 | 0 E0 41 0 0 E0 41 0 0 E8 41 0 0 E8 41 0 | A A A A 230 | 0 F0 41 0 0 F0 41 0 0 F8 41 0 0 F8 41 0 | A A A A 240 | 5400ns 577 bytes
  • 40.
    40 CBOR public void serializeCbor()throws IOException, CborException { ByteArrayOutputStream baos = new ByteArrayOutputStream(); CborBuilder di = new CborBuilder(); ArrayBuilder ab = di.addArray(); for (float x = 0 ; x < 32 ; x++) { ab.addArray() .add(x) .add(x) .end(); } List<DataItem> curves = di.build(); new CborEncoder(baos).encode(new CborBuilder() .add(UUID.randomUUID().toString()) .add(Instant.now().getEpochSecond()) .add(Instant.now().getEpochSecond()) .add(42.23) .add(curves.get(0)) .add(curves.get(0)) .build()); byte[] encodedBytes = baos.toByteArray(); }
  • 41.
    CBOR 41 0 |24 3539 38 32 31 33 63 36 2D 64 65 37 37 2D 34 |$598213c 6-de77-4 10 |33 63 63 2D 62 64 38 61 2D 30 31 37 30 30 64 35 |3cc-bd8a -01700d5 20 |30 62 62 35 39 1A 5B BB C4 CC 1A 5B BB C4 CC FB |0bb59 [ [ 30 |40 45 1D 70 A3 D7 A 3D 98 20 82 F9 0 0 F9 0 |@E p = 40 | 0 82 F9 3C 0 F9 3C 0 82 F9 40 0 F9 40 0 82 | < < @ @ 50 |F9 42 0 F9 42 0 82 F9 44 0 F9 44 0 82 F9 45 | B B D D E 60 | 0 F9 45 0 82 F9 46 0 F9 46 0 82 F9 47 0 F9 | E F F G 70 |47 0 82 F9 48 0 F9 48 0 82 F9 48 80 F9 48 80 |G H H H H 80 |82 F9 49 0 F9 49 0 82 F9 49 80 F9 49 80 82 F9 | I I I I 90 |4A 0 F9 4A 0 82 F9 4A 80 F9 4A 80 82 F9 4B 0 |J J J J K A0 |F9 4B 0 82 F9 4B 80 F9 4B 80 82 F9 4C 0 F9 4C | K K K L L B0 | 0 82 F9 4C 40 F9 4C 40 82 F9 4C 80 F9 4C 80 82 | L@ L@ L L C0 |F9 4C C0 F9 4C C0 82 F9 4D 0 F9 4D 0 82 F9 4D | L L M M M D0 |40 F9 4D 40 82 F9 4D 80 F9 4D 80 82 F9 4D C0 F9 |@ M@ M M M E0 |4D C0 82 F9 4E 0 F9 4E 0 82 F9 4E 40 F9 4E 40 |M N N N@ N@ F0 |82 F9 4E 80 F9 4E 80 82 F9 4E C0 F9 4E C0 82 F9 | N N N N 100 |4F 0 F9 4F 0 82 F9 4F 40 F9 4F 40 82 F9 4F 80 |O O O @ O@ O 110 |F9 4F 80 82 F9 4F C0 F9 4F C0 98 20 82 F9 0 0 | O O O 120 |F9 0 0 82 F9 3C 0 F9 3C 0 82 F9 40 0 F9 40 | < < @ @ 130 | 0 82 F9 42 0 F9 42 0 82 F9 44 0 F9 44 0 82 | B B D D 140 |F9 45 0 F9 45 0 82 F9 46 0 F9 46 0 82 F9 47 | E E F F G 150 | 0 F9 47 0 82 F9 48 0 F9 48 0 82 F9 48 80 F9 | G H H H 160 |48 80 82 F9 49 0 F9 49 0 82 F9 49 80 F9 49 80 |H I I I I 170 |82 F9 4A 0 F9 4A 0 82 F9 4A 80 F9 4A 80 82 F9 | J J J J 180 |4B 0 F9 4B 0 82 F9 4B 80 F9 4B 80 82 F9 4C 0 |K K K K L 190 |F9 4C 0 82 F9 4C 40 F9 4C 40 82 F9 4C 80 F9 4C | L L@ L@ L L 1A0 |80 82 F9 4C C0 F9 4C C0 82 F9 4D 0 F9 4D 0 82 | L L M M 1B0 |F9 4D 40 F9 4D 40 82 F9 4D 80 F9 4D 80 82 F9 4D | M@ M@ M M M 1C0 |C0 F9 4D C0 82 F9 4E 0 F9 4E 0 82 F9 4E 40 F9 | M N N N@ 1D0 |4E 40 82 F9 4E 80 F9 4E 80 82 F9 4E C0 F9 4E C0 |N@ N N N N 1E0 |82 F9 4F 0 F9 4F 0 82 F9 4F 40 F9 4F 40 82 F9 | O O O@ O@ 1F0 |4F 80 F9 4F 80 82 F9 4F C0 F9 4F C0 14000ns 509 bytes
  • 42.
    Battle Results 42 Message sizeTime needed Cross- Platform JSON 1537 60000nsYes Java Serialization (not cross-language) 1022 5000nsNo Protocol Buffers 817 2100nsYes Captain Proto 664 5200nsYes CBOR 509 14000nsYes AVRO 577 5700nsYes
  • 43.
  • 44.