The document provides advice for optimizing data transferring between client and server. It recommends minimizing the amount of data transferred by using techniques like gzip compression, caching, sending only partial responses with needed fields, and using binary formats like Protocol Buffers which are smaller and faster than JSON. It also suggests optimizing data structures, using streams for parsing, and benchmarking different approaches to determine the best solution.

1. Data transferring
faster, stronger, better and not harder

2. Disclaimer
Data transferring optimizations requires both, client side and server side
development, could be hard to implement and it could break your system (and
may be life)
...But it worth to be done

3. Is it really a problem?

4. Is it really a problem?

5. General advice
● GZIP
● Cache (!!!)
● Partial requests/responses
● Minimize data structure
● Use streams where possible
● Use Binary formats

6. General advice
● GZIP
● Cache (!!!)
● Partial requests/responses
● Minimize data structure
● Use streams where possible
● Use Binary formats

7. GZIP
● Network cost vs CPU
● Measure it

8. General advice
● GZIP
● Cache (!!!)
● Partial requests/responses
● Minimize data structure
● Use streams where possible
● Use Binary formats

9. Cache
● Client DB
● Network client level cache
● Server side support (ETag, Cache-Control, etc.)
● Use push notifications for cache invalidating

10. General advice
● GZIP
● Cache (!!!)
● Partial requests/responses
● Minimize data structure
● Use streams where possible
● Use Binary formats

11. Partial requests/responses

12. Partial requests/responses
● Transfer only what you really need
● Control field count (Data Diet)
● Pagination is our friend

13. Partial requests/responses
https://www.googleapis.com/demo/v1?key=YOUR-API-KEY

14. Partial requests/responses
https://www.googleapis.com/demo/v1?key=YOUR-API-KEY
{ "kind": "demo",
...
"items": [
{
"title": "First title",
"comment": "First comment.",
"characteristics": {
"length": "short",
"accuracy": "high",
"followers": ["Jo", "Will"],
}, ... ]
}

15. Partial requests/responses
https://www.googleapis.com/demo/v1?key=YOUR-API-KEY&fields=kind,items(title,characteristics/length)

16. Partial requests/responses
https://www.googleapis.com/demo/v1?key=YOUR-API-KEY&fields=kind,items(title,characteristics/length)
{ "kind": "demo",
"items": [
{
"title": "First title",
"characteristics": {
"length": "short"
}
}, ... ]
}

17. Partial requests/responses
{ "kind": "demo",
...
"items": [
{
"title": "First title",
"comment": "First comment.",
"characteristics": {
"length": "short",
"accuracy": "high",
"followers": ["Jo", "Will"],
}, ... ]
}
{ "kind": "demo",
"items": [
{
"title": "First title",
"characteristics": {
"length": "short"
}
}, ... ]
}

18. Partial requests/responses
{ "kind": "demo",
...
"items": [
{
"title": "First title",
"comment": "First comment.",
"characteristics": {
"length": "short",
"accuracy": "high",
"followers": ["Jo", "Will"],
}, ... ]
}
{ "kind": "demo",
"items": [
{
"title": "First title",
"characteristics": {
"length": "short"
}
}, ... ]
}

19. Fields parameter syntax summary
● Use a comma-separated list to select multiple fields.
● Use a/b to select a field b that is nested within field a; use a/b/c to select a
field c nested within b.
● Exception: For API responses that use "data" wrappers, where the response
is nested within a data object that looks like data: { ... }, do not include "data"
in the fields specification. Including the data object with a fields specification
like data/a/b causes an error. Instead, just use a fields specification like a/b.

20. Fields parameter syntax summary
● Use a sub-selector to request a set of specific sub-fields of arrays or objects
by placing expressions in parentheses "( )".
For example: fields=items(id,author/email) returns only the item ID and author's
email for each element in the items array. You can also specify a single sub-field,
where fields=items(id) is equivalent to fields=items/id.

21. Fields parameter syntax summary
● Use wildcards in field selections, if needed.
For example: fields=items/pagemap/* selects all objects in a pagemap.

22. Use PATCH (yes, there is a HTTP Verb “PATCH”)
● https://tools.ietf.org/html/rfc5789
● If you’ve changed one field - send one field
● Retrofit support included ;)
● Could be used with Partial Responses

23. General advice
● GZIP
● Cache (!!!)
● Minimize data
● Minimize data structure
● Use streams where possible
● Use Binary formats

24. Minimize data structure
● Remove spaces
● Remove long field’ names
● Remove braces
● Remove data (see previous slides)

25. Make you data homogeneous
What is an Homogenous Collection?
Before...
[
{"a":"A","b":"B"},
{"a":"C","b":"D"},
{"a":"E","b":"F"}
]
After…
[2,"a","b","A","B","C","D","E","F"]

26. Tools
● CJSON Algorithm
● JSON.hpack Algorithm
● Brain (use it carefully)

27. General advice
● GZIP
● Cache (!!!)
● Partial requests/responses
● Minimize data structure
● Use streams where possible
● Use Binary formats

28. Why streams?
● Start parsing from the first byte
● Reduce memory usage
● Most of the tools support it by default

29. General advice
● GZIP
● Cache (!!!)
● Partial requests/responses
● Minimize data structure
● Use streams where possible
● Use Binary formats

30. Binary formats
● Fast
● Small
● Optimized for every platform
● Static typed

31. Binary formats
● Protobuf
● Thrift (Never used it :( )
● Cap’n proto

32. Protobuf
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}

33. Protobuf
Person john = Person.newBuilder()
.setId(1234)
.setName("John Doe")
.setEmail("jdoe@example.com")
.build();
output = new FileOutputStream(args[0]);
john.writeTo(output);

34. Protobuf
● Advanced android support (Retrofit, Wire, etc.)
● Obj-c/Swift/JS are also supported
● Easy to implement on server side

35. Cap’n proto
● Faster than Protobuf
● Same author
● No Obj-C/Swift support (use C++ and suffer)
● No Android specific realization (Java only)
● BETA

36. Why it worth to check
● Incremental reads
● Random access
● Mmap support
● Tiny generated code
● Tiny runtime library

37. Why you shouldn’t use it in production
● Still BETA
● Has not yet had a security review
● The Cap’n Proto programming interface may still change in ways that break
existing code
● Performance (“Currently it only beats Protobufs in realistic-ish end-to-end
benchmarks by around 2x-5x. We can do better”)

38. Cap’n proto: Schema
struct Person {
name @0 :Text = “Bob”;
birthdate @3 :Date;
email @1 :Text = “default@value.com”;
phones @2 :List(PhoneNumber);
struct PhoneNumber {
number @0 :Text;
type @1 :Type;
enum Type {
mobile @0;
home @1;
work @2;
}
}
}

39. Cap’n proto: Schema
● Types come after names
● @N annotations
● # Comments also are supported (should appear after the declaration)

40. Cap’n proto: Built-in Types
● Void: Void
● Boolean: Bool
● Integers: Int8, Int16, Int32, Int64
● Unsigned integers: UInt8, UInt16, UInt32, UInt64
● Floating-point: Float32, Float64
● Blobs: Text, Data
● Lists: List(T)

41. Cap’n proto: Unions
struct Shape {
area @0 :Float64;
union {
circle @1 :Float64; # radius
square @2 :Float64; # width
}
}

42. Cap’n proto: Unions
● Unions members are numbered in the same number space as fields
● “Void” is useful if no value is needed
● By default, when a struct is initialized, the lowest-numbered field in the union
is “set”

43. Cap’n proto: Groups
struct Person {
# ...
# Note: This is a terrible way to use groups, and meant
# only to demonstrate the syntax.
address :group {
houseNumber @8 :UInt32;
street @9 :Text;
city @10 :Text;
country @11 :Text;
}
}

44. Cap’n proto: Groups
● Behaves as one field of other structure (address :Address)
● Group is not separated object!
● Essentially, a group is just a namespace
● Useful with Unions

45. Cap’n proto: Groups
struct Shape {
area @0 :Float64;
union {
circle :group {
radius @1 :Float64;
}
rectangle :group {
width @2 :Float64;
height @3 :Float64;
}
}
}

46. Cap’n proto: Other features
● Dynamically-typed Fields
● Enums
● Interfaces (RPC)
● Generic Types
● Generic Methods (RPC)
● Constants
● Nesting, Scope, and Aliases
● Imports
● Annotations

47. Cap’n proto: Other features

48. Summary
● Don’t transfer data at all
● If you should transfer something - get only part that you need
● ...then minimize data structure of that part
● ...then compress it
● ...then optimize data parsing
● ...then cache result it and see p.1
● Use binary data!

50. Thanks!
URLs:
https://capnproto.org/
https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?
hl=en
https://developers.google.com/site-verification/v1/performance
https://developers.google.com/protocol-buffers/
Contact me:
Twitter: @AntonMinashkin
Email: anton.minashkin@outlook.com
FB: https://www.facebook.com/anton.minashkin.1