The document summarizes a presentation about using Protocol Buffers and Redis together. It discusses how Protocol Buffers provide strict data types, versioning, and serialization/deserialization benefits. It then outlines Redis key patterns using namespaces, versions, data categories and identifiers. Examples are provided to show how Protocol Buffers messages can be stored in Redis using these key patterns, including storing connections data in sets, sorted sets and individual messages. Benefits discussed include structure, readability, testing and abstraction.

1. PRESENTED
BY
The Happy Marriage of Redis and Protobuf
A true story about Communication, Keys and Give and Take
Scott Haines - @newfront
Principal Software Engineer, Twilio

2. PRESENTED
BY
1 Protocol Buffers
What are protocol buffers? Why I love them and why you should too.
2 Redis Key Patterns
How your redis key space can be modeled to add rules and structure.
3 Redis + Protobuf
Together forever or at least for now! Use cases and mental model
Agenda:

3. PRESENTED
BY
Why use them?
• Protocol Buffers give you strict
types and compile time
guarantees
• Versioning your API / Pipeline /
Data can be done by sticking to a
release version of your schema
• Rely on *Maven Release for your
versioning
• Interoperable with most major
programming languages
(java/scala/c++/go/obj-c/node-
js/python and more)
Protocol Buffers

4. PRESENTED
BY
Why use them?
• Save time with Automatically
generated builder classes. (Just
compile the .proto)
• Optimized Message format with
lighting fast Serialization /
Deserialization (SerDe)
Protocol Buffers
@newfront

5. PRESENTED
BY
Why use them?
• *Backwards and Forwards
compatible. Cause that is the way
things should be.
• Supports new schema members
through the use of unknown
fields rather than throwing
Exceptions when parsing.
• We use for our Data Pipelines at
Twilio
Protocol Buffers
@newfront

6. PRESENTED
BY
Why use them?
• Speaks JSON.
• Use protobuf inside applications,
still expose JSON endpoints or
move to GRPC and use end-to-end
protobuf.
Protocol Buffers
@newfront

7. PRESENTED
BY
Why use them?
• Speaks JSON.
• Use protobuf inside applications,
still expose JSON endpoints or
move to GRPC and use end-to-end
protobuf.
Protocol Buffers
@newfront

8. PRESENTED
BY
Redis Key Patterns
@newfront
{namespace}.{version}.{data-
category}.{identifier}

9. PRESENTED
BY
Redis Key Patterns
@newfront
Namespace
• Stores the common root identifier
for a given key space collection
• Can be used to control key space
access across multi-tenant
systems with a little server-side
wizardry
• Ensures key space collisions don’t
occur
{namespace}.{version}.{data-
category}.{identifier}

10. PRESENTED
BY
Redis Key Patterns
@newfront
Version
• Things change.
• Use key versioning the same way
you would version an API
• Maintains Data Access
compatibility for the data stored
within each Key
{namespace}.{version}.{data-
category}.{identifier}

11. PRESENTED
BY
Redis Key Patterns
@newfront
DataType / Category
• Denotes logically the data /
collection of data stored within
the key or collection
• Using this like a SQL table or API
route helps form the mental
model
• Use singular or plurals to denote
the intended usage pattern. Eg:
connection or connections, users
or user.
• Plural keys back collections of
pointers
{namespace}.{version}.{data-
category}.{identifier}

12. PRESENTED
BY
Redis Key Patterns
@newfront
Identifier
• Use to denote a subset of data in
a collection by unique id
• Or a specific key by unique id.
Example
vi.v1.connections.us1 which we will
see more about in just a sec.
{namespace}.{version}.{data-
category}.{identifier}

13. PRESENTED
BY
@newfront

14. PRESENTED
BY
Redis + Protobuf
@newfront
• Accounts connect to Twilio to
perform actions like Making a Call
or Sending a Text Message.
• Assume a Connection is one of
many events but that all events
begin with a connection.

15. PRESENTED
BY
Redis + Protobuf
@newfront
• The root key here is used like a
traditional list API endpoint to
view connection regions.
• Use a simple set Redis data
structure
• Acts like /v1/connections if you
fronted this with an API

16. PRESENTED
BY
Redis + Protobuf
@newfront
• Using the root key and one of the
set members (us1) gives us a key
pointer.
• Using the prior key
(vi.v1.connections) + (us1) points
to new data structure.

17. PRESENTED
BY
Redis + Protobuf
@newfront
• Connections have temporal
information so use a sorted set
(or *redis stream)
• Using the timestamp uuid and
connection zone from the
Connection protobuf to add
set members to the sorted set.

18. PRESENTED
BY
Redis + Protobuf
@newfront
• Set members are the unique
identifiers of each connection,
but not the physical binary
data.

19. PRESENTED
BY
Redis + Protobuf
@newfront
• Now store the actual binary
data using the singular
connection root key +
identifier
• Stores the original Connection
protobuf as binary serialized
data
• Equivalent API route would be
/v1/connection/{uuid}
• Keeping track of prefix, version
and category can get tricky…

20. PRESENTED
BY
Redis + Protobuf
@newfront
• Introducing the RedisKey
pattern.
• Adds structure and enforces
convention.
• Can be used to generate a
binary or non-binary key
space.
• *Extend for GDPR friendly
rules like added expiry or
access controls
Protobuf Message

21. PRESENTED
BY
Redis + Protobuf
@newfront
• Simplify generation of keys for
your Redis data
• Supports versioning of your
key space or extension for
generators for different
Namespaces.

22. PRESENTED
BY
Redis + Protobuf
@newfront
• Store DataType to Protobuf
mappings
• Mix with RedisKeys to make
powerful Redis interaction
layers

23. PRESENTED
BY
Redis + Protobuf
@newfront
• Build abstractions using
RedisKeys and
TypeRegistry
• Abstract away the Redis
connection and sharding or
connecting to different
physical clusters

24. PRESENTED
BY
Redis + Protobuf
@newfront
• Simplify your unit tests.
• Share common libraries
and modules across *all of
your projects
• Reduces the time to
production for new
projects

25. PRESENTED
BY
Redis + Protobuf
@newfront
• Readability in your code
goes way up.
• Access patterns in the code
make sense

26. PRESENTED
BY
Redis + Protobuf
@newfront
• Extend RedisSource to
support pagination, bulk
fetch (mget), bulk add
(mset), you name it.
• Bring your favorite redis
library to the table (*I use
Jedis and just learned
about redis4cats)
• Extend this pattern to read
or write from RedisStreams
(XADD, *see different talks
for more info or read docs)

27. PRESENTED
BY
Redis + Protobuf
@newfront
• Abstract away only what
you want.
• Pooled Resources can leak,
patch them up.

28. PRESENTED
BY
Redis + Protobuf
@newfront
• Use what makes sense for
your project.
• Wrap more complex
components and still
enable engineering
freedom and creativity

29. Thank You!