1.
1
How Kroger Embraced a
“Schema First”
Philosophy in Building
Real-time Data Pipelines

2.
Rob Hammonds Rob Hoeting Lauren McDonald

3.
460,000
Associates
Company-Wide
$121.2
Billion
2018 Total Sales
2,764
Supermarkets &
Multi-Department
Stores
Serving
Customers in
35
States and The
District of Columbia

4.
We are evolving!

5.
This is just a data
swamp.
My report just broke!
How do I use the data?
Where do I find the data?
I want this new business
feature by Friday.
Let's use AI and streaming
analytics to solve all our
problems!
I’m spending too
much time spraying
data to everyone!
I had to roll back my
prod release because
my data change broke
someone!

6.
Event Streaming Platform
Tenets
Thou shalt democratize data
Thou shalt model business processes
Thou shalt have a high developer
experience

7.
Avro to the
rescue!

8.
Schema-First
Development
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "StoreID",
"type": "int"
},
{
"name": "Location",
"type": "string"
}
]
}
StoreCreated.avsc StoreCreated.newBuilder()
.setStoreName(“Kroger-Cincinnati”)
.setStoreID(“513”)
.setLocation(“1014 Vine Street”)
.build()
Serialization
Framework

9.
Ugh…How do I makes sense out of all this?

10.
Let’s make the events
composable!
Address: {
Street, City, State, ZipCode
}
Person: {
firstName,
lastName,
address: Address,
}
Role: {
reportsTo
title
}
EmployeeHiredEvent {
employee: Person
role: Role
startDate: date
salary: float
}

11.
Class Generation and
Publishing
Register the Schema
Compatibility Checking

12.
Bootstrap all the things!
#LazyDevelopersAreTheBestDevelopers

13.
My pipeline
isn’t
registering the
schemas
I register
schemas but
my jar isn’t
being
published
I’m failing
compatibility
but I HAVE to
change this
field name
The bootstrap
script isn’t
working for
me!

14.
Producer
v.1
Producer
v.2
Avro Files
v.1
Avro Files
v.2
Consumer
v.1
Consumer
v.2
Schema v.1 Schema v.2
Full Compatibility
Backward
Compatibility
Forward
Compatibility
Schema Evolution

15.
Schema Editor
Event Schema Lifecycle
V1
V2
V2
TIME
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "SID",
"type": "int"
}
]
}
StoreCreated.avsc – v1
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "SID",
"type": "int"
},
{
"name": "Location",
"type": "string",
"default": "Cincinnati"
}
]
}
StoreCreated.avsc – v2
PRODUCTION
Schema Registry
Full Compatibility
STAGE
Schema Registry
Full Compatibility
DEVELOPMENT
Schema Registry
Full Compatibility

16.
Schema Editor
Event Schema Lifecycle
V1
V2
V2
TIME
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "SID",
"type": "int"
}
]
}
StoreCreated.avsc – v1
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "SID",
"type": "int"
},
{
"name": "Location",
"type": "string",
"default": "Cincinnati"
}
]
}
StoreCreated.avsc – v2
V3
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "StoreID",
"type": "int"
},
{
"name": "Location",
"type": "string",
"default": "Cincinnati"
}
]
}
StoreCreated.avsc – v3
Changing the name of an
attribute breaks compatibility!
PRODUCTION
Schema Registry
Full Compatibility
STAGE
Schema Registry
Full Compatibility
DEVELOPMENT
Schema Registry
Full Compatibility

17.
I accidently named a
field wrong. Seems
rather harsh, don’t you
think?
I’m blocked.
Hey Robs and Lauren,
can you delete this
schema and clear my
topic?

18.
Schema Editor
Event Schema Lifecycle
V1
V2
TIME
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "StoreID",
"type": "int"
},
{
"name": "Location",
"type": ”string"
}
]
}
V2 V2 V3
StoreCreated.avsc – v2
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [
{
"name": "StoreName",
"type": "string"
},
{
"name": "StoreID",
"type": "int"
},
{
"name": "Location",
"type": "com.kroger.commons.Location"
}
]
}
StoreCreated.avsc – v3
You can’t change the type of an
attribute!
PRODUCTION
Schema Registry
Full Compatibility
STAGE
Schema Registry
Full Compatibility
DEVELOPMENT
Schema Registry
Full Compatibility

19.
How do we protect clients
in production, but improve
the DevX of schema
development?

20.
Schema Editor
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [{
"name": "StoreName",
"type": "string"
}, {
"name": "StoreID",
"type": "int"
},{
"name": "Location",
"type": ”com.kroger.commons.Location"
} }
StoreCreated.avsc – v2.0.0
TIME
Major Version Compatibility (MVC)
2.0.0
PRODUCTION
Schema Registry
Full Transitive
Compatibility
STAGE
Schema Registry
No Compatibility
DEVELOPMENT
Schema Registry
No Compatibility
2.0.0
2.0.0
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [{
"name": "StoreName",
"type": "string"
}, {
"name": "StoreID",
"type": "int"
},{
"name": "Location",
"type": "com.kroger.commons.Location"
},{
"name": "StoreManager",
"type": "com.kroger.commons.Person",
"doc": "Manager of the store"
} }
StoreCreated.avsc – v2.0.1
2.0.1
When adding an attribute, make
it nullable and add a default!

21.
Schema Editor
TIME
2.0.0
2.0.1
2.1.0
2.0.0
2.0.0
2.1.0
Major Version Compatibility (MVC)
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [{
"name": "StoreName",
"type": "string"
}, {
"name": "StoreID",
"type": "int"
},{
"name": "Location",
"type": "com.kroger.commons.Location"
},{
"name": "StoreManager",
"type": ["com.kroger.commons.Person",
"null"],
"default": "null",
"doc": "Manager of the store"
} }
StoreCreated.avsc – v2.0.1
PRODUCTION
Schema Registry
Full Transitive
Compatibility
STAGE
Schema Registry
No Compatibility
DEVELOPMENT
Schema Registry
No Compatibility

22.
Schema Editor
TIME
2.0.0
2.0.1
2.1.0
2.0.0
2.0.0
2.1.0 2.1.1
Major Version Compatibility (MVC)
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [{
"name": "Store",
"type": "string"
}, {
"name": "StoreID",
"type": "int"
},{
"name": "Location",
"type": ”com.kroger.commons.Location"
},{
"name": "StoreManager",
"type": ["com.kroger.commons.Person",
"null"],
"default": "null",
"doc": "Manager of the store"
} }
StoreCreated.avsc – v2.1.1
You can’t change the name of an
attribute, but you can add an alias!
PRODUCTION
Schema Registry
Full Transitive
Compatibility
STAGE
Schema Registry
No Compatibility
DEVELOPMENT
Schema Registry
No Compatibility

23.
Schema Editor
TIME
2.0.0
2.0.1
2.1.0
2.0.0
2.0.0
2.1.0
Major Version Compatibility (MVC)
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [{
"name": "StoreName",
"alias" : "Store",
"type": "string"
}, {
"name": "StoreID",
"type": "int"
},{
"name": "Location",
"type": " com.kroger.commons.Location"
},{
"name": "StoreManager",
"type": ["com.kroger.commons.Person",
"null"],
"default": "null",
"doc": "Manager of the store”
} }
StoreCreated.avsc – v2.1.1
2.1.1
PRODUCTION
Schema Registry
Full Transitive
Compatibility
STAGE
Schema Registry
No Compatibility
DEVELOPMENT
Schema Registry
No Compatibility

24.
Schema Editor
TIME
2.0.0
2.0.1
2.1.0
2.0.0
2.0.0
2.1.0
Major Version Compatibility (MVC)
{
"type": "record",
"name": "Store",
"doc": "This is an avro schema for a store",
"namespace": "com.kroger",
"fields": [{
"name": "StoreName",
"alias": "Store",
"type": "string"
}, {
"name": "StoreID",
"type": "int"
},{
"name": "Location",
"type": "com.kroger.commons.Location"
},{
"name": "StoreManager",
"type": ["com.kroger.commons.Person",
"null"],
"default": "null",
"doc": "Manager of the store"
} }
StoreCreated.avsc – v2.1.1
2.1.1
PRODUCTION
Schema Registry
Full Transitive
Compatibility
STAGE
Schema Registry
No Compatibility
DEVELOPMENT
Schema Registry
No Compatibility

25.
Lets automate all
the things…
#lazydevelopersarethebestdevelopers

26.
What are we improving again?
Cross-platform Schema Development
Schema Composition
Schema Registration
JAR building
Schema Lifecycle

27.
Development
Schema Registry
Stage
Schema Registry
Production
Schema Registry
ELMR
Global Schema and
Metadata Store
ELMR
Artifactory
JAR files
CI/CD Build Server
Java Plugin
ELMR – Event Lifecycle Management Repository

28.
Event Streaming Becomes a Thing

29.
The Business

30.
No problem, I just sent
you a 500 line Avro
schema file which has
everything you need.
Can you help me
understand all the
fields in those
inventory events?

31.
How can these
people live like this?

32.
How we
improved data
discovery...
#LazyDevelopersAreTheBestDevelopers

33.
ELMR
UI

34.
Development
Schema Registry
Stage
Schema Registry
Production
Schema Registry
ELMR
Global Schema and
Metadata Store
ELMR
Artifactory
JAR files
CI/CD Build Server
Java Plugin
ELMR-UI

35.
Artifact
Discovery

36.
Development
Schema Registry
Stage
Schema Registry
Production
Schema Registry
ELMR
Global Schema and
Metadata Store
ELMR
Artifactory
JAR files
CI/CD Build Server
Java Plugin
ELMR-UI

37.
Event Discovery & Socialization
Event Lifecycle Management
CI/CD Automation
Avro Standard
Events Streaming
A Review of our Journey

38.
What’s next?

39.
Future Ideas
INTUITIVE SCHEMA
EDITING
GLOBAL STRUCTURES
AND FIELDS
EXTENDED ATTRIBUTES
AND VALIDATION

40.
Thank You!!!
@RHammonds1 @RobHoeting @Lew181818