In this talk, you’ll learn how we’re taking Comcast’s Technology and Product group’s massive, heterogeneous set of data collection systems and centralizing on a single platform built around Kafka. These data collection systems are used for everything from business analytics, to near-real time operations, to executive reporting. We’ll go over what it takes to wrangle streaming data across an enterprise, including the need for, and our approaches to: Schema management, both at schema creation time and when schema evolution is required Data ingest and cleansing Multi-datacenter collection and failover How we use the same data stream for many different purposes, across many different teams

1. Towards a Stream-Centered Enterprise
Kafka Summit 2016
Gabriel Commeau

2. Stream Data Platform Overview

3. Data Ingest

4. Multi-Datacenter Deployment

5. o Avro versus json, protobuf, xml, thrift
o Some guidelines for Avro schema:
• Document it all
• Use explicit names
• Provide default value
• Avoid complex union type
• If possible avoid byte arrays
o Use:
• Common core schemas
• Proper nesting
• Enum + String
Schema Management

6. Avro Schema Evolution
Enum
• Reader can add value
• Writer can remove value
Don’t
• Change namespace
• Change name
• Unless default value
• Change type
Do
• Use default value
• Use union type
containing null
Union Type
• Change type order ok
• Reader can add type
• Writer can remove type
Compatibility
Ok
• Change order
• Change, add, remove,
reorder alias