Wikipedia’s Event Data Platform, Or: JSON Is Okay Too With Andrew O...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Event Data
Platform
or, JSON is
okay too

Andrew Otto
Principal Software Engineer
at The Wikimedia Foundation
(previously an SRE)

The Plan

The Plan
What makes Wikimedia unique?

The Plan
What makes Wikimedia unique?
A typical event streaming stack.

The Plan
What makes Wikimedia unique?
Why this stack doesnʼt work for WMF.
A typical event streaming stack.

The Plan
What makes Wikimedia unique?
Why this stack doesnʼt work for WMF.
WMFʼs solution.
A typical event streaming stack.

not just
Wikipedia

What makes
unique?

Free and open source
CC-by-SA 4.0, Myleen Hollero

Data should be public!
CC-by-SA 4.0, Zack McCune
Privacy should be protected

Distributed
CC-by-SA 4.0, Myleen Hollero

Conﬂict!

A typical Kafka
event streaming
stack

event bus

Producers &
Consumers
client libraries

Producers &
Consumers
Stream
Processing
(Kafka Streams, Flink, etc.)
stream processing
framework

Kafka
REST Proxy
Producers &
Consumers
Stream
Processing
(Kafka Streams, Flink, etc.)
HTTP API

Kafka
REST Proxy
Connectors
Kafka
Connect
Producers &
Consumers
Stream
Processing
(Kafka Streams, Flink, etc.)
ingestion f...

Kafka
REST Proxy
Connectors
Kafka
Connect
Producers &
Consumers
Schema Registry
Avro
Stream
Processing
(Kafka Streams, Fli...

Kafka
REST Proxy
Connectors
Kafka
Connect
Producers &
Consumers
Schema Registry
Avro
Stream
Processing
(Kafka Streams, Fli...

Free and
open
source
“As an organization,
we strive to use open
source tools over
proprietary ones”
https://foundation.wik...

1. Free Redistribution
The license shall not restrict any party from selling
or giving away the software as a component of...

Conﬂuent Community
License

“[...] does not allow you to
provide the software as a SaaS
oﬀering [...]”
Conﬂuent Community License

“data we oﬀer for re-use
MUST use clearly
speciﬁed data schemas
and SHOULD be based on
widely used open
standards”
https:/...

“Our software architecture and
development processes
SHOULD be geared towards
building an ecosystem of
third-party re-user...

Kafka
REST Proxy
Connectors
Kafka
Connect
Producers &
Consumers
Schema Registry
Avro
Stream
Processing
(Kafka Streams, Fli...

To use Avro you need:
● An Avro implementation in your
language
Apache Avro

To use Avro you need:
● An Avro implementation in your
language
● An Avro schema
Apache Avro

Avro at rest
Binary data at rest in a ﬁle:
schema is in the ﬁle
source: https://www.clairvoyant.ai/blog/big-data-file-form...

In streams, each message is binary data.
Avro in streams

In streams, each message is binary data.
To read, you need the Avro schema.
Avro in streams

In streams, each message is binary data.
To read, you need the Avro schema.
But, how to get it?
Avro in streams

Conﬂuentʼs solution
source: https://rajcrk.medium.com/kafka-avro-and-schema-registry-introduction-f265e556da2d
Avro in str...

Conﬂuentʼs solution
Avro in streams
source: https://rajcrk.medium.com/kafka-avro-and-schema-registry-introduction-f265e556...

Conﬂuentʼs solution
Avro in streams
source: https://rajcrk.medium.com/kafka-avro-and-schema-registry-introduction-f265e556...

Conﬂuentʼs solution
1. Schemas registered in a centralized schema registry and
assigned unique integer schema ID.
2. Conﬂu...

Conﬂuentʼs solution
Avro in streams
source: https://rajcrk.medium.com/kafka-avro-and-schema-registry-introduction-f265e556...

Wikimediaʼs software development
model is distributed
Schema Registry & Avro in streams

● Open Source
● Public Data
● Distributed
Making
choices
Constraints

Kafka
REST Proxy
Connectors
Kafka
Connect
Producers &
Consumers
Schema Registry
Avro
Stream
Processing
(Kafka Streams, Fli...

Alternatives?
If not Avro,
then what?

What about
good olʼ JSON &
JSONSchema?
Alternatives?

JSON and
Event Schemas

What are schemas for?

Schemas are the data contract.
What are schemas for?
Like an API spec, but for data.

What are schemas for?
1. Data validation
2. Data integration
Schemas are needed for

$Producing - need to ensure data is valid Schemas are for: Validation { "user_id": 123, "create_dt": “2022-10-05T12:00:00Z”...$

$Automating ingestion - need to know data types Schemas are for: Integration { "user_id": 123, "create_dt": “2022-10-05T12:...$

Can we use JSONSchema?
1. Validation 2. Integration
?

JSON is (arguably) more
ubiquitous than Avro
JSON serialization

No special tooling to read JSON
serialized data
JSON serialization

$JSON is only ʻlooselyʼ schemaed JSON serialization { "name": "Bobby Tables", "age": 12, "height": 1.7, "create_dt": "2022-...$

$Used to validate JSON data JSONSchema { "name": "Bobby Tables", "age": 12, "height": 1.7, "create_dt": "2022-10-05T12:00:0...$

Can be used for automated
integration
JSONSchema
{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"age": ...

Can we use JSONSchema?
1. Validation
��
2. Integration

Avro and Conﬂuent Schema
Registry do fancy stuﬀ
Can we use JSONSchema?

Schema distribution
Conﬂuent Schema Registry + Avro features

Schema distribution
Conﬂuent Schema Registry + Avro features
Schema versioning

Schema evolution
Schema distribution
Conﬂuent Schema Registry + Avro features
Schema versioning

JSONSchema
distribution

Schema URIs
JSONSchemas
have ʻmeta
schemasʼ

Schema URIs
schema of this schema
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": ...

Schema URIs
{
"$schema": "http://schema.company.org/entity/user.json",
"name": "Bobby Tables",
"age": 12,
"height": 1.7
}
...

Schema URIs
fully qualiﬁed URL in $schema
{
"$schema": "http://schema.company.org/entity/user.json",
"name": "Bobby Tables...

Schema URIs
{
"$schema": "http://schema.company.org/entity/user.json",
"name": "Bobby Tables",
"age": 12,
"height": 1.7
}
...

Schema Repositories
Repository > Registry

Git is great at
decentralization.
Schema Repositories

Schemas in git
Schema Repositories

Schema Repositories
{
"$schema": "/entity/user.json",
"name": "Bobby Tables",
"age": 12,
"height": 1.7
}
Only path part of...

Schema Repositories
{
"$schema": "/entity/user.json",
"name": "Bobby Tables",
"age": 12,
"height": 1.7
}
This isnʼt enough...

Schema Repositories
Preﬁx $schema with a base address
"file:///local/path/to/schema_repository" + $schema
"https://schema....

JSONSchema
versioning

Versioned Schemas
Code changes

Versioned Schemas
so does data
Code changes

Versioned Schemas
Donʼt use git for versioning of
schemas

Versioned Schemas
/entity/user/1.1.0.json

Versioned Schemas
title keyword is used to associate
diﬀerent schema versions
/entity/user/1.1.0.json
{
"title": "entity/u...

Versioned Schemas
A schemaʼs unique identity is declared
using $id keyword
/entity/user/1.1.0.json
{
"$id": "/entity/user/...

Versioned Schemas
Event records set
$schema to match
their schemaʼs $id
/entity/user/1.1.0.json
{
"$id": "/entity/user/1.1...

JSONSchema
evolution

JSONSchema Evolution
source: Paul Downey - https://www.flickr.com/photos/psd/3352842349

JSONSchema compatibility rule
Only allowed to add new
optional ﬁelds

● No ﬁeld type changes
JSONSchema compatibility rule

● No ﬁeld type changes
● No ﬁeld deletions
JSONSchema compatibility rule

● No field type changes
● No field deletions
● No field renames
JSONSchema compatibility rule

JSONSchema versioning and evolution
/entity/user/1.0.0.json
/entity/user/1.1.0.json
/entity/user/1.1.1.json
/entity/user/1...

jsonschema-tools

jsonschema-tools
/entity/user/current.json
a single ﬁle for developers to edit

jsonschema-tools
/entity/user/current.json
/entity/user/1.1.0.json
{
"$id": "/entity/user/1.1.0",
"title": "entity/user",
...

jsonschema-tools
/entity/user/current.json
/entity/user/1.0.0.json
/entity/user/1.1.0.json
/entity/user/1.1.1.json
…

jsonschema-tools
tests
entity/user
Major Version 1
✓ 1.1.0 must be compatible with 1.0.0
✓ 1.1.1 must be compatible with 1...

Summary

Summary
JSONSchema distribution

Summary
JSONSchema distribution
/entity/user/1.1.1.json
JSONSchema versioning

Summary
JSONSchema distribution
JSONSchema evolution
managed with jsonschema-tools
/entity/user/1.1.1.json
JSONSchema vers...

Summary
JSONSchema for validation

Summary
JSONSchema for validation
JSONSchema for integration

Summary
PROS CONS
JSON is easy to use Ineﬃcient text format (but compresses well)
JSONSchema is common outside of ʻbig dat...

● Event Platform blog post series
● Event Platform Docs
● schema.wikimedia.org- event schemas service and UI
● stream.wiki...

Questions?
- Andrew Otto
otto@wikimedia.org