1.
EVOLVING YOUR ANALYTICS
STACK WITH YOUR BUSINESS
SNOWPLOW - LONDON MEETUP #4

2.
SNOWPLOW - LONDON MEETUP #4
BUSINESSES ARE CONSTANTLY EVOLVING…
▸ Your products (apps & platforms) change
▸ Your questions should change too
▸ It’s critical that the analytics stack can evolve with your
business

3.
SNOWPLOW - LONDON MEETUP #4
SELF-DESCRIBING DATA EVENT DATA MODELING+
EVOLVING EVENT DATA PIPELINE
HOW?

4.
SELF-DESCRIBING DATA
PART 1

5.
SNOWPLOW - LONDON MEETUP #4
NO TWO COMPANIES ARE ALIKE

6.
SNOWPLOW - LONDON MEETUP #4
DEFINE YOUR OWN EVENTS AND ENTITIES
Events
Entities
‣ Build castle
‣ Form alliance
‣ Declare war
‣ Player
‣ Game
‣ Level
‣ Castle
‣ View product
‣ Buy product
‣ Deliver product
‣ Product
‣ Customer
‣ Basket
‣ Vehicle

7.
"description": "Schema for a fighter context",
"vendor": "com.ufc",
"name": “fighter",
"version": “1-0-2“,
"properties": {
"FirstName": {"type": "string"},
"LastName": {"type": "string"},
"Nickname": {"type": "string"},
"FacebookProfile": {"type": "string"},
"WeightLbs": {"type": ["integer", "null"]},
"Record": {"type": “string", "pattern": "^[0-9]+-[0-9]+-[0-9]+$"}
}
}
SNOWPLOW - LONDON MEETUP #4
YOU THEN DEFINE A SCHEMA FOR EACH EVENT AND ENTITY
I DON’T DO EVENTS
THAT AREN’T SCHEMA’ED

8.
SNOWPLOW - LONDON MEETUP #4
YOU THEN DEFINE A SCHEMA FOR EACH EVENT AND ENTITY
"schema": "iglu:ufc/fighter/jsonschema/1-0-2",
"data": {
"FirstName": “Daniel”
"LastName": “Cormier”,
"Nickname": “DC”,
"FacebookProfile": “Daniel-Cormier”,
"TwitterName": “dc_mma”,
"WeightLbs": 205
}
}

9.
SNOWPLOW - LONDON MEETUP #4
THE SCHEMAS CAN THEN BE USED IN A NUMBER OF WAYS
▸ Validate the data (important for data quality)
▸ Load the data into tidy tables in your data warehouse
▸ Make it easy / safe to write downstream data processing
application (e.g. for real-time users)

10.
EVENT DATA MODELING
PART 2

11.
SNOWPLOW - LONDON MEETUP #4
WHAT IS EVENT DATA MODELING?
▸ Event data modeling is the process of using business logic
to aggregate over event-level data to produce 'modeled'
data that is simpler for querying.

12.
SNOWPLOW - LONDON MEETUP #4
MODELED VS UNMODELED DATA
event 1
event n
…
Users
Sessions
…
Funnels
IMMUTABLE.
UNOPINIATED. HARD TO CONSUME. NOT
MUTABLE
AND OPINIONATED. EASY TO CONSUME.

13.
SNOWPLOW - LONDON MEETUP #4
IN GENERAL, EVENT DATA MODELING IS PERFORMED ON THE COMPLETE EVENT STREAM
▸ Late arriving events can change the way you understand
earlier arriving events
▸ If we change our data models: this gives us the flexibility
to recompute historical data based on the new model

14.
EVOLVING THE DATA PIPELINE
PART 3

15.
SNOWPLOW - LONDON MEETUP #4
HOW DO WE HANDLE PIPELINE EVOLUTION?
▸ Businesses change over time
▸ The events that occur are going to change
▸ Use of the data will change
▸ Insight -> more questions -> more insight -> more
questions
▸ Two types of evolution: push and pull
BUSINESSES ARE NOT STATIC, SO EVENT PIPELINES SHOULD NOT BE EITHER

16.
SNOWPLOW - LONDON MEETUP #4
PUSH EXAMPLE:
▸ If data is self-describing it is easy to add an additional
sources
▸ Self-describing data is good for managing bad data and
pipeline evolution
I’M
AN EMAIL SEND
EVENT AND I HAVE
INFORMATION ABOUT THE
RECIPIENT (EMAIL

17.
SNOWPLOW - LONDON MEETUP #4
ANSWERING THE QUESTION:
1. EXISTING DATA MODEL
SUPPORTS ANSWER
2. NEED TO UPDATE DATA
MODEL
3. NEED TO UPDATE DATA
MODEL AND DATA COLLECTION

18.
SNOWPLOW - LONDON MEETUP #4
SELF-DESCRIBING DATA AND THE ABILITY TO RECOMPUTE DATA MODELS ARE ESSENTIAL TO ENABLE PIPELINE EVOLUTION
SELF-DESCRIBING DATA RECOMPUTE DATA MODELS ON ENTIRE DATA SET
‣ Updating existing events and entities in a
backward compatible way e.g. add
optional new fields
‣ Update existing events and entities in a
backwards incompatible way e.g. change
field types, remove fields, add
compulsory fields
‣ Add new event and entity types
‣ Add new columns to existing derived
tables e.g. add new audience
segmentation
‣ Change the way existing derived tables
are generated e.g. change
sessionization logic
‣ Create new derived tables

19.
QUESTIONS?
SNOWPLOW - LONDON MEETUP #4