Spark Summit East 2016, New York City: Talk by Josef Adersberger (@adersberger, CTO at QAware). Abstract: Users leave thousands of traces per second on a successful ecommerce site. It’s very pragmatic to analyse and react on this trace event stream in realtime. This is called clickstream analysis. In the talk I’ll present a software architecture based on Apache Spark which is able to process thousands of clickstream events per second. A product based on this architecture is in production since mid 2015 and is still performing well. The building blocks of the architecture beside Spark are Kafka to handle the inbound event stream, Spark Streaming for initial stream processing and Parquet as serialization format. I argue why we’ve chosen these technologies and what experiences we had in developing, launching and operating the product.

1. CLICKSTREAM ANALYSIS WITH
SPARK – UNDERSTANDING
VISITORS IN REAL-TIME
Dr. Josef Adersberger
QAware GmbH, Germany

2. THE CHALLENGE

3. One Kettle to Rule ‘em All
Web Tracking Ad Tracking
ERP CRM

4. One Kettle to Rule ‘em All
Retention Reach
Monetarization
steer …
 Campaigns
 Offers
 Contents

6. THE CONCEPTS
by Randy Paulino

7. The First Sketch
(= real-time)
Tableau

8. User Journey Analysis
C V VT VT VT C X
C V
V V V V V V V
C V V C V V V
VT VT V V V VT C
V X
Event stream: User journeys:
Web / Ad tracking
KPIs:
 Unique users
 Conversions
 Ad costs / conversion value
 …
V
X
VT
C Click
View
View Time
Conversion

9. THE ARCHITECTURE

10. „Larry & Friends“ Architecture
Collector Aggregation
SQL DB
Runs not well for more
than 1 TB data in terms of
ingestion speed, query time
and optimization efforts

11. by adweek.com
Nope.
Sorry, no Big Data.

12. „Hadoop & Friends“ Architecture
Collector
Batch Processor
[Hadoop]
Collector Event Data Lake Batch Processor Analytics DB
JSON
Stream
Aggregation
takes too long
Cumbersome
programming model
(can be solved with
pig, cascading et al.)
Not
interactive
enough

13. Nope.
Too sluggish.

14. κ-Architecture
Collector Stream Processor
Analytics DB
Persistence
JSON
Stream
Cumbersome
programming model
Over-engineered: We only need
15min real-time ;-)
Stateful aggregations (unique x,
conversions) require a separate DB
with high throughput and fast
aggregations & lookups.

15. λ-Architecture
Collector
Event Processor
Event Data Lake Batch Processor
Analytics DB
Speed Layer
Batch Layer
JSON
Stream
Cumbersome
programming model
Complex
architecture
Redundant
logic

16. Feels Over-Engineered…
http://www.brainlazy.com/article/random-nonsense/over-engineered

17. The Final Architecture*
*) Maybe called μ-architecture one day ;-)

18. Functional Architecture
Strange Events
IngestionRaw Event
Stream
Collection Events Processing Analytics
Warehouse
Fact
Entries
Atomic Event
Frames
Data Lake
Master Data Integration
 Buffers load peeks
 Ensures message
delivery (fire & forget
for client)
 Create user journeys and
unique user sets
 Enrich dimensions
 Aggregate events to KPIs
 Ability to replay for schema
evolution
 The representation of truth
 Multidimensional data
model
 Interactive queries for
actions in realtime and
data exploration
 Eternal memory for all
events (even strange
ones)
 One schema per event
type. Time partitioned.
class Analytics Model
«fact»
WebFact
«dimension»
Zeit
«dimension»
Kampagne
Jahr
Quartal
Monat
Woche
Tag
Stunde
Minute
Kunde
+ Land: String
Partner
«dimension»
Tracking
Tracking Group
SensorTag
+ Typ: SensorTagType
Platzierung
+ Format: ImageSize
+ Kostenmodell: KostenmodellArt
Werbemittel
+ AdGroup: String
+ Format: ImageSize
+ Größe: KiloBytes
+ LandingPage: URL
+ Motif: URL
Kampagne
«dimension»
Client
Kategorie
Dev ice
+ Bezeichner: String
+ Hersteller: String
+ Typ: String
Browser
+ Typ: String
+ Version: int
«dimension»
Ausspielort
LandRegion
Stadt
«dimension»
Kanal
Kanal
«dimension»
Vermarktung
«enumeration»
SensorTagType
ORDER_TAG
MASTER_TAG
CUSTOM_TAG
Betriebssystem
+ Typ: String
+ Version: Version
⦁ Dimension: Unabhängiges Prädikat auf Metriken bei der Analyse ("kann isoliert darüber nachdenken / isoliert dazu Analysen
fahren")
⦁ Hierarchie: Sub-Prädikat auf Metriken. Erzeugt mehr als eine (zueinander diskunkte) Teilmengen der Metriken. Entspricht den gängigen
Drill-Down-Pfaden in den Reports bzw. den Batch-Aggregate-Up-Pfaden in der Aggregationslogik. Semantische Unterstrukturen: "ist Teil
von & kann nicht existieren ohne".
⦁ Asssoziation: Nicht verwendet. Separates Stammdatenmodell.
⦁ Attribut: Ermöglicht eine weitere (querschneidende) Einschränkung der Metrikmenge ergänzend zu den Hierarchien.
Domain
Website
Tracking Site
Vermarkter
Auslieferungs-
Domain
Referral
«enumerati...
KostenmodellArt
CPC
CPM
CPO
CPA
«abstract»
DimensionValue
+ id: int
+ name: String
+ sourceId: String
WebsiteFact
+ Bounces: int
+ Verweildauer: float
+ Visits: int
BasicAdFact
+ Clicks: int
+ Sichtbare Views: int
+ Validierte Clicks: int
+ View (angefragt): int
+ View (ausgeliefert): int
+ View (gemessen): int
«dimension»
Produkt
Shop
Produkt
+ Produktkategorie: String
«dimension»
Zeitfenster
Letzte X Tage
«dimension»
User
User Segment
«dimension»
Order
OrderStatus
+ Status: OrderStatus
«enumeration»
OrderStatus
IN_BEARBEITUNG
ERFOLGREICH (AKTIVIERT)
ABGELEHNT
NICHT_IN_BEARBEITUNG
UniquesFact
+ Unique Clicks: int
+ Unique Users: int
+ Unique Views: int
AdCostFact
+ CPC: int
+ Kosten: float
Conv ersionFact
+ PC: int
+ PR: int
+ PV: int
+ Umsatz PC: float
+ Umsatz PR: float
+ Umsatz PV: float
AdVisibilityFact
+ Sichtbarkeitsdauer: float
Activ atedOrderFact
+ Orders: int
+ Umsatz: float
TrackingFact
+ Orders: int
+ Page Impressions: int
+ Umsatz: float
X = {7, 14, 28, 30}
 Fault tolerant message handling
 Event handling: Apply schema, time-partitioning, De-dup, sanity
checks, pre-aggregation, filtering, fraud detection
 Tolerates delayed events
 High throughput, moderate latency (~ 1min)

19. Series Connection of Streaming
and Batching - all based on Spark.
IngestionRaw Event
Stream
Collection Event Data Lake Processing Analytics
Warehouse
Fact
Entries
SQL Interface
Atomic Event
Frames
 Cool programming model
 Uniform dev&ops
 Simple solution
 High compression ratio due to
column-oriented storage
 High scan speed
 Cool programming model
 Uniform dev&ops
 High performance
 Interface to R out-of-the-box
 Useful libs: MLlib, GraphX, NLP, …
 Good connectivity (JDBC,
ODBC, …)
 Interactive queries
 Uniform ops
 Can easily be replaced
due to Hive Metastore
 Obvious choice for
cloud-scale messaging
 Way the best throughput
and scalability of all
evaluated alternatives

20. LESSONS LEARNED
by: http://hochmeister-alpin.at

21. Technology Mapping
https://github.com/qaware/big-data-landscape
User Interface
Data Lake
Data Warehouse
Ingestion
Processing
Data Science Interactive Analysis Reporting & Dashboards
Data Sources
Analytics
Micro Analytics Services instead of reporting
servers.
Charting Libraries:Dashboards:
Analytics Frontends
Algorithm Libraries
Structured Data Lake: The eternal memory.
Efficient data serialization formats:
 Integated compression
 Column-oriented storage
 Predicate pushdown
Distributed Filesystem or NoSQL DB
Data Workflows ETL Jobs Massive
Parallelization
Pig Open Studio
Data Logicstics Stream
Processing
NewSQL: SQL meets NoSQL.
Polyglott Persistence
Index Machines: Fast aggregation and search.
In-Memory Databases: Fast access.
Time Series Databases
Atlas

22. Polyglott Analytics
Data Lake
Analytics
Warehouse
SQL
lane
R
lane
Timeseries
lane
Reporting Data Exploration
Data Science

23. Micro Analytics Services
Microservice
Dashboard
Microservice …

24. No Retention Paranoia
Data Lake
Analytics
Warehouse
 Eternal memory
 Close to raw events
 Allows replays and refills
into warehouse
Aggressive forgetting with clearly defined
retention policy per aggregation level like:
 15min:30d
 1h:4m
 …
Events
Strange Events

25. Continuous Tuning
IngestionRaw Event
Stream
Collection Event Data Lake Processing Analytics
Warehouse
Fact
Entries
SQL Interface
Atomic Event
Frames
Load
Generator Throughput & latency probes

26. In Numbers
Overall dev effort until the first release: 250 person days
Dimensions: 10 KPIs: 26
Integrated 3rd party systems: 7
Inbound data volume per day: 80GB
New data in DWH per day: 2GB
Total price of cheapest cluster which is able to handle production load:

28. THANK YOU.
@adersberger
josef.adersberger@qaware.de

29. Bonus Topic: Roadmap
SparkKafka HDFS
IaaS
Simplify Ops with Mesos
Faster aggregation & easier updates
with Spark-on-Solr
http://qaware.blogspot.de/2015/06/solr-with-sparks-or-how-to-submit-spark.html

30. Bonus Topic: Smart Aggregation
Ingestion Event Data Lake Processing Analytics
Warehouse
Fact
Entries
Analytics
Atomic Event
Frames
1 2
3

31. Architecture follows requirements
class Analytics Model
«fact»
WebFact
«dimension»
Zeit
«dimension»
Kampagne
Jahr
Quartal
Monat
Woche
Tag
Stunde
Minute
Kunde
+ Land: String
Partner
«dimension»
Tracking
Tracking Group
SensorTag
+ Typ: SensorTagType
Platzierung
+ Format: ImageSize
+ Kostenmodell: KostenmodellArt
Werbemittel
+ AdGroup: String
+ Format: ImageSize
+ Größe: KiloBytes
+ LandingPage: URL
+ Motif: URL
Kampagne
«dimension»
Client
Kategorie
Device
+ Bezeichner: String
+ Hersteller: String
+ Typ: String
Browser
+ Typ: String
+ Version: int
«dimension»
Ausspielort
LandRegion
Stadt
«dimension»
Kanal
Kanal
«dimension»
Vermarktung
«enumeration»
SensorTagType
ORDER_TAG
MASTER_TAG
CUSTOM_TAG
Betriebssystem
+ Typ: String
+ Version: Version
⦁ Dimension: Unabhängiges Prädikat auf Metriken bei der Analyse ("kann isoliert darüber nachdenken / isoliert dazu Analysen
fahren")
⦁ Hierarchie: Sub-Prädikat auf Metriken. Erzeugt mehr als eine (zueinander diskunkte) Teilmengen der Metriken. Entspricht den gängigen
Drill-Down-Pfaden in den Reports bzw. den Batch-Aggregate-Up-Pfaden in der Aggregationslogik. Semantische Unterstrukturen: "ist Teil
von & kann nicht existieren ohne".
⦁ Asssoziation: Nicht verwendet. Separates Stammdatenmodell.
⦁ Attribut: Ermöglicht eine weitere (querschneidende) Einschränkung der Metrikmenge ergänzend zu den Hierarchien.
Domain
Website
Tracking Site
Vermarkter
Auslieferungs-
Domain
Referral
«enumerati...
KostenmodellArt
CPC
CPM
CPO
CPA
«abstract»
DimensionValue
+ id: int
+ name: String
+ sourceId: String
WebsiteFact
+ Bounces: int
+ Verweildauer: float
+ Visits: int
BasicAdFact
+ Clicks: int
+ Sichtbare Views: int
+ Validierte Clicks: int
+ View (angefragt): int
+ View (ausgeliefert): int
+ View (gemessen): int
«dimension»
Produkt
Shop
Produkt
+ Produktkategorie: String
«dimension»
Zeitfenster
Letzte X Tage
«dimension»
User
User Segment
«dimension»
Order
OrderStatus
+ Status: OrderStatus
«enumeration»
OrderStatus
IN_BEARBEITUNG
ERFOLGREICH (AKTIVIERT)
ABGELEHNT
NICHT_IN_BEARBEITUNG
UniquesFact
+ Unique Clicks: int
+ Unique Users: int
+ Unique Views: int
AdCostFact
+ CPC: int
+ Kosten: float
ConversionFact
+ PC: int
+ PR: int
+ PV: int
+ Umsatz PC: float
+ Umsatz PR: float
+ Umsatz PV: float
AdVisibilityFact
+ Sichtbarkeitsdauer: float
ActivatedOrderFact
+ Orders: int
+ Umsatz: float
TrackingFact
+ Orders: int
+ Page Impressions: int
+ Umsatz: float
X = {7, 14, 28, 30}
act Processing
Processing
WebsiteOrderActivationAdCostConversionUniques+
Overlap
AdVisibilityBasicAdsTracking
Dimensionsv erzeichnis
aktualisieren
Page
Impressions
und Orders
zählen
Aggregation
TrackingFact
Ad
Impressions
zählen
Aggregation
BasicAdFact
CTR und
Sichtbarkeitsrate
berechnen
Inferenz der
Sichtbarkeitsdauer
Aggregation
AdVisibilityFact
Dimensionsraum
aufspannen
Pro Vektor die Ev ents und
die dort enthaltenen UserIds
und Interaktionsarten
ermitteln
Menge der UserIds pro
Interaktionsart erzeugen und deren
Mächtigkeit bestimmen
UniquesFact
User Journeys
erstellen (bzw.
LV/LC pro User)
Attribution v on Conv ersion
auf Basis v orkonfiguriertem
Conv ersion Modell
Conv ersions
zählen (PV, PC, PR)
Warenkorbwert ermitteln
bei Order-Conv ersion
(PV, PC, PR)
Aggregation
Conv ersionFact
Kosten pro
Ev ent
ermitteln Aggregation
Nicht zuweisbare
Kosten ermitteln
CostFact
Inferenz der Visits
Visits zählen
Bounces zählen
Analyse der Verweildauer
Aggregation
WebsiteFact
Order Status
ermitteln
Anzahl der nicht getrackten Orders ermitteln
Aggregation
Activ atedOrderFact
:Analytics Warehouse:Event Data Lake
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
«flow»
Data Processing WorkflowMultidimensional Data Model

32. Sample results
Geolocated and gender-
specific conversions.
Frequency of visits
Performance of an ad campaign