The document discusses the challenges of handling billions of events per minute at Twitter. It describes Twitter's event log pipeline architecture using microservices, event aggregation with Scribe, and event processing with MapReduce. The pipeline was later modified to use streaming technologies to reduce latency, including Apache Beam and Google Cloud services. Latency for different dataset types ranges from seconds to minutes. Future challenges include scaling further for volume and spikes while maintaining fast failure recovery times.

1. Event Log
Pipeline @Twitter
Challenges of handling Billions
of events per minute
Systems @Scale Fall 2021
Lohit VijayaRenu
@lohitvijayarenu

2. #
TAGHASHT
AG#HASH
#HASHTAG
#
TAG#HASH
DataPlatform @Twitter
#Startups #Yahoo!
#Hadoop
Lohit VijayaRenu
@lohitvijayarenu
9:35AM · Oct 13, 2021

3. Example of
slide
● User interactions generate events
● Events are grouped as Datasets
● Datasets for Data processing & Analytics
Event Log Pipeline
Event Log Pipeline

4. 1B 5B
~500 datasets
● Scale
● Resource requirement
~800 datasets
Events per
minute
Events per
minute
Challenges

5. Example of
slide
Architecture
Microservices
>100k instances
Clients
Events

6. Example of
slide
Architecture
Microservices
>100k instances
Clients
Event
Aggregation
Scribe
~3K instances
Events

7. Example of
slide
Architecture
Microservices
>100k instances
Clients
Event
Aggregation
Scribe
~3K instances
MapReduce
>10K jobs
Event
Processing
Events

8. Example of
slide
Architecture
Microservices
>100k instances
Clients
Event
Aggregation
Scribe
~3K instances
MapReduce
>10K jobs
HDFS, Presto, Scalding
>100PBs of data
Event
Processing
Event Log
Management
Events Datasets

9. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
● Per aggregator scaling challenges
● No multi tenancy for datasets
● Eco system integration
● Difficult to maintain and improve
Event Aggregation
Problem

10. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
● Moved to open source Flume
● Per aggregator improvements
● Microbatch, memory management,...
● Optimize resource usage
● Streaming at network speed
Event Aggregation

11. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
● Memory spikes before and after
Event Aggregation

12. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
● Data tiers with priorities
● Dataset groups
● Aggregator groups
● Dynamic scaling
Event Aggregation

13. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
● Scaling Aggregation
Event Aggregation
Dataset 1
Dataset 2
Dataset 3
Aggregator Group 1
Aggregator Group 2
Service
Discovery
Dataset Storage

14. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
Event Processing
Problem
● Thousands of process jobs
● Multitenancy
● Long tail reducers
● Unpredictable run time (traffic surge)

15. Example of
slide
Event Processing
● Apache Tez based processor
● Data tier processors
● Dynamic Hash based partition

16. Example of
slide
Event Processing
● Apache Tez based processor
● Data tier processors
● Dynamic Hash based partition
Raw Events
Partition
Partition
Partition

17. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
Migration
● Transparent migration
● More than 5 Billion events per
minute
● Per dataset scaling
● Reliable and fault tolerant framework

18. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
Migration
● Transparent migration
● More than 5 Billion events per
minute
● Per dataset scaling
● Reliable and fault tolerant framework
But…….
Latency!!!

19. Example of
slide
Near real time
Near real time
event streaming
● Analytics on real time user interactions
● End to end latency of minutes
● Data storage with real time insertion

20. Example of
slide
Architecture
Microservices
>1k instances
Clients
Events

21. Example of
slide
Architecture
Microservices
>10k instances
Clients
Streaming
Aggregation
Flume, Google
PubSub
Events

22. Example of
slide
Architecture
Microservices
>10k instances
Clients
Streaming
Aggregation
Flume, Google
PubSub
Apache Beam,
Google Dataflow
Streaming
Processor
Events

23. Example of
slide
Architecture
Microservices
>10k instances
Clients
Streaming
Aggregation
Flume, Google
PubSub
Apache Beam,
Google Dataflow
BigQuery, Google Cloud
Storage, Amplitude
Streaming
Processor
Real time
insertion
Events Datasets

24. Example of
slide
Latency
Dataset Type Event Latency
p90 p95 p99
Dataset 1 (very small datasets) 0.42 sec 0.48 sec 0.59 sec
Dataset 2 (Smaller datasets) 2.6 sec 2.8 sec 8.4 sec
Dataset 3 (Large datasets) 22.1 sec 29.9 sec 67.7 sec
Dataset 4 (Large datasets 5 min micro batch) 6.3 min 6.5 min 7.2 min
● Promising results
● Independent pipelines

25. Example of
slide
Example of Statement
Slide. Lorem ipsum dolor
sit amet, consectetur
adipiscing elit. Text size
should be between 80 pt to
140 pt depending on
paragraph length.
New challenges and future
● Scale for volume and spikes
● Stream processing and ingestion at
scale
● Faster catch up on failures
● Change Event Log Pipeline to Event
Streaming Pipelines
Follow @TwitterEng
@TwitterCareers

26. Thank You!
26
Log Ingestion and Replication
Data Platform Infrastructure
Data Platform SRE