This presentation examines use cases for event-driven data processing and explains Streamlio's technology and how it applies to handling streaming event data.

1. Engineering Real Time
Event Driven Processing
Karthik Ramasamy, Co-founder
Streamlio

2. Information Age
!2
Ká !
Event data is key

3. Increasingly Connected World
!3
Internet of Things
30 B connected devices by 2020
Health Care
153 Exabytes (2013) -> 2314 Exabytes
(2020)
Machine Data
40% of digital universe by 2020
Connected Vehicles
Data transferred per vehicle per month
4 MB -> 5 GB
Digital Assistants (Predictive Analytics)
$2B (2012) -> $6.5B (2019) [1]
Siri/Cortana/Google Now
Augmented/Virtual Reality
$150B by 2020 [2]
Oculus/HoloLens/Magic Leap
Ñ
+
>

4. Use Cases

5. Observations
• Fight spammy content, engagements, and behaviors in Twitter
• Spam campaign comes in large batch
• Despite randomized tweaks, enough similarity among spammy entities are
preserved
Requirement
• Real time - a competition with spammers (i.e) “detect” vs “mutate”
• Generic - need to support all common feature representations
Product Safety
!5

6. Product Safety - System Overview
!6
KV store for
clustering
Messaging
System (Event
Bus)
Similarity Clustering (Heron)

7. Real Time Ads
!7
KV store
Messaging
System (Event
Bus)
Ads Serving (Heron)
Ads Prediction (Heron)
Impressions
Spend
Ads Analytics (Heron)
Engagements
Spend
Ads Requests
Ads Responses
Impressions
Spend

8. Connected Cars
!8
KV store for
clustering
Messaging
System
Traffic Patterns
Messaging
System
Data Capture/Filter
Fuel Efficiency

9. Architectural Pattern

10. Recurring Pattern
!10
ProcessMessaging
Storage
Data Ingestion Data Processing
Results StorageData Storage
Data
Serving

11. State of the World
!11
Aggregation
Systems
Messaging
Systems
Result
Engine
HDFS
Queryable
Engines

12. Towards Unification and Simplification
!12
Interactive
Querying
Storm API Streamlets SQL
Application
Builder
Pulsar
API
BK/
HDFS
API
Metadata
Management
Operational
Monitoring
Chargeback
Security
Authentication
Quota
Management
Kafka
API

13. Apache Pulsar

14. Apache Pulsar highlights
!14
Stream-Native Functions
*NEW*
Apply processing functions on
data, fully managed by Pulsar
Multi-tenancy
A single cluster can support
many tenants and use cases
High throughput
Millions of messages/s in a
single partition
Durability
Data replicated and synced
to disk
Geo-replication
Out of box support for
geographically distributed
applications
Unified messaging model
Support both Topic & Queue
semantic in a single model
Delivery Guarantees
At least once, at most once
and effectively once
Low Latency
Low publish latency of 5ms
at 99pct
Scalability
Supports millions of topics in
a single cluster

15. Pulsar Architecture
!15
Serving
Brokers can be added independently
Traffic can be shifted quickly across brokers
Storage
Bookies can be added independently
New bookies will ramp up traffic quickly

16. Segment Centric Architecture
!16

17. Pulsar multi-datacenter replication
!17
Geo-replication
Asynchronous replication
Integrated in the broker message flow
Simple configuration to add/remove regions
Topic (T1) Topic (T1)
Topic (T1)
Subscription
(S1)
Subscription
(S1)
Producer
(P1)
Consumer
Producer
(P3)
Producer
(P2)
Consumer
Data Center A Data Center B
Data Center C

18. Apache Heron

19. Apache Heron design goals
!19
Efficiency
Reduce resource
consumption
Support for diverse
workloads
Throughput vs latency
sensitive
Support for multiple
semantics
At most once, At least once,
Effectively once
Native Multi-Language
Support
C++, Java, Python
Task Isolation
Ease of debug-ability/isolation/
profiling
Support for back pressure
Topologies should be self
adjusting
Use of containers
Runs in schedulers - Kubernetes &
DCOS & many more
Multi-level APIs
Procedural, Functional and Declarative
for diverse applications
Diverse deployment models
Run as a service or pure library

20. Heron data model
!20
%
%
%
%
%
Spout 1
Spout 2
Bolt 1
Bolt 2
Bolt 3
Bolt 4
Bolt 5

21. Writing Heron topologies
!21
Procedural - Low Level API
Directly write your spouts and bolts
Functional - Mid Level API
Use of maps, flat maps, transform, windows
Declarative - SQL (in the works)
Use of declarative language - specify what you
want, system will figure it out.
,
%

22. Topology execution
!22
Topology Master
ZK
Cluster
Stream
Manager
I1 I2 I3 I4
Stream
Manager
I1 I2 I3 I4
Logical Plan,
Physical Plan and
Execution State
Sync Physical Plan
DATA CONTAINER DATA CONTAINER
Metrics
Manager
Metrics
Manager
Metrics
Manager
Health
Manager
MASTER
CONTAINER

23. Heron topology examples
!23

24. Heron topology scale
!24
CONTAINERS - 1 TO 600
INSTANCES - 10 TO 6000

25. • No more pages during midnight for Heron team
• Very rare incidents for Heron customer teams
• Easy to debug during incident for quick turn around
• Reduced resource utilization saving cost
Heron impact at Twitter
!25

26. Apache BookKeeper

27. Computation across batch/streaming is similar
• Expressed as DAGS
• Run in parallel on the cluster
• Intermediate results need not be materialized
• Functional/Declarative APIs
Storage is the key
• Messaging/Storage are two faces of the same coin
• They serve the same data
Observations
!27

28. • Be able to write and read streams of records with low latency,
storage durability
• Data storage should be durable, consistent and fault tolerant
• Enable clients to stream or tail ledgers to propagate data as
they’re written
• Store and provide access to both historic and real-time data
Storage Requirements
!28

29. Using BookKeeper
!29

30. Companies using the technology
!30