Enterprises are increasingly building applications to take advantage of real-time workloads, and Kubernetes is fast becoming the de-facto scheduler and orchestrating system for distributed systems. In this session, Karthik Ramasamy will show how easy it is to deploy Apache Pulsar for queuing and streaming and Twitter Heron for stream processing using Kubernetes and show how end-to-end data pipelines are built. Pulsar and Heron are at the core of Streamlio's end-to-end real-time platform, which is ideally suited to enterprises building next generation real time pipelines

1. Streaming Pipelines in Kubernetes
Using Apache Pulsar, Heron and
BookKeeper
Karthik Ramasamy
Cofounder
Streamlio

2. 2
Information Age
Event Data is key
Ká !

3. 3
Increasingly Connected World
Internet of Things
30 B connected devices by 2020
Health Care
153 Exabytes (2013) -> 2314 Exabytes (2020)
Machine Data
40% of digital universe by 2020
Connected Vehicles
Data transferred per vehicle per month
4 MB -> 5 GB
Digital Assistants (Predictive Analytics)
$2B (2012) -> $6.5B (2019) [1]
Siri/Cortana/Google Now
Augmented/Virtual Reality
$150B by 2020 [2]
Oculus/HoloLens/Magic Leap
Ñ
!+
>

4. 4
Traditional Batch Processing
Challenges
Introduces too much “decision latency”
Responses are delivered “a[er the fact”
Maximum value of the idenfied situaon is lost
Decisions are made on old and stale data
Data at Rest
Store Analyze Act

5. 5
The New Era: Streaming Data/Fast Data
Events are analyzed and processed in real-me as they arrive
Decisions are mely, contextual and based on fresh data
Decision latency is eliminated
Data in moon
Modern Data Pipelines Streaming Data Pipelines

6. 6
Streaming Data Pipeline
A Streaming Data Pipeline
captures events for analysis,
process as they arrive and
produce con=nuous results

7. 7
Streaming Data Pipeline
Data Lake
Product Features
Adhoc Features
Counng
Machine Learning
Extract Transform Load (ETL)
Click Data
Engagement Data
Email sends
User events

8. Use Cases

9. 9
Product Safety
Observa=ons
Fight spammy content, engagements, and behaviors in Twieer
Spam campaign comes in large batch
Despite randomized tweaks, enough similarity among spammy enes are
preserved
Requirement
Real me - a compeon with spammers (i.e) “detect” vs “mutate”
Generic - need to support all common feature representaons

10. 10
Product Safety - System Overview
KV store for
clustering
Messaging
System
(Event Bus)
Similarity Clustering (Heron)

11. 11
Real Time Ads
KV store
Messaging
System
(Event Bus)
Ads Serving (Heron)
Ads Predicon (Heron)
Impressions
Spend
Ads Analycs (Heron)
Engagements
Spend
Ads Requests
Ads Responses
Impressions
Spend

12. 12
Connected Cars
KV store for
clustering
Messaging
System
Traffic Paeerns
Messaging
System Data Capture/Filter
Fuel Efficiency

13. 13
Streaming Use Cases
Algorithmic trading
Online fraud detecon
Geo fencing
Proximity/locaon tracking
Intrusion detecon systems
Traffic management
Real me recommendaons
Churn detecon
Internet of things
Social media/data analycs
Gaming data feed
IoT Edge Analycs

14. Architectural Pattern

15. 15
Recurring Pattern
ProcessMessaging
Storage
Data Ingeson Data Processing
Results StorageData Storage
Data
Serving

16. 16
State of the World
Aggregation
Systems
Messaging
Systems
Result
Engine
HDFS
Queryable
Engines

17. 17
Towards Unification and Simplification
Interactive
Querying
Storm API Streamlets SQL
Application
Builder
Pulsar
API
BK/
HDFS
API
Kubernetes
Metadata
Management
Operational
Monitoring
Chargeback
Security
Authentication
Quota
Management
Kafka
API

18. Apache Pulsar

19. 19
Why Apache Pulsar?
Ordering
Guaranteed ordering
Mul=-tenancy
A single cluster can support many
tenants and use cases
High throughput
Can reach 1.8 M messages/s in a
single par==on
Durability
Data replicated and synced to disk
Geo-replica=on
Out of box support for geographically
distributed applica=ons
Unified messaging model
Support both Topic & Queue
seman=c in a single model
Delivery Guarantees
At least once, at most once and effec=vely
once
Low Latency
Low publish latency of 5ms at 99pct
Highly scalable
Can support millions of topics

20. 20
Pulsar Architecture
Separa=on of Storage and Serving
SERVING
Brokers can be added independently
Traffic can be shifted quickly across brokers
STORAGE
Bookies can be added independently
New bookies will ramp up traffic quickly

21. 21
Segment Centric Architecture

22. 22
Broker Failure Recovery
• Topic is reassigned to
an available broker
based on load
• Can reconstruct the
previous state
consistently
• No data needs to be
copied
• Failover handled
transparently by client
library

23. 23
Bookie Failure Recovery
• After a write failure,
BookKeeper will
immediately switch write to
a new bookie, within the
same segment.

24. 24
Bookie Failure Recovery
• In background, starts a
many-to-many recovery
process to regain the
configured replication
factor

25. 25
Pulsar Architecture
Geo Replica=on
GEO REPLICATION
Asynchronous replication
Integrated in the broker message flow
Simple configuration to add/remove regions
Topic (T1) Topic (T1)
Topic (T1)
Subscripon
(S1)
Subscripon
(S1)
Producer
(P1)
Consumer
(C1)
Producer
(P3)
Producer
(P2)
Consumer
(C2)
Data Center A Data Center B
Data Center C

26. 26
Pulsar in Production
3+ years
Serves 2.3 million topics
100 billion messages/day
Average latency < 5 ms
99% 15 ms (strong durability guarantees)
Zero data loss
80+ applicaons
Self served provisioning
Full-mesh cross-datacenter replicaon -
8+ data centers

27. Apache Heron

28. 28
Heron Design Goals
Efficiency
Reduce resource consumpon
Support for diverse workloads
Throughput vs latency sensive
Support for mul=ple seman=cs
Atmost once, Atleast once,
Effecvely once
Na=ve Mul=-Language Support
C++, Java, Python
Task Isola=on
Ease of debug-ability/isolaon/profiling
Support for back pressure
Topologies should be self adjusng
Use of containers
Runs in schedulers - Kubernetes & DCOS &
many more
Mul=-level APIs
Procedural, Funconal and Declarave for
diverse applicaons
Diverse deployment models
Run as a service or pure library

29. 29
Heron Data Model
%
%
%
%
%
Spout 1
Spout 2
Bolt 1
Bolt 2
Bolt 3
Bolt 4
Bolt 5

30. 30
Writing Heron Topologies
Procedural - Low Level API
Directly write your spouts and bolts
Functional - Mid Level API
Use of maps, flat maps, transform, windows
Declarative - SQL (in the works)
Use of declarave language - specify what you
want, system will figure it out.
,
%

31. 31
Topology Execution
Topology Master
ZK
Cluster
Stream
Manager
I1 I2 I3 I4
Stream
Manager
I1 I2 I3 I4
Logical Plan,
Physical Plan and
Execution State
Sync Physical Plan
DATA CONTAINER DATA CONTAINER
Metrics
Manager
Metrics
Manager
Metrics
Manager
Health
Manager
MASTER
CONTAINER

32. 32
Heron @Twitter
LARGEST CLUSTER
100’s of TOPOLOGIES
BILLIONS OF MESSAGES100’s OF TERABYTESREDUCED INCIDENTS
GOOD NIGHT SLEEP
3X - 5X reduction in resource usage

33. 33
Heron Topology Complexity

34. 34
Heron Topology Scale
CONTAINERS - 1 TO 600 INSTANCES - 10 TO 6000

35. 35
Heron Happy Facts :)
! No more pages during midnight for Heron team
" Very rare incidents for Heron customer teams
# Easy to debug during incident for quick turn around
$ Reduced resource ulizaon saving cost

36. Apache BookKeeper

37. 37
Observations
Computaon across batch/streaming is similar
Expressed as DAGS
Run in parallel on the cluster
Intermediate results need not be materialized
Funconal/Declarave APIs
Storage is the key
Messaging/Storage are two faces of the same coin
They serve the same data

38. 38
Storage Requirements
Requirements for a real-=me storage placorm
Be able to write and read streams of records with low latency, storage durability
Data storage should be durable, consistent and fault tolerant
Enable clients to stream or tail ledgers to propagate data as they’re wrieen
Store and provide access to both historic and real-me data

39. 39
Apache BookKeeper - Stream Storage
A storage for log streams
Replicated, durable storage of log streams
Provide fast tailing/streaming facility
Opmized for immutable data
Low-latency durability
Simple repeatable read consistency
High write and read availability

40. 40
Record
Smallest I/O and Address Unit
A sequence of invisible records
A record is sequence of bytes
The smallest I/O unit, as well as the unit of address
Each record contains sequence numbers for addressing

41. 41
Logs
Two Storage Primi=ves
Ledger: A finite sequence of records.
Stream: An infinite sequence of records.

42. 42
Ledger
Finite sequence of records
Ledger: A finite sequence of records that gets terminated
A client explicitly close it
A writer who writes records into it has crashed.

43. 43
Stream
Infinite sequence of records
Stream: An unbounded, infinite sequence of records
Physically comprised of mulple ledgers

44. 44
Bookies
Stores fragment of records
Bookie - A storage server to store data records
Ensemble: A group of bookies storing the data records of a ledger
Individual bookies store fragments of ledgers

45. 45
Using BookKeeper
Durable Messaging, Scalable Compute and Stream Storage

46. 46
Companies using BookKeeper
Enterprise Grade Stream Storage

47. 47
Companies using the projects
Enterprise Grade Stream Storage

48. 48
Experiences with Kubernetes
Use of naked pods for Heron containers
Never gets rescheduled
Have to create names similar to Heron container (topology-name-shard-id)
Have to keep track of every pod
Cannot use deployment
Reproducible shard id
Deployments produce different ids a[er a pod failure
Topology names have to be lowercase
First Implementa=on

49. 49
Experiences with Kubernetes
Use of stateful sets
Nice mapping form stateful sets to container shard ids
Inially used defaults
Very long startup me
Very long shutdown me
Long pod restart me (during machine failure & pod failure)
Second Implementa=on

50. 50
Experiences with Kubernetes
Invesgaon pointed out us to
Default pod management policy is set to OrderedReady
Default rescheduling pod on a node failure is set to 5 mins
Soluons
Sets the toleraons to 10 seconds for restarts
node.kubernetes.io/not-ready
node.alpha.kubernetes.io/notReady
node.alpha.kubernetes.io/unreachable
Second Implementa=on

51. 51
Experiences with Kubernetes
Dynamically updang the job
Patching stateful sets -
Kubernetes Java Client library has bug
Soluons
Waing for a fix!
Second Implementa=on

52. 52
Experiences with Kubernetes
Submidng User jobs
Heron API
Server
Job 1
Job 2
Job 3User
Needs a way to talk to Kubernetes
API server
Solu=on - Run a slim side container with kubectl proxy

53. 53
Ingesting Data to Pulsar
Pulsar Broker 1 Pulsar Broker 2 Pulsar Broker 3
Bookie 1 Bookie 2 Bookie 3 Bookie 4 Bookie 5
Apache BookKeeper
Apache Pulsar
Producer Consumer
Internal IPs

54. 54
Ingesting Data to Pulsar
Pulsar Broker 1 Pulsar Broker 2 Pulsar Broker 3
Bookie 1 Bookie 2 Bookie 3 Bookie 4 Bookie 5
Apache BookKeeper
Apache Pulsar
Producer Consumer
Load Balancer
Pulsar Proxy 1 Pulsar Proxy 3Pulsar Proxy 1

55. 55
Finally…

57. 57
GET IN TOUCH
C O N T A C T U S
@kramasamy
955 Alma Street, Palo Alto, CA
karthik@streaml.io