The document discusses the journey towards building a real-time data platform capable of handling 2.5 million events per second. It describes migrating Spark processing from on-premises CDH to AWS EMR to improve scalability. Fault tolerance was added through batch processing in Spark and auto-recovery capabilities. Backpressure was enabled through Spark streaming, HDFS, and pulling data into Vertica to prevent overloading downstream systems. Monitoring was enhanced with a separate application to track pipeline metrics. The final platform achieved the performance goals through these architectural changes.

1. ULTIMATE JOURNEY TOWARDS
REALTIME DATA PLATFORM
2.5M / s
BORIS TROFIMOV @ SIGMA SOFTWARE

2. Leading DWH @ Oath:
Major expertise Big Data and Enterprise
Cofounder of Odessa JUG
Passionate follower of Scala
Associate professor at ONPU
ABOUT ME

3. WHERE IS BIG DATA?

4. API
BACK OFFICE
CUSTOMER
WEB PORTAL
MOBILE APPS
INEGRATION
POINTS
INTRODUCING DATA PLATFORM

5. DOMAIN SERVICE
API
BACK OFFICE
CUSTOMER
WEB PORTAL
MOBILE APPS
MICROSERVICE
CORE DOMAIN SERVICES
INEGRATION
POINTS
DOMAIN SERVICE
DOMAIN SERVICE
DOMAIN SERVICE
INTRODUCING DATA PLATFORM

6. DOMAIN SERVICE
API
BACK OFFICE
CUSTOMER
WEB PORTAL
MOBILE APPS
MICROSERVICE
INFRASTRUCTURE SERVICES
SERVICE
DISCOVERY
SHARED
CONFIG
DOMAIN
DEPENDENCY
MANAGEMENT
ACL
MANAGEMENT
MICROSERVICE
CORE DOMAIN SERVICES
INEGRATION
POINTS
DOMAIN SERVICE
DOMAIN SERVICE
DOMAIN SERVICE
INTRODUCING DATA PLATFORM

7. DOMAIN SERVICE
API
BACK OFFICE
CUSTOMER
WEB PORTAL
MOBILE APPS
MICROSERVICE
INFRASTRUCTURE SERVICES
SERVICE
DISCOVERY
SHARED
CONFIG
DOMAIN
DEPENDENCY
MANAGEMENT
ACL
MANAGEMENT
MICROSERVICE
CORE DOMAIN SERVICES
INEGRATION
POINTS
DOMAIN SERVICE
DOMAIN SERVICE
DOMAIN SERVICE
INTRODUCING DATA PLATFORM
BIG DATA ?

8. API
DATA PLATFORM
BACK OFFICE
CUSTOMER
WEB PORTAL
MOBILE APPS
MICROSERVICE
INFRASTRUCTURE SERVICES
SERVICE
DISCOVERY
SHARED
CONFIG
DOMAIN
DEPENDENCY
MANAGEMENT
ACL
MANAGEMENT
MICROSERVICE
CORE DOMAIN SERVICES
INEGRATION
POINTS
DOMAIN SERVICE
DOMAIN SERVICE
DOMAIN SERVICE
DOMAIN SERVICE
INTRODUCING DATA PLATFORM

9. DATA PLATFORM
INTRODUCING DATA PLATFORM

10. DATA PLATFORM
3rd PARTY
PROVIDERS
PLATFORM
COMPONENTS
INTRODUCING DATA PLATFORM
events

11. DATA PLATFORM
INTRODUCING DATA PLATFORM
3rd PARTY
PROVIDERS
PLATFORM
COMPONENTS
REPORTING
ANALYTICS
Major mission: organize data
events

12. ZOOMING IN DATA PLATFORM
INGESTION
MODULE
REPORTING
SERVICE
WAREHOUSE
VALIDATION
ENRICHMENT
MODULE
RAW DATA
AGGREGATIONS
MODULE
FACTS
RAW DATA
DIMENSIONS
ANALYTICS
MODULE
CONFIGURATION
MODULE
DIMENSION
UPDATER

13. based on © https://blogs.msdn.microsoft.com/agile/2012/07/26/cqrs-journey-guidance-project-released/

14. 7 JUNE

15. OUR DOMAIN
CORE
PLATFORM
DATA
PLATFORM
VIDEO PLAYERS
CONTENT OWNERS
END USERS

16. OUR DOMAIN
S3 Data Lake
5 PB
Vertica
500 TB
Raws/Table
600 B
Events/Sec
2.5 M
Files/Hour/Pipeline
15 K
Data/Daly
25 TB
DATA LAKE PROCESSING

17. ORIGINAL PIPELINE
DATA PLATFORM
VERTICAS3 HADOOPNGINX
REPORTING
SERVICE
DATA LAG ~1h

18. UNITED LAMBDA PLATFORM
DATA PLATFORM
KAFKA SPARK MEMSQL
VERTICAS3 HADOOPNGINX
REPORTING
SERVICE
DATA LAG ~2m
DATA LAG ~1h

19. UNITED LAMBDA PLATFORM
DATA PLATFORM
KAFKA SPARK MEMSQL
VERTICAS3 HADOOPNGINX
REPORTING
SERVICE
DATA LAG ~2m
DATA LAG ~1h

20. WHAT WAS GOOD
DATA DELIVERY TIME 2 min
FINE ON PROD SCALE @ THAT TIME -- 150K/s
PAINFUL SCALE UP TO 1M/s

21. НЕЛЬЗЯ ПРОСТО ТАК ВЗЯТЬ
И ВЫРАСТИ В 20 РАЗ

22. WHY WE NEEDED CHANGES
ROCKY SCALING
Adding/removing nodes to CDH YARN requires yarn restart and downtime for apps
Tricky to build quick sandboxes
The latest Memsql release 5.X It was not able to operate cluster with > 80 nodes
Max supported rate limit 1M events/s, while Business required 2.5M/s
ZERO TOLERANCE
EC2 faulty nodes could make Spark or Memsql get stuck for a while
Buggy HA, even one faulty node could break entire Memsql cluster, make to recreate database and lose data
PUSH approach to write data to Memsql
MONITORING & ALERTING
Find the most relevant metrics
Eliminate FALSE POSITIVE and FALSE NEGATIVE errors

23. ON A WAY TO 2.5 M / S

25. MIGRATING SPARK TO EMR
EASY CREATE, EASY DESTROY
• easy to … make bill cost a fortune
MULTIPLE EMR CLUSTERS
• Separating concerns and Isolation
• Better to run single application per EMR cluster
• Simplified auto-scaling rules
STATELESS EMR CLUSTERS
• Do not use local HDFS

26. CAUTION, EMR!
EASY TO ALLOCATE AND EASY TO LOSE EMR NODE
• Concerns mostly m4.4xl as the most popular instance type
LOSING MASTER NODE – LOSING ENTIRE CLUSTER
• Hard to build reliable platform involving multiple AZ [see Fleets model]
• Develop one-step evacuation procedure to another EMR
LUCK OF LACK ON SPECIFIC INSTANCE TYPE
• Can be mitigated by fleets model

27. DEPLOYMENT DETAILS
MASTER
TASK TASK TASK
…
EMR CLUSTER [YARN]
TASK TASK TASK
…

28. DEPLOYMENT DETAILS
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]
TASK TASK TASK
…

29. DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
TASK TASK TASK
…

30. DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
LOCAL
YARN
CONFIG
TASK TASK TASK
…

31. DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
TASK TASK TASK
…
LOCAL
YARN
CONFIG

32. DEPLOYMENT DETAILS
SPARK BINARIES
MASTER
S3
YARN
CONFIG
(zip)
TASK TASK TASK
…
EMR CLUSTER [YARN]DOCKER CONTAINER
DRIVER APP
TASK TASK TASK
…
LOCAL
YARN
CONFIG
RANCHER

33. CDH vs EMR
E M RC D H
Cannot scale out/in on demand Is able to scale out/in on demand
No extra cost (for community
license)
Extra ~30% to EC2 costs
Per second billing (!)
Adding machines to CDH
requires restarting Yarn
No Yarn restart
Easy configuration management
via CM
Limited configuration available
during EMR creation
Classic Yarn cluster Ordinary Yarn under hood, imposes
EMR-driven way to deploy apps
Single CDH per AZ EMR cluster on demand as unit
of clustering

35. MAKING SPARK WRITE FASTER
USING CUSTOM HADOOP COMMITER
• FileOutputCommiter committer with V2 option to exclude file moving in HDFS/S3
WRITE DATAFRAME TO HDFS FIRST
• Spark writes to HDFS directly into partitioned folder and registers new partition in
Hive

36. WRITING FASTER – FILE FORMATS
MOST STABLE PERFORMANCE ON ORC UNCOMPRESSED
• spark apps writes raw data in ORC
• presto reads ORC and writes aggregations in ORC
• replication uses ORC to send delta to Vertica
BEST PERFORMANCE ON HDFS BLOCK SIZE AND STRIP 64M
• Thankfully to strict retention policy 6 hours
ENABLING hive.orc.use-column-names=true
• simplifies Spark app, allowing to write dataframe as is, presto accesses columns by name
• allows to evolve/modify schema for dataframe and database independently

38. SPARK PERFORMANCE
ONE EXECUTOR PER YARN NODE
• for better cpu and cache utilization, using 16 vcores (aligning to m4.4xl)
ALIGN RDD PARTITIONS TO VCORES
• Repartition data we read from Kakfa [address if there is a skew in kafka partitions]
SPLIT PROCESSING BATCH INTERVAL ONTO RESPONSIBILITY ZONES
• Control each interval separately
FETCH FROM KAFKA ENRICHMENT WRITE TO HIVE
1 minute
8 seconds 20 seconds 20 seconds
STUFF/OVERHEAD
12 seconds

40. FRIENDLY REMINDER
DATA PLATFORM
KAFKA SPARK MEMSQL
NGINX
REPORTING
SERVICE

41. INTRODUCING PRESTO
DATA PLATFORM
KAFKA SPARK PRESTO
VERTICANGINX
REPORTING
SERVICE
DATA LAG ~3m

42. UNDER HOOD
Aggregations and replications are running every minute
Presto uses dimensions hosted outside. Using Memsql with realtime
updates
VERTICA
NODE
REPORTING
SERVICE
SPARK, EMR
HDFS
NODE
NODE
COLLOCATED HDFS/PRESTO
PRESTO
REPLICATORS
JENKINS SCHEDULER
MEMSQL

44. FAULT TOLERANCE
EMR FLEETS MODEL
• New feature
• Allows to focus on cores instead of machines
• Allows provisioning nodes over multiple AZ
SPARK SPECULATION & BLACK LISTING
• Faulty nodes is total disaster (c)
• Spark Feature request to introduce minimal speculation interval (conflict with DirectCommiter)

45. FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1 BATCH 2 BATCH 1

46. FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
BATCH 3
PRESTO
BATCH 1
HDFS/HIVE
AGGREGATION TABLE
BATCH 2
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1

47. FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1 BATCH 2 BATCH 1
BATCH 3

48. FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates microbatch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 3
BATCH 1 BATCH 2 BATCH 1
BATCH 3

49. FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates micro-batch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACT TABLE
BATCH 2
BATCH 3
PRESTO
HDFS/HIVE
AGGREGATION TABLE
REPLICATOR
VERTICA
AGGREGATION TABLE
BATCH 1 BATCH 2 BATCH 1
BATCH 3

50. FAULT TOLERANCE
EVENT/BATCH SOURCING
• Spark associates micro-batch with batch_id [timestamp]
• Batch_id is partitioned Hive column
• Aggregating and replicating only missed batches
• In case of failures after restart every component shall auto-recover without data losses
SPARK
BATCH 1
HDFS/HIVE
RAW FACTS
BATCH 2
BATCH 3
PRESTO
HDFS/HIVE
AGGREGATED FACTS
REPLICATOR
VERTICA
AGGREGATED FACTS
BATCH 1 BATCH 2 BATCH 1
BATCH 3
BATCH 2
BATCH 3

53. BACKPRESSURE ENABLED
DATA PLATFORM
KAFKA SPARK PRESTO
VERTICANGINX REPORTING
SERVICE

54. BACKPRESSURE ENABLED
KAFKA SPARK PRESTO
VERTICANGINX REPORTING
SERVICE
SPARK STREAMING BACKPRESSURE
• MUST HAVE for variable rate
• FEATURE contributed to Spark master with back pressure initial max rate for direct mode

55. BACKPRESSURE ENABLED
KAFKA SPARK PRESTO
VERTICANGINX REPORTING
SERVICE
HDFS VALVE
• HDFS between Spark and Presto
• Retention policy 12h

56. BACKPRESSURE ENABLED
KAFKA SPARK PRESTO
VERTICANGINX REPORTING
SERVICE
PULL WRITE
• Using Vertica’s query COPY from HDFS to let Vertica read data with own rate

57. BACKPRESSURE ENABLED
KAFKA SPARK PRESTO
VERTICANGINX REPORTING
SERVICE
KAFKA OUTAGES
• Lua writes events directly to Kafka
• Unsent events stored locally and sent to S3
• NiFi periodically rends that data back to Kafka

59. I AM A GOD
I HAVE NO IDEA
WHAT’S GOING ON

60. MONITORING FUNDAMENTALS
FUNDAMENTAL REALTIME METRICS
• IN RATE
• OUT RATE
• CURRENT LAG
• ERRORS RATE
• BATCH PROCESSING TIME
• PIPELINE LATENCY
SEPARATED APP INTRODUCED [ aka BANDARLOG ]
• Tracks offsets for kafka, and Hive/Presto and Vertica
• Standalone application
• Open sourced soon
USING DATADOG
• Dashboards, monitors

61. DASHBOARD EXAMPLE [INGESTION]

62. DASHBOARD EXAMPLE [AGGREGATIONS]

64. WHAT WE HAVE ACHIEVED
SCLABLE PRODUCTION
• Ability to grow further beyond 1M/s up to 2.5M
STABLE PRODUCTION ENVIRONMENT
• fault tolerant components, easier to recover
LESS EXPENSIVE
• Smaller Spark cluster (-50%)
• Presto cluster is smaller than Memsql-driven one (30%)
SIMPLIFIED MAINTENANCE
• Auto recovery and scaling
• No wakeups over night

65. THANK YOU