Talks about our journey to stream processing; as part of the talk I shared with the audience the specifics of the solution we built on AWS cloud and pointers for others to help them think through their own use-cases.

1. engineering.deltax.com
Building a Real-time Stream
Processing Pipeline
Akshay Surve, CTO DeltaX
akshay@deltax.com / @ak47surve
Hastag: #awsblr #meetup

2. engineering.deltax.com
● 12 years
○ Shipping Ideas, Making Mistakes, GTD
○ Marathons / Hackathons / *-athon :)
● Co-founded DeltaX in 2013
○ Ad-tech / Product Startup
○ 300+ advertisers across India, APAC and US.
About Me
2

3. engineering.deltax.com
Agenda
● Use-case
● Processing Models
● Old Batch Processing Architecture
○ Challenges
● Goals
● Moving Blocks for a Stream Processing Model
○ Kinesis Data Firehose
○ Amazon ElasticSearch
○ Amazon Athena
● Review New Stream Processing Architecture 3

4. engineering.deltax.com
Use-case
● Ad Tracking & Ad Serving
● Cloud Architecture
4

5. engineering.deltax.com
Use-case
- Ad Tracking & Ad-serving
5

6. engineering.deltax.com
Use-case
- Ad Tracking & Ad-serving
6

7. engineering.deltax.com
Use-case
- Ad Tracking & Ad-serving
Advertiser
7

8. engineering.deltax.com
Use-case
- Ad Tracking & Ad-serving
Event
8

9. engineering.deltax.com
Use-case
- Ad Tracking & Ad-serving
Timestamp
9

10. engineering.deltax.com
Use-case
- Cloud Architecture
10

11. engineering.deltax.com
● Batch Processing
● Stream Processing
Processing Models
11

12. engineering.deltax.com
● Batch Processing
Processing Models
Input OutputBatch Job(s)
12

13. engineering.deltax.com
● Stream Processing
Processing Models
Queue
Stream
Processor
Output
13

14. engineering.deltax.com
● Batch vs Stream
Processing Models
Batch Stream
High Latency Low Latency
Static Files Event Streams
Snapshot Continuous Window
14

15. engineering.deltax.com
Batch Processing
15

16. engineering.deltax.com
Batch Processing (Close-up)
16

17. engineering.deltax.com
Batch Processing (Challenges)
● Modelled around batch processing and not stream processing
● Ingesting JSON files in bulk isn’t natural for SQL - JSON parsing > SQL
tables
● Varied levels of aggregations - campaign, ad, device, geo + unique metrics
● Future roadmap - userid cookie pool across advertisers; exchange based
cookie matching, etc. become challenges in itself
17

18. engineering.deltax.com
● Stream processing as a paradigm suits our use case the best
● Easy to maintain or managed service in the cloud would be ideal
● Developer friendly and peace of mind was of utmost importance
● Being able to ingest streaming data and query summaries was important
● Good to have a way to run batch processing framework for machine learning,
data crunching, and analysis
Goals
18

19. engineering.deltax.com
● Amazon Athena
● Amazon Elasticsearch
● Kinesis Data Firehose
Moving Blocks
19

20. engineering.deltax.com
20

21. engineering.deltax.com
Amazon Athena
21

22. engineering.deltax.com
Amazon Athena
● Persistent Store
● DDL
● Query
22

23. engineering.deltax.com
Amazon Athena
● Persistent Store (AWS S3)
○ Text files, e.g., CSV, raw logs
○ Apache Web Logs, TSV files
○ JSON (simple, nested)
○ Compressed files
○ Columnar formats such as Apache Parquet & Apache ORC
23

24. engineering.deltax.com
Amazon Athena
● Persistent Store (AWS S3)
○ JSON events
24

25. engineering.deltax.com
● DDL (Apache Hive)
Amazon Athena
25

26. engineering.deltax.com
● DDL (Apache Hive)
Amazon Athena
26

27. engineering.deltax.com
Amazon Athena
● Query Engine (Presto query engine)
○ In Memory
○ ANSI SQL Compliant
27

28. engineering.deltax.com
● Query Engine (Presto query engine)
○ In Memory
○ ANSI SQL Compliant
Amazon Athena
28

29. engineering.deltax.com
● Query Engine (Presto query engine)
○ In Memory
○ ANSI SQL Compliant
Amazon Athena
29

30. engineering.deltax.com
● Serverless
● No spin-up time
● Query data directly from S3
● ANSI SQL
Amazon Athena (Advantages)
30

31. engineering.deltax.com
● Queries run fast
Amazon Athena (Advantages)
31

32. engineering.deltax.com
Amazon Elasticsearch
32

33. engineering.deltax.com
Amazon Elasticsearch
● ELK Stack (Searching, Log monitoring)
● Seamless Ingestion (Document-based model)
● Real-time queries (even during ingestion; 30s refresh window; immutability)
● Meant for search; Efficient for time-series (will discuss why?)
33

34. engineering.deltax.com
Amazon Elasticsearch
- Document that gets ingested
34

35. engineering.deltax.com
Elasticsearch (Internals)
● Elasticsearch Index
○ Inverted Index
○ Doc Values
35

36. engineering.deltax.com
Elasticsearch (Internals)
Deeper into an Elasticsearch Index
36

37. engineering.deltax.com
Elasticsearch (Internals)
● Deeper into an Elasticsearch Index - Inverted Index
○ The quick brown fox jumped over the lazy dog
○ Quick brown foxes leap over lazy dogs in summer
37

38. engineering.deltax.com
Elasticsearch (Internals)
Deeper into an Elasticsearch Index - Doc Values
● column-oriented fashion that is way more efficient for sorting and
aggregations
● Filesystem optimized
38

39. engineering.deltax.com
● Integration with AWS ecosystem
Amazon Elasticsearch (Advantages)
39

40. engineering.deltax.com
Amazon Elasticsearch (Advantages)
● Cluster Management (scale out/up)
40

41. engineering.deltax.com
Amazon Elasticsearch (Advantages)
● Monitoring & Alerts
41

42. engineering.deltax.com
Amazon Elasticsearch (Advantages)
● Snapshot Recovery / Backup to S3
● Elasticsearch Upgrades (could be made smoother)
42

43. engineering.deltax.com
Amazon Elasticsearch (Advantages)
● Integration with AWS ecosystem
● Cluster Management (scale out/up)
● Monitoring & Alerts
● Snapshot Recovery / Backup to S3
● Elasticsearch Upgrades
43

44. engineering.deltax.com
Kinesis Data Firehose
44

45. engineering.deltax.com
Kinesis
45

46. engineering.deltax.com
Kinesis Data Firehose
46

47. engineering.deltax.com
Kinesis Data Firehose
● Streaming Data Processing
● Multiple destinations - S3, Redshift, ES
● Intermediate Record transformations (using AWS Lambda) before delivery to
the destination
○ Ip2location
○ Enrich flow
○ Ua-parser
● Combine with Kinesis Analytics
47

48. engineering.deltax.com
Kinesis Data Firehose (source)
48

49. engineering.deltax.com
Kinesis Data Firehose (transformation)
49

50. engineering.deltax.com
Kinesis Data Firehose (destination)
50

51. engineering.deltax.com
Kinesis Data Firehose (ES config options)
51

52. engineering.deltax.com
Kinesis Data Firehose (ES destination)
Node.js (tracker) >
52

53. engineering.deltax.com
Kinesis Data Firehose (Advantages)
● Cloud Offering
53
Source: https://blog.ippon.tech/spark-storm-s
xd-comparison/

54. engineering.deltax.com
Kinesis Data Firehose (Advantages)
● Pluggability
54
Source: https://www.slideshare.net/AmazonWebServices/aws-reinvent-
2016-analyzing-streaming-data-in-realtime-with-amazon-kinesis-analytics-
bdm304

55. engineering.deltax.com
Kinesis Data Firehose
(Architecture)
55

56. engineering.deltax.com
Architecture
(Old vs New)
56

57. engineering.deltax.com
Stats
● Data: ~12 GB / day (peaks of 32 GB/day)
57

58. engineering.deltax.com
“The cloud is not a silver bullet”
silver bullet ~ noun
‘a simple and seemingly magical solution to a complicated problem’
Twitter - @ak47suve #awsblr #meetup
Email - akshay@deltax.com
Blog - engineering.deltax.com
58