From talk given on June 10, 2020 at the DevTalks Reimagined conference. ABSTRACT: Serverless is an amazing beast of a technology. With it, you can quickly build and deploy incredible systems. You get out-of-the-box scalability and flexibility.Nevertheless, with great power comes great(er) cost!In this talk, you’ll learn about building a huge data pipeline using Serverless architecture, and how to tame the beast.After this session, you will understand the pitfalls to avoid and the great powers to exploit.

1. Opher Dubrovsky & Avner Tamir
June 10, 2020
DevTalks Reimagined
Taming the Beast !!

2. 3
● Amazing powers of serverless
● Pitfalls
● How not to get burnt

3. 4
A system - DataOut
250 Billion events / day
Serverless

4. 5
Who We Are
Big Data dev lead
Loves data and challenges
Opher
Dubrovsky
Avner
Tamir
Director of Cloud
Operations
Automate everything !

5. 6
Nielsen Identity Engine
● eXelate (startup) acquired by Nielsen in 2015
● A Data company
● Machine learning models for insights
● Data used for (its a DMP):
○ Targeting (campaigns)
○ Business decisions

6. 7
Nielsen Identity Engine in numbers
~300B events/day
~60TB/day
S3
~3000 nodes/day
$500,000/month

7. 8
Outline
○ Serverless
○ Problem & solution architecture
○ Pitfalls we encountered & solutions
○ Q&A

8. Serverless

9. 10
Running Code on Servers
Some Code
__________
__________
__________
__________
Steps to Run
1. Deploy & Install
2. Start server
3. Run code
4. Maintain
5. Scale
Costs $$
Hassle
Costs $$
Hassle

10. 11
Serverless
Some Code
__________
__________
__________
__________
Steps to Run
1. Deploy
2. Start server
3. Run code
4. Stop server
5. Delete code
6. Reuse server for
something elseBenefits:
● Save $$
● Less management hassle
● Automatic scaling
● Highly parallelizable

11. 12
Unexpected Benefit
Code Improvements === Cost Reduction
improvements
$$

12. 13
Our System: DataOut

13. 14
Goals:
● BigData pipeline - 24/7 stream
● Sends data to ad platforms
● Daily data:
○ 200 platforms * 400,000 segments * millions of users
○ == 120 Billion Events / Day
Has to be:
● Scalable
● Cost effective
● Meet SLAs
● Transparent

14. 15
Our Solution - Serverless Infrastructure
Input Output

15. 16
Our Solution - Serverless Infrastructure
Input Output

16. 17
Our Solution - Serverless Infrastructure
Input Output

17. 18
What Makes It Attractive ?
Data
Processing
Costs

18. 19
Classical Server Systems Are
Built for the worst case Many hours are like this
Wasted $$$$$$

19. 20
What Makes It Attractive ?
Data
Processing
Costs
Savings

20. 21
Benefits of Serverless
● Scale up / down
● Run in parallel (short processing time, SLA)
● Save when not in use
● Optimizations translate to savings
● Simple to maintain

21. 22
We Got
Superpowers

22. 23
Some Numbers
● Average Day - 120 Billion events
● Highest Day - 250 Billion events
○ 17 million files
○ 55 TB of data
● Hourly Scaling -
○ 1 TB ←→ 6 TB
○ Scales up/down by a factor of 6 !!

23. 24
BUT - Life is Not That Easy

24. 25
Pitfall 1 - Clash of clans
● Easy is risky, easy and manual is crazy
● Too many breaking changes
● Too many hands
● No orchestration

25. 26
First Shot Solution
● Blocking permissions
● BUT - CloudOps became a bottleneck
Created tension and ARGUMENTS !!

26. 27
Second Shot
● Automate anything and everything…..
● Give developers access to run all automation tools
Reduced tensions - developers could now work !!

27. 28
Clash of Clans - Peace Conference
● Created a shared task force (Cloud & Data engineers)
● Define & Implement holistic solution
○ Tools (Terraform & Jenkins)
○ Processes (CI & CD)
○ Deployments (Canary, gradual, full)

28. 29
Clash of Clans - New Era
● Developers own production!
● CloudOps maintain tools and approve changes

29. 30
Pitfall 2 - Costs
● Optimization will save $$$
● BUT - what is the reverse of optimization ?
BUGS !!

30. 31
BUG 1 - Reprocessing

31. 32
BUG 1 - Reprocessing 38 Times !!
● Monitor tasks
● If task is “in-progress” for too long - we reprocess
Time
Task
Task
Task
Task
Task
Task
Task Processing Time
Reprocess
Threshold
Reprocess
Tasks

32. 33
The Bug
● We deployed a version that had a lower threshold
● Many normal tasks were reprocessed (38 times)
Time
Task
Task
Task
Task
Task
Task
Task Processing Time
Reprocess
Threshold
Reprocess
Tasks
Task
Task

33. 34
Result - Costs

34. 35
Total $ Damage - $3413
Item Cost
Total cost of 4 bug days $ 7,442
Average 4 day costs $ 4,029
Net $ damage $ 3,413

35. 36
Solution for Reprocessing
● Task validations to prevent multi processing
● Stick to your procedures !!!

36. 37
BUG 2 - Lambda Throttling

37. 38
BUG 2 - Lambda Throttled Queue
SNS
Work
Manager
Workers
FanOut
DB
High load → causing queued invocations

38. 39
BUG 2 - Lambda Throttled Queue
SNS
Increased
concurrency
Workers
FanOut

39. 40
AWS Cost Report

40. 41
Total $ Damage - $2555
Item Cost
Total cost of 5 bug days $ 6,055
Average day costs (of 5 days) $ 3,500
Net $ damage $ 2,555

41. 42
Solution for Escalating Costs
● New monitoring & alerts on:
○ number of invocations
○ Measure costs in real time and alert

42. 43
Pitfall 3 - DDoS Attack on Partners
Our system scales up and down as traffic comes in !!
BUT - the world does not work like that !!

43. 44
Email from Partner
Subject: You’re killing our systems
Message: can you guys check…….
……….
Data going up 1 → 6x over the day

44. 45
Solution for DDoS
Throttling
Adslfjk alsdf al;sdjf aljfds
j;laj fda;sdflj a;lksdfj
;alksdjf ;alksdjf a;lkjdsf
;alksjdf a;lsdkfj a;lskdjf
;alsdkfj a;lsdkjf a;ldjkf
a;ldkjf ;alsdj a;ldkjf a;ldjkf
Adslfjk alsdf al;sdjf aljfds j;laj fda;sdflj a;lksdfj ……….
We can throttle on:
● MB/sec
● Items/sec
● HTTP requests/sec , connections, …..

45. 46
Summary - Avoiding the Pitfalls
● Measure all risk dimensions
● In our system they are:
○ Cost
○ Throughput
○ Load
● Quickly alert on deviations
● Build detailed dashboards on the above
● Create circuit breakers in the system based on alerts !!

46. 47
Summary - Serverless
● Amazing powers and scale
● Great cost optimizations
● Parallelization
BUT - BEWARE of escalating costs
and be ready for the UNEXPECTED !!

47. Copyright © 2017 The Nielsen Company. Confidential and proprietary.
Questions