Yes, it matters! (we are talking about databases, don’t worry). Learn how to combine some of the most amazing AWS services to build a petabyte scale database able to store and retrieve billions of records per day without spending a lot of money. Best of all, we won’t need a single server running!

1. Lucas Ceballos
Size does matter@ nerdear.la 2018
1

2. Lucas Ceballos
Size does matter@ nerdear.la 2018
2
SIZEDOES MATTER

3. Lucas Ceballos
Size does matter@ nerdear.la 2018
2
SIZEDOES MATTER
HOW TO BUILD A SERVERLESS
PETA-BYTE SCALE DATABASE

4. Lucas Ceballos
Size does matter@ nerdear.la 2018
3
Lucas Ceballos
linkedin.com/in/lucas-ceballos
Digital HouseHeadway UTN

5. Lucas Ceballos
Size does matter@ nerdear.la 2018
3
Lucas Ceballos
linkedin.com/in/lucas-ceballos
Digital HouseHeadway UTN

6. Lucas Ceballos
Size does matter@ nerdear.la 2018
4
DOITLATER,DOITBETTER
Como procesar mas de 100k eventos por segundo y no morir en el intento

7. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 5
CONTEXT
CHAPTER 1

8. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 6
P O W E R E D B Y T E C H N O L O G Y

9. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 7
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE

10. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER
Click IDImpression ID
7
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE

11. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 7
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t

12. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 7
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min 1-∞min

13. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 7
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min 1-∞min

14. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 7
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min AttributionWindow

15. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 8
THEPROBLEM
CHAPTER 2

16. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 9
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

17. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 9
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

18. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 9
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

19. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 9
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

20. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 9
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

21. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 10
{“_id”:"1808210149a523932a4ceaed","version":5,"earn":
0,"spent":0,"converted":false,"campaignId":
280392,"categoryGroup":"mainstream","publisherId":"1046
6_581_34_863796162656735232","adSlot":null,"siteId":nul
l,"sellerId":null,"formatId":null,"sourceUid":"59af4b01
e9cbfd600085ca9df0e709ed","appId":"","bundleId":null,"d
eviceId":null,"ip":"170.79.119.28","referer":null,"user
Agent":"Mozilla/5.0(Linux;Android8.0;N820Build/
KOT49H)AppleWebKit/537.36(KHTML,likeGecko)Chrome/
60.0.3112MobileSafari/
537.36","countryCode":"BR","countryName":"Brazil","regi
on":"RioGrandedoNorte","city":"Natal","isp":"JDSMedeiro
sProvedorDeInternet-
Me","connectionType":"wifi","latitude":-5.8002,"longitu
de":-35.2107,"device":"N820","deviceType":"mobile","bro
wser":"Chrome","browserVersion":"60.0.3112","os":"Andro
id","osVersion":"8.0","payout":3.23,"cost":
1.87,"share":0,"impressions":0,"clicks":
1,"conversions":
0,"createdAt":"20180821014947","updatedAt":"20180821014
947","date":"2018-08-21T04:49:47.205Z","datetime":"2018
-08-2101:49:47","dateUpdated":"2018-08-21T04:49:47.205Z
","callbackUrl":"http://click.foo-bar.com/fass/?
atoken=f8ad6505fa1b53fc948c8543dc0afc7a05296200&version
=1&c=tracker.postback&click_id={sid}"}
CLICK TAXONOMY
JSON
FORMAT
1KB
NO COMP.
650B
GZIP

22. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 11
1808210149a523932a4ceaed
CLICK ID
Date &Time (up to minute) Random Hexadecimal String

23. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 12
500.000.000
x 650
x 30
Dailyclicks
Avg. click size
Days/month
~9 TB/month

24. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 13
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

25. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 13

26. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 13
COST
2
1 3
LATENCYSIZE

27. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 14
FIRSTATTEMPT
CHAPTER 3

28. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 15
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

29. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 15
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

30. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 15
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

31. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 16
10TB
x 3
x 1
x 102,4
500M clicks/month
Mongo replicas
Months Retention
GP2 cost/tb/month
~3000 USD/month

32. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 17

33. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 18
ANEWHOPE
CHAPTER 4

34. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 19
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA

35. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 19
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory

36. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 19
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory Costly

37. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 20
THINKOUTOFTHEBOX
CHAPTER 5

38. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 21
t1 t2
attribution window

39. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 21
t1 t2
attribution window
24H

40. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 21
t1 t2
attribution window
24H
90%<24H
10%>24H

41. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 21
t1 t2
attribution window
1H

42. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 21
t1 t2
attribution window
1H
70%<1H
30%>1H

43. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 22
“LAGRIETA”
FINAL CHAPTER

44. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 23
H T
Last hourclicks
C LD
> 1 hourclicks
FREQUENTLY
ACCESSED
SMALL
NUMBER
UNFREQUENTLY
ACCESSED
HUGE
NUMBER
+ +
= =

45. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 23
H T
Last hourclicks
C LD
> 1 hourclicks
+ +
= =
Fridge

46. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 24
UNDER THE HOOD
EC2
EC2
EC2
EC2
Incoming
Clicks
EC2
EC2
EC2
EC2
Incoming
Conversions
H T H T
C LDC LD
Fridge

47. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 25
H T
EC2
EC2
EC2
EC2
Incoming
Clicks
CSV
EC2
EC2
EC2
EC2
Incoming
Conversions
(450b)
1 hour~ 9GB
Elasticache

48. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 26
EC2
EC2
EC2
EC2
Incoming
Clicks
C LD
JSON
1m BUFFERING
PARQUET
Fridge

49. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 27
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

50. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 27
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

51. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 27
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

52. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 28
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge

53. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 28
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed

54. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 28
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed

55. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 29
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

56. Lucas Ceballos
Size does matter@ nerdear.la 2018
SIZEDOESMATTER 30
443
+ 300
+ 290
+ 140
S3 Storage
Elasticache
Athena
Lambdas
~1173 USD/month

57. Lucas Ceballos
Size does matter@ nerdear.la 2018
31
www.headwaydigital.comlucas@headwaydigital.com
Gracias!Q&A
lucas-ceballos
We’re hiring!