YouTube video: https://www.youtube.com/watch?v=PliThXFP3dM Yes, it matters! (we are talking about databases, don’t worry). Learn how to combine some of the most amazing AWS services to build a petabyte scale database able to store and retrieve billions of records per day without spending a lot of money. Best of all, we won’t need a single server running!

1. Lucas Ceballos
Size does matter@ Dataops BCN 2019
1

2. Lucas Ceballos
Size does matter@ Dataops BCN 2019
2
Lucas Ceballos
linkedin.com/in/lucas-ceballos
CTO @ Headway Born @ ArgentinaCTO @ Smadex

3. Lucas Ceballos
Size does matter@ Dataops BCN 2019
3
SIZEDOES MATTER

4. Lucas Ceballos
Size does matter@ Dataops BCN 2019
3
SIZEDOES MATTER
HOW TO BUILD A SERVERLESS
PETA-BYTE SCALE DATABASE

5. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 4
CHAPTER 1
CHAPTER 2
CHAPTER 3-5
CHAPTER 6
CONTEXT
PROBLEM
ATTEMPTS
Q&A

6. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 5
CONTEXT
CHAPTER 1

7. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 6
D I G I T A L A N D A P P G R O W T H M A R K E T I N G C O M P A N Y

8. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 7
“We help our clients to grow
through digital advertising”

9. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 8
DeliverYou

10. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 9
NEWUSERS
USERACQUISITION
NEWORDERS
USERRE-ENGAGEMENT

11. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE

12. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER
Click IDImpression ID
10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE

13. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t

14. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min 1-∞min

15. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min 1-∞min

16. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min AttributionWindow

17. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 11
Intheworldofdigitalattribution,anattributionwindow
isatimeperiodduringwhicheligibleconversionevents
canbeclaimed.Theconversioneventscanbeanapp
install,addedtocart,orpurchase.
ATTRIBUTIONWINDOW:

18. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

19. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

20. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

21. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

22. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

23. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 13
1808210149a523932a4ceaed
CLICK ID
Date &Time (up to minute) Random Hexadecimal String

24. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 14
{“_id”:"1808210149a523932a4ceaed","version":5,"earn":
0,"spent":0,"converted":false,"campaignId":
280392,"categoryGroup":"mainstream","publisherId":"1046
6_581_34_863796162656735232","adSlot":null,"siteId":nul
l,"sellerId":null,"formatId":null,"sourceUid":"59af4b01
e9cbfd600085ca9df0e709ed","appId":"","bundleId":null,"d
eviceId":null,"ip":"170.79.119.28","referer":null,"user
Agent":"Mozilla/5.0(Linux;Android8.0;N820Build/
KOT49H)AppleWebKit/537.36(KHTML,likeGecko)Chrome/
60.0.3112MobileSafari/
537.36","countryCode":"BR","countryName":"Brazil","regi
on":"RioGrandedoNorte","city":"Natal","isp":"JDSMedeiro
sProvedorDeInternet-
Me","connectionType":"wifi","latitude":-5.8002,"longitu
de":-35.2107,"device":"N820","deviceType":"mobile","bro
wser":"Chrome","browserVersion":"60.0.3112","os":"Andro
id","osVersion":"8.0","payout":3.23,"cost":
1.87,"share":0,"impressions":0,"clicks":
1,"conversions":
0,"createdAt":"20180821014947","updatedAt":"20180821014
947","date":"2018-08-21T04:49:47.205Z","datetime":"2018
-08-2101:49:47","dateUpdated":"2018-08-21T04:49:47.205Z
","callbackUrl":"http://click.foo-bar.com/fass/?
atoken=f8ad6505fa1b53fc948c8543dc0afc7a05296200&version
=1&c=tracker.postback&click_id={sid}"}
CLICK TAXONOMY
JSON
FORMAT
1KB
NO COMP.
650B
GZIP

25. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 15
BIGGERATTRIBUTIONWINDOW
LONGERCLICK RETENTION PERIOD
=

26. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 16
THEPROBLEM
CHAPTER 2

27. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 17
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

28. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 18
500.000.000
x 650
x 30
Dailyclicks
Avg. click size
Days/month
~9 TB/month

29. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 19
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

30. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 19

31. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 19
COST
2
1 3
LATENCYSIZE

32. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 20
FIRSTATTEMPT
CHAPTER 3

33. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

34. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

35. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

36. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 22

37. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 22

38. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 23
2NDATTEMPT
CHAPTER 4

39. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 24
2ND ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

40. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 25
FASTSCALABLE FIXED SIZE COSTLY
PROS CONS

41. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 26
10TB
x 3
x 1
x 102,4
500M clicks/month
Mongo replicas
Months Retention
GP2 cost/tb/month
~6500 USD/month

42. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 27
ANEWHOPE
CHAPTER 4

43. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 28
REQUIREMENTS
KEYVALUE CHEAP TB SCALE FAST

44. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA

45. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory

46. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory Costly

47. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 30

48. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 30

49. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 31
THINKOUTOFTHEBOX
CHAPTER 5

50. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window

51. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
24H

52. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
24H
90%<24H
10%>24H

53. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
1H

54. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
1H
70%<1H
30%>1H

55. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 33
“DIVIDEANDCONQUER”
FINAL CHAPTER

56. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 34
H T
Last hourclicks
C LD
> 1 hourclicks
FREQUENTLY
ACCESSED
SMALL
NUMBER
INFREQUENTLY
ACCESSED
HUGE
NUMBER
+ +
= =

57. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 34
H T
Last hourclicks
C LD
> 1 hourclicks
+ +
= =
Fridge

58. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 35
UNDER THE HOOD
EC2
EC2
EC2
EC2
Incoming
Clicks
EC2
EC2
EC2
EC2
Incoming
Conversions
H T H T
C LDC LD
Fridge

59. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 36
H T
EC2
EC2
EC2
EC2
Incoming
Clicks
CSV
EC2
EC2
EC2
EC2
Incoming
Conversions
(450b)
1 hour~ 9GB
Elasticache
HADES

60. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 37
EC2
EC2
EC2
EC2
Incoming
Clicks
C LD
JSON
1m BUFFERING
PARQUET
Fridge
ALFREDDO

61. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 38
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

62. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 38
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

63. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 38
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

64. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 39
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge

65. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 39
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed

66. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 39
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed

67. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 40
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

68. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 41
443
+ 300
+ 290
+ 140
S3 Storage
Elasticache
Athena
Lambdas
~1173 USD/month

69. Lucas Ceballos
Size does matter@ Dataops BCN 2019
42
www.headwaydigital.comlucas@headwaydigital.com
Moltes Gracies!Q&A
lucas-ceballos
We’re hiring!