The document presents the keynote by Lucas Ceballos at DataOps BCN 2019, focusing on the importance of size in building a serverless petabyte-scale database. It covers topics like digital growth marketing, click attribution windows, and various database alternatives such as Redis and DynamoDB. The presentation also discusses technical implementations and cost considerations involved in managing high traffic data environments.

1. Lucas Ceballos
Size does matter@ Dataops BCN 2019
1

2. Lucas Ceballos
Size does matter@ Dataops BCN 2019
2
Lucas Ceballos
linkedin.com/in/lucas-ceballos
CTO @ Headway Born @ ArgentinaCTO @ Smadex

3. Lucas Ceballos
Size does matter@ Dataops BCN 2019
3
SIZEDOES MATTER

4. Lucas Ceballos
Size does matter@ Dataops BCN 2019
3
SIZEDOES MATTER
HOW TO BUILD A SERVERLESS
PETA-BYTE SCALE DATABASE

5. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 4
CHAPTER 1
CHAPTER 2
CHAPTER 3-5
CHAPTER 6
CONTEXT
PROBLEM
ATTEMPTS
Q&A

6. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 5
CONTEXT
CHAPTER 1

7. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 6
D I G I T A L A N D A P P G R O W T H M A R K E T I N G C O M P A N Y

8. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 7
“We help our clients to grow
through digital advertising”

9. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 8
DeliverYou

10. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 9
NEWUSERS
USERACQUISITION
NEWORDERS
USERRE-ENGAGEMENT

11. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE

12. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER
Click IDImpression ID
10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE

13. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t

14. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min 1-∞min

15. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min 1-∞min

16. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 10
IMPRESSION
+50K / SEG
CLICK
+5K / SEG
CONVERSION
+15K / HR
AD LIFE CYCLE
△t△t
<1min AttributionWindow

17. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 11
Intheworldofdigitalattribution,anattributionwindow
isatimeperiodduringwhicheligibleconversionevents
canbeclaimed.Theconversioneventscanbeanapp
install,addedtocart,orpurchase.
ATTRIBUTIONWINDOW:

18. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

19. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

20. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

21. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

22. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 12
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

23. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 13
1808210149a523932a4ceaed
CLICK ID
Date &Time (up to minute) Random Hexadecimal String

24. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 14
{“_id”:"1808210149a523932a4ceaed","version":5,"earn":
0,"spent":0,"converted":false,"campaignId":
280392,"categoryGroup":"mainstream","publisherId":"1046
6_581_34_863796162656735232","adSlot":null,"siteId":nul
l,"sellerId":null,"formatId":null,"sourceUid":"59af4b01
e9cbfd600085ca9df0e709ed","appId":"","bundleId":null,"d
eviceId":null,"ip":"170.79.119.28","referer":null,"user
Agent":"Mozilla/5.0(Linux;Android8.0;N820Build/
KOT49H)AppleWebKit/537.36(KHTML,likeGecko)Chrome/
60.0.3112MobileSafari/
537.36","countryCode":"BR","countryName":"Brazil","regi
on":"RioGrandedoNorte","city":"Natal","isp":"JDSMedeiro
sProvedorDeInternet-
Me","connectionType":"wifi","latitude":-5.8002,"longitu
de":-35.2107,"device":"N820","deviceType":"mobile","bro
wser":"Chrome","browserVersion":"60.0.3112","os":"Andro
id","osVersion":"8.0","payout":3.23,"cost":
1.87,"share":0,"impressions":0,"clicks":
1,"conversions":
0,"createdAt":"20180821014947","updatedAt":"20180821014
947","date":"2018-08-21T04:49:47.205Z","datetime":"2018
-08-2101:49:47","dateUpdated":"2018-08-21T04:49:47.205Z
","callbackUrl":"http://click.foo-bar.com/fass/?
atoken=f8ad6505fa1b53fc948c8543dc0afc7a05296200&version
=1&c=tracker.postback&click_id={sid}"}
CLICK TAXONOMY
JSON
FORMAT
1KB
NO COMP.
650B
GZIP

25. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 15
BIGGERATTRIBUTIONWINDOW
LONGERCLICK RETENTION PERIOD
=

26. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 16
THEPROBLEM
CHAPTER 2

27. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 17
UNDER THE HOOD
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Conversions
W R
attribution window

28. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 18
500.000.000
x 650
x 30
Dailyclicks
Avg. click size
Days/month
~9 TB/month

29. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 19
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

30. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 19

31. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 19
COST
2
1 3
LATENCYSIZE

32. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 20
FIRSTATTEMPT
CHAPTER 3

33. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

34. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

35. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 21
FIRST ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

36. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 22

37. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 22

38. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 23
2NDATTEMPT
CHAPTER 4

39. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 24
2ND ATTEMPT
EC2
EC2
EC2
EC2
EC2
EC2
EC2
EC2
Incoming
Clicks
Incoming
Events
W R

40. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 25
FASTSCALABLE FIXED SIZE COSTLY
PROS CONS

41. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 26
10TB
x 3
x 1
x 102,4
500M clicks/month
Mongo replicas
Months Retention
GP2 cost/tb/month
~6500 USD/month

42. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 27
ANEWHOPE
CHAPTER 4

43. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 28
REQUIREMENTS
KEYVALUE CHEAP TB SCALE FAST

44. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA

45. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory

46. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 29
ALTERNATIVES
REDIS MEMCACHED DYNAMODB CASSANDRA
In Memory Costly

47. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 30

48. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 30

49. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 31
THINKOUTOFTHEBOX
CHAPTER 5

50. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window

51. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
24H

52. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
24H
90%<24H
10%>24H

53. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
1H

54. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 32
t1 t2
attribution window
1H
70%<1H
30%>1H

55. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 33
“DIVIDEANDCONQUER”
FINAL CHAPTER

56. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 34
H T
Last hourclicks
C LD
> 1 hourclicks
FREQUENTLY
ACCESSED
SMALL
NUMBER
INFREQUENTLY
ACCESSED
HUGE
NUMBER
+ +
= =

57. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 34
H T
Last hourclicks
C LD
> 1 hourclicks
+ +
= =
Fridge

58. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 35
UNDER THE HOOD
EC2
EC2
EC2
EC2
Incoming
Clicks
EC2
EC2
EC2
EC2
Incoming
Conversions
H T H T
C LDC LD
Fridge

59. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 36
H T
EC2
EC2
EC2
EC2
Incoming
Clicks
CSV
EC2
EC2
EC2
EC2
Incoming
Conversions
(450b)
1 hour~ 9GB
Elasticache
HADES

60. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 37
EC2
EC2
EC2
EC2
Incoming
Clicks
C LD
JSON
1m BUFFERING
PARQUET
Fridge
ALFREDDO

61. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 38
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

62. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 38
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

63. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 38
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

64. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 39
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge

65. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 39
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed

66. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 39
JSON
PARQUET
Athena
PRESTO dt=2018-08-21-01-51
dt=2018-08-21-01-50
dt=2018-08-21-01-49
dt=2018-08-21-01-48
dt=2018-08-21-01-47
Fridge
1808210149a523932a4ceaed

67. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 40
EC2
EC2
EC2
EC2
Incoming
Conversions
C LD
SQS Athena Clicks
H TSQS
Retry
Fridge

68. Lucas Ceballos
Size does matter@ Dataops 2019
SIZEDOESMATTER 41
443
+ 300
+ 290
+ 140
S3 Storage
Elasticache
Athena
Lambdas
~1173 USD/month

69. Lucas Ceballos
Size does matter@ Dataops BCN 2019
42
www.headwaydigital.comlucas@headwaydigital.com
Moltes Gracies!Q&A
lucas-ceballos
We’re hiring!