Download to read offline
Size does matter - How to build a serverless petabyte scale database
- 1. Lucas Ceballos Size doesmatter@ nerdear.la 2018 1
- 2. Lucas Ceballos Size doesmatter@ nerdear.la 2018 2 SIZEDOES MATTER
- 3. Lucas Ceballos Size doesmatter@ nerdear.la 2018 2 SIZEDOES MATTER HOW TO BUILD A SERVERLESS PETA-BYTE SCALE DATABASE
- 4. Lucas Ceballos Size doesmatter@ nerdear.la 2018 3 Lucas Ceballos linkedin.com/in/lucas-ceballos Digital HouseHeadway UTN
- 5. Lucas Ceballos Size doesmatter@ nerdear.la 2018 3 Lucas Ceballos linkedin.com/in/lucas-ceballos Digital HouseHeadway UTN
- 6. Lucas Ceballos Size doesmatter@ nerdear.la 2018 4 DOITLATER,DOITBETTER Como procesar mas de 100k eventos por segundo y no morir en el intento
- 7. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 5 CONTEXT CHAPTER 1
- 8. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 6 P O W E R E D B Y T E C H N O L O G Y
- 9. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 7 IMPRESSION +50K / SEG CLICK +5K / SEG CONVERSION +15K / HR AD LIFE CYCLE
- 10. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER Click IDImpression ID 7 IMPRESSION +50K / SEG CLICK +5K / SEG CONVERSION +15K / HR AD LIFE CYCLE
- 11. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 7 IMPRESSION +50K / SEG CLICK +5K / SEG CONVERSION +15K / HR AD LIFE CYCLE △t△t
- 12. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 7 IMPRESSION +50K / SEG CLICK +5K / SEG CONVERSION +15K / HR AD LIFE CYCLE △t△t <1min 1-∞min
- 13. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 7 IMPRESSION +50K / SEG CLICK +5K / SEG CONVERSION +15K / HR AD LIFE CYCLE △t△t <1min 1-∞min
- 14. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 7 IMPRESSION +50K / SEG CLICK +5K / SEG CONVERSION +15K / HR AD LIFE CYCLE △t△t <1min AttributionWindow
- 15. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 8 THEPROBLEM CHAPTER 2
- 16. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 9 UNDER THE HOOD EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Conversions W R attribution window
- 17. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 9 UNDER THE HOOD EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Conversions W R attribution window
- 18. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 9 UNDER THE HOOD EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Conversions W R attribution window
- 19. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 9 UNDER THE HOOD EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Conversions W R attribution window
- 20. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 9 UNDER THE HOOD EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Conversions W R attribution window
- 21. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 10 {“_id”:"1808210149a523932a4ceaed","version":5,"earn": 0,"spent":0,"converted":false,"campaignId": 280392,"categoryGroup":"mainstream","publisherId":"1046 6_581_34_863796162656735232","adSlot":null,"siteId":nul l,"sellerId":null,"formatId":null,"sourceUid":"59af4b01 e9cbfd600085ca9df0e709ed","appId":"","bundleId":null,"d eviceId":null,"ip":"170.79.119.28","referer":null,"user Agent":"Mozilla/5.0(Linux;Android8.0;N820Build/ KOT49H)AppleWebKit/537.36(KHTML,likeGecko)Chrome/ 60.0.3112MobileSafari/ 537.36","countryCode":"BR","countryName":"Brazil","regi on":"RioGrandedoNorte","city":"Natal","isp":"JDSMedeiro sProvedorDeInternet- Me","connectionType":"wifi","latitude":-5.8002,"longitu de":-35.2107,"device":"N820","deviceType":"mobile","bro wser":"Chrome","browserVersion":"60.0.3112","os":"Andro id","osVersion":"8.0","payout":3.23,"cost": 1.87,"share":0,"impressions":0,"clicks": 1,"conversions": 0,"createdAt":"20180821014947","updatedAt":"20180821014 947","date":"2018-08-21T04:49:47.205Z","datetime":"2018 -08-2101:49:47","dateUpdated":"2018-08-21T04:49:47.205Z ","callbackUrl":"http://click.foo-bar.com/fass/? atoken=f8ad6505fa1b53fc948c8543dc0afc7a05296200&version =1&c=tracker.postback&click_id={sid}"} CLICK TAXONOMY JSON FORMAT 1KB NO COMP. 650B GZIP
- 22. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 11 1808210149a523932a4ceaed CLICK ID Date &Time (up to minute) Random Hexadecimal String
- 23. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 12 500.000.000 x 650 x 30 Dailyclicks Avg. click size Days/month ~9 TB/month
- 24. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 13 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Events W R
- 25. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 13
- 26. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 13 COST 2 1 3 LATENCYSIZE
- 27. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 14 FIRSTATTEMPT CHAPTER 3
- 28. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 15 FIRST ATTEMPT EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Events W R
- 29. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 15 FIRST ATTEMPT EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Events W R
- 30. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 15 FIRST ATTEMPT EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Incoming Clicks Incoming Events W R
- 31. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 16 10TB x 3 x 1 x 102,4 500M clicks/month Mongo replicas Months Retention GP2 cost/tb/month ~3000 USD/month
- 32. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 17
- 33. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 18 ANEWHOPE CHAPTER 4
- 34. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 19 ALTERNATIVES REDIS MEMCACHED DYNAMODB CASSANDRA
- 35. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 19 ALTERNATIVES REDIS MEMCACHED DYNAMODB CASSANDRA In Memory
- 36. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 19 ALTERNATIVES REDIS MEMCACHED DYNAMODB CASSANDRA In Memory Costly
- 37. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 20 THINKOUTOFTHEBOX CHAPTER 5
- 38. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 21 t1 t2 attribution window
- 39. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 21 t1 t2 attribution window 24H
- 40. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 21 t1 t2 attribution window 24H 90%<24H 10%>24H
- 41. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 21 t1 t2 attribution window 1H
- 42. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 21 t1 t2 attribution window 1H 70%<1H 30%>1H
- 43. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 22 “LAGRIETA” FINAL CHAPTER
- 44. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 23 H T Last hourclicks C LD > 1 hourclicks FREQUENTLY ACCESSED SMALL NUMBER UNFREQUENTLY ACCESSED HUGE NUMBER + + = =
- 45. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 23 H T Last hourclicks C LD > 1 hourclicks + + = = Fridge
- 46. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 24 UNDER THE HOOD EC2 EC2 EC2 EC2 Incoming Clicks EC2 EC2 EC2 EC2 Incoming Conversions H T H T C LDC LD Fridge
- 47. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 25 H T EC2 EC2 EC2 EC2 Incoming Clicks CSV EC2 EC2 EC2 EC2 Incoming Conversions (450b) 1 hour~ 9GB Elasticache
- 48. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 26 EC2 EC2 EC2 EC2 Incoming Clicks C LD JSON 1m BUFFERING PARQUET Fridge
- 49. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 27 EC2 EC2 EC2 EC2 Incoming Conversions C LD SQS Athena Clicks H TSQS Retry Fridge
- 50. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 27 EC2 EC2 EC2 EC2 Incoming Conversions C LD SQS Athena Clicks H TSQS Retry Fridge
- 51. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 27 EC2 EC2 EC2 EC2 Incoming Conversions C LD SQS Athena Clicks H TSQS Retry Fridge
- 52. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 28 JSON PARQUET Athena PRESTO dt=2018-08-21-01-51 dt=2018-08-21-01-50 dt=2018-08-21-01-49 dt=2018-08-21-01-48 dt=2018-08-21-01-47 Fridge
- 53. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 28 JSON PARQUET Athena PRESTO dt=2018-08-21-01-51 dt=2018-08-21-01-50 dt=2018-08-21-01-49 dt=2018-08-21-01-48 dt=2018-08-21-01-47 Fridge 1808210149a523932a4ceaed
- 54. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 28 JSON PARQUET Athena PRESTO dt=2018-08-21-01-51 dt=2018-08-21-01-50 dt=2018-08-21-01-49 dt=2018-08-21-01-48 dt=2018-08-21-01-47 Fridge 1808210149a523932a4ceaed
- 55. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 29 EC2 EC2 EC2 EC2 Incoming Conversions C LD SQS Athena Clicks H TSQS Retry Fridge
- 56. Lucas Ceballos Size doesmatter@ nerdear.la 2018 SIZEDOESMATTER 30 443 + 300 + 290 + 140 S3 Storage Elasticache Athena Lambdas ~1173 USD/month
- 57. Lucas Ceballos Size doesmatter@ nerdear.la 2018 31 www.headwaydigital.comlucas@headwaydigital.com Gracias!Q&A lucas-ceballos We’re hiring!