Data Lessons Learned at Scale
•
1 like•934 views
Charlie Reverte, VP of Engineering at AddThis, discusses lessons learned from processing large-scale web data. AddThis processes data from 14 million domains, including 100 billion monthly page views and 50,000 events per second. Reverte outlines challenges around distributed ID generation, counting unique values, joining distributed data, sampling large datasets, and deploying systems that invalidate over 1.4 billion browser caches. He advocates for loose coupling between systems using approaches like Kafka for asynchronous event logging. Reverte also discusses techniques for columnar compression, tunable quality of service, and open sourcing Hydra, AddThis' custom processing system optimized for real-time data.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Data Lessons Learned at Scale
Similar to Data Lessons Learned at Scale (20)
Recently uploaded
Recently uploaded (20)
Data Lessons Learned at Scale
- 1. Charlie Reverte VP Engineering @numbakrrunch Data Lessons Learned at Scale
- 2. @numbakrrunch Topic Half of the work that it takes to do data science is plumbing and wrangling Here are some lessons we’ve learned..
- 3. @numbakrrunch About AddThis We make tools for websites
- 4. @numbakrrunch Our Data We process website data ● Visitation ● Sharing ● Following ● Content Classification And use it to improve the site ● Content Recommendation ● Personalization ● Analytics
- 5. @numbakrrunch At Scale... ● 14 million domains ● 100 billion views/month ● 50k events/sec ● 160k concurrent firewall sessions ● 500k unique ganglia metrics
- 6. @numbakrrunch Distributed ID Generation ● Session IDs are generated in the browser ● We concatenate time and a random value Hex: 4f6934b6f54bd7c1 Base64: T2k0to403VS ● Time-bounded probabilistic uniqueness ○ (m 2) / n = 0.142 collisions/sec (at 35k rq/sec) ● Naturally time ordered, built-in DoB Compare to Twitter Snowflake https://github.com/twitter/snowflake/ time rand 63 31 0
- 7. @numbakrrunch Counting Things ● Cardinality ● Set membership ● Top-k elements ● Frequency ● Estimate when possible ● Sample when possible ● Often streaming vs. batch ● Mergeability is a big plus ○ Distributed counting ○ Checkpointing Stream-lib: https://github.com/clearspring/stream-lib http://highlyscalable.wordpress. com/2012/05/01/probabilistic-structures-web- analytics-data-mining/
- 8. @numbakrrunch Joining Data ● Value of data increases with higher dimensionality ○ Geo, user profile, page attributes, external data ● Join and de-normalize data when you ingest ○ Disk is cheap ● Join your data in client-side storage ○ Browsers as a lossy distributed database ● Oceans of data in the cloud.. “The value is in the join” (or something like that) https://github.com/stewartoallen
- 9. @numbakrrunch Sharding and Sampling ● Choose your shard keys wisely ○ High cardinality field to reduce lumpiness ○ What do you need to co-locate ○ Storage is cheap, multiple copies? ● Shards also useful for sampling ○ Complete data subsets ● Can yield statistical significance ○ Depending on the question
- 10. Deployment ● Continuous Deploy? ● Deploying our javascript costs $3k ○ Have to invalidate 1.4B browser caches ○ Several hours to flush to browsers (clench) ● 2PB of CDN data served per month ● Have DDOSed ourselves ○ Very interesting bugs ● Simulation is weak ○ The internet is a dirty place ○ Embrace incremental deploys
- 11. @numbakrrunch The Log Jay Kreps: “Real-time data’s unifying abstraction” ● Centralized logging ● Loosely coupled consumers Divide your dependencies: ● Synchronous - 0mq ● Asynchronous - Kafka Distributed event logging ● Does determinism matter? Log format durability? ● Protobuf? http://bit.ly/thelog
- 12. @numbakrrunch Columnar Compression ● Columnar storage techniques for row data ● Better compressor efficiency ● Different compressors per column ● >20% size savings ● https://github.com/addthis/columncompressor ○ by @abramsm Time IP UID URL Geo Time IP UID URL Geo Input Data Stored Data Block Size
- 13. @numbakrrunch Tunable QoS Cassandra URL Store ● We scrape and classify 20M URLs/day ● 750 million active records ● 2.2B reads/day ● Variable cache TTLs ○ Depending on write rate per record ● Global TTL knob ○ Turn up to reduce load for maintenance ○ Turn down to improve responsiveness 6 CDN cache
- 14. Hydra Our custom processing system Optimized for real-time data Just open sourced: https://github.com/addthis/hydra Go see @csby’s talk Great Hall North @3:55pm
- 15. @numbakrrunch Summary ● Are you more like the post office or the bank? ● Look for good-enough answers ● Fight your nerd tendency for perfect ○ I’m still struggling with this