Riak at shareaholic

  • 15,954 views
Uploaded on

Slides from my talk on using Riak at Shareaholic

Slides from my talk on using Riak at Shareaholic

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
15,954
On Slideshare
0
From Embeds
0
Number of Embeds
18

Actions

Shares
Downloads
42
Comments
2
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Riak @ Robby Grossman robby@shareaholic.com @freerobby
  • 2. AgendaShareaholic: Product & TechWhy Riak: The Search for a Big Data StoreTransitioning to RiakRiak Use CasesDeploying to EC2
  • 3. What’s ?
  • 4. Browser Tools
  • 5. Sharing Buttons
  • 6. Recommendations
  • 7. Social Analytics
  • 8. Monthly @ Thousands of developers hitting API Hundreds of thousands of publishers Tens of millions of shares & clicks Hundreds of millions of pageviews & events
  • 9. Tech @JRuby on Rails (via Torquebox)MySQL (Master, Read Slave)Elastic MapReduce (similar to Hadoop)RedisFormerly Mongo, Now Riak
  • 10. Why Not Mongo?Working set needs to fit in memoryGlobal write lock blocks all queriesdespite not having transactions/joinsStandbys not “hot”
  • 11. Why Riak?
  • 12. Next @Options: Goals: HBase Linear scalability Cassandra Full-text search Riak Flexible indexing Easier Devops
  • 13. HBasePros Cons Battle tested Complex Architecture High performance SPOFs Requires Hive for Indexing/Querying Expensive to deploy at small scale
  • 14. CassandraPros Cons Native secondary Known users all indices domain experts Linear scalability Search requires Lucene Tunable CAP Heavy Weight MapReduce
  • 15. RiakPros Cons Operationally simpler Multi-data center replication requires Linear scalability Enterprise product Integrated search leveldb puts high strain on CPU Secondary indices Tunable CAP Vector clocks solve time-sync problems
  • 16. From Mongo to Riak
  • 17. Migration GoalsNo time where database goes “offline”Product parity throughout migration
  • 18. Migration Process1. App writes to Mongo and Riak2. Verify data integrity3. Import historical data4. App reads from Riak5. Decommission Mongo
  • 19. Use Cases
  • 20. Share APISave shared contentUses MapReduce topopulate user dashboard
  • 21. RecommendationsSets of related pagesGenerated on-demand
  • 22. Publisher AnalyticsGenerated nightly via HadoopTypical stored “document” (JSON)80kb-1Mb
  • 23. Riak Successes
  • 24. MapReduceHandy for queryingRuns at “web page speed”.Easy to re-reduce for complex queriesEasy to test via CURL
  • 25. Tunable CAP @ Replication: primary/secondary authority Read failure tolerance: speed/consistency Write failure tolerance
  • 26. Full Text SearchBuilt on LuceneMake user content searchableMake arbitrary keys queryable“Just turn it on”Hiccup: corrupt merge indexes
  • 27. Query Example Who’s our oldest user who’s shared something in the last minute?curl -XPOST http://localhost:8098/mapred -H Content-Type: application/json -d { "inputs": { "bucket":"links", "query":"timestamp:[1346350877 TO 1346350937}" //60 second period }, "query":[ {"map":{"language":"javascript","source":"function(riakObject) { return [[Riak.mapValuesJson(riakObject)[0].user_id]]; }"}}, {"reduce":{"language":"javascript", "name":"Riak.reduceMin" // [[2],[5],[9],[13]] => [[2]] }} ]} [[2197]]
  • 28. Riak on EC2
  • 29. In a NutshellEC2 specs poorly proportioned for leveldbMultiple AZs in one location works wellScale vertically for better latency & consistencyScale horizontally for more throughput/$
  • 30. BenchmarksTop Graph: c1.medium (1.7G, 5 CPU)Middle: m1.large (7.5G, 4 CPU)Bottom: cc1.4xlarge (23G, 33.5 CPU)
  • 31. Throughput
  • 32. Latency (Typical)
  • 33. Latency (Worst Case)
  • 34. Calculationsc1.medium (1.7G, 5 CPU)1758 IOPS/$-hrWorst 1% of queries: 300ms/800msm1.large (7.5G, 4 CPU)1167 IOPS/$-hrWorst 1% of queries: 110ms/200mscc1.4xlarge (23G, 33.5 CPU)872 IOPS/$-hrWorst 1% of queries: 47ms/139ms
  • 35. Benchmark Takeaways You can’t go “by spec” IO is limiting factor RAM never limiting factor for 1% of keyspace to be in memory
  • 36. Fin. Questions?Thanks: We’re Hiring! Tom Santero Robby Grossman Justin Sheehy robby@shareaholic.com Ryan Zezeski @freerobby Reid Draper #freenode riak crew
  • 37. Fin.