Successfully reported this slideshow.

10 EZ Steps to SOLR Domination - Berlin Buzzwords 2012

676 views

Published on

2012-06-05 presentation at Berlin Buzzwords

Published in: Technology
  • Be the first to comment

  • Be the first to like this

10 EZ Steps to SOLR Domination - Berlin Buzzwords 2012

  1. 1. SOLR Power FTW10 EZ Steps to SOLR Domination Alex Pinkin @apinkin
  2. 2. Alex Pinkin● Director of Engineering, Data and Infrastructure, Bazaarvoice @apinkin
  3. 3. Bazaarvoice● SaaS company powering user generated content on thousands of web sites● 5+ billion page views per month● 300+ billion impressions
  4. 4. SOLR Stats Documents 350 MM Index size 330 GB AVG QPS 3,400 MAX QPS 12,500 AVG Response time 7 ms 95P Response time 14 ms
  5. 5. Initial State - back in 2007... ● SoR: MySQL ● Search: MySQL ● Numerous indexes ● Numerous aggregates
  6. 6. Problem 1FULL TEXT SEARCH
  7. 7. Solution 1: FT Search● SOLR Master + Hot Backup● Single index
  8. 8. Problem 2● Reads overwhelming master● No R/W isolation
  9. 9. Solution 2● R/W separation
  10. 10. Problem 3● QPS Growing
  11. 11. Solution 3● Add more slaves
  12. 12. Problem 4● Number of Clients is growing
  13. 13. Solution 4● Multiple Deployment Groups
  14. 14. Problem 5● MySQL still in trouble: Too many Indexes, Aggregates
  15. 15. Solution 5● Offload filtering and sorting to SOLR● Add more slaves● Add more RAM
  16. 16. Problem 6● Query latency is getting worse
  17. 17. Solution 6● Multiple cores● Core management tools● Transparent data migration
  18. 18. Problem 7MySQL is still in trouble ● Too many aggregate tables ● Error-prone aggregation logic
  19. 19. Solution 7● Off-load simple aggregates to SOLR● Use Faceting
  20. 20. Problem 8● SOLR is GCing itself to death● 99p Latency = :-(
  21. 21. Solution 8● GC tuning ○ Concurrent Low Pause Collector ○ Parallel Young Generation Collector ○ Incremental mode
  22. 22. Problem 9● Disaster Recovery (Multi-DC)
  23. 23. Solution 9● Cross-DC replication
  24. 24. Problem 10● Schema Changes... "ALTER TABLE OF DOOM" @josh_wills
  25. 25. Solution 10● Bittorrent for the rescue!
  26. 26. Remaining Challenges● Elasticity● HA: failover● Schema changes
  27. 27. Questions @apinkin

×