Downloaded 31 times
![Elasticsearch Deployments
Internal Search
- 216 Internal Blogs - 750k docs [3 GB]
Support Documents
- KNN Link Prediction - 1.7m docs [14 GB]
Polldaddy
- Word Clouds/Freq Response - 39m docs [9 GB]
WordPress.com VIP Search
- KFF.org - 18m docs [99 MB]
- NY Post - 600k docs [2.3 GB]
WordPress.com - ~800m docs [4 TB]
- Related Posts - 48 mil reqs/day
- search.wordpress.com - 3 mil reqs/day
Tuesday, February 25, 14](https://image.slidesharecdn.com/elasticsearchatautomattic-140225130408-phpapp02/85/Elasticsearch-at-Automattic-5-320.jpg)
![Related Posts Relevancy
Great With Long Content
{ "more_like_this":{
"fields":["mlt_content"],
"like_text":"Scaling Elasticsearch Part 1: Overview
ElasticSearch scaling Search We recently launched
Related Posts across WordPress.com, so its time to
pop the hood and take a look at what ended up in
our engine... ",
"percent_terms_to_match":0.08,
"boost_terms":5,
"analyzer": "en_analyzer"
}}
Tuesday, February 25, 14](https://image.slidesharecdn.com/elasticsearchatautomattic-140225130408-phpapp02/85/Elasticsearch-at-Automattic-34-320.jpg)
Uploaded byGreg Brown
Elasticsearch at Automattic
The document details the performance and scaling challenges faced by Automattic's Elasticsearch deployments, highlighting a significant amount of data and operations such as indexing and querying across various platforms. Key improvements in indexing speed and real-time goals are discussed, with an emphasis on handling failures and optimizing queries. It also addresses issues related to slow queries and the need for effective language analyzers to enhance relevancy in search results.
More Related Content
Similar to Elasticsearch at Automattic
Elasticsearch at Automattic
- 1.
- 2. Greg Ichneumon Brown Data Wrangler atAutomattic http://gibrown.wordpress.com @gregibrown greg@automattic.com Tuesday, February 25, 14
- 3.
- 4. 1 Billion Monthly Uniques Tuesday,February 25, 14
- 5. Elasticsearch Deployments Internal Search -216 Internal Blogs - 750k docs [3 GB] Support Documents - KNN Link Prediction - 1.7m docs [14 GB] Polldaddy - Word Clouds/Freq Response - 39m docs [9 GB] WordPress.com VIP Search - KFF.org - 18m docs [99 MB] - NY Post - 600k docs [2.3 GB] WordPress.com - ~800m docs [4 TB] - Related Posts - 48 mil reqs/day - search.wordpress.com - 3 mil reqs/day Tuesday, February 25, 14
- 6. Overview of RelatedPosts Our “10X Improvements” - Indexing - Querying Our Open Issues Tuesday, February 25, 14
- 7. Related Posts Search withinjust the one blog Tuesday, February 25, 14
- 8. WordPress.com Total Elasticsearch Operations Operation RoutedQueries 23 mil Global Queries 2 mil Docs Indexed 13 mil Docs Updated 10 mil Docs Deleted 2.5 mil Delete By Query Tuesday, February 25, 14 Ops/Day 250k
- 9. Global Cluster DC1 1 Master DC2 DC3 1Master 14 Data Tuesday, February 25, 14 14 Data 1 Master 14 Data
- 10. Our Secret ToScaling Routed Queries All Posts for each Blog are on the same Shard Tuesday, February 25, 14
- 11. Global Index 7 Indices 10mil Blogs per Index 25 Shards per Index 175 Shards Total Tuesday, February 25, 14
- 12. Overview of RelatedPosts Our “10X Improvements” - Indexing - Querying Our Open Issues Tuesday, February 25, 14
- 13. 20% Improvements Don’t solvescaling problems Tuesday, February 25, 14
- 14. Indexing Entangling Elasticsearch with ExistingSystems Tuesday, February 25, 14
- 15. Bulk Indexing 1.0 44Days to Index all Posts (estimated) Tuesday, February 25, 14
- 16. Bulk Indexing Problems -Overhead: Spent too much time starting indexing jobs WordPress.com has 500 mil MySQL tables. - High DB Load: Corner Cases. Blogs with 1+ mil followers. - High DB Load: Indexing sequentially doesn’t spread the load. - High DB Load: Heavy load on archive DBs. Tuesday, February 25, 14
- 17. Bulk Indexing Today 12.0? 4Days to Index all Posts (running right now) Tuesday, February 25, 14
- 18. Real Time Indexing TheHardest Part! Tuesday, February 25, 14
- 19. Real Time Goals 1)Eventually Consistent 2) Minimize Bulk Re-indexing 3) Normally updated < 1 minute Tuesday, February 25, 14
- 20. Real Time Goals 1)Eventually Consistent 2) Minimize Bulk Re-indexing 3) Normally updated < 1 minute Bulk reindexed 3 times in 5 months. One intentional, Two during system upgrades. Tuesday, February 25, 14
- 21. Stuff Fails 1) Humans 2)Hardware 3) Elasticsearch (steady improvements) Combinations of the above. Tuesday, February 25, 14
- 22. Hardware Problems 1) Detectand Track Down Servers 2) Prioritize Queries over Indexing 3) Throttle Indexing Jobs - any issues: block bulk changes to blogs - >10 min: block doc updates - >20 min: block all indexing Tuesday, February 25, 14
- 23. Real Time Failures 1)Auto Retry Failed Indexing Jobs 2) Indexing Queue for Failures 3) Scrolling Queries to Find Bad Docs Tuesday, February 25, 14
- 24. Cluster Restarts Indexing acrossreplicas is non-deterministic Segments diverge Slows Restart Time Tuesday, February 25, 14
- 25. Simplistic Example Docs Shard 1 merges Primary Replica Segments w/identical checksums Tuesday, February 25, 14 Only first segment is identical
- 26. After Bulk Index Everysegment is out of sync! Tuesday, February 25, 14
- 27. Our Bulk IndexingProcedure 1) Bulk Index All Docs 2) Optimize the index 3) Rolling Restart (sync segments) 4) Future restarts will be much faster. - Play with recovery settings - SSDs? => use noop Linux scheduling Tuesday, February 25, 14
- 28. Indexing It’s all abouthandling Failures Tuesday, February 25, 14
- 29. Overview of RelatedPosts Our “10X Improvements” - Indexing - Querying Our Open Issues Tuesday, February 25, 14
- 30. Querying Test and Iterate Tuesday,February 25, 14
- 31. Related Posts Query Startedwith MoreLikeThis API. Did not scale well enough. Tuesday, February 25, 14
- 32. MLT API 1) GetDocument 2) Analyze Document 3) Search for Similar Docs Tuesday, February 25, 14
- 33. MLT API vsMLT Query MLT API MLT Query 147 req/sec 1062 req/sec 40% CPU 30% CPU 306 ms median latency 49.5 ms median latency All processing by ES Tuesday, February 25, 14 Build query in PHP
- 34. Related Posts Relevancy GreatWith Long Content { "more_like_this":{ "fields":["mlt_content"], "like_text":"Scaling Elasticsearch Part 1: Overview ElasticSearch scaling Search We recently launched Related Posts across WordPress.com, so its time to pop the hood and take a look at what ended up in our engine... ", "percent_terms_to_match":0.08, "boost_terms":5, "analyzer": "en_analyzer" }} Tuesday, February 25, 14
- 35. MLT Query Relevancy Usematch or multi_match for short content. Average Related Posts CTR Tuesday, February 25, 14
- 36. Language Analyzers arabic, armenian,basque, brazilian, bulgarian, catalan, chinese, czech, danish, dutch, english, finnish, french, galician, german, greek, hindi, hungarian, indonesian, italian, japanese, korean, norwegian, persian, portuguese, romanian, russian, spanish, swedish, turkish, thai Tuesday, February 25, 14
- 37. Related Posts Relevancy HowImportant is using the correct Language Analyzer? Tuesday, February 25, 14
- 38. Related Posts Relevancy HowImportant is using the correct Language Analyzer? Doubled Click Through Rate Tuesday, February 25, 14
- 39. Unfortunately Increased Slow Queries (>1second) by 10x still worth it. Tuesday, February 25, 14
- 40. Global Query Performance search.wordpress.com Tuesday,February 25, 14
- 41. Parent-Child Filtering Blog Doc public:true|false Post Doc title: “...” content: “...” Tuesday, February 25, 14
- 42. has_parent Filter Querying AcrossAll Shards With has_parent Without has_parent 7.6 req/sec 17.5 req/sec 75% CPU 50% CPU 503 ms median latency 207 ms median latency Requires more Indexing Tuesday, February 25, 14
- 43. Indexing: Optimize to HandleFailures Querying: Test and Iterate Tuesday, February 25, 14
- 44. Overview of RelatedPosts Our “10X Improvements” - Indexing - Querying Our Open Issues Tuesday, February 25, 14
- 45. Open Issues Slow Queries(> 1 second) Getting Better. Shards are too big. Tuesday, February 25, 14
- 46. Open Issues What doesit take to scale? 3x Data 5x Queries Tuesday, February 25, 14
- 47. Open Issues Elasticsearch forNatural Language Processing? At Scale. On Live Data. Tuesday, February 25, 14
- 48.