SF ElasticSearch Meetup 2012.10.03
•Download as PPTX, PDF•
1 like•811 views
Some thoughts on scaling ElasticSearch, especially related to index building and optimizing for query performance.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to SF ElasticSearch Meetup 2012.10.03
Similar to SF ElasticSearch Meetup 2012.10.03 (20)
Recently uploaded
Recently uploaded (20)
SF ElasticSearch Meetup 2012.10.03
- 1. Scaling ElasticSearch SF Meetup 2012.10.03 Sushant Shankar sushant.shankar@33across.com
- 2. Agenda • Why we need a search engine • Monitoring • Index Building • Query Performance
- 3. Who is asdfas >600,000 Publishers Machine Learning and Graph algorithms to: - Build advertising segments - Extract insights out of social and interest data - Target via high-performance distributed systems that integrate with our advertising partners Website | Facebook | Twitter
- 4. Why we really need a search engine Batch! Good for complicated tasks (Machine Learning, Graph Algorithms, etc.) … …
- 5. INDEX BUILDING 1 WEEK → 3 HOURS
- 6. Mappers to build index 6 nodes, 24GB RAM 16GB for ES service 4 cores 3x 1.5TB drive >1TB/index Build index (replicated) using MR job ~300M documents and Bulk API ~5KB / document ~3 hours
- 9. Parameter Optimization Amount bulk indexed Time taken CPU util. Mem util. Disk I/O Network # Shards
- 10. Index Building: Learnings • Bulk API • No replicas • 2 shards / CPU • 10,000 documents (users) per indexing request • Refresh off (index.refresh_interval = -1)
- 11. QUERY PERFORMANCE 5 MINUTES 10 SECONDS
- 12. Query Performance: Learnings • 1-2 Replicas (and for reliability) • Turn refresh on again (5s default) • Warm up effect (Index Warm up API 0.20+) • Optimize API • Simulate multiple users
- 13. Warm Up: load into memory and cache
- 14. Other cool features • Custom Scoring functions • Scripts – MVEL, Python • Facets • Exploring: • Real-time indexing • Indexing images, files, etc. • Parent-child relationships
- 15. QUERIES?
- 16. Index Building over time
Editor's Notes
- Collect information over 1B users internationally – text copied from over 600K publisher sites, images, searches, pages visitedDifferent slices of data – now!