SF ElasticSearch Meetup 2013.04.06 - Monitoring


Published on

Using monitoring tools Zabbix for systems-level monitoring of ElasticSearch and SPM (http://sematext.com/spm/elasticsearch-performance-monitoring/index.html) for ElasticSearch-specific monitoring. Using these tools was crucial was optimizing index building performance as well as query performance. Some general tips for index building and query performance.

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • http://www.zabbix.com/ - ‘’Enterprise class monitoring solution for everyone’
  • http://www.zabbix.com/ - ‘’Enterprise class monitoring solution for everyone’
  • Collect information over 1B users internationally – text copied from over 600K publisher sites, images, searches, pages visitedDifferent slices of data – now!
  • SF ElasticSearch Meetup 2013.04.06 - Monitoring

    1. 1. Monitoring tools for ElasticSearch SF Meetup 2013.03.06 Sushant Shankar Shyam Kuttikkad
    2. 2. • Why and how we use ElasticSearch• Monitoring – Tools – Index Building – Query Performance
    3. 3. Who is asdfas• Social Sharing and Content Discovery platform – We help >600,000 publishers with content distribution, user engagement, and advertising monetization – 450 Fortune 1000 brand marketers leverage our unique social signals to deliver impactful advertising• We develop Machine Learning algorithms operating on Big Data to: – Provide content sharing insights to Publishers – Build customized audience segments for advertising campaigns – Extract actionable insights out of social and interest datawww.33Across.comwww.tynt.com
    4. 4. Data firehose of 30B monthly events, 1.25B cookies - Interaction with web content - Shares – images, copies - Searches Build, understand, analyze Real-time view ElasticSearch! Social Audiences Behavior Context Knowledge
    5. 5. Production ElasticSearch clusterHardware6 nodes, 24GB RAM16GB for ES service4 cores3x 1.5TB driveIndex Build index>1TB/index using MR job(replicated) and Bulk API~300M documents~5KB / document~3 hours
    6. 6. System monitoring using Zabbix Index Build
    7. 7. ElasticSearch specific monitoring using SPMScalable Performance Monitoring (http://sematext.com/spm/index.html)• Index stats – Total/Refreshed/Merged documents• Shards – Total/Active/Relocating/Initializing• Search - Request rate and latency• Cache – {Filter, field} cache {count, evictions, size}• Machine – CPU, Memory, JVM, GC, Network, Disk
    8. 8. Index Building Optimization using Zabbix and SPMAmount bulk indexed Time taken CPU util. Mem util. Disk I/O Network # Shards
    9. 9. in practice…
    10. 10. Debugging and Validating using SPM
    11. 11. Index Building: Learnings• 2 shards / CPU• 10,000 documents (users) per indexing request• Bulk API for our use case• No replicas• Refresh off (index.refresh_interval = -1)
    12. 12. Query Performance: Learnings• 1-2 Replicas (and for reliability)• Turn refresh on again (5s default)• Warm up effect (Index Warm up API 0.20+)• Optimize API• Simulate multiple users
    13. 13. QUERIES?
    14. 14. Sushant Shankarsushant.shankar@33across.com Shyam Kuttikkadshyam.kuttikkad@33across.com
    15. 15. Why we really need a search engine Batch! Good for complicated tasks (Machine Learning, Graph Algorithms, etc.) … …
    16. 16. Warm Up: load into memory and cache
    17. 17. Other cool features• Custom Scoring functions• Scripts – MVEL, Python• Facets• Exploring:• Real-time indexing• Indexing images, files, etc.• Parent-child relationships