Download as PDF, PPTX
Uploaded byAll Things Open
Building Real-Time Search at MailChimp
The document presents a technical overview of building a real-time search application at Mailchimp, detailing the challenges of querying and indexing search data. It explores the potential use of Elasticsearch, Kafka for event streaming, and the indexing process for various data models. The results highlight improvements in query response times and a successful implementation without support tickets post-launch.
More Related Content
Similar to Building Real-Time Search at MailChimp
Building Real-Time Search at MailChimp
- 1. Building Real-Time Search atMailchimp Kevin Xu Software Engineer
- 3.
- 4.
- 5. 2018 SEARCH APP (downstream) search queries indexingrequests (every 5 min) SQLite Search
- 6.
- 7. 2018 SEARCH APP Problem: Indexing indexingrequests (every ???)
- 8. 2018 SEARCH APP search queries indexingrequests (every ???) SQLite Search
- 9. 2018 Considering a Replacement -Use cases: full-text search, logging/log analysis, events and metrics - Fast queries - Scale horizontally
- 10.
- 11.
- 12. 2018 Capturing Events fromMC US1 US19
- 13. 2018 Possible Solution: DirectIndexing?
- 14.
- 15.
- 16. Proprietary & Confidential2018 16 Kafka: Topics
- 17.
- 18.
- 19.
- 20. 2018 App-Layer Filtering config.searchable_model: “Contact”; config.searchable_model:“Campaign”; ... $contact = new Contact(“Ben”); $contact->save(); // onSave()
- 21. 2018 Write to File,Ship to Kafka
- 22. 2018 Indexing to ES PHPConsumers ?
- 23.
- 24.
- 25. 2018 No Order, NoProblem
- 26.
- 27.
- 28. 2018 Queries > 1s 9/26/1710/10/17 NumberofQueries release!
- 29.
- 30. 2018 30 250msMedian (p50)Query Response Time
- 31. 2018 31 400msp95 QueryResponse Time
- 32. 2018 32 19Number ofElasticsearch clusters
- 33. 2018 33 373 billion Totaldocs across all ES clusters
- 34. 2018 34 93,000 Total Changelogevents per second (peak)
- 35. 2018 35 3.7 minutes AverageTime to Index
- 36. 2018 36 0Support ticketspost-launch
- 37. Proprietary & Confidential2018 37 What Now? - Explore other applications of this infrastructure - Ongoing technical challenges - Data Drift, Consumer Lag
- 38. Proprietary & Confidential2018 Thanks! 38 kevin.xu@mailchimp.com