Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Real-Time Search at MailChimp

217 views

Published on

Presented at All Things Open 2018
Presented by Kevin Xu with MailChimp
10/23/18 - 4:00 PM - Case Study/Demo Track

Published in: Technology
  • I have always found it hard to meet the requirements of being a student. Ever since my years of high school, I really have no idea what professors are looking for to give good grades. After some google searching, I found this service ⇒ www.HelpWriting.net ⇐ who helped me write my research paper.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • You have to choose carefully. ⇒ www.WritePaper.info ⇐ offers a professional writing service. I highly recommend them. The papers are delivered on time and customers are their first priority. This is their website: ⇒ www.WritePaper.info ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Building Real-Time Search at MailChimp

  1. 1. Building Real-Time Search at Mailchimp Kevin Xu Software Engineer
  2. 2. 2018 JAN ‘17 OCT ‘17
  3. 3. 2018 MC Technical Overview US1 - US19
  4. 4. 2018 SEARCH APP (downstream) search queries indexing requests (every 5 min) SQLite Search
  5. 5. 2018 SEARCH APP Problem: Querying search queries
  6. 6. 2018 SEARCH APP Problem: Indexing indexing requests (every ???)
  7. 7. 2018 SEARCH APP search queries indexing requests (every ???) SQLite Search
  8. 8. 2018 Considering a Replacement - Use cases: full-text search, logging/log analysis, events and metrics - Fast queries - Scale horizontally
  9. 9. 2018 Elasticsearch Docs
  10. 10. 2018 Querying ES
  11. 11. 2018 Capturing Events from MC US1 US19
  12. 12. 2018 Possible Solution: Direct Indexing?
  13. 13. 2018 Adding a Message Queue
  14. 14. 2018 Real-Time Streaming
  15. 15. Proprietary & Confidential 2018 16 Kafka: Topics
  16. 16. 2018 Kafka: Partitions
  17. 17. 2018 Connecting the Dots
  18. 18. 2018 Tracking Events All Changes Searchable Changes
  19. 19. 2018 App-Layer Filtering config.searchable_model: “Contact”; config.searchable_model: “Campaign”; ... $contact = new Contact(“Ben”); $contact->save(); // onSave()
  20. 20. 2018 Write to File, Ship to Kafka
  21. 21. 2018 Indexing to ES PHP Consumers ?
  22. 22. 2018 Generating Documents
  23. 23. 2018 No Ordering Guarantees!
  24. 24. 2018 No Order, No Problem
  25. 25. 2018 PHP Consumers
  26. 26. 2018 PHP Consumers
  27. 27. 2018 Queries > 1s 9/26/17 10/10/17 NumberofQueries release!
  28. 28. 2018 release! Queries > 2sNumberofQueries 9/26/17 10/10/17
  29. 29. 2018 30 250msMedian (p50) Query Response Time
  30. 30. 2018 31 400msp95 Query Response Time
  31. 31. 2018 32 19Number of Elasticsearch clusters
  32. 32. 2018 33 373 billion Total docs across all ES clusters
  33. 33. 2018 34 93,000 Total Changelog events per second (peak)
  34. 34. 2018 35 3.7 minutes Average Time to Index
  35. 35. 2018 36 0Support tickets post-launch
  36. 36. Proprietary & Confidential 2018 37 What Now? - Explore other applications of this infrastructure - Ongoing technical challenges - Data Drift, Consumer Lag
  37. 37. Proprietary & Confidential 2018 Thanks! 38 kevin.xu@mailchimp.com

×