(Elastic)search in big data

1,964 views
1,795 views

Published on

The place of Elasticsearch in Big Data landscape.

(Elastic)search in big data

  1. 1. (Elastic)search in Big Data Radu Gheorghe @radu0gheorghe @sematext
  2. 2. What is “search in Big Data”? Challenges? Some solutions? How does Elasticsearch do it? Agenda
  3. 3. Search Expectations
  4. 4. headphones for iPhone 4, iPhone 5, iPhone 6 and iPhone 7 iPhone 5 iPhone 4 Relevancy... iphone
  5. 5. iphone iphone 5 Institute of Public Health ...and autocomplete... iph
  6. 6. No results found for “iphnoe” iPhone 5 iPhone 4 … and fuzziness... iphnoe
  7. 7. Did you mean “iPhone”? iPhone 5 iPhone 4 ...and corrections... iphnoe shows results anyway
  8. 8. iPhone 5 iPhone 4 iPhone 3 Galaxy S4 ...and similar terms... iphone
  9. 9. iPhone 5 iPhone 4 ...and don’t forget the statistics! iphone ☑ iOS ☐ other ☑ <100RON ☐ 100-200RON ☐ >200RON
  10. 10. Wait. Fancy search == Big Data?
  11. 11. Fancy stuff isn’t free iphone ☑ iOS ☐ other ☑ <100RON ☐ 100-200RON ☐ >200RON N requests for autocomplete Did you mean... iPhone 5 iPhone 4 iPhone 3 Galaxy S4 1 request for each of the stats 1 request for synonyms, 1 for exact matches, etc 1 request for corrections
  12. 12. Distributed search. When one server doesn’t cut it
  13. 13. Log Search web_server01 database01 backend01 search engine 10:01 - webapp - DB connect error 10:00 - DB - I/O error error
  14. 14. Log Analytics unique IPs: 7584 iPhone 5 iPhone 4 Galaxy S4 best sellers Romania: 200 France: 150 Hungary: 120 users per country revenue per day
  15. 15. Distributed search solutions Elasticsearch Solr Others: SenseiDB, Sphinx… SaaS: CloudSearch, Logsene... built on top of Lucene
  16. 16. Document-oriented Lucene awesome: index & store data, relevancy, fuzzy, suggesters... ...all wrapped up in JSON over HTTP Elasticsearch
  17. 17. Aggregations revenue per day unique IPs: 7584
  18. 18. Aggregations revenue per day unique IPs: 7584 Romania: 200 France: 150 Hungary: 120 unique IPs per country
  19. 19. Aggregations revenue per day Romania: 200 France: 150 Hungary: 120 unique IPs per country unique IPs per country per day Romania unique IPs: 7584
  20. 20. Node 1
  21. 21. Node 1
  22. 22. Node 1 Node 2
  23. 23. Node 1 Node 2
  24. 24. Node 1 Node 2 Node 3
  25. 25. Node 1 Node 2 Node 3
  26. 26. Node 1 Node 2 Node 3
  27. 27. Node 1 Node 2
  28. 28. Node 1 Node 2
  29. 29. Big Data distributed search search and real- time analytics
  30. 30. Big Data distributed search search and real- time analytics more search features
  31. 31. Big Data distributed search search and real- time analytics more search features clients usage (logs)
  32. 32. Thank you! radu.gheorghe@sematext.com @radu0gheorghe @sematext Big Data distributed search search and real- time analytics more search features clients usage (logs)

×