Search and Analytics (using Elasticsearch)

6,080 views

Published on

Slides from Costin Leau's talk on Search and Analytics (using Elasticsearch) at the 18th Big Data London meetup

Published in: Technology

Search and Analytics (using Elasticsearch)

  1. 1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch and Analytics(using Elasticsearch)Costin Leau
  2. 2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedWhy search?
  3. 3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch – what’s the big deal?Basic/Metadata retrieval“Find banks with more then (x) accounts”
  4. 4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch – what’s the big deal?Basic/Metadata retrieval“Find banks near my location”
  5. 5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch – What we’re all about
  6. 6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch categoriesBasic/Metadata retrievalFull-text searchHighlightingGeolocationFuzzy search (“did-you-mean”)Natural Language
  7. 7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch categoriesBasic/Metadata retrievalFull-text searchHighlightingGeolocationFuzzy search (“did-you-mean”)Natural Languagedata storessearch engines
  8. 8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited‘Players’ in the search marketSearch engines- Google/Bing/Yahoo!/Ask.com/Yandex/BaiduOpen-Source- Sphinx- Apache Lucene- Elasticsearch- Solr- SenseiEnterprise Search- Oracle Endeca / MDEX- HP Autonomy- Exalead- IBM Enterprise Search
  9. 9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedElasticsearch
  10. 10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedElasticsearchOpen-Source Search & Analytics engine- Structured & Unstructured Data- Real Time- Analytics capabilities (facets)- REST basedDistributed- Designed for the Cloud- Designed for Big Data
  11. 11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedElasticsearchOpen-Source Search & Analytics engine- Structured & Unstructured Data- Real Time- Analytics capabilities (facets)- REST basedDistributed- Designed for the Cloud- Designed for Big DataLightweight
  12. 12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedElasticsearchOpen-Source Search & Analytics engine- Structured & Unstructured Data- Real Time- Analytics capabilities (facets)- REST basedDistributed- Designed for the Cloud- Designed for Big DataLightweightPopular: >200K downloads/month
  13. 13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedUsers
  14. 14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedUsers
  15. 15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedPlatform Adoptionhttp://www.thoughtworks.com/radar#platforms 2013
  16. 16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedPlatform Adoptionhttp://www.thoughtworks.com/radar#platforms 2013
  17. 17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedUse Case - Text Searchhttps://github.com/blog/1381-a-whole-new-code-search
  18. 18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearches 50,000,000 venues every day usingElasticsearchUse Case - Geolocation
  19. 19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedUse Case – Support/Reporting
  20. 20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedUse Case - Centralized Logging
  21. 21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedUse Case - Pure Analytics
  22. 22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedSearch and Big Data
  23. 23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedA Holistic View of a Big Data SystemETLRealTimeStreamsUnstructured Data (HDFS)RT SemistructuredDatabase(hBase,Cassandra,Mongo)Big SQL(Greenplum,AsterData,Etc…)BatchProcessingReal-TimeProcessing(s4, storm)Analytics
  24. 24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedA Holistic View of a Big Data SystemETLRealTimeStreamsUnstructured Data (HDFS)RT SemistructuredDatabase(hBase,Cassandra,Mongo)Big SQL(Greenplum,AsterData,Etc…)BatchProcessingAnalyticsReal-TimeProcessing(s4, storm)
  25. 25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedHadoop eco-systemHadoop Distributed File System (HDFS)Map Reduce Framework (MapRed)
  26. 26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedHadoop eco-systemHadoop Distributed File System (HDFS)Map Reduce Framework (MapRed)
  27. 27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedElasticsearch + Hadoop0102030405060M/R Pig HiveRaw w/ ES0102030405060M/R Pig HiveRaw w/ ESWriting Reading / Querying
  28. 28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedExplore data through(Elastic)Search
  29. 29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibitedThank you!@costinlhttp://www.elasticsearch.org/

×