Downloaded 17 times
![{
"name":"Manoj Mohan",
"email":"manoj@intelligrape.com",
"company": "Intelligrape Software Ltd",
"role": "Develop customized web solutions",
"partTimeHobby":["Photography",”Sketching”]
}
speaker.dump()](https://image.slidesharecdn.com/indicthreads-delhi13-elasticsearchbymanojmohan-130903061656-/75/BigData-Search-Simplified-with-ElasticSearch-3-2048.jpg)
![Percolating !!!
TestPercolator
DistanceAlarm
Elastic Index
“name”:”John Doe”,
“distance” : 120
{ "ok" : true, "matches" : ["distanceAlarm"]}](https://image.slidesharecdn.com/indicthreads-delhi13-elasticsearchbymanojmohan-130903061656-/75/BigData-Search-Simplified-with-ElasticSearch-43-2048.jpg)
Uploaded byTO THE NEW | Technology
BigData Search Simplified with ElasticSearch
The document discusses Elasticsearch as a powerful, distributed, open-source search server designed to manage big data challenges such as searching through structured and unstructured data. It highlights key features like dynamic schema updates, support for complex documents, and an efficient indexing process to improve data retrieval. Additionally, the text compares Elasticsearch with other search engines and emphasizes its capabilities in handling data-related tasks beyond simple search functionalities.
More Related Content
Similar to BigData Search Simplified with ElasticSearch
More from TO THE NEW | Technology
BigData Search Simplified with ElasticSearch
- 2.
- 3. { "name":"Manoj Mohan", "email":"manoj@intelligrape.com", "company": "IntelligrapeSoftware Ltd", "role": "Develop customized web solutions", "partTimeHobby":["Photography",”Sketching”] } speaker.dump()
- 4. Agenda Big Data Search Contenders Elasticsearch -Elastic Whaaat ?? - Some cool features - Analyzing Data More than just search Unchartered Territory
- 5.
- 6. What is BigData Highly subjective term used when human/technical infra unable to cope with company's data needs Rapid data consumption volume means todays “Big” is tomorrows ”Normal” The Big V's – Velocity, Volume, Variety, Veracity Small but complex is still “Big”
- 7. Is it reallyBig Data ?? Elasticsearch: Big Data Search Simplified “If you aren’t taking advantage of big data, then you don’t have big data, you have just a pile of data” Jay Parikh, VP of Infrastructure Facebook
- 8. Challenges with BigData Search Data can be both structured and unstructured Need to search across exabytes of data Data needs to be collated and normalized first to get accurate results Fine tuning and refining search can take time Needs an easily scalable solution
- 9. Big Data Search Contenders Elasticsearch -Elastic Whaaat ?? - Some cool features - Analyzing Data More than just search Unchartered Territory Looking Back ….
- 10.
- 11.
- 12. Lucene Started in 1999,joined Apache family in 2001 High-performance, full-featured text search engine library written entirely in Java Used by adding a jar to your application
- 13. Apache Solr Scalable anddistributed solution Uses Lucene under the hood Provides a http-wrapper over Lucene marketing itself as ready-to-use solution. Adds XML/JSON support, caching, replication, sharding Deployable in any servlet container – Tomcat, Jetty, Resin
- 14. Sphinx Cannot be hitdirectly from your web-app Aligns closely with SQL/PostgreSQL database Scalable and Distributed Searching High performance No partial updates No Restful API or data replication
- 15. Big Data Search Contenders Elasticsearch -Elastic Whaaat ?? - Some cool features - Analyzing Data More than just search Unchartered Territory Looking Back ….
- 16. Elasticsearch: Big Data SearchSimplified
- 17.
- 18. Elastic Whaaat ....?? ElasticSearch is a distributed, RESTful, free/open source search server Based on Apache Lucene Developed by Shay Banon, written in Java Latest and Greatest - 0.90.3
- 19. Really Elastic !! Highperformance ● Document oriented and schema free Built distributed from the ground up Support for complex documents Support for multitenancy RestfulAPI Supports dynamic schema updates
- 20. Push Replication Replica created aftersuccessful indexing Index a documentP: Primary Shard R: Replica
- 21. Node Auto Discovery Elasticsearch:Big Data Search SimplifiedRelocated New node detected automatically Shards and replicas are now distributed taking into account the new node 20
- 22. Elasticsearch: Big Data SearchSimplified Ping check Data nodes ping master to check if master is alive A new master is elected among the child nodes automatically Become master Fail-Safe
- 23.
- 24. Indexing it Right!!! The first step to make big data useful is to identify the relevant data. Lets index 2 documents 1) I know I am a really great developer 2) I develop excellent Excuses
- 25. Indexing it Right!!! 1) I know I am a really great developer 2) I develop excellent Excuses Hmmm... Lets index everything ...
- 26. Indexing it Right!!! 1) I know I am a really great developer 2) I develop excellent Excuses Wait .... Do we actually need words like “I”, “a”, “am” etc. c
- 27. Indexing it Right!!! 1) I know I am a really great developer 2) I develop excellent Excuses Wait .... Some words “stem” from others.
- 28. 1) I knowI am a really great developer 2) I develop excellent Excuses Wait .... Some words are simply synonyms of others Indexing it Right !!!
- 29. Indexing it Right!!! 1) I know I am a really great developer 2) I develop excellent Excuses Wait .... Case Insensitive ??
- 30. Pre Process Strategy Dropstopwords (Stopword Analyzer) Lowercase everything (Lowercase Filter) Reduce words to stems (Stemming Analyzer) Consider synonyms (Synonym Filter)
- 31. Big Data Search Contenders Elasticsearch -Elastic Whaaat ?? - Some cool features - Analyzing Data More than just search Unchartered Territory Looking Back ….
- 32. More than justsearch ...
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40. What is Percolation? “Reversedsearch” Instead of storing documents, and then searching them with queries ..... store queries and “percolate” documents through them
- 41. Elasticsearch Percolator Register anumber of queries and determine which of them match a particular document Business rules of your application. Helping you decide on the “nature” of the document Commonly used for alerts
- 42. Percolating !!! Distance >100 distanceAlarm TestPercolator distanceAlarm Elastic Index
- 43. Percolating !!! TestPercolator DistanceAlarm Elastic Index “name”:”JohnDoe”, “distance” : 120 { "ok" : true, "matches" : ["distanceAlarm"]}
- 44. Big Data Search Contenders Elasticsearch -Elastic Whaaat ?? - Some cool features - Analyzing Data More than just search Unchartered Territory Looking Back ….
- 45.
- 46. Elasticsearch and NOSQLDatabase Stores documents or JSON data CRUD High performance -24 billion records queried in 900 ms Unlike SOLR original source is retained Built in support for Sharding and Replicas Versioning and Commits
- 47. What IF ?? Removethe hassle of maintaining 2 data stores Simply use ES as a NOSQL db with advanced out of the box search capabilities Has limitations.. but can be worked around
- 48.
- 49. Contact us Our Office ClientLocation Here’s how TOTHENEW helps your customers outsource across the globe using BIG DATA! Click Here To Know More! Have more queries related to BIG DATA? Talk To Our Experts
- 50.