The power of polyglot searching
•
1 like•117 views
Presentation at Big Data Universe 2.0 in Budapest 2017.05.18. In the previous years we have got the Polyglot Persistence. This is a fancy term which means that when storing data, it is best to use multiple data storage technologies, chosen based upon the way data is being used by. If we have multiple persistence, then sometimes we need polyglot operations. One of the most popular use case in Big Data is searching. Almost all websites provide a search function to their users, to be able to find what they are looking for. Usually it is an Apache Lucene based solution, like Elasticsearch or Solr. I will show you how to enrich this kind of searching with the power of graph based searches, and implement a polyglot search functionality, where the results are based on the cooperation of a search engine and a graph based real time recommendation.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The power of polyglot searching
Similar to The power of polyglot searching (20)
More from GraphAware
More from GraphAware (20)
Recently uploaded
Recently uploaded (20)
The power of polyglot searching
- 1. GraphAware® The power of polyglot searching Janos Szendi-Varga graphaware.com @graph_aware
- 2. Most frequently used UI element GraphAware® Search Go
- 3. Evolution of Internet Search https://moz.com/blog/the-evolution-of-search
- 4. Slide from BDU 2016
- 5. We started to be Polyglot Big data architecture is not a vision We hired Data Scientists We started to index things (Lucene) We started to use Solr, ElasticSearch, etc It became the part of our Big Data architecture We introduced Search Infrastructure Evolution in corporate search GraphAware®
- 6. The fundamental of search infrastructure GraphAware® ?
- 7. They are aggregate oriented databases, they have limitations when it comes to connected data Typical setup: Two users searching for the same thing will get the same results They are in the search 3.0-4.0 phase They are superstars of Full text search We need to extend this with Graph-aided search We have to boost some Search Hit (c`mon It is a recommender system) We have to filter out or degrade the score We need Things, not Strings!!444!!!négy!!! Challenges GraphAware®
- 8. Example of graph-based search GraphAware®
- 9. “A knowledge graph is a multi-relational graph composed of entities as nodes and relationships as edges with different types that describe facts in the world." Knowledge graph GraphAware® It is about “understanding the world as you and I do”.
- 12. Search infrastructure should be easily integrated into existing architecture New data sources should be easily added Should support the strategic goals e.g. Search driven e-commerce Scalable Should provide personalised results Simple interface Requirements of searching and KG GraphAware®
- 13. Take a graph database (Neo4j, Cayley, OntoText GraphDB, etc.) Graph construction: Knowledge extraction from the internet open data grabbing from text (NLP) from current databases (Master Data) from logs Knowledge Graph Construction Have a good graph model Connect the things together Steps to build KG GraphAware®
- 16. Apache Kafka for streaming pipelines Product topic Search topic Feedback topic Spark on the processing side Neo4j on the consuming side CQRS (Command Query Responsibility Segregation) pattern Push to ElasticSearch with GraphAware plugin Neo4j Transaction Handler (afterCommit) You can define mappings to ES Parts of the architecture GraphAware®
- 17. Success story 1. • Sharing Tribal Knowledge inside the company • >20 offices • >3000 employees • Data sources: • Tableau dashboards (4000) • Knowledge posts (>1000) • Superset charts and dashboards (>6000) • Experiments and metrics (>5000) GraphAware®https://www.slideshare.net/ChristopherWilliams24/20170108scaling-tribalknowledge
- 18. Success story 2. •Half-century of collective NASA engineering knowledge •It is called Lessons Learned database •They use it in Mars mission project GraphAware® Impact: “Neo4j saved well over two years of work and one million dollars of taxpayers funds.” “When we had the [Apollo 1] fire, we took a step back and said okay, what lessons have we learned from this horrible tragedy? Now let’s be doubly sure that we are going to do it right the next time. And I think that fact right there is what allowed us to get Apollo done in the ‘60s.” —Dr. Christopher C. Kraft, Jr., Director of Flight Operations
- 19. Neo4j ElasticSearch GraphAware modules: Neo4j to ElasticSearch ElasticSearch Plugin NLP plugin Github: github.com/graphaware Open data Resources GraphAware®
- 20. GraphAware® It is not a rocket science! Anonymous NASA scientist