Elasticsearch Meetup - August 2018 - SocialCops
•Download as PPTX, PDF•
0 likes•15 views
This document discusses near real-time data synchronization between a primary datastore like MongoDB and Elasticsearch by tailing the primary's operation logs. It explains why this is useful for faster search, analytics and aggregations. Key terms are introduced like replica sets, oplogs and tailable cursors. Concepts covered include locating the logs, reading them continuously and putting the data into scalable stores. Considerations for implementation involve preventing overload, failover and minimizing lag. A demo of oplog tailing is provided.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Elasticsearch Meetup - August 2018 - SocialCops
Similar to Elasticsearch Meetup - August 2018 - SocialCops (20)
Recently uploaded
Recently uploaded (20)
Elasticsearch Meetup - August 2018 - SocialCops
- 1. Aayush Sarva & Gaurav Sehgal Backend Engineers @ SocialCops Near real-time data synchronization between your primary datastore and Elasticsearch
- 2. - Why bother? - Understanding some terms - Grasping underlying concepts - Possibilities - Demo Overview
- 3. Get to insights, faster. - Faster search & discovery - Powerful analytics & visualizations - Efficient & scalable data aggregations Elasticsearch - distributed, RESTful search & analytics engine Why
- 4. - MongoDB - Replica Set - Operation logs (Oplogs) - Tailable cursor / Oplog tailing Terminology
- 5. - Locate your database operation logs - Reliably & continuously read them - Put them into efficient & scalable data stores - Build utilities on top of these data stores Concepts
- 6. - Preventing over-burdening the database - Failover mechanisms - Extensibility - Minimise lag between data stores Considerations
Editor's Notes
- Start!
- MongoDB => document-oriented NoSQL database Replica set => Group of mongod processes maintaining a copy of the same data for redundancy and high availability. Oplog => Log of database operations. Special capped collection keeping a rolling record of all operations that modify data. Oplog tailing => Conceptually the same as the Unix "tail -f" command. Once you've reached the end of the result set, the cursor will not be closed, rather it will continue to wait forever for new data and when it arrives, return that too.
- We use MongoDB extensively at SocialCops. Faster search & discovery - “simple search”, “fuzzy search”, “aggregations”, “auto-completion” A one size fits all database doesn’t fit anyone. - Werner Vogel (CTO - Amazon.com)
- MongoDB => document-oriented NoSQL database Replica set => Group of mongod processes maintaining a copy of the same data for redundancy and high availability. Oplog => Log of database operations. Special capped collection keeping a rolling record of all operations that modify data. Oplog tailing => Conceptually the same as the Unix "tail -f" command. Once you've reached the end of the result set, the cursor will not be closed, rather it will continue to wait forever for new data and when it arrives, return that too.
- As aayush spoke about what and why. First, tailing oplog of mongodb. It works like a listener. Here we can specify which collection of which database we want to tail. Whenever a new entry being made in specified collection this listner will throw an event with data. Second, connector. Here you can do anything with your data. You can do transformation and pre-processing befor pushing to ES. It can be from finding the basic distributions like mean, median to applying ml algos. Last step that's obvious pushing data to Es. Here you have to take care of few thing like index convention, mapping and data structure etc. Okay, with this architecture in mind. Lets start with a demo. For the purpose of our insertion system. Im gonna use our inhouse tool. Collect - A Data Collection Tool. Inspired from richard Hendricks, silicon valley. I hope everyone has heard about it. I have created a web form called tabs or spaces. Its a basic web form to know whether you like tabs and spaces. For visualisation purpose, i have created a simple widget for distribution piwered by es. Its a basic page built using high chart . Basic aggregation query is running to fetch data. As you see its 0,0 now. But i have entered something. Its because we haven't made the connector yet. I already curate a basic connector script. Lets quickly understand what's happening here. At the high level it just tail through oplog and push the same data to es. In mongodb oploag is a capped Collection which stores in local.oplog.rs collection. Getting the current time stamp. Putting query. First one is cursor type that help to create the tailable cursor with the connection always open second it oplog_replay that help to find the ts we are looking for faster. Explain the code. Okay let's start with the real demo. For that i need your help. Here's the link. Choose you favourite whether you like tabs or spaces while coding. But be honest because either way Richard Hendricks gonna find out who you are and What you like.