Elasticsearch provides a quick and easy method to aggregate data, whether you want to use it for simplifying your search across multiple depots and databases, or as part of your analytics stack. Getting the data from your transactional engines into Elasticsearch is something that can be achieved within your application layer with all of the associated development and maintenance costs. Instead, offload the operation and simplify your deployment by using direct data replication to handle the insert, update and delete processes. AGENDA - Basic replication model - How to concentrate data from multiple sources - How the data is represented within Elasticsearch - Customizations and configurations available to tailor the data format - Filters and data modifications available

1. Tungsten Replicator for Elasticsearch - Real-time data
loading from Oracle and MySQL into Elasticsearch

2. Topics
In today’s webinar, we will discuss:
• Why Elasticsearch?
• How replication into Elasticsearch works
• Customizations and configurations
• Future direction
2

3. Why Elasticsearch
• Elasticsearch is a REST-based search and analytics engine
• Provides a quick method to store/search data and link back to source
• Very fast
• Great at doing very large scale analytics and aggregations
• Able to combine different forms and data (structured, unstructured, metrics)
• Easily scalable
• Provides different buckets and different types to allow multiple data repos

4. Data Collection and Organization with Elasticsearch
Tungsten
Replicator
Index
Index
Index

5. How Replication Works
DBMS
Logs
Download transactions
via network
Elasticsearch Applier
THL = Events + Metadata
Redo Logging
Master Replicator:
Extractor
THL
Slave Replicator:
Applier
THL
Redo ReaderGenerated
PLOG
Extractor gets event
data by reading the
PLOG files generated by
the Redo Reader (PLOG
generation requires the
DBMS service to be
online).

6. What Tungsten Replicator Does to Apply into Elasticsearch
• Takes an incoming row and converts it to a JSON document
• By default:
– Incoming schema/database name is used as the index name
– Incoming table name is used as the type
• Can also be explicitly set to an index/type to concentrate data (or use filters)
• Key is taken from the primary key information
– Can be converted/merged from multiple keys and formats
• Document is constructed with metadata and record information
• INSERTS create the data
• UPDATES update the existing data using the key information
• DELETE delete the record (but this can be disabled)

7. Message Structure
{
"_index" : "test",
"_type" : "messages",
"_id" : "99999",
"_version" : 1,
"found" : true,
"_source" : {
"msg" : "Hello Elasticsearch",
"committime" : "2017-06-23 19:02:22.0",
"id" : "99999",
"source_table" : "messages",
"source_schema" : "test"
}
}

8. Customizable Elements
• Documentation ID format
– pkey, pkeyus, tspkey, tspkeyus
• Embedded information
– Commit time
– Schema/table source information
• Ignore delete or update errors
• Whether to use schema and table names as index and type
• Custom index name and type names
• Whether to use a self generated ID or one derived from primary key

9. Demo

10. Future Direction
• Further customization of the structures and stored information
• Integration with the data combination and concentration tools
• DDL integration and control of the structures
– Currently TRUNCATE and CREATE/DROP TABLE are not replicated
• Improved data structures and record metadata

11. Next Steps
• If you are interested in knowing more about Tungsten Replicator and would like to try it out for
yourself, please contact our sales team who will be able to take you through the details and
setup a POC – sales@continuent.com
• Read the documentation at http://docs.continuent.com
• Follow us on Twitter @ continuent or www.facebook.com/Continuent
• Subscribe to our Tungsten University YouTube channel! http://tinyurl.com/TungstenUni
14

12. For more information, contact us:
Eero Teerikorpi
Founder, CEO
eero.teerikorpi@continuent.com
+1 (408) 431-3305
Eric Stone
COO
eric.stone@continuent.com
MC Brown
VP Products
mc.brown@continuent.com
Petri Virsunen
VP Marketing
petri.virsunen@continuent.com
+1 (408) 806-9860