https://www.bigdataspain.org/2016/program/thu-migration-coexistence-between-relational-nosql-databases.html https://www.youtube.com/watch?v=mM9pchslCn4&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&index=56&t=18s

1. ©2015 Couchbase Inc. 1

2. Migration and Coexistence between
RDBMS and NoSQL databases
Manuel Hurtado
Solutions Engineer

3. ©2015 Couchbase Inc. 3
What is NoSQL?

4. ©2015 Couchbase Inc. 4
Dynamo
October 2007
Cassandra
August 2008
Bigtable
November 2006
Voldemort
February 2009
Very few organizations want to (fewer can) build and maintain database software technology.
But every organization participating in the digital economy needs this technology.
Why NoSQL
New data solutions to address these needs:
 No schema required before inserting data
 No schema change required to change data format
 Auto-Sharding without application participation
 Distributed queries
 ManagedCache
 Data Synchronization ( multi-datacenter)

5. ©2015 Couchbase Inc. 5
What is NoSQL?
 No SQL? No.
 Not only SQL? Not really.
 Non relational?Yes.
 Distributed (most)
– Scaled out, not up
• Elasticity and commodity hardware
– Partitioned and replicated
• Scalability, performance, availability
 Schema-less (most)
– Flexible model
– JSON (some)

6. ©2015 Couchbase Inc. 6
World of Operational Databases
6
Graph
Stores
Neo4j
RDBMS/Transactional
Stores
Oracle, DB2, SQLServer,
MySql, Postgres
Caching
Stores
Coherence
, Redis,
Gemfire
Full-text
Stores
Lucene, SOLR,
ElasticSearch
Key/Value
Stores
Hbase,
Cassandra,
Riak, Dynamo
Document Stores
Marklogic, mongoDB,
Cloudant,
DocumentDB

7. ©2015 Couchbase Inc. 7
Why migrate from RDBMS to NoSQL?

8. ©2015 Couchbase Inc. 8
What Is Driving Use of NoSQL?

9. ©2015 Couchbase Inc. 9
What’s the problem with relational?
Scalability and Performance
 Single server
 Fixed capacity
 Under-provisioned
 Not enough resources
 Poor performance
 Over-provisioned
 Too many resources
 Unnecessary cost

10. ©2015 Couchbase Inc. 10
How does NoSQL solve it?
Scalability and Performance
 Cluster
 Commodity hardware
 Variable capacity
 Add resources
 On demand
 Without downtime
 High availability

11. ©2015 Couchbase Inc. 11
Agility: Changing the data model
Relational database
• Modify the database schema
• Modify the application code (e.g.
Java)
• Modify the interface (e.g. HTML5/JS)
Document database
• Modify the interface (e.g. HTML5/JS)

12. ©2015 Couchbase Inc. 12
Identifying the right application
Have one or more of the following characteristics or requirements:
 Iterate faster
 Send and receive JSON
 Provide low latency at any throughput
 Support many concurrent users
 Supports users anywhere and everywhere
 Be available 24x7
 Store terabytes of data
 Read and write to multiple data centers
Service
RDMBS
Service Service
NoSQL
Application

13. ©2015 Couchbase Inc. 13
Identifying the right application
Examples
• Product Catalog Service
• AssetTracking Service
• Content Management Service
• Application Configuration Service
• Customer Management Service
• File or Streaming Metadata Service
Service
RDMBS
Service Service
NoSQL
Application

14. ©2015 Couchbase Inc. 14
How can you introduce NoSQL with microservices?
 Introduce NoSQL with a single service
 Could be a new service, could be an existing one
 Update other services one at a time
 Where it makes sense…
Polyglot!

15. ©2015 Couchbase Inc. 15
How can you introduce NoSQL with caching?
 Deploy as mission-critical cache
 Primary source of engagement
 Handles all external reads & writes
 Results in less RDMBS costs (hardware)
 Results in less MIPS (mainframe usage)
 Leverage streaming replication to sync

16. ©2015 Couchbase Inc. 16
Modeling you Data

17. ©2015 Couchbase Inc. 17
Data Modeling
 An ecommerce order in relational world

18. ©2015 Couchbase Inc. 18
Data Modeling
 Embed vs Refer

19. ©2015 Couchbase Inc. 19
Data Modeling
 Embed vs Refer

20. ©2015 Couchbase Inc. 20
Modeling your data: Fixed vs. self-describing schema

21. ©2015 Couchbase Inc. 21
Modeling your data:The flexibility of JSON
Same document type,
Different fields
• Different types
• Optional
• On-demand

22. ©2015 Couchbase Inc. 22
Modeling your data: Strategies and best practices
If… Then…
Relationship is one-to-one or one-to-many Store related data as nested objects
Relationship is many-to-one or many-to-many Store related data as separate documents
Data reads are mostly parent fields Store children as separate documents
Data reads are mostly parent + child fields Store children as nested objects
Data writes are mostly parent or child (not both) Store children as separate documents
Data writes are mostly parent and child (both) Store children as nested objects

23. ©2015 Couchbase Inc. 23
What is Couchbase Server?

24. ©2015 Couchbase Inc. 24
What’s Couchbase?
Couchbase is the company behind Couchbase Server & Couchbase Mobile
• Open source JSON database
• Founded 2010
• 500+ enterprise customers globally
Some of our customers:
Couchbase Server can be deployed as:
Document database Key-value store Distributed cache

25. ©2015 Couchbase Inc. 25
Top Use Cases
Profile Management Personalization 360o Customer View Internet of Things
Content Management Catalog Real Time Big Data Digital Communication
Mobile Applications
High Availability Caching

26. ©2015 Couchbase Inc. 26
What is Couchbase Server?
The Scalability of NoSQL + The Flexibility of JSON + The Power of SQL
Develop withAgility Operate atAny Scale
 Elastic scalability
 Consistent high performance
 Always-on availability
 Multi-Data Center Deployment
 Simple & Powerful Administration
 Enterprise-grade Security
 Easier, faster development
 Flexible data modeling
 Powerful querying & indexing
 SQL integration & migration
 Big data integration
 Mobile / IoT Extensions

27. [
{
"brewery": "BrewDog Ltd",
"beers": 4,
},
{
"brewery": "Founders Brewing"
"beers": 4,
},
…
]
SELECT brewery.name AS brewery, count(*) AS beers
FROM `beer-sample` beer
INNER JOIN `beer-sample` brewery
ON KEYS beer.brewery_id
WHERE beer.type = "beer" AND
beer.style = "American-Style Imperial Stout" AND
brewery.type = "brewery"
GROUP BY brewery.name
HAVING count(*) > 2
ORDER BY beers DESC, brewery ASC;
CREATE INDEX beer_styles ON `beer-sample`(style)WHERE type = "beer”;

28. ©2015 Couchbase Inc. 28
Integrating RDBMS and NoSQL

29. ©2015 Couchbase Inc. 29
The reality:The Data Layer Landscape
Challenges
 Legacy Systems
 DifferentTechnologies
 Data Ownership
 Bulk loading
 Data synchronization
RDBMS
NoSQL
RDBMS
RDBMS
NoSQL
NoSQL
NoSQL
HadoopMainframe

30. ©2015 Couchbase Inc. 30
The reality: it is integration time!
Challenges
 Legacy Systems
 DifferentTechnologies
 Data Ownership
 Bulk loading
 Data synchronization

31. ©2015 Couchbase Inc. 31
Integrating RDBMS and NoSQL
Tasks
 Export/Import/Bulk loading
 Data synchronization
 Data Modeling: NoSQL- RDBMS gap
Tools
 Export/Import/Bulk loading
 SDK Client API
 ETLTools (Talend, Informatica, Pentaho,...)
RDBMS
NoSQL
RDBMS
RDBMSNoSQL
NoSQL
NoSQL
HadoopMainframe

32. ©2015 Couchbase Inc. 32
Integrating RDBMS and NoSQL: Demo 1
Demo: Bulk loading from Oracle to Couchbase
 Custom tool based on SDKAPI
 Takes care of data types
 https://github.com/mahurtado/oracle2couchbase

33. ©2015 Couchbase Inc. 33
Integrating RDBMS and NoSQL: Demo 1
Demo: Bulk loading from Oracle to Couchbase

34. ©2015 Couchbase Inc. 34
Integrating RDBMS and NoSQL: Demo 2
Demo: Bulk loading from PostgreSQL to Couchbase
 Use Case: Product Catalog
 Export to csv from PostgreSQL utility
 Import using Couchbase utility: cbbackupmgr

35. ©2015 Couchbase Inc. 35
Integrating RDBMS and NoSQL: Demo 3
Demo: Synchronize data from Oracle to Couchbase
 UsingOracle Golden Gate
 Golden Gate Java Adapter Using Couchbase SDK
 https://github.com/mahurtado/CouchbaseGoldenGateAdapter

36. ©2015 Couchbase Inc. 36
Integrating RDBMS and NoSQL: Demo 4
Demo: Synchronize data from Couchbase to PostgreSQL
 UsingCouchbase Kafka connector
 Kafka consumer writes on PostgreSQL
RealTime changes
Couchbase
Kafka
Connector
Kafka
Consumer

37. Thank you!