DataOps barcelona - MySQL 8.0 document store: NoSQL with all the benefits of a transactional RDBMS

MySQL 8.0
a Document Store with all the benefits of a transactional RDBMS

Frédéric Descamps - MySQL Community Manager - Oracle
3 / 120

Save the date !
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
4 / 120

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purpose only, and may not
be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be
relied up in making purchasing decisions. The development, release and timing of any features or functionality described
for Oracle´s product remains at the sole discretion of Oracle.
5 / 120

about me - http://about.me/lefred
Who am I ?
6 / 120

Frédéric Descamps
@lefred
MySQL Evangelist
Hacking MySQL since 3.23
devops believer
living in Belgium 🇧🇪
http://lefred.be

7 / 120

RDBMS & MySQL
Relational Databases
8 / 120

Data Integrity
9 / 120

Data Integrity
normalization
10 / 120

Data Integrity
normalization
constraints (foreign keys, ...)
11 / 120

Data Integrity
normalization
Atomicity, Consistency, Isolation, Durability
12 / 120

Data Integrity
normalization
ACID compliant
13 / 120

Data Integrity
normalization
ACID compliant
transactions
14 / 120

Data Integrity
normalization
ACID compliant
transactions
SQL
15 / 120

Data Integrity
normalization
ACID compliant
transactions
SQL
powerfull query language
16 / 120

NoSQL & MySQL
NoSQL Databases
17 / 120

NoSQL or Document Store
Schemaless
18 / 120

Schemaless
no schema design, no normalization, no foreign keys, no data types, ...
19 / 120

Schemaless
very quick initial development
20 / 120

Schemaless
Flexible data structure
21 / 120

Schemaless
embedded arrays or objects
22 / 120

Schemaless
valid solution when natural data can´t be modelized optimaly into a relational model
23 / 120

Schemaless
objects persistance without the use of any ORM - mapping oobject-oriented
24 / 120

Schemaless
JSON
25 / 120

Schemaless
JSON
close to frontend
26 / 120

Schemaless
JSON
close to frontend
native in JS
27 / 120

Schemaless
JSON
close to frontend
native in JS
easy to learn
28 / 120

How DBAs see data
29 / 120

How DBAs see data How Developers see data
{
"GNP" : 249704,
"Name" : "Belgium",
"government" : {
"GovernmentForm" :
"Constitutional Monarchy, Federation",
"HeadOfState" : "Philippe I"
},
"_id" : "BEL",
"IndepYear" : 1830,
"demographics" : {
"Population" : 10239000,
"LifeExpectancy" : 77.8000030517578
},
"geography" : {
"Region" : "Western Europe",
"SurfaceArea" : 30518,
"Continent" : "Europe"
}
}
30 / 120

What if there was a way to provide both SQL and NoSQL on one
stable platform that has proven stability on well know
technology with a large Community and a diverse ecosystem ?
31 / 120

DBMS or NoSQL ?
32 / 120

DBMS or NoSQL ?
Why not both ?
33 / 120

The MySQL Document Store !
SQL is now optional !
34 / 120

Developers:
[x] schemaless
[x] rapid prototying/simpler APIs
[x] document model
[x] transactions
Operations:
[x] performance management/visibility
[x] robust replication, backup, restore
[x] comprehensive tooling ecosystem
[x] simpler application schema upgrades
Business Owner:
[x] don't lose my data == ACID trx
[x] capture all my data = extensible/schemaless
[x] product on schedule/time to market = rapid
developement
A solution for all
35 / 120

MySQL Document Store
the Solution
36 / 120

37 / 120

Built on the
MySQL JSON
Data type and
Proven MySQL
Server
Technology
Provides a schema flexible JSON Document Store
No SQL required
No need to define all possible attributes, tables, etc.
Uses new X DevAPI
Can leverage generated column to extract JSON values into materialized columns
that can be indexed for fast SQL searches.
Document can be ~1GB
It's a column in a row of a table
It cannot exceed max_allowed_packet
Allows use of modern programming styles
No more embedded strings of SQL in your code
Easy to read
Also works with relational Tables
Proven MySQL Technology
38 / 120

X DevAPI Connectors for
C++, Java, .Net, Node.js, Python, PHP
working with Communities to help them supporting it too
New MySQL Shell
Command Completion
Python, JavaScrips & SQL modes
Admin functions
New Util object
A new high-level session concept that can scale from single MySQL Server to
a multiple server environment
Non-blocking, asynchronous calls follow common language patterns
Supports CRUD operations
39 / 120

Starting using MySQL DS in few minutes

40 / 120

MySQL Shell Info
41 / 120

Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
42 / 120

We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data.
43 / 120

We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data.
44 / 120

Dump and load using MySQL Shell & Python
this example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000;
db.restaurants. nd().shellPrint()'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json
45 / 120

Dump and load using MySQL Shell & Python
this example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000;
db.restaurants. nd().shellPrint()'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json
46 / 120

47 / 120

Let´s query
48 / 120

Let´s query
That´s too much records to show it here... let´s limit it
49 / 120

50 / 120

Some more examples
51 / 120

Some more examples
Let´s add a selection criteria:
52 / 120

Using IN...
53 / 120

Syntax slightly different than MongoDB
54 / 120

Syntax slightly different than MongoDB
> db.restaurants. nd({"cuisine": "French",
"borough": { $not: /^Manhattan/} },
{"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2)
{ "borough" : "Queens", "cuisine" : "French",
55 / 120

CRUD operations
56 / 120

CRUD operations for collections
Add a document
57 / 120

Modify a document
58 / 120

Remove a document
59 / 120

MySQL Document Store Objects Summary
60 / 120

Document Store Full ACID !
It relies on the proven MySQL InnoDB´s strength & robustness:
61 / 120

innodb_ ush_log_at_trx_commit = 1
62 / 120

innodb_doublewrite = ON
63 / 120

sync_binlog = 1
64 / 120

sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
65 / 120

sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
We do care about your data !
66 / 120

MySQL Document Store supports transactions
67 / 120

MySQL Document Store supports transactions
68 / 120

69 / 120

what about my old SQL ?
The hidden part of the iceberg
70 / 120

JSON datatype is behind the scene
native datatype (since 5.7.8)
71 / 120

JSON values are stored in MySQL tables using UTF8MB4
72 / 120

Conversion from "native" SQL types to JSON values
73 / 120

JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)
74 / 120

Generated/virtual columns
75 / 120

Indexing JSON data
76 / 120

Indexing JSON data
Foreign Keys to JSON data
77 / 120

Indexing JSON data
Foreign Keys to JSON data
SQL Views to JSON data
78 / 120

How does the MySQL Document Store work ?
79 / 120

What does a collection look like on the server ?
80 / 120

_id Every document has a unique identifier called the document ID, which can be thought
of as the equivalent of a table´s primary key. The document ID value can be manually
assigned when adding a document. If no value is assigned, a document ID is
generated and assigned to the document automatically !
Use getDocumentId() or getDocumentIds() to get _ids(s)

81 / 120

Mapping to SQL examples
createCollection('mycollection')
82 / 120

CREATE TABLE `test`.`mycoll` (
doc JSON,
_id VARCHAR(32)
GENERATED ALWAYS AS (doc->>'$._id') STORED
PRIMARY KEY
) CHARSET utf8mb4;
83 / 120

doc JSON,
_id VARCHAR(32)
PRIMARY KEY
) CHARSET utf8mb4;
mycollection.add({'test': 1234})
84 / 120

doc JSON,
_id VARCHAR(32)
PRIMARY KEY
) CHARSET utf8mb4;
mycollection.add({'test': 1234})
INSERT INTO `test`.`mycoll` (doc)
VALUES (
JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf',
'test',1234)
);
85 / 120

Mapping to SQL examples (2)
mycollection.find("test > 100")
86 / 120

Mapping to SQL examples (2)
mycollection.find("test > 100")
SELECT doc
FROM `test`.`mycoll`
WHERE (JSON_EXTRACT(doc,'$.test') > 100);
87 / 120

SQL and JSON Example
88 / 120

same as:
89 / 120

same as:
we can use it in SQL:
90 / 120

Indexes in Document Store
It's also possible to create indexes without using SQL syntax:
91 / 120

Indexes in Document Store
It's also possible to create indexes without using SQL syntax:
92 / 120

SQL and JSON Example (2): validation
93 / 120

94 / 120

95 / 120

SQL and JSON Example (3): explain
96 / 120

SQL and JSON Example (3): explain
97 / 120

SQL and JSON Example (4): add index
98 / 120

SQL and JSON Example (4): add index
99 / 120

SQL and JSON Example (5): arrays
100 / 120

101 / 120

102 / 120

103 / 120

NoSQL as SQL
JSON_TABLE
104 / 120

NoSQL as SQL (2)
JSON_TABLE
105 / 120

NoSQL as SQL - aggregation
106 / 120

NoSQL as SQL - aggregation (2)
107 / 120

SQL and NoSQL - JOINS
108 / 120

109 / 120

110 / 120

SQL and NoSQL - JOINS (2)
111 / 120

112 / 120

113 / 120

SQL and NoSQL - and back to JSON Document
114 / 120

NoSQL or SQL
JS> session.sql("select doc->>'$.name', doc->>'$.cuisine'
from restaurants where doc->>'$.cuisine'='Italian' limit 2")
+----------------------------+-------------------+
| doc->>'$.name' | doc->>'$.cuisine' |
+----------------------------+-------------------+
| Philadelhia Grille Express | Italian |
| Isle Of Capri Resturant | Italian |
+----------------------------+-------------------+
2 rows in set (0.0287 sec)
JS> db.restaurants. nd("cuisine='Italian'"). elds('name','cuisine').limit(2)
[
{
"cuisine": "Italian",
"name": "Philadelhia Grille Express"
},
{
"cuisine": "Italian",
"name": "Isle Of Capri Resturant"
}
]
2 documents in set (0.0015 sec)
115 / 120

what do I gain ?
Conclusion
116 / 120

schemaless
flexible data structure
easy to start (CRUD)
Conclusion
This is the best of the two worlds in one product !
Data integrity
ACID Compliant
Transactions
SQL
117 / 120

118 / 120

http://lefred.be/content/top-10-reasons-for-nosql-with-mysql/
119 / 120

Thank you !
Any Questions ?
share your 💕 for MySQL on social media using @mysql #DataOpsBarcelona
120 / 120

DataOps barcelona - MySQL 8.0 document store: NoSQL with all the benefits of a transactional RDBMS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to DataOps barcelona - MySQL 8.0 document store: NoSQL with all the benefits of a transactional RDBMS

Similar to DataOps barcelona - MySQL 8.0 document store: NoSQL with all the benefits of a transactional RDBMS (20)

More from Frederic Descamps

More from Frederic Descamps (20)

Recently uploaded

Recently uploaded (20)

DataOps barcelona - MySQL 8.0 document store: NoSQL with all the benefits of a transactional RDBMS