PyConX - Python & MySQL 8.0 Document Store

5/2/2019 Python & MySQL 8.0 Document Store
ﬁle:///home/fred/ownCloud/Presentations/ORACLE/PyconX/Python e MySQL 8.0 Document Store/Python e MySQL 8.0 Document Store.html#49 1/104
1 / 104

2 / 1042 / 104


Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purpose only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. The development, release and timing of any features or
functionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2019 Oracle and/or its affiliates. All rights reserved.
3 / 104

about me - http://about.me/lefred
Who am I ?
4 / 104

Frédéric Descamps
@lefred
MySQL Evangelist
Hacking MySQL since 3.23
devops believer
living in Belgium 🇧🇪
http://lefred.be

5 / 104

MySQL and JSON
How MySQL works with JSON
6 / 104

JavaScript Object Notation (JSON)
{
"_id": "5ad5b645f88c5bb8fe3fd337",
"name": "Morris Park Bake Shop",
"grades": [
{
"date": "2014-03-03T00:00:00Z",
"grade": "A",
"score": 2
},
],
"address": {
"coord": [
-73.856077,
40.848447
],
"street": "Morris Park Ave",
"zipcode": "10462",
"building": "1007"
},
"borough": "Bronx",
"cuisine": "Bakery",
"restaurant_id": "30075445"
}
7 / 104

8.0
MySQL has Native JSON Support
JSON data type since MySQL 5.7
8 / 104

8.0
Stored as binary object
9 / 104

8.0
Support for partial updates in MySQL 8.0
10 / 104

8.0
Support for partial updates in MySQL 8.0
28 JSON functions
11 / 104

8.0
... and support JSON Paths
$.address.coord[*]
$ the root of the document
12 / 104

8.0
$.address.coord[*]
. path leg separator
13 / 104

8.0
$.address.coord[*]
* wildcard:
14 / 104

8.0
$.address.coord[*]
* wildcard:
.* all members in the object
[*] all values in the array
[prefix]**suffix path beginning with prefix and ending with suffix
15 / 104

MySQL Document Store
the Solution
16 / 104

Built on
the MySQL
JSON Data
type and
Proven
MySQL
Server
Technology
Provides a schema flexible JSON Document Store
No SQL required
No need to define all possible attributes, tables, etc.
Uses new X DevAPI
Can leverage generated column to extract JSON values into
materialized columns that can be indexed for fast SQL searches.
Document can be ~1GB
It's a column in a row of a table
It cannot exceed max_allowed_packet
Allows use of modern programming styles
No more embedded strings of SQL in your code
Easy to read
Also works with relational Tables
Proven MySQL Technology
17 / 104

X Protocol Connectors
18 / 104

X DevAPI Developed from scratch for modern development
Supports:
SQL
NoSQL - JSON documents
NoSQL - SQL tables
Uniform API across programming languages
We provide connectors for
C++, Java, .Net, Node.js, Python, PHP
working with Communities to help them supporting it too
Supported in the MySQL Shell
Full ACID support
Savepoints
Connection pools (as of 8.0.13)
19 / 104

PY
JS
SQL
>
MySQL 8.0
Upgrade Checker
Auto Completion
&
Command History
Output
Formats
(table, json, tabs)
Prompt
Themes
Batch
Execution
5.7
8.0
JS
Document
Store
X dev API
SQL CLI InnoDB
Cluster
importJSON
JSJS
20 / 104

Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
21 / 104

Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
22 / 104

23 / 104

24 / 104

Let´s query
25 / 104

Let´s query
That´s too much records to show here...
26 / 104

Let´s query
27 / 104

Let´s query
Let´s add a limit to it
28 / 104

29 / 104

Some more examples
30 / 104

Some more examples
Let´s add a selection criteria:
31 / 104

Using IN...
32 / 104

Syntax slightly different than MongoDB
33 / 104

Syntax slightly different than MongoDB
> db.restaurants. nd({"cuisine": "French",
"borough": { $not: /^Manhattan/} },
{"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2)
{ "borough" : "Queens", "cuisine" : "French",
"name" : "La Baraka Restaurant" }
{ "borough" : "Queens", "cuisine" : "French",
"name" : "Air France Lounge" }
34 / 104

CRUD operations
35 / 104

CRUD operations for collections
Add a document
36 / 104

Add a document
collection.add({ name: 'fred', age: 42 })
.add({ name: 'dave', age: 23 })
.execute()
collection.add([
{ name: 'dimo', age: 50 },
{ name: 'kenny', age: 25 }
]).execute()
37 / 104

Modify a document
38 / 104

Modify a document
collection.modify('name = :name')
.bind('name', 'fred')
.set('age', 43)
.sort('name ASC')
.limit(1)
.execute()
collection.modify('name = :name')
.patch({ age: 43, active: false })
.sort('name DESC')
.limit(1)
.execute()
39 / 104

Remove a document
40 / 104

Remove a document
collection.remove('name = :name')
.sort('age ASC')
.limit(1)
.execute()
41 / 104

MySQL Document Store Objects Summary
42 / 104

All you need to know is here:
https://dev.mysql.com/doc/x-devapi-userguide/en/crud-operations-overview.html
43 / 104

MySQL Connector Python
The Python connector for MySQL
44 / 104

NoSQL
SQL

MySQL Connector/Python 8
GA since April 2018
45 / 104

NoSQL
SQL

GA since April 2018
Maintained by Oracle
46 / 104

NoSQL
SQL

GA since April 2018
Dual license
47 / 104

NoSQL
SQL

GA since April 2018
Dual license
Support MySQL Server 5.5, 5.6, 5.7 and
8.0
48 / 104

NoSQL
SQL

GA since April 2018
Dual license
8.0
SQL and NoSQL support
49 / 104

NoSQL
SQL

GA since April 2018
Dual license
8.0
Table and JSON Document support
50 / 104

NoSQL
SQL

GA since April 2018
Dual license
8.0
Table and JSON Document support
All you need to know is here: https://dev.mysql.com/doc/connector-python/en/
51 / 104

MySQL Connector/Python 8 - 3 APIs
Choice of Three APIs
API Python Module Comment
PEP249 Python Database API mysql.connector The traditional API
C Extension API _mysql_connector Similar to the MySQL C API
MySQL X DevAPI mysqlx New in 8.0, both SQL and NoSQL
52 / 104


MySQL Connect/Python and Django
MySQL Connector/Python includes also a mysql.connector.django module that
provides a Django back end for MySQL
DATABASES = {
'default': {
'NAME': 'user_data',
'ENGINE': 'mysql.connector.django',
'USER': 'mysql_user',
'PASSWORD': 'password',
'OPTIONS': {
'autocommit': True,
},
}
}
53 / 104

MySQL Connector/Python 8 - installation
easiest, use pip:
use a MySQL repository for your distribution (fedora, ubuntu, debian, ...)
54 / 104

MySQL Connector/Python 8 - installation (2)
or go to https://dev.mysql.com/downloads/connector/python/
55 / 104

What time is it ?
Time to code !
56 / 104

Some coding...
import mysqlx
session = mysqlx.get_session( {
'host': 'localhost', 'port': 33060,
'user': 'fred', 'password': 'fred' })
db = session.get_schema('docstore')
col = db.get_collection('all_recs')
restaurants = col. nd("cuisine='Italian'"). elds("name, cuisine, borough")
.limit(3).execute()
for restaurant in restaurants.fetch_all():
print restaurant
session.close()
57 / 104

If it was not yet obvious, you can now use
MySQL without SQL !
58 / 104

OK we have Document Store, CRUD and ACID
but what makes MySQL Document Store unique ?
59 / 104

Challenge: list the best restaurant of each type
of food and show the top 10, with the best one
first !

don't forget that all these restaurants are just JSON documents
60 / 104

NoSQL as SQL - aggregation
61 / 104

62 / 104

63 / 104

import mysqlx
session = mysqlx.get_session( {
'host': 'localhost', 'port': 33060,
'user': 'fred', 'password': 'fred' })
db = session.get_schema('docstore')
col = db.get_collection('all_recs')
restaurants = col. nd("cuisine='Italian'"). elds("name, cuisine, borough")
.limit(3).execute()
for restaurant in restaurants.fetch_all():
print restaurant
result = session.sql('''WITH cte1 AS (SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]"
COLUMNS (score INT PATH "$.score")) AS r) AS avg_score
FROM docstore.all_recs) SELECT *, RANK() OVER
(PARTITION BY cuisine ORDER BY avg_score DESC) AS `rank`
FROM cte1 ORDER BY `rank`, avg_score DESC LIMIT 10''').execute()
for restaurant in result.fetch_all():
print "%s - %s - %d " % (restaurant[0], restaurant[1], restaurant[2])
session.close()
64 / 104

Output
65 / 104

And back to JSON
And of course it's possible to return the result as a JSON document:
result = session.sql('''select JSON_PRETTY(JSON_ARRAYAGG(JSON_OBJECT(
"name", name, "cuisine", cuisine,
"score", avg_score)))
from (WITH cte1 AS (SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score)
FROM JSON_TABLE(doc, "$.grades[*]"
COLUMNS (score INT PATH "$.score"))
AS r) AS avg_score
FROM docstore.all_recs) SELECT *, RANK() OVER
(PARTITION BY cuisine ORDER BY avg_score DESC)
AS `rank`
FROM cte1 ORDER BY `rank`, avg_score
DESC LIMIT 10) as t''').execute()
row = result.fetch_one()
print row[0]
66 / 104

And back to JSON - output
67 / 104

some extras...
The hidden part of the iceberg
68 / 104

_id Every document has a unique identifier called the document ID. It can
be manually assigned when adding a document or generated and
assigned to the document automatically !
Since MySQL 8.0.11, the _id is generated by the server. To get the list
of automatically generated IDs use the
result.getGeneratedIDs() method.
The _id is made of 3 parts, all hex encoded:
prefix
timestamp when the MySQL Server instance was started
auto-increment counter
Globally unique IDs
Optimized for InnoDB storage
Examples
00005cc17b700000000000000003
69 / 104

Document Store Full ACID !
It relies on the proven MySQL InnoDB´s strength & robustness:
70 / 104

innodb_ ush_log_at_trx_commit = 1
71 / 104

innodb_doublewrite = ON
72 / 104

sync_binlog = 1
73 / 104

sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-
COMMITTED|...
74 / 104

sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-
COMMITTED|...
We do care about your data !
75 / 104

What time is it now ?
Time for some more
examples !
76 / 104

More fun with The Walking Dead
77 / 104

Example: All Episodes of Season 1
78 / 104

Example: All First Episodes of Each Season
79 / 104

Example: Speed Up Episode's Name Lookup
80 / 104

81 / 104

82 / 104

Example: Transaction
83 / 104

Example: Do more with SQL
List the amount of episodes by season:
84 / 104

Example: Do more with SQL (2)
Episode statistics for each season:
85 / 104

Example: Do more with SQL (3)
Number of days between episodes:
86 / 104

JSON Data validation
It's possible to have constraints between collections (or tables):
87 / 104

JSON Data validation (2)
Let's see in SQL how the 2 collections look like:
88 / 104

Let's add one virtual column in each collection:
89 / 104

Now we can index the reference and create the constraint:
90 / 104

91 / 104

And now even in pure NoSQL CRUD, we can see the data validation in action:
92 / 104

JSON Data validation (6) - NEW 8.0.16
Since MySQL 8.0.16, CHECK constraints are now supported !
93 / 104

94 / 104

95 / 104

what do I gain ?
Conclusion
96 / 104

schemaless
flexible data structure
easy to start (CRUD)
Conclusion
This is the best of the two worlds in one product !
Data integrity
ACID Compliant
Transactions
SQL
97 / 104

98 / 104

8.0
http://lefred.be/content/top-10-reasons-for-
nosql-with-mysql/
99 / 104

8.0
https://www.slideshare.net/lefred.descamps/mysql-
shell-the-best-dba-tool
https://github.com/lefred/mysql-
shell-mydba
https://github.com/lefred/mysql-
shell-innotop
More with MySQL Shell & Python
100 / 104

MySQL Shell Reporting Framework - 8.0.16
Now it's also possible to create user-defined reports directly in the Shell. They can be
written in Python and called the same way on every operating systems. No need for extra
libraries !
More Info:
https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-reporting.html
https://mysql.wisborg.dk/2019/04/26/mysql-shell-8-0-16-built-in-reports/
https://mysql.wisborg.dk/2019/04/27/mysql-shell-8-0-16-user-defined-reports/
101 / 104

Credits
Thank you Olivier Dasini for some TV Show examples:
http://dasini.net/blog/2019/04/02/mysql-json-document-store/
Thank you Jesper Wisborg Krogh for inspiration extending the MySQL Shell
102 / 104

103 / 104

Thank you !
Any Questions ?
share your 💕 for MySQL on social media using @mysql #MySQL8isGreat
#MySQL
104 / 104

PyConX - Python & MySQL 8.0 Document Store

PyConX - Python & MySQL 8.0 Document Store

Recommended

More Related Content

What's hot

What's hot(20)

Similar to PyConX - Python & MySQL 8.0 Document Store

Similar to PyConX - Python & MySQL 8.0 Document Store(20)

More from Frederic Descamps

More from Frederic Descamps(20)

Recently uploaded

Recently uploaded(20)

PyConX - Python & MySQL 8.0 Document Store