2. Python and MySQL 8.0 Document
Store
DEV5966
Frédéric Descamps
Community Manager
MySQL
September 2019
2 / 111
3.
Safe Harbor
The following is intended to outline our general product direction. It is intended for information purpose only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any
features or functionality described for Oracle's product may change and remains at the sole discretion of Oracle
Corporation.
Statement in this presentation relating to Oracle's future plans, expectations, beliefs, intentions and ptospects are
"forward-looking statements" and are subject to material risks and uncertainties. A detailed discussion of these
factors and other risks that a ect our business is contained in Oracle's Securities and Exchange Commission
(SEC) lings, including our most recent reports on Form 10-K and Form 10-Q under the heading "Risk Factors".
These lings are available on the SEC's website or on Oracle's website at h p://www.oracle.com/investor. All
information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any
statement in light of new information or future events.
Copyright @ 2019 Oracle and/or its affiliates.
3 / 111
8. 8.0
MySQL has Native JSON Support
JSON data type since MySQL 5.7
Copyright @ 2019 Oracle and/or its affiliates.
8 / 111
9. 8.0
MySQL has Native JSON Support
JSON data type since MySQL 5.7
Stored as binary object
Copyright @ 2019 Oracle and/or its affiliates.
9 / 111
10. 8.0
MySQL has Native JSON Support
JSON data type since MySQL 5.7
Stored as binary object
Support for partial updates in MySQL 8.0
Copyright @ 2019 Oracle and/or its affiliates.
10 / 111
11. 8.0
MySQL has Native JSON Support
JSON data type since MySQL 5.7
Stored as binary object
Support for partial updates in MySQL 8.0
30 JSON functions
Copyright @ 2019 Oracle and/or its affiliates.
11 / 111
12. 8.0
... and support JSON Paths
$.address.coord[*]
$ the root of the document
Copyright @ 2019 Oracle and/or its affiliates.
12 / 111
13. 8.0
... and support JSON Paths
$.address.coord[*]
$ the root of the document
. path leg separator
Copyright @ 2019 Oracle and/or its affiliates.
13 / 111
14. 8.0
... and support JSON Paths
$.address.coord[*]
$ the root of the document
. path leg separator
* wildcard:
Copyright @ 2019 Oracle and/or its affiliates.
14 / 111
15. 8.0
... and support JSON Paths
$.address.coord[*]
$ the root of the document
. path leg separator
* wildcard:
.* all members in the object
[*] all values in the array
[pre x]**su x path beginning with
pre x and ending with su x
Copyright @ 2019 Oracle and/or its affiliates.
15 / 111
17. Built on
the MySQL
JSON Data
type and
Proven
MySQL
Server
Technology
Provides a schema exible JSON Document Store
No SQL required
No need to de ne all possible a ributes, tables, etc.
Uses new X DevAPI
Can leverage generated column to extract JSON values into
materialized columns that can be indexed for fast SQL searches.
Document can be ~1GB
It's a column in a row of a table
It cannot exceed max_allowed_packet
Allows use of modern programming styles
No more embedded strings of SQL in your code
Easy to read
Also works with relational Tables
Proven MySQL Technology
Copyright @ 2019 Oracle and/or its affiliates.
17 / 111
19. X DevAPI Developed from scratch for modern development
Supports:
SQL
NoSQL - JSON documents
NoSQL - SQL tables
Uniform API across programming languages
We provide connectors for
C++, Java, .Net, Node.js, Python, PHP
working with Communities to help them supporting it too
Supported in the MySQL Shell
Full ACID support
Savepoints
Connection pools (as of 8.0.13)
Copyright @ 2019 Oracle and/or its affiliates.
19 / 111
20. PY
JS
SQL
>
MySQL 8.0
Upgrade Checker
Auto Completion
&
Command History
Output
Formats
(table, json, tabs)
Prompt
Themes
Batch
Execution
5.7
8.0
JS
Document
Store
X dev API
SQL CLI InnoDB
Cluster
importJSON
JSJS
parallel
importTABLE
data
Copyright @ 2019 Oracle and/or its affiliates.
20 / 111
21. Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
Copyright @ 2019 Oracle and/or its affiliates.
21 / 111
22. Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
Copyright @ 2019 Oracle and/or its affiliates.
22 / 111
40. CRUD operations for collections
Remove a document
Copyright @ 2019 Oracle and/or its affiliates.
40 / 111
41. CRUD operations for collections
Remove a document
collection.remove('name = :name')
.bind('name', 'fred')
.sort('age ASC')
.limit(1)
.execute()
Copyright @ 2019 Oracle and/or its affiliates.
41 / 111
42. MySQL Document Store Objects Summary
Copyright @ 2019 Oracle and/or its affiliates.
42 / 111
43. All you need to know is here:
h ps://dev.mysql.com/doc/x-devapi-userguide/en/crud-operations-overview.html
Copyright @ 2019 Oracle and/or its affiliates.
43 / 111
44. MySQL Connector Python
The Python connector for MySQL
Copyright @ 2019 Oracle and/or its affiliates.
44 / 111
48. NoSQL
SQL
MySQL Connector/Python 8
GA since April 2018
Maintained by Oracle
Dual license
Support MySQL Server 5.5, 5.6, 5.7 and 8.0
Copyright @ 2019 Oracle and/or its affiliates.
48 / 111
49. NoSQL
SQL
MySQL Connector/Python 8
GA since April 2018
Maintained by Oracle
Dual license
Support MySQL Server 5.5, 5.6, 5.7 and 8.0
SQL and NoSQL support
Copyright @ 2019 Oracle and/or its affiliates.
49 / 111
50. NoSQL
SQL
MySQL Connector/Python 8
GA since April 2018
Maintained by Oracle
Dual license
Support MySQL Server 5.5, 5.6, 5.7 and 8.0
SQL and NoSQL support
Table and JSON Document support
Copyright @ 2019 Oracle and/or its affiliates.
50 / 111
51. MySQL Connector/Python 8 - 3 APIs
Choice of Three APIs
API Python Module Comment
PEP249 Python Database API mysql.connector The traditional API
C Extension API _mysql_connector Similar to the MySQL C API
MySQL X DevAPI mysqlx New in 8.0, both SQL and NoSQL
All you need to know is here: h ps://dev.mysql.com/doc/connector-python/en/
Copyright @ 2019 Oracle and/or its affiliates.
51 / 111
52. DATABASES = {
'default': {
'NAME': 'user_data',
'ENGINE': 'mysql.connector.django',
'USER': 'mysql_user',
'PASSWORD': 'password',
'OPTIONS': {
'autocommit': True,
},
}
}
MySQL Connect/Python and Django
MySQL Connector/Python includes also a mysql.connector.django module that provides a
Django back end for MySQL
Copyright @ 2019 Oracle and/or its affiliates.
52 / 111
53. MySQL Connector/Python 8 - installation
easiest, use pip:
use a MySQL repository for your distribution (fedora, ubuntu, debian, ...)
Copyright @ 2019 Oracle and/or its affiliates.
53 / 111
54. MySQL Connector/Python 8 - installation
or go to h ps://dev.mysql.com/downloads/connector/python/
Copyright @ 2019 Oracle and/or its affiliates.
54 / 111
55. What time is it ?
Time to code !
Copyright @ 2019 Oracle and/or its affiliates.
55 / 111
56. Some coding...
import mysqlx
session = mysqlx.get_session( {
'host': 'localhost', 'port': 33060,
'user': 'fred', 'password': 'fred' })
db = session.get_schema('docstore')
col = db.get_collection('all_recs')
restaurants = col. nd("cuisine='Italian'"). elds("name, cuisine, borough")
.limit(3).execute()
for restaurant in restaurants.fetch_all():
print restaurant
session.close()
Copyright @ 2019 Oracle and/or its affiliates.
56 / 111
57. If it was not yet obvious, you can now use
MySQL without SQL !
Copyright @ 2019 Oracle and/or its affiliates.
57 / 111
58. OK we have Document Store, CRUD and ACID
but what makes MySQL Document Store
unique ?
Copyright @ 2019 Oracle and/or its affiliates.
58 / 111
59. Challenge: list the best restaurant of each type
of food and show the top 10, with the best one
first !
don't forget that all these restaurants are just JSON documents
Copyright @ 2019 Oracle and/or its affiliates.
59 / 111
60. NoSQL as SQL - aggregation
Copyright @ 2019 Oracle and/or its affiliates.
60 / 111
61. NoSQL as SQL - aggregation
Copyright @ 2019 Oracle and/or its affiliates.
61 / 111
62. NoSQL as SQL - aggregation
Copyright @ 2019 Oracle and/or its affiliates.
62 / 111
63. import mysqlx
session = mysqlx.get_session( {
'host': 'localhost', 'port': 33060,
'user': 'fred', 'password': 'fred' })
db = session.get_schema('docstore')
col = db.get_collection('all_recs')
restaurants = col. nd("cuisine='Italian'"). elds("name, cuisine, borough")
.limit(3).execute()
for restaurant in restaurants.fetch_all():
print restaurant
result = session.sql('''WITH cte1 AS (SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]"
COLUMNS (score INT PATH "$.score")) AS r) AS avg_score
FROM docstore.all_recs) SELECT *, RANK() OVER
(PARTITION BY cuisine ORDER BY avg_score DESC) AS `rank`
FROM cte1 ORDER BY `rank`, avg_score DESC LIMIT 10''').execute()
for restaurant in result.fetch_all():
print "%s - %s - %d " % (restaurant[0], restaurant[1], restaurant[2])
session.close()
Copyright @ 2019 Oracle and/or its affiliates.
63 / 111
65. And back to JSON
And of course it's possible to return the result as a JSON document:
result = session.sql('''select JSON_PRETTY(JSON_ARRAYAGG(JSON_OBJECT(
"name", name, "cuisine", cuisine,
"score", avg_score)))
from (WITH cte1 AS (SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score)
FROM JSON_TABLE(doc, "$.grades[*]"
COLUMNS (score INT PATH "$.score"))
AS r) AS avg_score
FROM docstore.all_recs) SELECT *, RANK() OVER
(PARTITION BY cuisine ORDER BY avg_score DESC)
AS `rank`
FROM cte1 ORDER BY `rank`, avg_score
DESC LIMIT 10) as t''').execute()
row = result.fetch_one()
print row[0]
Copyright @ 2019 Oracle and/or its affiliates.
65 / 111
66.
And back to JSON - output
Copyright @ 2019 Oracle and/or its affiliates.
66 / 111
67. some extras...
The hidden part of the iceberg
Copyright @ 2019 Oracle and/or its affiliates.
67 / 111
68. _id Every document has a unique identi er called the document ID. It can
be manually assigned when adding a document or generated and
assigned to the document automatically !
Since MySQL 8.0.11, the _id is generated by the server. To get the list
of automatically generated IDs use the result.getGeneratedIDs()
method.
The _id is made of 3 parts, all hex encoded:
pre x
timestamp when the MySQL Server instance was started
auto-increment counter
Globally unique IDs
Optimized for InnoDB storage
Examples
00005cc17b700000000000000003
Copyright @ 2019 Oracle and/or its affiliates.
68 / 111
69. Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
Copyright @ 2019 Oracle and/or its affiliates.
69 / 111
70. Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
innodb_ ush_log_at_trx_commit = 1
Copyright @ 2019 Oracle and/or its affiliates.
70 / 111
71. Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
innodb_ ush_log_at_trx_commit = 1
innodb_doublewrite = ON
Copyright @ 2019 Oracle and/or its affiliates.
71 / 111
72. Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
innodb_ ush_log_at_trx_commit = 1
innodb_doublewrite = ON
sync_binlog = 1
Copyright @ 2019 Oracle and/or its affiliates.
72 / 111
73. Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
innodb_ ush_log_at_trx_commit = 1
innodb_doublewrite = ON
sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
Copyright @ 2019 Oracle and/or its affiliates.
73 / 111
74. Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
innodb_ ush_log_at_trx_commit = 1
innodb_doublewrite = ON
sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
We do care about your data !
Copyright @ 2019 Oracle and/or its affiliates.
74 / 111
75. What time is it now ?
Time for some more
examples !
Copyright @ 2019 Oracle and/or its affiliates.
75 / 111
76. More fun with The Walking Dead
Copyright @ 2019 Oracle and/or its affiliates.
76 / 111
77. Example: All Episodes of Season 1
Copyright @ 2019 Oracle and/or its affiliates.
77 / 111
78. Example: All First Episodes of Each Season
Copyright @ 2019 Oracle and/or its affiliates.
78 / 111
79. Example: Speed Up Episode's Name Lookup
Copyright @ 2019 Oracle and/or its affiliates.
79 / 111
80. Example: Speed Up Episode's Name Lookup
Copyright @ 2019 Oracle and/or its affiliates.
80 / 111
81. Example: Speed Up Episode's Name Lookup
Copyright @ 2019 Oracle and/or its affiliates.
81 / 111
83. Example: Do more with SQL
List the amount of episodes by season:
Copyright @ 2019 Oracle and/or its affiliates.
83 / 111
84. Example: Do more with SQL (2)
Episode statistics for each season:
Copyright @ 2019 Oracle and/or its affiliates.
84 / 111
85. Example: Do more with SQL (3)
Number of days between episodes:
Copyright @ 2019 Oracle and/or its affiliates.
85 / 111
86. JSON Data validation
It's possible to have constraints between collections (or tables):
Copyright @ 2019 Oracle and/or its affiliates.
86 / 111
87. JSON Data validation (2)
Let's see in SQL how the 2 collections look like:
Copyright @ 2019 Oracle and/or its affiliates.
87 / 111
88. JSON Data validation (3)
Let's add one virtual column in each collection:
Copyright @ 2019 Oracle and/or its affiliates.
88 / 111
89. JSON Data validation (4)
Now we can index the reference and create the constraint:
Copyright @ 2019 Oracle and/or its affiliates.
89 / 111
90. JSON Data validation (4)
Copyright @ 2019 Oracle and/or its affiliates.
90 / 111
91. JSON Data validation (5)
And now even in pure NoSQL CRUD, we can see the data validation in action:
Copyright @ 2019 Oracle and/or its affiliates.
91 / 111
92. JSON Schema validation - NEW 8.0.16
Since MySQL 8.0.16, CHECK constraints are now supported !
Copyright @ 2019 Oracle and/or its affiliates.
92 / 111
93. JSON Schema validation - NEW 8.0.16
Since MySQL 8.0.16, CHECK constraints are now supported !
Copyright @ 2019 Oracle and/or its affiliates.
93 / 111
94. JSON Schema validation - NEW 8.0.16
Since MySQL 8.0.16, CHECK constraints are now supported !
Copyright @ 2019 Oracle and/or its affiliates.
94 / 111
95. JSON Schema validation (2) - NEW 8.0.17
New JSON_SCHEMA_VALID function:
Copyright @ 2019 Oracle and/or its affiliates.
95 / 111
96. JSON Schema validation (2) - NEW 8.0.17
New JSON_SCHEMA_VALID function:
Copyright @ 2019 Oracle and/or its affiliates.
96 / 111
97. JSON Schema validation (3) - NEW 8.0.17
Knowing why thanks to the new JSON_SCHEMA_VALIDATION_REPORT function:
Copyright @ 2019 Oracle and/or its affiliates.
97 / 111
98. JSON Schema validation (4) - NEW 8.0.17
And nally, mixing JSON_SCHEMA_VALID with CHECK CONSTRAINT:
Copyright @ 2019 Oracle and/or its affiliates.
98 / 111
99. JSON Schema validation (4) - NEW 8.0.17
And nally, mixing JSON_SCHEMA_VALID with CHECK CONSTRAINT:
Copyright @ 2019 Oracle and/or its affiliates.
99 / 111
100. what do I gain ?
Conclusion
Copyright @ 2019 Oracle and/or its affiliates.
100 / 111
101. schemaless
exible data structure
easy to start (CRUD)
Conclusion
This is the best of the two worlds in one product !
Data integrity
ACID Compliant
Transactions
SQL
Copyright @ 2019 Oracle and/or its affiliates.
101 / 111
104. MySQL Shell Reporting Framework - 8.0.16
Now it's also possible to create user-de ned reports directly in the Shell. They can be
wri en in Python and called the same way on every operating systems. No need for extra
libraries !
More Info:
h ps://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-reporting.html
h ps://mysql.wisborg.dk/2019/04/26/mysql-shell-8-0-16-built-in-reports/
h ps://mysql.wisborg.dk/2019/04/27/mysql-shell-8-0-16-user-de ned-reports/
Copyright @ 2019 Oracle and/or its affiliates.
104 / 111
105. MySQL Shell Extentions & Plugins - 8.0.17
Since 8.0.17, MySQL Shell has been extended to allow you to write your own extensions
and plugins !
More Info:
h ps://mysqlserverteam.com/mysql-shell-plugins-introduction/
h ps://lefred.be/content/overview-on-mysql-shell-8-0-17-extensions-plugins-and-
how-to-write-yours/
Copyright @ 2019 Oracle and/or its affiliates.
105 / 111
106. Credits
Thank you Olivier Dasini for some TV Show examples:
h p://dasini.net/blog/2019/04/02/mysql-json-document-store/
Thank you Jesper Wisborg Krogh for inspiration extending the MySQL Shell
Copyright @ 2019 Oracle and/or its affiliates.
106 / 111
110. Upgrade to MySQL 8.0
It's time to upgrade to MySQL 8.0, the
fastest MySQL adoption release ever !
Copyright @ 2019 Oracle and/or its affiliates.
110 / 111
111. Session Survey
Help us make the content even be er.
Please complete the session survey in
the Mobile App.
Copyright @ 2019 Oracle and/or its affiliates.
111 / 111