Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Document Store - A Document Store with all the benefits of a Transactional RDBMS

216 views

Published on

SQL + NoSQL = MySQL
MySQL Document Store allows developers to work with SQL relational tables and schema-less JSON collections. To make that possible MySQL has created the X Dev API which puts a strong focus on CRUD by providing a fluent API allowing you to work with JSON documents in a natural way. The X Protocol is a highly extensible and is optimized for CRUD as well as SQL API operations.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

MySQL Document Store - A Document Store with all the benefits of a Transactional RDBMS

  1. 1. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store Copyright © 2018, Oracle and/or its affiliates. All rights reserved. A Document Store with all the benefits of a Transactional RDBMS Olivier Dasini MySQL Principal Solutions Architect EMEA olivier.dasini@oracle.com Twitter : @freshdaz Blog : http://dasini.net/blog
  2. 2. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  3. 3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 3 Me, Myself & I  MySQL Geek  Addicted to MySQL for 15+ years!  Playing with databases for 20+ years  MySQL Writer, Blogger and Speaker  Also former : DBA, Consultant, Architect, Trainer, ...  MySQL Principal Solutions Architect EMEA at Oracle  Stay tuned! :  Twitter: @freshdaz  Blog: http://dasini.net/blog Olivier DASINI
  4. 4. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 4 The world's most popular open source database
  5. 5. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Powers Social 5
  6. 6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Powers eCommerce 6
  7. 7. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Powers FinTech 7
  8. 8. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Powers SaaS 8
  9. 9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Powers the Cloud 9
  10. 10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 10 • Data Integrity – Normalization – Constraints (Foreign keys, …) • Atomicity, Consistency, Isolation, Durability – ACID Compliant – Transactions • SQL – Powerful Query Language – Schema, Table, Row – SELECT / UPDATE / DELETE / INSERT Relational Databases 10
  11. 11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 11 • An object that can represent structured data • Structure is implicit in the document; usually no external/central schema • Compact, popular and standardized • Can be represented natively in many languages (JavaScript,Python, etc.) 11 • JSON is a lightweight data-interchange format. • It is easy for humans to read and write. • It is easy for machines to parse and generate. • It is based on a subset of the JavaScript Programming. • Language, Standard ECMA-262 3rd Edition - December 1999. • Use for storing and exchanging data. What is a JSON Document ? JavaScript Object Notation
  12. 12. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 12 JSON Support • Native File Format – Standardized as ECMA-404 (http://json.org) – Binary Storage • Generated Columns • 20+ Functions  Search Functions  Aggregations Functions • Query structured data and semi-structured JSON data
  13. 13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 13 JSON_ARRAY_APPEND() JSON_ARRAY_INSERT() JSON_ARRAY() JSON_CONTAINS_PATH() JSON_CONTAINS() JSON_DEPTH() JSON_EXTRACT() JSON_INSERT() JSON_KEYS() JSON_LENGTH() JSON_MERGE[_PRESERVE]() JSON_OBJECT() JSON_QUOTE() JSON_REMOVE() JSON_REPLACE() JSON_SEARCH() JSON_SET() JSON_TYPE() JSON_UNQUOTE() JSON_VALID() JSON_PRETTY() JSON_STORAGE_SIZE() JSON_STORAGE_FREE() JSON_ARRAYAGG() JSON_OBJECTAGG() JSON_MERGE_PATCH() JSON_TABLE() JSON Functions Perform operations on JSON documentsPerform operations on JSON documents https://dev.mysql.com/doc/refman/8.0/en/json-functions.html
  14. 14. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 14 • Schemaless – No schema design, No normalization, No Foreign Keys, No data types, … – Very quick initial development • Flexible data structure – Embedded arrays or objects – Valid solution when natural data can’t be modeled optimally into a relational model – Objects persistence without the use of any ORM – Mapping Object-Oriented • JSON – Close to frontend – Native in JavaScript – Easy to learn NoSQL : Document Store 14
  15. 15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | How DBAs see data 15 How Devs see data { " G N P " : 2 7 3 1 , " _ i d " : " M T Q " , " N a m e " : " M a r t i n i q u e " , " I n d e p Ye a r " : n u l l , " g e o g r a p h y " : { " R e g i o n " : " C a r i b b e a n " , " C o n t i n e n t " : " N o r t h A m e r i c a " , " S u r f a c e A r e a " : 11 2 8 } , " g o v e r n m e n t " : { " H e a d O f S t a t e " : " E m m a n u e l M a c r o n " , " G o v e r n m e n t F o r m " : " O v e r s e a s D e p a r t m e n t o f F r a n c e " } , " d e m o g r a p h i c s " : { " P o p u l a t i o n " : 3 9 5 0 0 0 , " L i f e E x p e c t a n c y " : 7 8 . 3 0 0 0 0 3 0 5 1 7 5 7 8 1 } }
  16. 16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 16 • Schema (Database) • Collection – Group of JSON documents. – Equivalent of table • Document – Set of key-value pairs in JSON format – Equivalent of row (tuple) Document Store Databases 16 • CRUD (basic functions of persistent storage) – CREATE – READ – UPDATE – DELETE TerminologiesTerminologies
  17. 17. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 17 Document Oriented Databases Usability & ScalabilityUsability & Scalability • Schemaless: No centralized database schema – Data model enforcement and validation (if any) at application layer – Simpler schema updates (no ALTER TABLE penalty) • NoSQL APIs: Simpler programming interfaces – No specialized language for queries and data manipulation – Complex queries handled at application layer (no complex SELECTs, JOINs) – Document in, document out, manipulations at client side • Scalability, but some drawbacks: – Limited database features (no foreign keys, no transactions, etc.) – Weak consistency guarantees
  18. 18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 18 Document Store An easy, straight forward way to work with JSON documents in MySQL
  19. 19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 19 RDBMS or NoSQL ? Why not both ?
  20. 20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | • One Extensible database – Do more with the MySQL database – Many can manage (with deep skills) – Stable – Cost-effective – Easy to move data between like database types – Fewer Drivers – Few Tools – SQL works, CRUD works – Operational and Analytical Together • Many different databases – Requires larger skill repertoire, more complex development … • Harder to find deep skills – Many Drivers – Many Tools – More effort to share and exchange data – It’s a lot more work – Operational and Analytical Separate One Database Many Models VS Many Databases Many Models 20
  21. 21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | NoSQL, MySQL, Why not… • Have both schema-less AND schema in the same technology stack? • One that checks all the boxes of all stakeholders: 21 Developers: [✔] Schemaless or/and Schema [✔] Rapid Prototyping/Simpler APIs [✔] Document Model [✔] Transactions Operations: [✔] Performance Management/Visibility [✔] Robust Replication, Backup, Restore [✔] Comprehensive Tooling Ecosystem [✔] Simpler application schema upgrades Business Owner: [✔] Don’t lose my data = ACID transactions [✔] Capture all my data = Extensible/Schemaless [✔] Products On Schedule/Time to Market = Rapid Development
  22. 22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | - A One Stop Shop • Expensive to manage many data stores • Better few databases – more flexibility – More developers • Can harness both NoSQL and SQL savvy – More DBAs • No shortage of highly experienced MySQL DBAs – Less training – don’t need to learn many products – Cross data store exchange – easier to move from from docs to tables etc. – One connector/driver needed for apps 22 Combining Relational and Document StoresCombining Relational and Document Stores
  23. 23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Document Store 23
  24. 24. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Document Store: Components • MySQL X PluginMySQL X Plugin • Introduces X Protocol for relational- and document operations • Maps CRUD operations to standard SQL (relational tables, JSON datatype and functions) • X ProtocolX Protocol • New MySQL client protocol based on top of industry standard (Protobuf) • Works for both, CRUD and SQL operations • InnoDB ClusterInnoDB Cluster • Read-Scaling, Write-Scaling, HA • X DevAPIX DevAPI • New, modern, async developer API for CRUD and SQL operations on top of X Protocol • Introduces Collections as new Schema obj. • MySQL ShellMySQL Shell • Offers interactive X DevAPI mode for app prototyping • MySQL ConnectorsMySQL Connectors • Support for X DevAPI for • JavaScript, Python, PHP, Java, C#, C++ 24
  25. 25. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Document Store: Architecture 27
  26. 26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 34 Document Store SQL is now optional ! - Write applications using X DevAPIX DevAPI https://insidemysql.com/mysql-8-0-welcome-to-the-devapi/
  27. 27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 35 Welcome to the X DevAPI! MySQL X devAPI connector for: • Java – https://insidemysql.com/connector-j-8-0-11-the-face-for-your-brand-new-document-oriented-database/ • .NET – https://insidemysql.com/introducing-connector-net-with-full-support-for-mysql-8-0/ • Node.JS – https://insidemysql.com/introducing-connector-node-js-for-mysql-8-0/ • C++ – https://insidemysql.com/what-is-new-in-connector-c-8-0/ • Python – https://insidemysql.com/using-mysql-connector-python-8-0-with-mysql-8-0/ • PHP – https://insidemysql.com/introducing-the-mysql-x-devapi-php-extension-for-mysql-8-0/ • ODBC – https://insidemysql.com/what-is-new-in-connector-odbc-8-0/ 35 Motivation • We are doing something that has not been done before • Document databases exist! Relational databases exist! • We even see databases that support relational and document querying over the same data set • However we have yet to see a relational database include a document model so that a user can use document objects alongside their existing relational data.
  28. 28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 36 Welcome to the X DevAPI! - PHP Example 36 Writing a program using MySQL 8.0 Document Store $user = 'my_user'; $passwd = 's0S3kR*T'; $host = 'localhost'; $port = '33060'; $connection_uri = 'mysqlx://'.$user.':'.$passwd.'@'.$host.':'.$port; $session = mysql_xdevapigetSession($connection_uri); $schema = $session->getSchema("test"); $collection = $schema->getCollection("my_collection"); $result = $collection->find("Name like :param")->bind(["param" => "Olivier"])->execute(); $docs = $result->fetchAll(); //… print results … $session->close(); Requirements / Installation https://dev.mysql.com/doc/apis-php/en/apis-php-mysql-xdevapi.setup.html https://insidemysql.com/introducing-the-mysql-x-devapi-php-extension-for-mysql-8-0/
  29. 29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 37 Welcome to the X DevAPI! - Python Example https://insidemysql.com/using-mysql-connector-python-8-0-with-mysql-8-0/ 37 Writing a program using MySQL 8.0 Document Store import mysqlx session = mysqlx.get_session({ "host": "localhost", "port": 33060, "user": "my_user", "password": "s0S3kR*T" }) schema = session.get_schema("test") collection = schema.get_collection("my_collection") result = collection.find("name like :param").bind("param", "Olivier").limit(1).execute() docs = result.fetch_all() print("Name: {0}".format(docs[0]["name"])) session.close() Connector/Python 8.0 installation shell> pip install mysql-connector-python
  30. 30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 38 Document Store – MySQL Shell An integrated development & administration shell where all MySQL products will be available through a common scripting interface
  31. 31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Shell 8.0.11+ 39Confidential - Oracle Internal 39 MySQL Server 5.7 MySQL 8.0 Upgrade Checker Prompt Themes Auto Completion & Command History MySQL Server 8.0 Document Store X DevAPI InnoDB ClusterSQL CLI Output Formats (Table, JSON, Tabbed) Batch Execution JavaScript Python SQL
  32. 32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Shell – Key Features 40 Interface for Development and Administration of MySQLInterface for Development and Administration of MySQL • Scripting for Javascript, Python, and SQL mode • Supports MySQL Standard and X Protocols • Document and Relational Models • CRUD Document and Relational APIs via scripting • Traditional Table, JSON, Tab Separated output results formats • Both Interactive and Batch operations
  33. 33. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 44 Document Store Under The Hood
  34. 34. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 45 js> session.createSchema('demo') js> use demo js> db.getCollections() js> db.createCollection('myCollection') js> db.getCollections() js> db.myCollection.add({"param1":"value1", "param2":"value2"}) js> db.myCollection.find() js> db.myCollection.find().limit(1) js> db.myCollection.find("_id = '00005af018430000000000000002'") js> db.myCollection.modify("_id = '1234'").set("param","value") js> db.myCollection.remove("_id = '1234'") js> session.startTransaction() js> … js> session.rollback() js> db.myCollection.createIndex("ageIdx",{fields:[{"field":"$.age","type":"INT",required:false}]}) Copyright @ 2018 Oracle and/or its affiliates. All rights reserved. MySQL Document Store “cheat sheet” CCreate RRead UUpdate DDelete Index TTransaction
  35. 35. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Documents and Collections • Collections are containers for documents – These documents share a purpose – Possibly share one or more indexes – Each collection has a unique name – Exists within a single schema • Within a Collection you can – Add(), Find(), Modify(), and Remove() - JSON documents • Collections can be – Create(), List(), Drop() 46
  36. 36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Operation Document Relational Create Collection.add() Table.insert() Read Collection.find() Table.select() Update Collection.modify() Table.update() Delete Collection.remove() Table.delete() 47 • Use SQL, CRUD APIs – Document (NoSQL) and Relational (SQL), or “All of the Above” – All of this is in addition to the Classic APIs Connectors include X DevAPI
  37. 37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Collection Search – find(), bind(), fields() • Supports many operators to specify searches – ||, &&, XOR, IS, NOT, BETWEEN, IN, LIKE, !=, <>, >, >=, <, <=, &, |, <<, >>, +, -, *, /, ~, %. • Searching - find() – db.CountryInfo.find("GNP > 500000 and demographics.Population < 100000000") – db.CountryInfo.find("GNP*1000000/demographics.Population > 30000") • Binding - bind() – db.CountryInfo.find("Name = :country").bind("country", "Italy") • Project Results – fields() – returns specific fields – db.CountryInfo.find("Name = :country").bind("country", "Italy") 48
  38. 38. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Setup 50 $ mysqlsh -u user -p --sql -e"SELECT PLUGIN_NAME, PLUGIN_STATUS, PLUGIN_DESCRIPTION FROM information_schema.plugins WHERE PLUGIN_NAME='mysqlx'" +-------------+---------------+--------------------+ | PLUGIN_NAME | PLUGIN_STATUS | PLUGIN_DESCRIPTION | +-------------+---------------+--------------------+ | mysqlx | ACTIVE | X Plugin for MySQL | +-------------+---------------+--------------------+ • MySQL 8.0.11 or higher • X plugin is enabled by default and is part of the MySQL Server • Check the X plugin installation: • Disabling at startup: – Set mysqlx=0 in your MySQL configuration file – Or by passing in either --mysqlx=0 or --skip-mysqlx when starting the MySQL server
  39. 39. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Shell 51 Interface for Development and Administration of MySQLInterface for Development and Administration of MySQL • Scripting for Javascript, Python, and SQL mode • Supports MySQL Standard and X Protocols • Document and Relational Models • CRUD Document and Relational APIs via scripting • Traditional Table, JSON, Tab Separated output results formats • Both Interactive and Batch operations
  40. 40. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 52 $ mysqlsh -u <user> -p … s MySQL Shell version 8.0.11 Session type: X Connection Id: 17 Default schema: Current schema: Current user: root@localhost SSL: Cipher in use: DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Using delimiter: ; Server version: 8.0.11-commercial MySQL Enterprise Server - Commercial Protocol version: X protocol Client library: 8.0.11 Connection: localhost via TCP/IP TCP port: 33060 Server characterset: utf8mb4 Schema characterset: utf8mb4 Client characterset: utf8mb4 Conn. characterset: utf8mb4 Uptime: 30 min 16.0000 sec • Following commands provides instructions to begin prototyping database applications interactively with MySQL Shell for Javascript • MySQL Shell for Python provide same functionalities
  41. 41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 53 h ===== Global Commands ===== help (?,h) Print this help. sql Switch to SQL processing mode. js Switch to JavaScript processing mode. py Switch to Python processing mode. source (.) Execute a script file. Takes a file name as an argument. Start multi-line input when in SQL mode. quit (q) Quit MySQL Shell. exit Exit MySQL Shell. Same as quit connect (c) Connect to a server. reconnect Reconnect with a server. option Manage MySQL Shell options. warnings (W) Show warnings after every statement. nowarnings (w) Don't show warnings after every statement. status (s) Print information about the current global connection. use (u) Set the current schema for the active session. rehash Update the auto-completion cache with database names. history View and edit command line history. For help on a specific command use the command as ? <command> ===== Global Objects ===== dba Enables you to administer InnoDB clusters using the AdminAPI. mysql Used to work with classic MySQL sessions using SQL. mysqlx Used to work with X Protocol sessions using the MySQL X DevAPI. session Represents the currently open MySQL session. shell Gives access to general purpose functions and properties. sys Gives access to system specific parameters. util Global object that groups miscellaneous tools like upgrade checker. For more help on a global variable use <var>.help(), e.g. dba.help()
  42. 42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 54 MySQL localhost:33060+ ssl JS> session <Session:root@localhost> mysql-js> session.help() ... The following functions are currently supported. - close Closes the session. - commit Commits all the operations executed after a call to startTransaction(). - createSchema Creates a schema on the database and returns the corresponding object. - dropSchema Drops the schema with the specified name. - getCurrentSchema Retrieves the active schema on the session. - getDefaultSchema Retrieves the Schema configured as default for the session. - getSchema Retrieves a Schema object from the current session through it's name. - getSchemas Retrieves the Schemas available on the session. - getUri Retrieves the URI for the current session. - help Provides help about this class and it's members - isOpen Returns true if session is known to be open. - quoteName Escapes the passed identifier. - releaseSavepoint Removes a savepoint defined on a transaction. - rollback Discards all the operations executed after a call to startTransaction(). - rollbackTo Rolls back the transaction to the named savepoint without terminating the transaction. - setCurrentSchema Sets the current schema for this session, and returns the schema object for it. - setFetchWarnings Enables or disables warning generation. - setSavepoint Creates or replaces a transaction savepoint with the given name. - sql Creates a SqlExecute object to allow running the received SQL statement on the target MySQL Server. - startTransaction Starts a transaction context on the server.
  43. 43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations Collection creation 55
  44. 44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 56 js> session.createSchema('doc') <Schema:doc> js> use doc Default schema `doc` accessible through db. js> db <Schema:doc> js> db.getCollections() [] js> db.createCollection('posts') <Collection:posts> js> db.getCollections() [ <Collection:posts> ] js> db.posts.add({"title":"MySQL 8.0 rocks", "text":"My first post!", "code": "42"}) Query OK, 1 item affected (0.0221 sec) js> db.posts.add({"title":"Polyglot database", "text":"Developing both SQL and NoSQL applications"}) Query OK, 1 item affected (0.0172 sec)
  45. 45. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations Find documents 58 https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-documents-find.html
  46. 46. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 59 js> db.posts.add("This is not a valid JSON document") CollectionAdd.add: Argument #1 expected to be a document, JSON expression or a list of documents (ArgumentError) js> db.posts.find() [ { "_id": "00005aec27830000000000000001", "code": "42", "text": "My first post!", "title": "MySQL 8.0 rocks" }, { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" } ] js> db.posts.find().fields("text") [ { "text": "My first post!" }, { "text": "Developing both SQL and NoSQL applications" } ]
  47. 47. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 60 js> db.posts.find().limit(1) [ { "_id": "00005aec27830000000000000001", "code": "42", "text": "My first post!", "title": "MySQL 8.0 rocks" } ] js> db.posts.add({"_id": "00005aec27830000000000000001", "code": "13"}) ERROR: 5116: Document contains a field value that is not unique but required to be js> db.posts.find().limit(1).skip(1) [ { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" } ]
  48. 48. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 61 js> db.posts.find("title = 'Polyglot database'") [ { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" } ] js> db.posts.find("title = :what").bind("what", "Polyglot database") [ { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" } ]
  49. 49. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 62 js> db.posts.find() [ { "_id": "00005aec27830000000000000001", "code": "42", "text": "My first post!", "title": "MySQL 8.0 rocks" }, { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" } ] js> db.posts.find().sort("text") /* same as : db.posts.find().sort("text asc")*/ [ { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" }, { "_id": "00005aec27830000000000000001", "code": "42", "text": "My first post!", "title": "MySQL 8.0 rocks" } ]
  50. 50. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations Document modification 63 https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-documents-modify.html
  51. 51. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 64 js> db.posts.find("_id = '00005aec27830000000000000002'") [ { "_id": "00005aec27830000000000000002", "text": "Developing both SQL and NoSQL applications", "title": "Polyglot database" } ] js> db.posts.modify("_id = '00005aec27830000000000000002'").set("title", "Hybrid database") Query OK, 1 item affected (0.0182 sec) js> db.posts.find("_id = '00005aec27830000000000000002'").fields("title", "text") [ { "text": "Developing both SQL and NoSQL applications", "title": "Hybrid database" } ] js> db.posts.modify("_id = '00005aec27830000000000000002'").set("author" ,"Daz").set("code" , 2018) Query OK, 1 item affected (0.0282 sec) js> db.posts.find("_id = '00005aec27830000000000000002'").fields("author","text","code") [ { "author": "Daz", "code": 2018, "text": "Developing both SQL and NoSQL applications" } ]
  52. 52. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 65 js> db.posts.modify("_id = '00005aec27830000000000000002'").set("autor", "Daz le magnifique") Query OK, 1 item affected (0.0117 sec) js> db.posts.find("_id = '00005aec27830000000000000002'") [ { "_id": "00005aec27830000000000000002", "author": "Daz", "autor": "Daz le magnifique", "code": 2018, "text": "Developing both SQL and NoSQL applications", "title": "Hybrid database" } ] js> db.posts.modify("_id = '00005aec27830000000000000002'").unset("autor") Query OK, 1 item affected (0.0123 sec) js> db.posts.find("_id = '00005aec27830000000000000002'") [ { "_id": "00005aec27830000000000000002", "author": "Daz", "code": 2018, "text": "Developing both SQL and NoSQL applications", "title": "Hybrid database" } ]
  53. 53. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations Full ACID 66 https://dev.mysql.com/doc/x-devapi-userguide/en/transaction-handling.html
  54. 54. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | It relies on the proven MySQL InnoDB’s strength & robustness: • innodb_flush_log_at_trx_commit = 1 • innodb_doublewrite = ON • sync_binlog = 1 • transaction_isolation = REPEATABLE-READ | READ-COMMITTED | ... We do care about your data! Document Store is Full ACID ACID transactions = Don’t lose my dataACID transactions = Don’t lose my data 67
  55. 55. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Document Store supports transactions ACID transactions = Don’t lose my dataACID transactions = Don’t lose my data js> session.startTransactionstartTransaction() // Start a transaction Query OK, 0 rows affected (0.0004 sec) js> db.posts.modify("_id ='00005aec27830000000000000002'").set("title","ACID Transaction").set("text","Don’t lose my data") js> db.posts.find("_id ='00005aec27830000000000000002'").fields(["title", "text"]) [ { "text": "ACID Transaction", "title": "Don’t lose my data" } ] js> db.restaurants.remove("_id='5ad5b645f88c5bb8fe3fd337'") js> db.restaurants.find("_id='5ad5b645f88c5bb8fe3fd337'") Empty set (0.0005 sec) js> session.rollback() // Rollback the transaction (if necessary e.g. in case of an error) Query OK, 0 rows affected (0.0862 sec) js> db.posts.find("_id ='00005aec27830000000000000002'").fields(["title", "text"]) [ { … [snip] … js> db.restaurants.find("_id='5ad5b645f88c5bb8fe3fd337'") [ { … [snip] … 69
  56. 56. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations Document deletion 70 https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-documents-remove.html
  57. 57. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 72 js> db.posts.add({"actions": {"type": 1, "site": "http://dasini.net/blog/fr"}}) js> db.posts.add({"actions": {"type": 2, "site": "http://dasini.net/blog/en"}}) js> db.posts.find().sort("actions.type desc").limit(1) [ { "_id": "00005aec2783000000000000000b", "actions": { "site": "http://dasini.net/blog/en", "type": 2 } } ] js> db.posts.remove("true").sort("actions.type desc").limit(1) Query OK, 1 item affected (0.1212 sec) js> db.posts.find('actions.type IN (1, 2)') [ { "_id": "00005aec2783000000000000000a", "actions": { "site": "http://dasini.net/blog", "type": 1 } } ] js> db.posts.remove('actions.type = 1') Query OK, 1 item affected (0.0192 sec) js> db.posts.find('actions.type IN (1, 2)') Empty set (0.0005 sec)
  58. 58. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations Indexes 73 https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-indexes-create.html
  59. 59. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 74 // Add a Non Unique Index js> db.posts.createIndex("idxAuthor", {fields: [{field: "$.author", type: "TEXT(20)"}]}) // Add a Unique Index js> db.posts.createIndex("idxCode", {fields: [{field: "$.code", type: "INT"}], unique: true}) js> db.posts.add({code: 42}) ERROR: 5116: Document contains a field value that is not unique but required to be // required:true => field is required to exist in the document i.e. NOT NULL js> db.posts.createIndex("idxTitle", {fields: [{field: "$.title", type: "TEXT(30)", required: true}]}) js> db.posts.add({code: 11}) ERROR: 5115: Document is missing a required field // Drop an Index js> db.posts.dropIndex("idxTitle") https://dev.mysql.com/doc/x-devapi-userguide/en/collection-indexing.html
  60. 60. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Using SQL with Session 75
  61. 61. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 76 // pics is a regular SQL table that contains JSON documents js> session.sql("SELECT * FROM pics") +----+------+---------------------+----------------------------------------------------------------+ | id | code | ts | doc | +----+------+---------------------+----------------------------------------------------------------+ | 1 | 42 | 2018-04-25 14:56:19 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 2 | 2018 | 2018-04-25 14:47:15 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | +----+------+---------------------+----------------------------------------------------------------+ // SQL join between a collection posts and a table pics js> session.sql("SELECT po.doc->'$.title' title, po.doc->'$.text' msg, ts time, pi.doc details FROM posts po INNER JOIN pics pi ON (pi.code = po.doc->>'$.code') WHERE pi.code = 2018") +-------------------+----------------------------------------------+---------------------+------------------------------------------------------------+ | title | msg | time | details | +-------------------+----------------------------------------------+---------------------+------------------------------------------------------------+ | "Hybrid database" | "Developing both SQL and NoSQL applications" | 2018-04-25 14:47:15 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | +-------------------+----------------------------------------------+---------------------+------------------------------------------------------------+ // Many other SQL commands available js> session.sql("CREATE … js> session.sql("CALL... js> session.sql("DROP... Using SQL with Session • The Session object has a sql() function that takes any SQL statement as a string • Possible to join collections with tables :-O
  62. 62. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations – SQL / Relational 77
  63. 63. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | CRUD operations – SQL / Relational Tables 78 • You can use MySQL Shell to manipulate not just JSON documents, but also relational tables. • Basic operations scoped by tables include: Operation Form Description db.name.insert() The insert() method inserts one or more records into the named table. db.name.select() The select() method returns some or all records in the named table. db.name.update() The update() method updates records in the named table. db.name.delete() The delete() method deletes one or more records from the named table.
  64. 64. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 79 js> db.getTables() [ <Table:pics> ] // INSERT Records into table pics js> db.pics.insert('code', 'doc').values(42, '{"data": {"tags": [1, 11], "type": "image", "location": "Fr"}}') js> db.pics.insert('code', 'doc').values(11, '{"data": {"tags": [2], "type": "image", "location": "Fr"}}') js> db.pics.insert('code', 'doc').values(11, '{"data": {"tags": [9, 99], "type": "sound"}}') // Select table pics js> db.pics.select() +----+--------+---------------------+----------------------------------------------------------------+ | id | code | ts | doc | +----+--------+---------------------+----------------------------------------------------------------+ | 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | | 3 | 11 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} | +----+--------+---------------------+----------------------------------------------------------------+ js> db.pics.select(["id", "ts", "doc"]) +----+---------------------+----------------------------------------------------------------+ | id | ts | doc | +----+---------------------+----------------------------------------------------------------+ | 1 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 2 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | | 3 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} | +----+---------------------+----------------------------------------------------------------+ js> db.pics.select(["ts", "doc"]).where("id = 1") +---------------------+----------------------------------------------------------------+ | ts | doc | +---------------------+----------------------------------------------------------------+ | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | +---------------------+----------------------------------------------------------------+
  65. 65. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 80 js> db.pics.select(["ts", "doc"]).where("id like :id").bind("id", 1) +---------------------+----------------------------------------------------------------+ | ts | doc | +---------------------+----------------------------------------------------------------+ | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | +---------------------+----------------------------------------------------------------+ js> db.pics.select(["ts", "doc"]).where('id IN (1, 2) AND ts < NOW()').orderBy(["ts desc"]).limit(2) +---------------------+----------------------------------------------------------------+ | ts | doc | +---------------------+----------------------------------------------------------------+ | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | +---------------------+----------------------------------------------------------------+ js> db.pics.select().where("doc->'$.data.type' = 'image'") +----+--------+---------------------+----------------------------------------------------------------+ | id | code | ts | doc | +----+--------+---------------------+----------------------------------------------------------------+ | 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | +----+--------+---------------------+----------------------------------------------------------------+ js> db.pics.select().where("doc->'$.data.tags[0]' = 9") +----+------+---------------------+----------------------------------------------+ | id | code | ts | doc | +----+------+---------------------+----------------------------------------------+ | 3 | 11 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} | +----+------+---------------------+----------------------------------------------+ // Is pics a view? js> db.pics.isView() False
  66. 66. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 81 js> db.pics.select() +----+--------+---------------------+----------------------------------------------------------------+ | id | code | ts | doc | +----+--------+---------------------+----------------------------------------------------------------+ | 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | | 3 | 11 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} | +----+--------+---------------------+----------------------------------------------------------------+ // UPDATE Table pics js> db.pics.update().set('code', 2018).where("doc->'$.data.type' = 'sound'") Query OK, 1 item affected (0.0208 sec) js> db.pics.select() +----+------+---------------------+----------------------------------------------------------------+ | id | code | ts | doc | +----+------+---------------------+----------------------------------------------------------------+ | 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} | | 3 | 2018 | 2018-04-25 12:31:09 | {"data": {"tags": [9, 99], "type": "sound"}} | +----+------+---------------------+----------------------------------------------------------------+ // DELETE records from table pics js> db.pics.delete().where("doc->'$.data.tags[0]' = 2") Query OK, 1 item affected (0.0512 sec) js> db.pics.select() +----+------+---------------------+----------------------------------------------------------------+ | id | code | ts | doc | +----+------+---------------------+----------------------------------------------------------------+ | 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} | | 3 | 2018 | 2018-04-25 12:12:01 | {"data": {"tags": [9, 99], "type": "sound"}} | +----+------+---------------------+----------------------------------------------------------------+ js> //db.pics.delete() // Drop all rows
  67. 67. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Relational / SQL / Analytics on MySQL Tables 82
  68. 68. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 84 SQL for running Complex and/or analytic queries • Relational Analytics – Collations are MySQL tables – So you can use ALL the power of MySQL’s SQL commands i.e. ● NATURAL [ INNER | CROSS | {LEFT|RIGHT} OUTER ] JOIN ● CTE ● Window functions ● JSON_ARRAYAGG() / JSON_OBJECTAGG() / JSON_PRETTY() ● …
  69. 69. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 85 // Calculate the average of the restaurant rating sql> SELECT rname AS Restaurant, Style, avg(rating) AS 'Average Rating' FROM restaurants, JSON_TABLE(doc, "$" columns( rname varchar(50) path "$.name", style varchar(50) path "$.cuisine", nested path "$.grades[*]" columns (rating int path "$.score" ))) as jt GROUP BY rname, style ORDER BY avg(rating) DESC; +--------------------------------+------------+----------------+ | Restaurant | Style | Average Rating | +--------------------------------+------------+----------------+ | Wendy'S | Hamburgers | 13.7500 | | Dj Reynolds Pub And Restaurant | Irish | 9.2500 | | Morris Park Bake Shop | Bakery | 8.2000 | +--------------------------------+------------+----------------+ SQL for running Complex and/or analytic queries • Simple query with JSON_TABLE – JSON_TABLE: Extracts data from a JSON document and returns it as a relational table having the specified columns
  70. 70. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 86 // Calculate the average of the restaurant rating (more elegant form) sql> SELECT doc->>"$.name" AS Restaurant, doc->>"$.cuisine" AS Cuisine, (SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT PATH "$.score")) AS r) AS 'Avg Rating' FROM restaurants ORDER BY 'Avg Rating' DESC; +--------------------------------+------------+------------+ | Restaurant | Cuisine | Avg Rating | +--------------------------------+------------+------------+ | Morris Park Bake Shop | Bakery | 8.2000 | | Wendy'S | Hamburgers | 13.7500 | | Dj Reynolds Pub And Restaurant | Irish | 9.2500 | +--------------------------------+------------+------------+ SQL for running Complex and/or analytic queries • Simple query with JSON_TABLE – JSON_TABLE: Extracts data from a JSON document and returns it as a relational table having the specified columns
  71. 71. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 87 // Calculate the average of the restaurant rating (Adding CTE and window functions) sql> WITH cte1 AS ( SELECT doc->>"$.name" AS Restaurant, doc->>"$.cuisine" AS Cuisine, (SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT PATH "$.score")) AS r) AS AvgScore FROM restaurants ) SELECT *, RANK() OVER (PARTITION BY cuisine ORDER BY AvgScore) AS `Rank` FROM cte1 ORDER BY `Rank`, AvgScore DESC; +--------------------------------+------------+----------+------+ | Restaurant | Cuisine | AvgScore | Rank | +--------------------------------+------------+----------+------+ | Wendy'S | Hamburgers | 13.7500 | 1 | | Dj Reynolds Pub And Restaurant | Irish | 9.2500 | 1 | | Morris Park Bake Shop | Bakery | 8.2000 | 1 | +--------------------------------+------------+----------+------+ SQL for running Complex and/or analytic queries • Commont Table Expression (CTE) : https://dev.mysql.com/doc/refman/8.0/en/with.html • Windows functions : https://dev.mysql.com/doc/refman/8.0/en/window-functions.html
  72. 72. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 94 Document Store Wrap-up
  73. 73. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 95 Document Store ✔ Built on Proven SQL/InnoDB/Replication technology ✔ Schema-less/Relational/Hybrid ✔ ACID/Transactions ✔ No SQL required - CRUD/JSON/Documents/Indexes ✔ Modern Dev API ✔ Modern/Efficient Protocol ✔ SQL Queries/Analytics over JSON Documents ✔ Transparent and Easy HA/Scaling with MySQL InnoDB Cluster 95
  74. 74. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Hybrid database that covers 4 core areas Relational MySQL Tables and SQL Relational MySQL Tables and SQL Relational Analytics MySQL Tables and SQL Relational Analytics MySQL Tables and SQL Document Store MySQL Collections and Table with JSON Datatype Key/Value MySQL Memcached Document Store MySQL Collections and Table with JSON Datatype Key/Value MySQL Memcached Mass Scale Analytics Sharded MySQL Relational and/or Document Store Mass Scale Analytics Sharded MySQL Relational and/or Document Store 96 Operational Analytical Relational/SQL NoSQL
  75. 75. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 97 Clients and Applications InnoDB Storage Engine mysqld process Firewall Audit Encryption Authentication Online Backup Monitoring Integration Support 3rd Party Tools3rd Party Tools3rd Party Tools3rd Party Tools3rd Party Tools NoSQLNoSQL Simple access patterns Compromise on consistency for performance Ad-hoc data format Simple operation SQLSQL Complex queries with joins ACID transactions Well defined schemas Rich set of tools Document Store Works transparently with MySQL Enterprise FeaturesWorks transparently with MySQL Enterprise Features
  76. 76. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 99 Resources Topic Link(s) Top 10 reasons for NoSQL with MySQL http://lefred.be/content/top-10-reasons-for-nosql-with-mysql/ MySQL as a Document Store https://dev.mysql.com/doc/refman/8.0/en/document-store.html MySQL Shell User Guide https://dev.mysql.com/doc/mysql-shell-excerpt/8.0/en/index.html MySQL Shell Documentation https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/ https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/ X Dev API User Guide https://dev.mysql.com/doc/x-devapi-userguide/en/ X Plugin https://dev.mysql.com/doc/refman/8.0/en/x-plugin.html MySQL Engineering Blog https://insidemysql.com/mysql-8-0-welcome-to-the-devapi/ https://insidemysql.com/mysql-document-store-crud-quick-start/ MySQL JSON Data Type https://dev.mysql.com/doc/refman/8.0/en/json.html Blogs http://dasini.net/blog/2018/07/23/30-mins-with-mysql-json-functions/ http://dasini.net/blog/2015/11/17/30-mins-avec-json-en-mysql/ http://dasini.net/blog/2015/11/30/json-et-colonnes-generees-avec-mysql http://mysqlserverteam.com/tag/json/ http://mysqlserverteam.com/category/docstore/
  77. 77. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 100
  78. 78. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Thank you!

×