Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

20171104 hk-py con-mysql-documentstore_v1

214 views

Published on

MySQL HA and Document Store with Python using MySQL Shell

Published in: Technology
  • Be the first to comment

  • Be the first to like this

20171104 hk-py con-mysql-documentstore_v1

  1. 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Using MySQL Shell High Availability + Document Store with Python Cross(X)over between NoSQL and SQL Ivan Ma 2017-11-04
  2. 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2
  3. 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Program Agenda MySQL Document Store Introduction Components of the MySQL Document Store The X DevAPI – A modern CRUD App Programming Interface Documents via SQL Behind the Scenes – CRUD to SQL Translation 1 2 3 4 5 3
  4. 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store Introduction 4
  5. 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Oriented Databases • Representing complex information, similar to an Object • JSON (=JavaScript Object Notation) – Compact, popular and standardized – Can be represented natively in many languages (JavaScript, Python etc) • Other popular encoding formats are XML, YAML etc What is a Document? 5 { "_id": "AUT", "Name": "Austria", "GNP": 211860, "IndepYear": 1918, "demographics": { "LifeExpectancy": 77.699, "Population": 8091800 }, "geography": { "Continent": "Europe", "Region": "Western Europe", "SurfaceArea": 83859 } } JSON Document Example
  6. 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Relational Databases vs Document Store Systems Relational Model Document Model - JSON 6 id name email city_id 3412 John Smith john@oracle.com 45 id city country_id 45 San Francisco US customer city { _id: 3412, name: "John Smith", email: john@oracle.com, city: "San Francisco", country: "US", orders: [ {date: "2017-08-24", total: 312.20}, {date: "2017-10-02", total: 24.95} ] } id id_customer date total 381 3412 2017-08-24 312.20 412 3412 2017-10-02 24.95 shop_order
  7. 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | The MySQL Document Store • Document oriented data storage for MySQL – Enables working with both, structured and unstructured data – Exposes new CRUD interface, while keeping SQL • Standard plugin for MySQL Server (since MySQL 5.7.12) – Today (2017-10-16 – MySQL 5.7.20 Released) – MySQL 8.0.3 RC • Store JSON documents in MySQL – Through traditional SQL interface or – New X DevAPI NoSQL interface (much easier to work with for documents) 7
  8. 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Components of the MySQL Document Store 8
  9. 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | API / Client Administration Server Side 9 MySQL Document Store - Components MySQL Connectors MySQL Shell X Plugin X DevAPI X Protocol MySQL Server JSON Datatype New JSON Syntax New JSON Functions
  10. 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | • MySQL Server 5.7 or 8.0 • JSON Datatype • MySQL X Plugin • Introduces X Protocol for relational- and document operations • Maps CRUD operations to standard SQL (relational tables, JSON datatype and functions) • X Protocol • New MySQL client protocol based on top of industry standard (Protobuf) • Works for both, CRUD and SQL operations • X DevAPI • New, modern, async developer API for CRUD and SQL operations on top of X Protocol • Introduces Collections as new Schema obj. • MySQL Shell • Offers interactive X DevAPI mode for app prototyping • MySQL Connectors • Support for X DevAPI and X Session 10 MySQL Document Store – Components
  11. 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11 X Plugin 33060 MySQL Shell X Protocol JavaScript / Python Node.js Application MySQL Connector/Node.js ... X DevAPI X DevAPI Windows Application MySQL Connector/Net X DevAPI MySQL Server MySQL Server MySQL Server MySQL Router MySQL Router X Plugin 33060 X Plugin 33060 MySQL InnoDB cluster MySQL Document Store
  12. 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store running on InnoDB Cluster • Document Store is tightly connected to InnoDB Cluster – Main goals: Ease-of-use & Scalability • Document Store opens an easier path to massive write scale-out – Sharding with the relational model is hard – Self contained documents can be sharded much easier 12
  13. 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Connector Application MySQL Connector Application MySQL Shell MySQL Connector Application MySQL Connector Application Running Doc Store on MySQL InnoDB Cluster MySQL InnoDB cluster MySQL Enterprise Monitor …
  14. 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14 Live Demo
  15. 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | The X DevAPI – A modern CRUD App Programming Interface 15
  16. 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • CRUD stands for the following 4 basic operations – CREATE – READ – UPDATE – DELETE CRUD Confidential – Oracle 16
  17. 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Examples • session.get_schemas() • mydb = session.get_schema(‘world_x’) • mydb.get_collections() 17
  18. 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Examples • session.get_schemas() • mydb = session.get_schema(‘world_x’) • mydb.get_collections() • mydb.help() 18
  19. 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | More examples with DOC operation with MySQL Shell • mydb.countryinfo.find() • mydb.create_collectin(‘mytest’); • mydb.mytest.add( …..) • mydb.mytest.find() 19
  20. 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • The Collection object represents a collection of JSON Documents Example: python products = db.create_collection('products'); Collections Confidential – Oracle 20
  21. 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • Add is used to add new JSON Documents to a Collection Example: products.add({"_id": 123, "name": "Vacuum Cleaner", "model": "UH20040", "brand": "Hoover", "price": 79.99}).execute(); products.add({"_id": 124, "name": "Ninja Master Prep", "brand": "Ninja", "model": "QB900B", "price": 34.88, "material": "Plastic", "color": "Silver"}).execute(); CRUD: Add Confidential – Oracle 21
  22. 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • Find is used to retrieve JSON Documents from a collection Example: products.find("price > 10 and price < 100").fields([”_id", "name", "price"]).sort(["name"]).execute(); CRUD: Find Confidential – Oracle 22
  23. 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • Modify is used to update existing JSON Documents of a Collection Example: products.modify(”_id = 123").set("dimensions.weight", 50).execute(); CRUD: Modify Confidential – Oracle 23
  24. 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • Remove is used to delete JSON Documents from a Collection Example: products.remove("_id = 123"); CRUD: Remove Confidential – Oracle 24
  25. 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • JavaScript / Node.js – Connector/Node.js 8.0.8-dmr • Get it from npm or https://dev.mysql.com/downloads/connector/nodejs/ • API Reference: https://dev.mysql.com/doc/dev/connector-nodejs/ • C#/.NET – Connector/Net 8.0.9-dmr • Get it from NuGet or http://dev.mysql.com/downloads/connector/net/ • API Reference: http://dev.mysql.com/doc/dev/connector-net • Java – Connector/J 8.0.8-dmr • Get it from http://dev.mysql.com/downloads/connector/j/ • API Reference: https://dev.mysql.com/doc/dev/connector-j/ 25 Supported Languages
  26. 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • Python – Connector/Python 8.0.5-dmr • Get it from http://dev.mysql.com/downloads/connector/python/ • API Reference: https://dev.mysql.com/doc/dev/connector-python/ • C++ – Connector/C++ 8.0.6-dmr • Get it from https://dev.mysql.com/downloads/connector/cpp/ • API Reference: https://dev.mysql.com/doc/dev/connector-cpp/devapi_ref.html • PHP – MySQL native driver for PHP (soon) 26 Supported Languages
  27. 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store - X DevAPI • X DevAPI Users Guide – https://dev.mysql.com/doc/x-devapi-userguide/en/ • Using MySQL as a Document Store – https://dev.mysql.com/doc/refman/5.7/en/document-store.html 27 More information
  28. 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Get ready to try the MySQL Document Store 28
  29. 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Getting Ready • Fast application prototyping using MySQL Shell – We will write a simple Document Store application in the MySQL Shell – We will create MySQL InnoDB Cluster of 3 nodes • Python creating MySQL InnoDB Cluster • Python to create document store on MySQL InnoDB Cluster – Accessing Document using CRUD – Accessing Document using SQL What is the goal of the demo? 29
  30. 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 30 Live Demo
  31. 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Documents via SQL 31
  32. 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Storage with SQL • Allows storage of JSON data in a MySQL table column • Validations on INSERT/UPDATE • Internal binary storage format allows fast key lookups inside documents • Use inline JSON in SQL expressions (compare JSON values etc.) CREATE TABLE mytable ( id INT PRIMARY KEY, data JSON, …); JSON Datatype Confidential – Oracle 32
  33. 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Storage with SQL • JSON data manipulation in SQL • Construct JSON values (JSON_OBJECT(), JSON_ARRAY()) • Extract information from JSON objects and arrays (JSON_EXTRACT()) • Modify JSON values (JSON_SET(), JSON_INSERT() etc.) • Inline JSON Path Syntax to reference values inside JSON data in SQL – [[schema.]table.]column->'$.field.array[0]' JSON Functions Confidential – Oracle 33
  34. 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Storage with SQL • Virtual columns allow indexes on JSON fields – Create a virtual column to "look into" a JSON document – Create index on the virtual column • Foreign keys can also be created on virtual columns Indexing JSON Data Confidential – Oracle 34
  35. 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Storage with SQL id name email city_id 3412 John Smith john@oracle.com 45 Use Case: Relational to JSON Data Confidential – Oracle 35 id city country_id 45 San Francisco US { id: 3412, name: "John Smith", email: john@oracle.com, city: "San Francisco", country: "US" } customer city
  36. 36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Storage with SQL Use Case: Relational to JSON Data Confidential – Oracle 36 SELECT JSON_OBJECT('id', cu.id, 'name', cu.name, 'email', cu.email, 'city', ci.city) as customer FROM customer cu JOIN city ci ON ci.id = cu.city_id
  37. 37. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Document Storage with SQL Use Case: Query JSON Data via SQL Confidential – Oracle 37 SELECT doc->'$.id' as id, doc->'$.name' as name, doc->'$.email' as email FROM customer WHERE doc->'$.city' = 'San Francisco'
  38. 38. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store Behind the Scenes 38
  39. 39. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How it Works • Applications use DevAPI connectors to write database operations in native code (instead of SQL) • Connector translates DevAPI operations to X protocol document requests • X Plugin translates document requests to SQL • Results sent back to application as JSON documents Confidential – Oracle 39
  40. 40. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 40 MySQL Document Store – How it works • Existing applications are not touched – Developers adopt DocStore only if they want to • X Protocol allows SQL and CRUD – SQL: SELECT * FROM myTable WHERE id = 12; – X DevAPI: db.myTable.find("id=12"); • Directly hooked into the optimizer – Bypass MySQL connection handling Storage Optimizer MySQL Connection handling X Plugin MySQL Server 5.7 & 8.0 330633060 MySQL Shell Existing Application MySQL Connector/ODBC SQL Result Classic MySQL ProtocolX Protocol JavaScript / Python Node.js Application MySQL C/Node.js ... X DevAPI X DevAPI Windows Application MySQL Connector/Net X DevAPI
  41. 41. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How it Works: Mapping to SQL CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4; mycollection = create_collection('mycollection') Confidential – Oracle 41
  42. 42. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How it Works: Mapping to SQL INSERT INTO `test`.`mycoll` (doc) VALUES ( JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf', 'test',1234) ); mycollection.add({'test': 1234}) Confidential – Oracle 42
  43. 43. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | How it Works: Mapping to SQL SELECT doc FROM `test`.`mycoll` WHERE (JSON_EXTRACT(doc,'$.test') > 100); mycollection.find("test > 100") Confidential – Oracle 43
  44. 44. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | MySQL Document Store Summary and Take Away 44 • New, modern way to develop database applications • Combine best of relational and document oriented models • MySQL InnoDB Cluster – Future proof for HA and scale-out deployments • Blogs: mysqlserverteam.com/category/docstore/ • https://dev.mysql.com/doc/refman/5.7/en/mysql-shell-tutorial-python.html
  45. 45. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Thank you! 45
  46. 46. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 47

×