Relational and Document Database with MySQL

2,275 views

Published on

MyNA@2016/05/30
About MySQLX

Published in: Software
  • Be the first to comment

Relational and Document Database with MySQL

  1. 1. Relational and Document Databases with MySQL MyNA(日本MySQLユーザ会)会 2016年5月 Twitter: @RDBMS
  2. 2. 本内容は個人の見解であり、 所属組織を代表するものではありません。
  3. 3. MySQLドキュメントストア 3
  4. 4. ドキュメントデータ • 形式・様式 (ex: XML, JSON) • ツリー構造 ( [], {} ) • スキーマレス +-----------------------------------------------------------------------------+ | body | +-----------------------------------------------------------------------------+ | {"id": 1, "name": "自転車", "price": 10000, "Conditions": ["NEW", 2015]} | | {“id”: 2, “name”: “テレビ", "price": 30000, "Conditions": ["USED", 2013]} | | {"id": 3, "name": "冷蔵庫", "price": 10438, "Conditions": ["NEW", 2015]} | | {"id": 4, "name": "冷蔵庫", "price": 50000, "Conditions": ["NEW", 2015]} | | {"id": 5, "name": "自転車", "price": 25000, "Conditions": ["NEW", 2015]} | +-----------------------------------------------------------------------------+ Document Store • 高速な検索 • 更新の最適化 • オペレーション 4 CREATE TABLE T_JSON_DOC (body json);
  5. 5. MYSQLのドキュメントストア機能拡張 5 • JSONサポート JSONデータ型(utf8mb4)・JSONファンクション Generated ColumnとIndexによる検索の高速化 MySQL 5.7.9(GA) • X Protocol MySQLサーバをドキュメントストアとして拡張する為に、 Xプラグイン(mysqlx)により実装 • X DevAPI SQL処理とドキュメントに対してのCRUD処理 Connector/Node.js, Connector/J, Connector/Netに実装 • mysqlsh コマンドラインクライアント (Javascript, Python, SQL) MySQL 5.7.12
  6. 6. JSONデータ  ネイティブJSONデータ型 (バイナリ形式)  Insert時のJSON構文バリデーション機能  組み込みJSON関数 (保存、検索、更新、操作)  ドキュメントにインデックス設定可能  SQLとの統合を容易にする新しいインライン構文  utf8mb4の文字セットとutf8mb4_binの照合 外部サイト /SNS モバイル デバイス コマース /ポータル その他 (data JSON); REST/JSON SELECT NAME,CountryCode from world.City where CountryCode ='JPN' limit 1; +-------+-------------+ | NAME | CountryCode | +-------+-------------+ | Tokyo | JPN | +-------+-------------+ SELECT JSON_OBJECT('CITY',NAME,'Country',CountryCode) from world.City where CountryCode ='JPN' limit 1; +------------------------------------------------+ | JSON_OBJECT('CITY',NAME,'Country',CountryCode) | +------------------------------------------------+ | {"CITY": "Tokyo", "Country": "JPN"} | +------------------------------------------------+ select feature from NEW57.features where json_extract(feature,'$.properties.STREET') = 'MARKET' limit 1¥G ************** 1. row ************** feature: {"type": "Feature", "geometry": {"type": "Polygon", "coordinates": [[[-122.39836263491878, 37.79189388899312, 0], [-122.39845248797837, 37.79233030084018, 0], [- 122.39768507706792, 37.7924280850133, 0], [- 122.39836263491878, 37.79189388899312, 0]]]}, "properties": {"TO_ST": "388", "BLKLOT": "0265003", "STREET": "MARKET", "FROM_ST": "388", "LOT_NUM": "003", "ST_TYPE": "ST", "ODD_EVEN": "E", "BLOCK_NUM": "0265", "MAPBLKLOT": "0265003"}}
  7. 7. MySQL 5.7.12 ~ コネクター, ドライバー, プロトコル MySQL Plugins X Protocol Plugin Memcached Plugin Core MySQL Connectors and Drivers X ProtocolStd Protocol Memcached driver X Protocol 33060 Std Protocol 3306 SQL API CRUD and SQL APIs Memcache Protocol X and Std Protocols MySQL Shell
  8. 8. X Protocol • 非同期APIサポート – 並列処理とバッチ処理をサポート – パイプライン方式 – 複数リクエストを送信, ラウンドトリップを削減 – CRUD ==大量に小さなPKを処理, 独立した複数のクエリーを処理 • ミドルウエアとの親和性 – ルーティング、シャーディング、読み取り書き込みスプリッティング (XSESSION) • オープンスタンダードの利用: TLS (Transport Layer Security), SASL(Simple Authentication and Security Layer), Protobuf (Protocol Buffers)等 8 Protobuf: https://developers.google.com/protocol-buffers/ +-------------+----------------+--------------------+ | PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_DESCRIPTION | +-------------+----------------+--------------------+ | mysqlx | 1.0 | X Plugin for MySQL | +-------------+----------------+--------------------+ The X Protocol focuses on: • extensibility • performance • security INSTALL PLUGIN mysqlx SONAME 'mysqlx.so';
  9. 9. X Protocol Query Time Client ServerNetwork Stage Time network path latency 1ms exectime 0. 1ms  クラッシックリクエスト/レスポンス Total: 4x path + 2x exectime = 4.2ms  パイプライン処理 Total: 2x path + 2x exectime = 2.2ms 9 Pipelining messages is a core feature of the Mysqlx Protocol. It sends messages to the server without waiting for a response to save latency. (no mandatory handshake) 参考) https://dev.mysql.com/doc/internals/en/x-protocol-messages-message-structure.html
  10. 10. X Protocol 10 エラーを無視して続行: Mysqlx.Expect::Open([-no_error]) 最初のエラーで失敗 : Mysqlx.Expect::Open([+no_error]) With expectations pipelined, the server will handle errors in a consistent, reliable way. In case error reporting isn't a major topic one can combine multi-row INSERT with pipelining and reduce the per-row network overhead. This is important in case the network is saturated. 詳細: https://dev.mysql.com/doc/internals/en/x-protocol.html Expectation Mysqlx.Expect::Open([-no_error])
  11. 11. 参考情報 Spec: http://dev.mysql.com/doc/internals/en/x-protocol.html Message Def: https://github.com/mysql/mysql-server/tree/5.7/rapid/plugin/x/protocol Protobuf: https://developers.google.com/protocol-buffers/ $ protoc -I --python_out=... .../mysql.proto 11  Pipeline https://dev.mysql.com/doc/internals/en/x-protocol-implementation-pipelining.html https://github.com/mysql/mysql-server/blob/5.7/mysql-test/suite/xplugin/t/crud_pipe.test  Expectations https://dev.mysql.com/doc/internals/en/x-protocol-expect-expectations.html https://github.com/mysql/mysql-server/blob/5.7/mysql-test/suite/xplugin/t/expect_noerror.test  独自クライアントを作成する場合  Pipeline及びExpectationsに関して
  12. 12. X DevAPI • X Pluginを有効にする事で、X Protocol経由で通信可能 • ドキュメントとテーブルのコレクションに対してのCRUD処理 • NoSQLライクな構文でドキュメントに対しCRUD処理可能 • Fluent API prod = sess.getSchema("prod") res = prod.users. find("$.name = 'Milk'"). fields(["name", "properties"]) X Plugin (MySQL) ⇔ X Protocol ⇔ X DevAPI (Driver) 12  MySQL Connector/node.js  MySQL Connector/J  MySQL Connector/Net  MySQL Shell
  13. 13. MySQL Connectors include X Dev API • Use SQL, CRUD APIs スキーマレスドキュメントおよびリレーショナルテーブルに対応 - Classic APIsに加えて、これらの全てが追加されます 13 Operation Document Relational Create Collection.add() Table.insert() Read Collection.find() Table.select() Update Collection.modify() Table.update() Delete Collection.remove() Table.delete() 参照) http://dev.mysql.com/doc/x-devapi-userguide/en/crud-operations-overview.html
  14. 14. 14 [root@misc01 nodejs]# cat sample_node_X_API.js const mysqlx = require('mysqlx'); mysqlx.getSession({ host: 'localhost', port: 33060, dbUser: 'demo_user', dbPassword: 'password' }).then(function (session) { return session.createSchema("test_schema").then(function (schema) { return schema.createCollection("myCollection"); }).then(function (collection) { return Promise.all([ collection.add( {baz: { foo: "bar"}},{foo: { bar: "baz"}}).execute(), collection.find("$.baz.foo == 'bar'").execute(function (row) {console.log("Row: %j", row); }).then(function (res) {console.log("Collection find done!");}), collection.remove("($.foo.bar) == 'baz'").execute().then(function () { console.log("Document deleted");}), collection.drop() ]); }).then(function () { return session.dropSchema("test_schema"); }).then(function () { return session.close(); }); }).catch(function (err) { console.log(err.stack); process.exit(); }); [root@misc01 nodejs]# node sample_node_X_API.js Row: {"_id":"630f0d3b-f6fd-1d99-6d80-a8e90352","baz":{"foo":"bar"}} Collection find done! Document deleted Connector:mysql-connector-nodejs-1.0.2.tar.gz
  15. 15. 2016-05-06T15:15:05.364983+09:00 37 Query /* xplugin authentication */ SELECT `authentication_string`, <SNIP> 2016-05-06T15:15:05.369200+09:00 37 Query CREATE DATABASE `test_schema` 2016-05-06T15:15:05.382450+09:00 37 Query CREATE TABLE `test_schema`.`myCollection` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED NOT NULL UNIQUE) CHARSET utf8mb4 ENGINE=InnoDB 2016-05-06T15:15:05.464364+09:00 37 Query INSERT INTO `test_schema`.`myCollection` (doc) VALUES ('{¥"baz¥":{¥"foo¥":¥"bar¥"},¥"_id¥":¥"c5d6964d-af2a-0b87-36aa- 4f5bc18b¥"}'),('{¥"foo¥":{¥"bar¥":¥"baz¥"},¥"_id¥":¥"3f106b27-e14a-31e0-5297-51da7f1c¥"}') 2016-05-06T15:15:05.481912+09:00 37 Query SELECT doc FROM `test_schema`.`myCollection` WHERE (JSON_EXTRACT(doc,'$.baz.foo') = 'bar') 2016-05-06T15:15:05.528923+09:00 37 Query DELETE FROM `test_schema`.`myCollection` WHERE (JSON_EXTRACT(doc,'$.foo.bar') = 'baz') 2016-05-06T15:15:05.551710+09:00 37 Query DROP TABLE `test_schema`.`myCollection` 2016-05-06T15:15:05.565824+09:00 37 Query DROP DATABASE `test_schema` 2016-05-06T15:15:05.643891+09:00 37 Quit General Log
  16. 16. MySQL Shell • 開発および管理用のシェルの統合 • 一般的なスクリプト・インターフェースを介して 利用可能な MySQLコマンドラインクライアント。 • PythonやJavaScriptなどのスクリプト言語で さまざまな製品と対話するための完全な開発用API 16 [root@misc01 admin]# mysqlsh --help | egrep -i "Start in" --sql Start in SQL mode using a node session. --sqlc Start in SQL mode using a classic session. --js Start in JavaScript mode. --py Start in Python mode. [root@misc01 admin]#  バッチ処理に利用可能 shell> mysqlsh --file code.js shell> mysqlsh < code.js shell> echo "show databases" | mysqlsh –sql  2016年5月現在:Development Release: 1.0.3
  17. 17. 17 [root@misc01 MID2016]# mysqlsh --uri demo_user@localhost/NEW57 -ppassword Creating an X Session to demo_user@localhost:33060/NEW57 Default schema `NEW57` accessible through db. Welcome to MySQL Shell 1.0.3 Development Preview ……. Currently in JavaScript mode. Use ¥sql to switch to SQL mode and execute queries. mysql-js> db.createCollection("x_posts"); <Collection:x_posts> mysql-js> db.x_posts.add({"title":"Hello World", "text":"This is the first post via mysqlx"}); Query OK, 1 item affected (0.01 sec) mysql-js> db.x_posts.find("title = 'Hello World'").sort(["title"]); [ { "_id": "baee9a744308e61168170800279cea3c", "text": "This is the first post via mysqlx", "title": "Hello World" } ] 1 document in set (0.00 sec) MySQL Shell Version 1.0.3 Development Preview session.getSchema(‘NEW57')
  18. 18. 2016-05-26T23:24:45.694581+09:00 11 Query /* xplugin authentication */ SELECT `authentication_string`, <SNIP> 2016-05-26T23:24:45.695786+09:00 11 Query show databases 2016-05-26T23:24:45.697994+09:00 11 Query select schema(), @@lower_case_table_names 2016-05-26T23:24:45.698455+09:00 11 Query select connection_id() 2016-05-26T23:24:45.698899+09:00 11 Query use `NEW57` 2016-05-26T23:24:45.699441+09:00 11 Query SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2 2016-05-26T23:24:45.701333+09:00 11 Query SHOW FULL TABLES FROM `NEW57` 2016-05-26T23:24:45.703443+09:00 11 Query SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2 2016-05-26T23:24:45.706091+09:00 11 Query SHOW FULL TABLES FROM `NEW57` 2016-05-26T23:25:15.862883+09:00 11 Query CREATE TABLE `NEW57`.`x_posts` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED NOT NULL UNIQUE) CHARSET utf8mb4 ENGINE=InnoDB 2016-05-26T23:25:38.026317+09:00 11 Query INSERT INTO `NEW57`.`x_posts` (doc) VALUES ('{¥"_id¥":¥"de3e8cb74d23e6112c1d0800279cea3c¥",¥"text¥":¥"This is the first post via mysqlx¥",¥"title¥":¥"Hello World¥"}') 2016-05-26T23:25:57.011923+09:00 11 Query SELECT doc FROM `NEW57`.`x_posts` WHERE (JSON_EXTRACT(doc,'$.title') = 'Hello World') ORDER BY JSON_EXTRACT(doc,'$.title')
  19. 19. root@localhost [NEW57]> SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2; +------------------------+---+ | table_name | c | +------------------------+---+ | Innovation_Day | 2 | | Innovation_Day_Confirm | 2 | | X_JSON | 2 | | x_posts | 2 | | X_PYTHON | 2 | +------------------------+---+ テーブル構造(doc json列+generated columnのPK(_id) から、通常のテーブルとドキュメントストアを判別 mysql-js> db.getCollections() { "Innovation_Day": <Collection:Innovation_Day>, "Innovation_Day_Confirm": <Collection:Innovation_Day_Confirm>, "X_JSON": <Collection:X_JSON>, "X_PYTHON": <Collection:X_PYTHON>, "x_posts": <Collection:x_posts> } mysql-js> db.getTables() { "T_GIS": <Table:T_GIS>, "T_JSON_DOC": <Table:T_JSON_DOC>, "T_JSON_DOC_TXT": <Table:T_JSON_DOC_TXT>, "T_TDE": <Table:T_TDE>, "citylots": <Table:citylots>, "employees": <Table:employees>, "employees_json": <Table:employees_json>, "employees_txt": <Table:employees_txt>,
  20. 20.  MySQL5.7~ ドキュメント関連機能の連携デモ JSONデータ型, MySQL Shell, X Protocol, 生成列, FTS Twitter API: (JSONデータ) https://api.twitter.com/1.1/xxxx localhost, port:33060 Schema: NEW57 myDb = mySession.getSchema('NEW57') 省略… timeline = json.loads(res.text) for tweet in timeline: myDb.X_PYTHON.add(tweet).execute() Shell> mysqlsh --py < demo_python_twitter.py +-------+--------------+------+-----+---------+------------------+ | Field | Type | Null | Key | Default | Extra | +-------+--------------+------+-----+---------+------------------+ | doc | json | YES | | NULL | | | _id | varchar(32) | NO | PRI | NULL | STORED GENERATED | | name | varchar(64) | YES | MUL | NULL | STORED GENERATED | | text | varchar(512) | YES | MUL | NULL | STORED GENERATED | +-------+--------------+------+-----+---------+------------------+ (1)Twitter APIに接続しJSONフォーマットのTweetを取得 (2) 取得したJSONデータを.addでテーブルに追加 (3) Generated ColumnやFTSを利用してデータ参照処理 OAuth1Session mysqlx 20
  21. 21. 有難うございました 21 MySQL5.7からの新機能を活用してみて下さい!! 問題・改善要求: https://bugs.mysql.com/

×