Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Relational and Document Database with MySQL

2,697 views

Published on

MyNA@2016/05/30
About MySQLX

Published in: Software
  • Be the first to comment

Relational and Document Database with MySQL

  1. 1. Relational and Document Databases with MySQL MyNA(日本MySQLユーザ会)会 2016年5月 Twitter: @RDBMS
  2. 2. 本内容は個人の見解であり、 所属組織を代表するものではありません。
  3. 3. MySQLドキュメントストア 3
  4. 4. ドキュメントデータ • 形式・様式 (ex: XML, JSON) • ツリー構造 ( [], {} ) • スキーマレス +-----------------------------------------------------------------------------+ | body | +-----------------------------------------------------------------------------+ | {"id": 1, "name": "自転車", "price": 10000, "Conditions": ["NEW", 2015]} | | {“id”: 2, “name”: “テレビ", "price": 30000, "Conditions": ["USED", 2013]} | | {"id": 3, "name": "冷蔵庫", "price": 10438, "Conditions": ["NEW", 2015]} | | {"id": 4, "name": "冷蔵庫", "price": 50000, "Conditions": ["NEW", 2015]} | | {"id": 5, "name": "自転車", "price": 25000, "Conditions": ["NEW", 2015]} | +-----------------------------------------------------------------------------+ Document Store • 高速な検索 • 更新の最適化 • オペレーション 4 CREATE TABLE T_JSON_DOC (body json);
  5. 5. MYSQLのドキュメントストア機能拡張 5 • JSONサポート JSONデータ型(utf8mb4)・JSONファンクション Generated ColumnとIndexによる検索の高速化 MySQL 5.7.9(GA) • X Protocol MySQLサーバをドキュメントストアとして拡張する為に、 Xプラグイン(mysqlx)により実装 • X DevAPI SQL処理とドキュメントに対してのCRUD処理 Connector/Node.js, Connector/J, Connector/Netに実装 • mysqlsh コマンドラインクライアント (Javascript, Python, SQL) MySQL 5.7.12
  6. 6. JSONデータ  ネイティブJSONデータ型 (バイナリ形式)  Insert時のJSON構文バリデーション機能  組み込みJSON関数 (保存、検索、更新、操作)  ドキュメントにインデックス設定可能  SQLとの統合を容易にする新しいインライン構文  utf8mb4の文字セットとutf8mb4_binの照合 外部サイト /SNS モバイル デバイス コマース /ポータル その他 (data JSON); REST/JSON SELECT NAME,CountryCode from world.City where CountryCode ='JPN' limit 1; +-------+-------------+ | NAME | CountryCode | +-------+-------------+ | Tokyo | JPN | +-------+-------------+ SELECT JSON_OBJECT('CITY',NAME,'Country',CountryCode) from world.City where CountryCode ='JPN' limit 1; +------------------------------------------------+ | JSON_OBJECT('CITY',NAME,'Country',CountryCode) | +------------------------------------------------+ | {"CITY": "Tokyo", "Country": "JPN"} | +------------------------------------------------+ select feature from NEW57.features where json_extract(feature,'$.properties.STREET') = 'MARKET' limit 1¥G ************** 1. row ************** feature: {"type": "Feature", "geometry": {"type": "Polygon", "coordinates": [[[-122.39836263491878, 37.79189388899312, 0], [-122.39845248797837, 37.79233030084018, 0], [- 122.39768507706792, 37.7924280850133, 0], [- 122.39836263491878, 37.79189388899312, 0]]]}, "properties": {"TO_ST": "388", "BLKLOT": "0265003", "STREET": "MARKET", "FROM_ST": "388", "LOT_NUM": "003", "ST_TYPE": "ST", "ODD_EVEN": "E", "BLOCK_NUM": "0265", "MAPBLKLOT": "0265003"}}
  7. 7. MySQL 5.7.12 ~ コネクター, ドライバー, プロトコル MySQL Plugins X Protocol Plugin Memcached Plugin Core MySQL Connectors and Drivers X ProtocolStd Protocol Memcached driver X Protocol 33060 Std Protocol 3306 SQL API CRUD and SQL APIs Memcache Protocol X and Std Protocols MySQL Shell
  8. 8. X Protocol • 非同期APIサポート – 並列処理とバッチ処理をサポート – パイプライン方式 – 複数リクエストを送信, ラウンドトリップを削減 – CRUD ==大量に小さなPKを処理, 独立した複数のクエリーを処理 • ミドルウエアとの親和性 – ルーティング、シャーディング、読み取り書き込みスプリッティング (XSESSION) • オープンスタンダードの利用: TLS (Transport Layer Security), SASL(Simple Authentication and Security Layer), Protobuf (Protocol Buffers)等 8 Protobuf: https://developers.google.com/protocol-buffers/ +-------------+----------------+--------------------+ | PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_DESCRIPTION | +-------------+----------------+--------------------+ | mysqlx | 1.0 | X Plugin for MySQL | +-------------+----------------+--------------------+ The X Protocol focuses on: • extensibility • performance • security INSTALL PLUGIN mysqlx SONAME 'mysqlx.so';
  9. 9. X Protocol Query Time Client ServerNetwork Stage Time network path latency 1ms exectime 0. 1ms  クラッシックリクエスト/レスポンス Total: 4x path + 2x exectime = 4.2ms  パイプライン処理 Total: 2x path + 2x exectime = 2.2ms 9 Pipelining messages is a core feature of the Mysqlx Protocol. It sends messages to the server without waiting for a response to save latency. (no mandatory handshake) 参考) https://dev.mysql.com/doc/internals/en/x-protocol-messages-message-structure.html
  10. 10. X Protocol 10 エラーを無視して続行: Mysqlx.Expect::Open([-no_error]) 最初のエラーで失敗 : Mysqlx.Expect::Open([+no_error]) With expectations pipelined, the server will handle errors in a consistent, reliable way. In case error reporting isn't a major topic one can combine multi-row INSERT with pipelining and reduce the per-row network overhead. This is important in case the network is saturated. 詳細: https://dev.mysql.com/doc/internals/en/x-protocol.html Expectation Mysqlx.Expect::Open([-no_error])
  11. 11. 参考情報 Spec: http://dev.mysql.com/doc/internals/en/x-protocol.html Message Def: https://github.com/mysql/mysql-server/tree/5.7/rapid/plugin/x/protocol Protobuf: https://developers.google.com/protocol-buffers/ $ protoc -I --python_out=... .../mysql.proto 11  Pipeline https://dev.mysql.com/doc/internals/en/x-protocol-implementation-pipelining.html https://github.com/mysql/mysql-server/blob/5.7/mysql-test/suite/xplugin/t/crud_pipe.test  Expectations https://dev.mysql.com/doc/internals/en/x-protocol-expect-expectations.html https://github.com/mysql/mysql-server/blob/5.7/mysql-test/suite/xplugin/t/expect_noerror.test  独自クライアントを作成する場合  Pipeline及びExpectationsに関して
  12. 12. X DevAPI • X Pluginを有効にする事で、X Protocol経由で通信可能 • ドキュメントとテーブルのコレクションに対してのCRUD処理 • NoSQLライクな構文でドキュメントに対しCRUD処理可能 • Fluent API prod = sess.getSchema("prod") res = prod.users. find("$.name = 'Milk'"). fields(["name", "properties"]) X Plugin (MySQL) ⇔ X Protocol ⇔ X DevAPI (Driver) 12  MySQL Connector/node.js  MySQL Connector/J  MySQL Connector/Net  MySQL Shell
  13. 13. MySQL Connectors include X Dev API • Use SQL, CRUD APIs スキーマレスドキュメントおよびリレーショナルテーブルに対応 - Classic APIsに加えて、これらの全てが追加されます 13 Operation Document Relational Create Collection.add() Table.insert() Read Collection.find() Table.select() Update Collection.modify() Table.update() Delete Collection.remove() Table.delete() 参照) http://dev.mysql.com/doc/x-devapi-userguide/en/crud-operations-overview.html
  14. 14. 14 [root@misc01 nodejs]# cat sample_node_X_API.js const mysqlx = require('mysqlx'); mysqlx.getSession({ host: 'localhost', port: 33060, dbUser: 'demo_user', dbPassword: 'password' }).then(function (session) { return session.createSchema("test_schema").then(function (schema) { return schema.createCollection("myCollection"); }).then(function (collection) { return Promise.all([ collection.add( {baz: { foo: "bar"}},{foo: { bar: "baz"}}).execute(), collection.find("$.baz.foo == 'bar'").execute(function (row) {console.log("Row: %j", row); }).then(function (res) {console.log("Collection find done!");}), collection.remove("($.foo.bar) == 'baz'").execute().then(function () { console.log("Document deleted");}), collection.drop() ]); }).then(function () { return session.dropSchema("test_schema"); }).then(function () { return session.close(); }); }).catch(function (err) { console.log(err.stack); process.exit(); }); [root@misc01 nodejs]# node sample_node_X_API.js Row: {"_id":"630f0d3b-f6fd-1d99-6d80-a8e90352","baz":{"foo":"bar"}} Collection find done! Document deleted Connector:mysql-connector-nodejs-1.0.2.tar.gz
  15. 15. 2016-05-06T15:15:05.364983+09:00 37 Query /* xplugin authentication */ SELECT `authentication_string`, <SNIP> 2016-05-06T15:15:05.369200+09:00 37 Query CREATE DATABASE `test_schema` 2016-05-06T15:15:05.382450+09:00 37 Query CREATE TABLE `test_schema`.`myCollection` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED NOT NULL UNIQUE) CHARSET utf8mb4 ENGINE=InnoDB 2016-05-06T15:15:05.464364+09:00 37 Query INSERT INTO `test_schema`.`myCollection` (doc) VALUES ('{¥"baz¥":{¥"foo¥":¥"bar¥"},¥"_id¥":¥"c5d6964d-af2a-0b87-36aa- 4f5bc18b¥"}'),('{¥"foo¥":{¥"bar¥":¥"baz¥"},¥"_id¥":¥"3f106b27-e14a-31e0-5297-51da7f1c¥"}') 2016-05-06T15:15:05.481912+09:00 37 Query SELECT doc FROM `test_schema`.`myCollection` WHERE (JSON_EXTRACT(doc,'$.baz.foo') = 'bar') 2016-05-06T15:15:05.528923+09:00 37 Query DELETE FROM `test_schema`.`myCollection` WHERE (JSON_EXTRACT(doc,'$.foo.bar') = 'baz') 2016-05-06T15:15:05.551710+09:00 37 Query DROP TABLE `test_schema`.`myCollection` 2016-05-06T15:15:05.565824+09:00 37 Query DROP DATABASE `test_schema` 2016-05-06T15:15:05.643891+09:00 37 Quit General Log
  16. 16. MySQL Shell • 開発および管理用のシェルの統合 • 一般的なスクリプト・インターフェースを介して 利用可能な MySQLコマンドラインクライアント。 • PythonやJavaScriptなどのスクリプト言語で さまざまな製品と対話するための完全な開発用API 16 [root@misc01 admin]# mysqlsh --help | egrep -i "Start in" --sql Start in SQL mode using a node session. --sqlc Start in SQL mode using a classic session. --js Start in JavaScript mode. --py Start in Python mode. [root@misc01 admin]#  バッチ処理に利用可能 shell> mysqlsh --file code.js shell> mysqlsh < code.js shell> echo "show databases" | mysqlsh –sql  2016年5月現在:Development Release: 1.0.3
  17. 17. 17 [root@misc01 MID2016]# mysqlsh --uri demo_user@localhost/NEW57 -ppassword Creating an X Session to demo_user@localhost:33060/NEW57 Default schema `NEW57` accessible through db. Welcome to MySQL Shell 1.0.3 Development Preview ……. Currently in JavaScript mode. Use ¥sql to switch to SQL mode and execute queries. mysql-js> db.createCollection("x_posts"); <Collection:x_posts> mysql-js> db.x_posts.add({"title":"Hello World", "text":"This is the first post via mysqlx"}); Query OK, 1 item affected (0.01 sec) mysql-js> db.x_posts.find("title = 'Hello World'").sort(["title"]); [ { "_id": "baee9a744308e61168170800279cea3c", "text": "This is the first post via mysqlx", "title": "Hello World" } ] 1 document in set (0.00 sec) MySQL Shell Version 1.0.3 Development Preview session.getSchema(‘NEW57')
  18. 18. 2016-05-26T23:24:45.694581+09:00 11 Query /* xplugin authentication */ SELECT `authentication_string`, <SNIP> 2016-05-26T23:24:45.695786+09:00 11 Query show databases 2016-05-26T23:24:45.697994+09:00 11 Query select schema(), @@lower_case_table_names 2016-05-26T23:24:45.698455+09:00 11 Query select connection_id() 2016-05-26T23:24:45.698899+09:00 11 Query use `NEW57` 2016-05-26T23:24:45.699441+09:00 11 Query SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2 2016-05-26T23:24:45.701333+09:00 11 Query SHOW FULL TABLES FROM `NEW57` 2016-05-26T23:24:45.703443+09:00 11 Query SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2 2016-05-26T23:24:45.706091+09:00 11 Query SHOW FULL TABLES FROM `NEW57` 2016-05-26T23:25:15.862883+09:00 11 Query CREATE TABLE `NEW57`.`x_posts` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED NOT NULL UNIQUE) CHARSET utf8mb4 ENGINE=InnoDB 2016-05-26T23:25:38.026317+09:00 11 Query INSERT INTO `NEW57`.`x_posts` (doc) VALUES ('{¥"_id¥":¥"de3e8cb74d23e6112c1d0800279cea3c¥",¥"text¥":¥"This is the first post via mysqlx¥",¥"title¥":¥"Hello World¥"}') 2016-05-26T23:25:57.011923+09:00 11 Query SELECT doc FROM `NEW57`.`x_posts` WHERE (JSON_EXTRACT(doc,'$.title') = 'Hello World') ORDER BY JSON_EXTRACT(doc,'$.title')
  19. 19. root@localhost [NEW57]> SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2; +------------------------+---+ | table_name | c | +------------------------+---+ | Innovation_Day | 2 | | Innovation_Day_Confirm | 2 | | X_JSON | 2 | | x_posts | 2 | | X_PYTHON | 2 | +------------------------+---+ テーブル構造(doc json列+generated columnのPK(_id) から、通常のテーブルとドキュメントストアを判別 mysql-js> db.getCollections() { "Innovation_Day": <Collection:Innovation_Day>, "Innovation_Day_Confirm": <Collection:Innovation_Day_Confirm>, "X_JSON": <Collection:X_JSON>, "X_PYTHON": <Collection:X_PYTHON>, "x_posts": <Collection:x_posts> } mysql-js> db.getTables() { "T_GIS": <Table:T_GIS>, "T_JSON_DOC": <Table:T_JSON_DOC>, "T_JSON_DOC_TXT": <Table:T_JSON_DOC_TXT>, "T_TDE": <Table:T_TDE>, "citylots": <Table:citylots>, "employees": <Table:employees>, "employees_json": <Table:employees_json>, "employees_txt": <Table:employees_txt>,
  20. 20.  MySQL5.7~ ドキュメント関連機能の連携デモ JSONデータ型, MySQL Shell, X Protocol, 生成列, FTS Twitter API: (JSONデータ) https://api.twitter.com/1.1/xxxx localhost, port:33060 Schema: NEW57 myDb = mySession.getSchema('NEW57') 省略… timeline = json.loads(res.text) for tweet in timeline: myDb.X_PYTHON.add(tweet).execute() Shell> mysqlsh --py < demo_python_twitter.py +-------+--------------+------+-----+---------+------------------+ | Field | Type | Null | Key | Default | Extra | +-------+--------------+------+-----+---------+------------------+ | doc | json | YES | | NULL | | | _id | varchar(32) | NO | PRI | NULL | STORED GENERATED | | name | varchar(64) | YES | MUL | NULL | STORED GENERATED | | text | varchar(512) | YES | MUL | NULL | STORED GENERATED | +-------+--------------+------+-----+---------+------------------+ (1)Twitter APIに接続しJSONフォーマットのTweetを取得 (2) 取得したJSONデータを.addでテーブルに追加 (3) Generated ColumnやFTSを利用してデータ参照処理 OAuth1Session mysqlx 20
  21. 21. 有難うございました 21 MySQL5.7からの新機能を活用してみて下さい!! 問題・改善要求: https://bugs.mysql.com/

×