HTTP Plugin for MySQL!

HTTP Plugin for MySQL!
HTTP Plugin for MySQL! Ulf Wendel, MySQL/Oracle
The speaker says... MySQL is the WebScaleSQL DBA darling: • The power and popularity of SQL • The – after all – deployment story • Scale-up: supports latest commodity hardware • Scale-out: asynchronous replication • Scale-out: in-memory, synchronous (2PC) replication • Scale-out: virtual synchronous, group based replication • Scale-out: sharding through Fabric Just, we forgot about the web developer!
shell> curl --user user:password -X PUT -d $'{"msg": "Welcome, JavaScript user!"}' http://new.mysql.com/doc/pres/http/key_message
The speaker says... Imagine, you are under such pressure that you have to revolutionize the way you work. Web developers do that. Infrastructure costs socialized through Open Source: web developers! Agile movement: web developers! Rusty database vendors don't deliver: Not Only SQL created! • Convenient APIs and relevant protocols • Really easy sharding • Schema flexibility and documents • Using their latest tools: JavaScript and JSON • No always failing short yet complicated ORM • No lobby standard ECISANO 5.7 Rev. 16 – buy now
Backend: It's LAMP. Period. Frontend: Dinosaur, it's 2014. Need a RESTful API. Backend: No time. And, no to framework Sendso...
The speaker says... Every other day two developers clash on the question of databases... MySQL cannot ignore the needs of web developers. MySQL cannot hide itself behind the argument some third party framework could be used – that failed badly for replication. MySQL cannot hesitate and hope legacy code would stop migrations. Web developers completed two revolutions. Replacing MySQL would be a minor one. DBAs won't stop them. What if MySQL speaks HTTP, stores JSON, returns JSON and preserves the richness of SQL?
Three HTTP APIs All APIs return JSON SQL CRUD: flat JSON documents mapped to tables Nested JSON documents stored in tables Key-Document semantics
The speaker says... Every other day, a proxy is desired to make MySQL more accessible: HTTP – PHP|Perl|Python|... - MySQL. All the proxies translate HTTP to SQL. SQL lets them offer any API. What if MySQL had a built-in HTTP to SQL proxy? Right from start, there could be a variety of APIs. Plain SQL over HTTP is a given. Everybody knows SQL, everybody understands the advantages of a declarative language, only SQL can expose the full feature set of MySQL. No attempt to dictate any whiteboard API: plain SQL. Google created AngularJS for CRUD applications. CRUD is a given. Nested JSON documents are a must.
http://labs.mysql.com Free (GPL), 80+ pages documentation, many examples
The speaker says... The built-in MySQL HTTP APIs are real. You can get the HTTP Plugin for MySQL from http://labs.mysql.com (September 2014). As always, it's free and GPL. It comes with a 80+ pages PDF document that has many examples on using it. The HTTP Plugin is for the JavaScript users, for those fighting corporate firewalls that allow no other protocols but HTTP, those who love web services and those who are tired of writing yet another RESTful plain scripting language proxy for MySQL! Next: Unboxing. Then a word on how it works, how it serves not only data but you. And finally an outlook.
UNBOXING: a seasonal s carf ready MySQL 5.7 Server
The speaker says... Autum is coming to the northern hemisphere but before the nights get longer than the days, before we all start knitting scarfs, it's time for world's first unboxing. In the box of the initial release is a special version of the MySQL Server. The server is based on the MySQL Server preview (5.7.5-m15) made available for MySQL Connect 2014. But, it has some additional feature tweaks. For now, you need this very server and no other. Why that is the case will be explained in a minute.
UNBOXING: a high tech H TTP Plugin for MySQL
The speaker says... MySQL stands for high tech: HTTP Plugin for MySQL! The treasure box Sakila (the MySQL mascot) accidently discovered at the beach contains the HTTP Plugin too. One slide on installation to come, then we talk how it works.
UNBOXING: plain vanilla... Install MySQL from the treasure box Create a SQL user for the plugin • Set password, grant access to SQL object Install the plugin • INSTALL PLUGIN myhttp SONAME 'libmyhttp.so' Edit my.cnf • Set default SQL user for plugin, HTTP basic auth, port, etc. Restart MySQL • If you edit first, then load – recall to use loose option prefix
The speaker says... The installation procedure is not different from any other plugin. Some stuff the plugin needs to work is to be set up first. The HTTP plugin runs SQL and does that – by default – as a certain user. Create that user. Personally, I prefer loading the plugin in the next step before making changes to the MySQL Server configuration file. This allows me not to bother about a server that barks about unkown settings. Once installed, the plugin begins listening to HTTP requests (default 8080). If you have a server running on that port, edit my.cnf first to set port, user etc., then start MySQL. Recall to use the „loose“ prefix trick. Then install the Plugin. The manual has the details... We get back to settings in the usage section after: how it works (now).
How? Extending MySQL! MySQL Core Deamon Plugin Deamon Plugin I_S Plugin Full-Text Parser Plugin Plugin Services User-defined functions Semisync Replication Plugin Audit Plugin Authentication Plugin Password-Validation Plugin Protocol Trace Plugin Storage Engine Plugin
The speaker says... The MySQL manual spends 80 pages educating readers how to extend MySQL: there are user-defined functions (UDF) and there are plugins. SQL can be extended with UDFs. Nine different types of plugins exist. Best known should be the storage engine plugins (InnoDB, Memory, NDB, …, 3rd party) and the authentication plugins (MySQL native, SHA256, …). The HTTP Plugin is a daemon plugin. A daemon plugins is a do-as-you-like library that is loaded into the MySQL Server. Plugins have access to half a dozen plugin service provided by the server (Memory, thread management, ...). The HTTP Plugin is build atop of new plugin services.
The Protocol deamon plugins HTTP Client Memcache driver MySQL Connector MySQL Daemon Plugins HTTP Plugin Memcache Plugin Core HTTP Protocol (JSON) Memcache Protocol (Binary) MySQL Client/Server Protocol (Binary)
The speaker says... Deamon plugins can start a network server and listen to an arbitrary protocols to serve data. In 2009, the year the equation NoSQL = Not Only SQL was proposed, the first community plugin did exactly that. For rich queries the customers used SQL, for fast but simple queries a home-brewed daemon plugin using lower level APIs. MySQL was at the forefront but few know it. The MySQL Server comes with a free Memcache Plugin for InnoDB since 5.6, so does MySQL Cluster. The customers and community desire was to kill any read access overhead: SQL parsing, network traffic. For write they desired ACID, in-place update and predictable performance under all circumstances.
The query language flexibility HTTP Client Memcache driver MySQL Connector MySQL Daemon Plugins HTTP Plugin Memcache Plugin Core SQL Key-Document Key-Value SQL
The speaker says... The HTTP Plugin Lab Release is not designed for maximum speed. It targets the JavaScript user. Today, it is just a demo of what APIs we could offer to the JavaScript user. The Lab Release does nothing you cannot do yourself for years already: all HTTP requests are mapped to plain SQL internally. But it does it for you. The day the APIs have been finalized we can boost the performance. We are also exposing the richness of SQL. There is no we know the best ORM and you must love it. Did they ever work? Use SQL or try the CRUD interface we sketched or store nested JSON documents. At this point, all we do is add a HTTP to SQL proxy to MySQL...
Your trouble vs. our trouble HTTP Client SQL, Key-Document over HTTP HTTP Plugin MySQL HTTP Client Some service over HTTP Webserver Script Language MySQL Connector MySQL
The speaker says... This Lab Release is no rocket science. Effectively, we are only announcing the intend to offer an HTTP interface as a convenience feature. We move the proxy you write today to make MySQL accessible through HTTP into the server – as an optional plugin. We take the trouble of thinking about security or performance. There is one drawback, assuming the APIs work for you. Our solution could be a little less scalable. A shard nothing architecture, an echo of the Unix way of having small independent programs, is often the best way to scale. But then, the external proxy has more connections and may be slower... Wait for it to stabilize before you stress test – not worth checking today. Let's talk using and APIs.
http://new.mysql.com/sql/pres/SELECT%20'Hello World!' http://new.mysql.com/crud/pres/http/hello http://new.mysql.com/doc/pres/http/hello
The speaker says... There are three APIs with some properties in common: • they have JSON all over • guess what: they all use HTTP, which sets limits • the same (yet unsecure) security pattern • they share the same restrictions The APIs differ by: • their query language • their current interest in REST... Common things first at the example of the SQL API.
The SQL endpoint and JSON shell> curl --user basic_auth_user:basic_auth_passwd --url "http://127.0.0.1:8080/sql/db/SELECT+1" [ { "meta":[ {"type":8,"catalog":"def","database":"","table":"", "org_table":"","column":"1","org_column":"","charset":63, "length":1,"flags":129,"decimals":0} ], "data":[ ["1"] ], "status":[{"server_status":2,"warning_count":0}] } ]
The speaker says... The SQL endpoint is our Connector/JavaScript. What you see is exactly what a standard MySQL Connector gets in reply to the query. But the standard MySQL Connector is using the binary MySQL Client/Server Protocol and all data would look funny. Because the same information is returned the HTTP client is not feature constrained. That is a sweet lie as we will see later but you get the idea. The HTTP client, the JavaScript user is not a second class client who is limited to restricted APIs up-front. All JSON is UTF-8. For now, the HTTP Plugins' web is UTF-8. Binary data and funny german umlauts are encoded using uxxxx notation, as json.org & friends ask for.
Dot zero: HTTP is stateless No session concept for the 1.0 • Stateless helps with scaling and reduces complexity • Some feature, e.g. SQL session variables, have limited value ACID but autocommit • Each web request is a transaction • Difficult to isolate multi-statement/business transactions shell> curl --user basic_auth_user:basic_auth_passwd --url "http://new.mysql.com/sql/db/SET+@no='session'" { "server_status": 2, "warning_count": 0, "affected_rows": 0, "last_insert_id": 0 }
The speaker says... The stateless nature of HTTP is no natural match for SQL. ACID transactions are an integral part of SQL systems. Transactions are units of work made of one or more statements. The SQL endpoint lets you carry out one SQL statement per request. Every new HTTP request is independent of the previous one, thus every statement forms its own transaction: it runs in autocommit mode. You get ACID properties: atomicy, consistency, isolation and durability. E.g. two concurrently executed web requests are properly isolated from each other. Sessions? Makes a lot sense to me! Not only for multi-statement/ business transactions.
Dot zero: user/security concept HTTP Client No SSL Authentication HTTP Plugin Preconfigured user MySQL HTTP Client SSL Authentication HTTP Plugin Request user MySQL SQL grants
The speaker says... The user/security concept is two-staged. First, a HTTP client authenticates itself towards the HTTP Plugin. Then, the HTTP Plugin executes SQL statements on behalf of the HTTP client as a certain SQL user. If no SSL is used, the HTTP Plugin logins in towards MySQL as a preconfigured SQL user. The HTTP world and the SQL world are decoupled. With secure connections (SSL) the user name and password obtained from the HTTP world can be passed to MySQL as login credentials. A HTTP client is just another MySQL client. Use the SQL permission system to restrict access!
HACKER Welcome! MySQL morons using HTTP Basic Authentication...
The speaker says... The initial dot zero Lab Release is using HTTP Basic Authentication all over. All resources – SSL and Non-SSL – are protected with basic auth. HTTP Basic Authentication is fast to implement. Everybody understands the examples because of their simplicity. That is perfect for a /preview/, that is perfect to kick of a discussion. Unfortunately HTTP Basic Authentication is one of the least secure choices one can make.
SSL: User/security concept HTTP Client SSL OpenID, OAuth... HTTP Plugin MySQL HTTP Client SSL Password exchange HTTP Plugin Authorization MySQL
The speaker says... How hard can it be to support any proper HTTP authentication method: a day for OAuth, two weeks for OpenID? The Lab Release uses HTTP Basic Auth as a vehicle that combines two steps, which is confusing but was fast to implement. For SSL connections, and only SSL connections, users should first authenticate themselves through some proper HTTP authentication method. Once authenticated, they shall be able to login as a MySQL user of their choice. We still need to fiddle out the details and we lack something here as the outlook will show. Next common properties of all APIs: restrictions.
Dot zero: restrictions No query cache support • It's HTTP – use a standard HTTP caching! • MySQL Support often suggests disabling the query cache No commercial thread pool plugin support • There are now HTTP network threads and SQL worker threads • HTTP layer can do multiplexing Not all MySQL pluggable auth methods supported • Some pluggable auth methods have handshakes • Handshake procedures require use of MySQL networking layers
The speaker says... The preview has some restrictions. They all boil down to the fact that building a proxy into MySQL is an all new requirement. Various modules in MySQL have not been designed for this new use case. How could they? It is a new use case... Do you mind being restricted to use MySQL SQL accounts that have been created with MySQL's default authentication plugin? Do you mind following a MySQL Support advice of not using the query cache? Recall, that you should deploy InnoDB with 80% of your memory dedicated to the buffer cache. Disk access is rare.
The SQL endpoint Unlimited: all SQL statements • DDL, DML and utility statements • Multi-resultset statements supported, e.g stored procedures Unlimited: everything the server returns • Meta data, query data, server status • Freedom to build your own Connector/JavaScript [{ "meta" : [ … ], "data" : [ … ], "status" : [ … ] }]
The speaker says... First class customers use the SQL endpoint. It has the most to offer. There are no restrictions on which SQL statements can be executed. Use the MySQL SQL permission system to set any, if needed. No limited white-board API is imposed on you. All the information the server returns in reply to a query is converted to JSON and sent to the client. A standard MySQL Connectors using the MySQL Client/Server Protocol gets the same information but in binary form. Because the same data is delivered, clients can develop their own Connectors, their own higher level APIs and data structures.
Unlimited result set meta data [{ "meta":[{ "type" : 4, "catalog" : "def", "database" : "pres", "table" : "sql_types", "org_table" : "sql_types", "column" : "col_float", "org_column": "col_float", "charset" : 63, "length" : 12, "flags" : 4097, "decimals": 31 }], "data" :[ … ], "status":[ … ] }]
The speaker says... On the slide is the result set meta data for SELECT col_float FROM sql_types. The full table definition of the example table sql_types is given in the manual, however, col_float is of the SQL type FLOAT. For every column in the result set you get one object in the meta array. Every object has the same members: there's the database name, the table name, the column name, the charset and so forth. Why all this result set meta data?
Types: string or NULL shell> php -r ' $m = new mysqli("host", "user", "password", "db", 3306); $m->query("SELECT col_float FROM sql_types")->fetch_assoc());' array(1) { ["col_float"]=> string(6) "0.9999" } shell> curl --user basic_auth_user:basic_auth_passwd --url "http://…/sql/db/SELECT+col_float+FROM+sql_types" [{ "meta":[ … ], "data":[ ["0.9999"], … ], "status":[ … ] }]
The speaker says... The SQL endpoint converts all SQL result data to JSON strings or JSON NULL. A conversion to string is the servers standard behaviour for non-prepared statements in the MySQL Client/Server Protocol too. This server implementation detail still shows in PHP. Still today, PHP returns numbers as strings. That made a lot sense in the last millennium. HTTP GET and POST data are strings. PHPs' dynamic typing was a relief for those doing web development in C before. There's still a point for sending strings today.
No type mapping, no loss How would the server know who consumes the data • Which rules? • Information loss allowed? SQL type JSON candidate Information loss SMALLINT Number Precision, range FLOAT Number Precision, range VARCHAR String Precision, charset DATE String Type, range BOOLEAN True, false NULL Null
The speaker says... The server does not know the type system of the client. If, for example, SQL data types had been mapped to JSON data types and no meta data had been included, information had been lost. Complicated? Choose between a Bugatti Veyron and a Suzuki Maruti 800. The Suzuki has simpler controls. The Suzuki costs less. The Suzuki is more fuel efficient. All speaks for the Suzuki but we won't give it to you. The SQL endpoint is as luxury and as complex as the Bugatti. Here's why: it's easy to use a hammer to shrink the Veyron to the size of the Suzuki. It's hard to make the Maruti go 400 km/h. Even when dropping from the sky – cW too high ;-).
JavaScript same origin policy JSON with padding • Optional escaping into string • CORS on the radar var ret = $.ajax({ url: ".../sql/db/SELECT%20'Hello world!'%20FROM%20DUAL", type: "GET", username: "basic_auth_user", password: "basic_auth_passwd", dataType : "jsonp", jsonp: "jsonp", success: function( json ) { alert("First row: " + json[0].data[0]); }, });
The speaker says... Browserside JavaScript has two options to overcome the same origin policy: JSON with padding (JSONP) and Cross-Origin Resource Sharding (CORS). JSONP is there for the Lab Release, CORS is missing. Add the GET parameter jsonp=<my_callback_name> to request to get JSONP replies. The result will be: my_callback_name(<json>). The slide shows an example for jQuery. Optionally, you may request the <json> to be returned as a string value, like so: my_callback_value("<json>"). If you need the latter, add the GET parameter jsonp_escape to the request URL.
HACKERs' heaven! Popcorn for everbody. MySQL in the web...
The speaker says... Ever noticed NoSQL systems offering HTTP interfaces? Peace: it is an optional plugin...! You decide whether you load it into MySQL or not. The HTTP Plugin currently lacks settings for disabling endpoints. Accidently exposing sensitive data, for example, because of inappropriate SQL permissions, is a security risk. We will talk about that at the end. The SQL endpoint is the unlimited first class customer API. Great power comes with great responsibility. Use it with greatest care! Next: the more restricted CRUD endpoint!
The CRUD endpoint Flat JSON objects mapped to tables • Always JSON objects towards the caller Limited query language • Data manipulation only: GET, PUT, DELETE • Key-document semantics • No data definition statements • No administrative statements Less network traffic • No result set meta data
The speaker says... The CRUD endpoint features simple key-document semantics for unnested JSON documents. Only data manipulation operations can be performed. Whereas the SQL endpoint allows no other HTTP methods but GET, the CRUD endpoint accepts GET, PUT and DELETE requests. In that sense it shows more love for REST. On the outside there are JSON documents (objects) only. On the inside all requests are mapped to SQL in a straightforward way. Let's see how.
The CRUD SQL mapping Document = SQL table row • Documents are flat, they are limited to scalar types • Underlying tables must have a single column primary key CREATE TABLE `simple` ( `id` int(11) NOT NULL AUTO_INCREMENT, `col_a` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 shell> curl --user basic_auth_user:basic_auth_passwd --url "http://.../crud/db/simple/1" {"id":"1","col_a":"Hello"}
The speaker says... It is that simple: SQL table rows are returned as JSON objects. Plain SQL tables serve as JSON document containers. Each document is derived from only table only. Because MySQL SQL tables support scalar types only, the corresponding JSON douments contain scalar types only such as JSON string, number, boolean or null. The CRUD endpoint does not support JSON documents containing arrays or objects. Every mapped table must have a single column primary key. The primary key is used for the key-document access logic.
GET = SELECT shell> curl … --url "http://../crud/db/sql_types/1" { "id" : "1", "col_char" : "CHAR(127)", "col_null" : null, "col_date" : "2014-08-21", "col_decimal": "123.45", "col_float" : "0.9999", "col_bigint" : "9223372036854775807" } shell> curl … --url 'http://../crud/db/simple/' { "errno" : 2000, "error" : "The request URL must include a primary key value" }
The speaker says... The initial version features key-document accesss only. Therefore, all GET requests must include a primary key value and follow the URL pattern protocol://server:port/crud/database/table/pk_value. Like the SQL endpoint, the CRUD endpoint returns all data either as a string or null. All the APIs are far from finalized. A possible future optimization would be a better mapping between SQL and JSON data types. For example, boolean values could be returned as true/false JSON constants.
Key-document only? No :-) ! Standard client, e.g. PHP Memcache driver MySQL Connector JS framework Rich SQL Tuneable K/V Fast K/V MySQL JS in browser Rich SQL HTTP Plugin Memcache Plugin Core
The speaker says... MySQL 5.7.5 handles up to ~625.000 point select SQL queries and ~1.100.000 comparable Memcache accesses per second. This clearly shows the tuning potential behind simplified key-value (document) style APIs. Standard MySQL clients need two drivers to profit from the rich query capabilities of SQL and the performance of K/V. The CRUD endpoint is untuned. First, there needs to be a proper API (not the current...), then one can optimize. Once optimized HTTP clients can choose by just changing the request URL slightly: sql/ vs. crud/ respectively doc/. Could not be better, could it? Back to the CRUD endpoint basics: HTTP PUT
PUT = REPLACE shell> curl … -X PUT -d '{"col_a":"Ahoy"}' --url "http://.../crud/db/simple/1" { "affected_rows": 2, "warning_count": 0 } shell> curl … --url "http://.../crud/db/simple/1" { "id" : "1", "col_a": "Ahoy" } shell> curl … -i -X PUT -d '' --url "http://.../crud/db/simple/1" HTTP/1.1 400 Bad Request …
The speaker says... HTTP PUT is mapped to SQL's REPLACE statement in the CRUD endpoint of the initial Lab Release. PUT expects a JSON object as input. The object must have one top level member for each column in the underlying SQL table that expects a value. If you fail to provide proper input, the HTTP Plugin for MySQL barks at you: 400 Bad Request. No attempt is being made to hide the SQL reply to the REPLACE statement. It is one of the many edges we failed to cut off for the development announcement release. Another edge is the lack of clever mapping like PUT = INSERT, PATCH = UPDATE. Adding that is about as hard for us as it is for you in your HTTP to MySQL proxy...
DELETE = DELETE, that's easy! shell> curl -i … --url "http://.../crud/db/simple/1" HTTP/1.1 200 OK … {"id":"1","col_a":"Ahoy"} shell> curl -i -X DELETE … --url "http://.../crud/db/simple/1" HTTP/1.1 200 OK shell> curl -i … --url "http://.../crud/db/simple/1" HTTP/1.1 404 Not Found … {"errno": 2000, "error":"Not found"}
The speaker says... HTTP DELETE is translated to SQL DELETE. That's it. There is some support for HTTP principles. For example, HTTP methods are mapped appropriately to data manipulation commands. Even HTTP headers are set: 200 OK, 400 Bad Request, 404 Not Found and so forth. This alone does not make a REST API and ticking of the checklist for the buzzword is the least (important). Next: nested JSON documents – SQL completely hidden.
The DOCUMENT endpoint Nested JSON objects mapped to tables • Always JSON objects towards the caller Limited query language • Key-document semantics • Data manipulation: GET, PUT, DELETE • Data definition: Table creation and removal • Optimistic locking No sign of SQL • No cryptic affected_rows ... • Leave room for possible optimizations
The speaker says... When it comes to HTTP details and access methods, CRUD and DOCUMENT are very much the same. DOCUMENT is just a tiny bit more. If you consider the design space for possible APIs of a HTTP Plugin that wants to speak JSON all over, you may notice that the two extremes are plain SQL and a JSON document store. In our initial Lab Release, the CRUD endpoint sits in the between the two extremes. The DOCUMENT endpoint is the „what if MySQL bend itself towards a JSON document store“. Stay seated: an API does not make MySQL a proper document store!
JSON goes into a BLOB shell> curl -i -X PUT --user … --url "http://.../doc/myhttp/new_table/" HTTP/1.1 201 Created … {"info": "Table created"} mysql> SHOW CREATE TABLE new_tableG *************************** 1. row *************************** Table: new_table Create Table: CREATE TABLE `new_table` ( `_id` varchar(36) NOT NULL, `_rev` bigint(20) unsigned NOT NULL DEFAULT '0', `_extra` blob NOT NULL, PRIMARY KEY (`_id`,`_rev`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8
The speaker says... Use HTTP PUT requests to create tables for storing documents, to insert documents and to update them. There are no restrictions on what kind of JSON objects the endpoint handles. The API accepts every valid JSON object. Internally, JSON documents are stored in a BLOB column of a standard SQL table. The primary key of the underlying SQL table is made of two columns: the id or key that appears in request URLs and a revision counter. The revision counter is managed by the HTTP plugin to implement optimistic locking.
Optimistic locking HTTP Client HTTP Client MySQL GET ../key {"_rev":1, …} {"_rev":"1", …} PUT ../key {"_rev":1, …} OK: _rev: 2 PUT ../key …1} Conflict!
The speaker says... Arguing that HTTP is stateless and reflecting that fact in the APIs is great if you are short on time. But then, without session support, without multi-statement transactions – all endpoints use autocommit – how can two clients avoid and detect conflicting wirtes? One can leave the task to the client, you the developer. One can consider multi-versioning. This raises at least a garbage collection problem because the server cannot know when old revisions may be removed. Or, the DOCUMENT endpoint supports optimistic locking – which just works if the clients cooperate. Means: a client must not modify the special JSON document object member _rev.
Solve conflicts by refetching shell> curl --user … --url ".../first_message" {"_id":"first_message","_rev":1,"msg": "No big news" } shell> curl -X PUT -d '{"_id": "first_message","_rev": 1,"msg": "Optimist?"}' --user … --url ".../first_message" {"info": "Document updated"} shell> curl -X PUT -d '{"_id":"first_message","_rev":1,"msg": "Optimist?"}' --user … --url ".../first_message" {"errno": 2000, "error": "Update failed. Your revision does not match the current revision"}
The speaker says... Here's an example of optimistic locking in action. Every JSON document managed by the DOCUMENT has two special members: _id and _rev. _id is the key of the document. _rev is the revision number used for optimistic locking. When a JSON document is written back to the database, the database compares its revision number with the revision stored in the database. Should they be equal, the write is accepted and the database increments the revision. If not, then someone must have updated the document already and the write is rejected. In case of a conflict, a client should fetch the latest version, merge the changes to solve the conflict and try again.
PUT = CREATE TABLE, REPLACE shell> curl -i -X PUT --user … --url "http://.../doc/db/another_table"; HTTP/1.1 201 Created {"info": "Table created"} shell> curl -i -X PUT -d '{"words": ["Hello", "world"]}' --user … --url "http://.../doc/db/another_table/key" HTTP/1.1 200 OK {"info": "Document added"} shell> curl -i -X PUT -d 'is this json?' --user … --url "http://.../doc/myhttp/another_table/key" HTTP/1.1 400 Bad Request {"errno":2000, "error":"Invalid JSON"}
The speaker says... For the REST, let's do a walkthrough the HTTP methods supported. In the initial development release, the DOCUMENT endpoint has slightly more features than the CRUD endpoint. The CRUD endpoint will certainly be updated. Not only AngularJS framework lovers shall have free choice between the two types of mapping from JSON to SQL, both APIs shall offer comparable feature sets. The DOCUMENT endpoint supports PUT for creating new tables, inserting nested JSON documents into them and updating documents. Nothing fancy but the optimistic locking. Invalid JSON is rejected.
GET = SELECT shell> curl --user … --url "http://.../doc/db/another_table/key" { "_id": "key", "_rev": "1", "words": ["Hello", "world"] } shell> curl --user … --url "http://.../doc/db/another_table/" { "another_table": [ { "_id": "key","_rev": "1","words": ["Hello", "world"] } ] } shell> curl -i --user … --url "http://.../doc/db/another_table" HTTP/1.1 400 Bad Request
The speaker says... A HTTP GET request against the DOCUMENT endpoint either returns one document identified by its key (id) or all documents from a table. Recall, that the access method is key-document only. For now, if you need filtering and the like, switch to the SQL endpoint. Change the request URL and use SQL for filtering.
Filtering? Indexing? Coming... See labs.mysql.com • JSON UDFs, work in progress: more functions than listed below • MySQL 5.7.4: Virtual columns, stay tuned on functional index Search • By value: JSON_SEARCH() • By key: JSON_CONTAINS_KEY(), JSON_EXTRACT() Modify • By value: JSON_MERGE() • By key: JSON_APPEND|REMOVE|REPLACE|SET()
The speaker says... With MySQL 5.7.4 it is finally possible to work on JSON using SQL. The release brings: • updated SQL functions for working on JSON • virtual columns • … and stay tuned on function based index The SQL functions let you search JSON and modify it. They fail to close the gap between the JSON and SQL data models but if you stay within the JSON model it works. The HTTP Plugin means JSON all over... Also, indexing JSON will boost search beyond key-document.
DELETE = DELETE shell> curl -X DELETE --user … --url "http://.../doc/myhttp/another_table/key" { "info": "Document removed" } shell> curl -X DELETE --user … --url "http://.../doc/myhttp/another_table/" { "info": "Table dropped" } shell> curl -X DELETE -i --user … --url "http://.../doc/myhttp/another_table/" HTTP/1.1 404 Not Found
The speaker says... To remove documents and tables issue HTTP DELETE requests. That's it – very similar to the CRUD endpoint.
Dear JavaScript users, please check the manual. Some examples already there, more coming.
Recap HTTP Protocol in addition to MySQL Protocol JSON all over Key-Document and SQL complementing each other SQL support for JSON getting 'real' with MySQL 5.7.4+
Hmm...? What's Sakila doing...
Sakila, you put it back in?
Hidden treasures? Labs leaks? Oh, no... back to the sea ?!
Yammy!
The speaker says... A MySQL Lab Releases can be anything from almost production ready to a development announcement. For example, there have been several prior releases of the SQL JSON functions, which means they should work reasonably well. The day MySQL 5.7.4 was released a colleguage was praised for contributing to the SQL standard around computed columns, which means they should work reasonably well... The HTTP Plugin 1.0.0 is a yammy development announcement release. It's all about discussing it early. That's why we have 80+ pages documentation from day one but edges in the APIs.
Leaving the safe harbour Glancing over the ocean of possibilities
The speaker says... More open talk: ideas. If you are a developer using MySQL, you may stop here. The presentation has explained what the HTTP Plugin is, why it was created, what it has to offer and what the current status is. What's left is a deeper look inside, how it's done inside MySQL. That's for MySQL experts who seeks to understand MySQL internals. Understanding them is a precondition to use them in a different context, tailored to the own needs.
Module A Module B Module C Module D Plugin API
The speaker says... The challenge in doing a HTTP Plugin is not the HTTP API itself. How hard can it be to add something developers do every day to MySQL? How hard can it be to add a Proxy to MySQL? Some say MySQL is not modular. Some say MySQL lacks clearly seperated modules in the code. Is MySQL a ball of woll with knotted fiber? Some say the code base shows its age. Is MySQL in the autum of its life, does it need a scarf? Well, RDBMS have undergone permanent refactoring in the past 50 years and they are sill around. NoSQL has its first wave yet to come. Think MongoDB and storage. (Insider: Steward, I took that photo prior to your blog post.)
Module A Module B My new API :-)
The speaker says... The problem is a different one. With every new feature request someone comes and invents a new module. Some say, the authors of the HTTP Plugin invent new non-existing modules every other day. There are two new plugin services inside the HTTP Plugin. Let's discover their potential beyond the HTTP Plugin.
Plugin Services HTTP Client SQL or any derived query method HTTP Plugin Output Formatter Web Server, ... SQL Execution Service Authentication Service MySQL
The speaker says... The HTTP Plugin is build atop of two plugin service. The first is an authentication service to log in a MySQL internal SQL execution thread as a certain MySQL user. Once authenticated and authorized, the MySQL internal SQL execution thread uses the SQL execution service. The results are returned as a binary string stream. The stream is processed by an output formatter, which is provided by the plugin calling the service. The plugins output formatter can serialize query results into any format. In case of the HTTP Plugin, its JSON. Could have made it BSON but which JavaScript client wants that...
Proxy: SQL over Protocol X Websocket Client SQL Websocket Plugin JSON Serialization MySQL HTTP Client SQL HTTP Plugin JSON Serialization SQL Execution Service Your Client SQL Your network Prot. Your serialization
The speaker says... The HTTP Plugin is a protocol and data serialization format SQL proxy built-in to MySQL. It is one that is tailored for the JavaScript user. This restricts the protocol options to HTTP and Websocket. In 2009 some MySQL high-end community users added an alternative, faster communication channel to MySQL for key-value style queries. Many expert users restrict their developers to a subset of SQL for performance reasons. The MySQL Client/Server Protocol has to support all SQL features and all programming language communities. Any ideas for optimized SQL centric protocols tailored to your users?
Proxy: for data models HTTP Client JSON Document HTTP Plugin SQL Rewrite Schema Mapping Util: e.g Catalog Relations MySQL
The speaker says... The HTTP Plugin is data model proxy. Closing the gaps between data models can be done in many ways. In its early days, when MySQL was still looking for its space in the market, the storage engine API was used to offer MySQL as SQL frontend for about anything. Here is a suggestion from the Connectors team! In a perfect world, the Connectors team understands developers and their motivation. Working at this level is easier, more can contribute. The RDBMS world tried data model mappings in the past: it was never fast. But, the systems where never fast to support new models either! Getting it right is hard. Time to market counts. Web developer = no mercy = they do revolutions...
Proxy: simplified shardig HTTP Client Key-document HTTP Plugin SQL Execution Any shard Target shard 301 Moved Which shard? Fabric
The speaker says... MySQL supports auto-everything sharding since a decade. It comes with MySQL Cluster. MySQL Cluster just learned active-active asynchronous replication with automatic conflict solution between synchronous clusters in different data centers! Problem is, the web user wants InnoDB. So we build MySQL Fabric. Fabric manages farms of MySQL servers and lets you shard data across them. Problem is, it is not transparent and requires special clients. Imagine you limit the scope to stateless HTTP and key document. Stateless means redirect is no pain. HTTP means you can use slim clients and 301. Document means cross-shard joins can be avoided... Extra hop? Pffft, look at MySQL Cluster.
Let's not loose focus: JS! V8! HTTP Client Improved security, more freedom HTTP Plugin app/ - v8 JavaScript sql/, crud/, docs/ - built-in SQL Execution Service Authentication Service MySQL
The speaker says... Let's not loose focus! The HTTP Plugin is for the developer not the DBA and database architect. That's why MySQL 5.7 brings true multi-master replication – its ease of use. Multi-master has scalability limits (~10...20 machines) but how many web developers are happy with a single of todays monster machines (6k Euro price point = 256GB RAM, 7TB disk, …) and bother only about dev features? The HTTP Plugin is specifically for the web and JavaScript developer. What could be more desired than adding the v8 JavaScript engine to the plugin. This also enables scripts to implement additional security rules. Rules that go beyond what the MySQL user system has to offer!
A world of options Make it happen, today!
The speaker says... It is a world of options. No decisions have been made about any of them. Looking forward to hear back or see you implement your ideas for MySQL.
THE END Contact: ulf.wendel@oracle.com
The speaker says... Thank you for your attendance! Upcoming shows: International PHP Conference (Munich), October 26 - 29, 2014 Your place? Just ask!

Check these out next

HTTP Plugin for MySQL!

Recommended

More Related Content

Slideshows for you(20)

Similar to HTTP Plugin for MySQL!(20)

More from Ulf Wendel(19)

Recently uploaded(20)

HTTP Plugin for MySQL!