Presentation at FOSS ASIA 2015 in Singapore, on March 15th 2015. This presentation explains benefit of "SQL and NoSQL" hybrid data management solution MySQL Cluster. MySQL Cluster is designed as high available and high scalable database cluster. In the same time MySQL Cluster can work as ACID compliant transactional Key-Value Datastore aka KVS #FOSSASIA

1. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
NoSQL
and
SQL
The
Best
of
Both
Worlds
梶山隆輔
/
Ryusuke
KAJIYAMA
MySQL
Sales
ConsulKng
Senior
Manager
MySQL
Global
Business
Unit

2. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Safe
Harbor
Statement
The
following
is
intended
to
outline
our
general
product
direcKon.
It
is
intended
for
informaKon
purposes
only,
and
may
not
be
incorporated
into
any
contract.
It
is
not
a
commitment
to
deliver
any
material,
code,
or
funcKonality,
and
should
not
be
relied
upon
in
making
purchasing
decisions.
The
development,
release,
and
Kming
of
any
features
or
funcKonality
described
for
Oracle’s
products
remains
at
the
sole
discreKon
of
Oracle.

3. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
The
Data
Management
Landscape
Source: 451Group

4. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Session
Agenda
• NoSQL
–
What
are
people
looking
for?
• RDBMS
–
What
advantages
do
they
sKll
have?
• How
MySQL
Delivers
the
Best
of
Both
Worlds
– MySQL
Cluster
• NoSQL
aZributes:
Scale-­‐out,
performance,
ease-­‐of-­‐use,
schema
flexibility,
on-­‐line
operaKons
• NoSQL
APIs
– Key-­‐Value
store
access
to
InnoDB
(Memcached)

5. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Key-­‐Value
Store

6. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Document
Store

7. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Graph
Database

8. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Massive
scalability
– No
applicaKon-­‐level
sharding
• Performance
• High
Availability/Fault
Tolerance
• Ease
of
use
– Simple
operaKons/administraKon
– Simple
APIs
– Quickly
evolve
applicaKon
&
schema
Scalability
Performance
HA
Ease
of
use
What
NoSQL
must
deliver

9. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

10. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
NoSQL
Simple
access
paZerns
Compromise
on
consistency
for
performance
Ad-­‐hoc
data
format
Simple
operaKon
SQL
Complex
queries
with
joins
ACID
transacKons
Well
defined
schemas
Rich
set
of
tools
• No
best
single
soluKon
fits
all
• Mix
and
match
SKll
a
role
for
SQL
(RDBMS)?
Scalability
Performance
HA
Ease
of
use
SQL/Joins
ACID
TransacBons

11. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
Overview
• Auto-­‐Sharding,
MulK-­‐Master
• ACID
Compliant,
OLTP
+
Real-­‐Time
AnalyKcs
HIGH
SCALE,
READS
+
WRITES
• Shared
nothing,
no
Single
Point
of
Failure
• Self
Healing
+
On-­‐Line
OperaKons
99.999%
AVAILABILITY
• In-­‐Memory
OpKmizaKon
+
Disk-­‐Data
• Predictable
Low-­‐Latency,
Bounded
Access
Time
REAL-­‐TIME
• Key/Value
+
Complex,
RelaKonal
Queries
• SQL
+
Memcached
+
JavaScript
+
Java
+
HTTP/REST
&
C++
SQL
+
NoSQL
• Open
Source
+
Commercial
EdiKons
• Commodity
hardware
+
Management,
Monitoring
Tools
LOW
TCO

12. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Who’s
Using
MySQL
Cluster?

13. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
Architecture
MySQL
Cluster
Data
Nodes
Clients
ApplicaKon
Layer
Data
Layer

14. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
Scaling
MySQL
Cluster
Data
Nodes
Clients
ApplicaKon
Layer
Data
Layer

15. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

16. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
HA
MySQL
Cluster
Data
Nodes
Clients
ApplicaKon
Layer
Data
Layer

17. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
On-­‐line
OperaKons
• Scale
the
cluster
(add
&
remove
nodes
on-­‐line)
• ReparKKon
tables
• Upgrade
/
patch
servers
&
OS
• Upgrade
/
patch
MySQL
Cluster
• Back-­‐Up
• Evolve
the
schema
on-­‐line,
in
real-­‐Kme

18. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

19. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
1.2
Billion
UPDATEs
per
Minute
• NoSQL
C++
API,
flexaSynch
benchmark
• 30
x
Intel
E5-­‐2600
Intel
Servers,
2
socket,
64GB
• ACID
TransacKons,
with
Synchronous
ReplicaKon
0
5
10
15
20
25
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
MillionsofUPDATEsperSecond
MySQL Cluster Data Nodes

20. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Memory
opKmized
tables
– Durable
– Mix
with
disk-­‐based
tables
• Massively
concurrent
OLTP
• Distributed
Joins
for
analyKcs
• Parallel
table
scans
for
non-­‐indexed
searches
• MySQL
Cluster
7.4
FlexAsych
– 200M
NoSQL
Reads/Second
3/15/15
30
MySQL
Cluster
7.4
NoSQL
Performance
200
Million
NoSQL
Reads/Second
Copyright
2015,
oracle
and/or
its
affiliates.
All
rights
reserved
!"!!!!
!50,000,000!!
!100,000,000!!
!150,000,000!!
!200,000,000!!
!250,000,000!!
2! 4! 6! 8! 10! 12! 14! 16! 18! 20! 22! 24! 26! 28! 30! 32!
Reads&per&second&
Data&Nodes&
FlexAsync&Reads&

21. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Memory
opKmized
tables
– Durable
– Mix
with
disk-­‐based
tables
• Massively
concurrent
OLTP
• Distributed
Joins
for
analyKcs
• Parallel
table
scans
for
non-­‐indexed
searches
• MySQL
Cluster
7.4
DBT2
BM
– 2.5M
SQL
Statements/Second
3/15/15
31
MySQL
Cluster
7.4
SQL
Performance
2.5M
SQL
Statements/Second
Copyright
2015,
oracle
and/or
its
affiliates.
All
rights
reserved
!"!!!!
!500,000!!
!1,000,000!!
!1,500,000!!
!2,000,000!!
!2,500,000!!
!3,000,000!!
2! 4! 6! 8! 10! 12! 14! 16!
SQL$Statements/sec$
Data$Nodes$
DBT2$SQL$Statements$per$Second$

22. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

23. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Scalability
!"
Performance
!"
HA
!"
Ease
of
use
!"
SQL/Joins
!"
ACID
TransacBons
!"

24. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

25. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
Auto-­‐Installer
• Fast
configuraKon
• Auto-­‐discovery
• Workload
opKmized
• Repeatable
best
pracKces
Specify
Workload
Auto-­‐
Discover
Define
Topology
Deploy

26. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

27. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Scalability
!"
Performance
!"
HA
!"
Ease
of
use
!"
SQL/Joins
!"
ACID
TransacBons
!"

28. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

29. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

30. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
NoSQL
Access
to
MySQL
Cluster
data
Apps
Apps
Apps
Apps
Apps
Apps
Apps
Apps
Apps
Apps
Apps
Apps
JPA
Cluster
JPA
PHP
Perl
Python
Ruby
JDBC
Cluster
J
JS
Apache
Memcached
MySQL
JNI
Node.JS
mod_ndb
ndb_eng
NDB
API
(C++)
MySQL
Cluster
Data
Nodes

31. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
7.1:
ClusterJ/JPA
• Domain
Object
Model
Persistence
API
(ClusterJ):
• Java
API
• High
performance,
low
latency
• Feature
rich
• JPA
interface
built
upon
this
new
Java
layer:
• Java
Persistence
API
compliant
• Implemented
as
an
OpenJPA
plugin
• Uses
ClusterJ
where
possible,
reverts
to
JDBC
for
some
operaKons
• Higher
performance
than
JDBC
• More
natural
for
most
Java
designers
• Easier
Cluster
adopKon
for
web
applicaKons
Apps
Apps
Apps
Apps
JPA
Cluster
JPA
JDBC
Cluster
J
MySQL
JNI
NDB
API
(C++)
MySQL
Cluster
Data
Nodes

32. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
7.2:
Memcached
• Memcached
is
a
distributed
memory
based
hash-­‐key/value
store
with
no
persistence
to
disk
• NoSQL,
simple
API,
popular
with
developers
• MySQL
Cluster
already
provides
scalable,
in-­‐
memory
performance
with
NoSQL
(hashed)
access
as
well
as
persistence
• Provide
the
Memcached
API
but
map
to
NDB
API
calls
• Writes-­‐in-­‐place,
so
no
need
to
invalidate
cache
• Simplifies
architecture
as
caching
&
database
integrated
into
1
Ker
• Access
data
from
exisKng
relaKonal
tables
Apps
Memcached
ndb_eng
NDB
API
(C++)
MySQL
Cluster
Data
Nodes

33. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Schema-­‐Free
apps
• Rapid
applicaKon
evoluKon
– New
types
of
data
constantly
added
– No
Kme
to
get
schema
extended
– Missing
skills
to
extend
schema
– IniKally
roll
out
to
just
a
few
users
– Constantly
adding
to
live
system

34. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Cluster
&
Memcached
–
Schema-­‐Free
<town:maidenhead,SL6>
key
value
<town:maidenhead,SL6>
key
value
Key
Value
town:maidenhead
SL6
generic
table
ApplicaKon
view
SQL
view

35. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Cluster
&
Memcached
-­‐
Configured
Schema
<town:maidenhead,SL6>
prefix
key
value
<town:maidenhead,SL6>
key
value
Prefix
Table
Key-­‐col
Val-­‐col
policy
town:
mapzip
town
code
cluster
Config
tables
town
...
code
...
maidenhead
...
SL6
...
mapzip
ApplicaKon
view
SQL
view

36. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Node.js
NoSQL
API
• NaKve
JavaScript
access
to
MySQL
Cluster
– End-­‐to-­‐End
JavaScript:
browser
to
app
&
DB
– Storing
and
retrieving
JavaScript
objects
directly
in
MySQL
Cluster
– Eliminate
SQL
transformaKon
• Implemented
as
a
module
for
node.js
– Integrates
Cluster
API
library
within
web
app
• Couple
high
performance,
distributed
apps,
with
high
performance
distributed
database
• OpKonally
routes
through
MySQL
Server
V8 JavaScript Engine
MySQL Cluster Node.js Module
MySQL Cluster Data Nodes
Clients

37. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
What
does
an
asynchronous
API
mean?
do_stuff1(data,
callback1)
callback1(results) do_stuff2(data,
callback2)
callback2(results)
Get
on
with
other
stuff
Get
on
with
other
stuff

38. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
7.3:
JavaScript
Connector
for
node
var nosql = require('mysql-js');
var annotations = new
nosql.TableMapping('towns').applyT
oClass(Town);
var dbProperties =
nosql.ConnectionProperties('ndb');
nosql.openSession(dbProperties,
annotations, onSession);
Modular connector with
various back-end
adapters:
• ndb: low-level native
access to MySQL Cluster
• mysql: access to any
MySQL server (translates
operations to SQL
statements)
Tutorial https://blogs.oracle.com/MySQL/entry/mysql_cluster_tutorial_nosql_api

39. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
var onSession = function(err, session) {
if (err) {console.log(err);
process.exit(0);} else {
var data = new Town('Maidenhead',
'Berkshire');
session.persist(data, onInsert, data,
session);
}
};
MySQL
Cluster
NoSQL
API
for
Node.js

40. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
NoSQL
API
for
Node.js
var onInsert = function(err, object, session) {
if (err) {console.log(err);} else {
console.log('Inserted: ' +
JSON.stringify(object));
session.find(Town, 'Maidenhead',
onFind);
}
};

41. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Cluster
NoSQL
API
for
Node.js
var onFind = function(err, result) {
if (err) {console.log(err);} else {
console.log('Found: ' +
JSON.stringify(result));
}
process.exit(0);
};

42. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
NoSQL
API
for
Node.js
&
FKs
FKs
enforced
on
all
APIs:
{ message: 'Error',
sqlstate: '23000',
ndb_error: null,
cause:
{message: 'Foreign key constraint violated: No parent row found [255]',
sqlstate: '23000',
ndb_error:
{ message: 'Foreign key constraint violated: No parent row found',
code: 255,
classification: 'ConstraintViolation',
handler_error_code: 151,
status: 'PermanentError' },
cause: null } }

43. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
5.6
Memcached
with
InnoDB
0
10000
20000
30000
40000
50000
60000
70000
80000
8 32 128 512
TPS
Client Connections
Memcached API
SQL
Clients
and
ApplicaBons
MySQL
Server
Memcached
Plug-­‐in
innodb_
memcached
local
cache
(opKonal)
Handler
API
InnoDB
API
InnoDB
Storage
Engine
mysqld
process
SQL
Memcached
Protocol
Up
to
9x
Higher
“SET
/
INSERT”
Throughput

44. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
InnoDB
as
a
NoSQL
Database
• A
NoSQL
product
such
as
Memcached
directly
accesses
InnoDB
• Simple
commands,
much
smaller
network
transmit
packages
• Persistent
storage
from
InnoDB
• Index
on
the
key
column
• Full
ACID
compliance
• Bypass
OpKmizer
and
QP
layer
of
MySQL
• Dual
access
of
data
(SQL
and
Memcached)
• Enabling
binlog
with
innodb_api_enable_binlog
• ConfiguraKon
Parameters
– daemon_memcached_r_batch_size
– daemon_memcached_w_batch_size

45. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
InnoDB
memcached
Plugin
• Supported
Plaqorm
– Linux,
Solaris,
and
Mac
OS
X
• Sosware
Prerequisites
– You
must
have
libevent
installed,
since
it
is
required
by
memcached
• Setng
Up
Required
Tables
– run
the
configuraKon
script
scripts/innodb_memcached_config.sql to
install
the
necessary
tables
• Installing
the
Daemon
Plugin
mysql> install plugin daemon_memcached soname "libmemcached.so”;

46. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
InnoDB
memcached
Plugin
• ConfiguraKon
tables
created
by
install
script
mysql> use innodb_memcache;
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_innodb_memcache |
+---------------------------+
| cache_policies |
| config_options |
| containers |
+---------------------------+
3 rows in set (0.01 sec)

47. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
InnoDB
memcached
Plugin
• The
mapping
is
done
through
specifying
corresponding
column
values
in
containers
table:
mysql> desc containers;
+------------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+-------+
| name | varchar(50) | NO | PRI | NULL | |
| db_schema | varchar(250) | NO | | NULL | |
| db_table | varchar(250) | NO | | NULL | |
| key_columns | varchar(250) | NO | | NULL | |
| value_columns | varchar(250) | YES | | NULL | |
| flags | varchar(250) | NO | | 0 | |
| cas_column | varchar(250) | YES | | NULL | |
| expire_time_column | varchar(250) | YES | | NULL | |
| unique_idx_name_on_key | varchar(250) | NO | | NULL | |
+------------------------+--------------+------+-----+---------+-------+
9 rows in set (0.02 sec)

48. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
InnoDB
memcached
Plugin
• Mapping
to
table
can
be
added
via
SQL
• To
switch
to
table
”db1/tbl1",
and
get
the
value
corresponding
to
“key_a”
in
"get
@@new_mapping_name.key”
format;
prefix
"@@"
is
required
for
signaling
a
mapped
table
change
• The
default
delimiter
that
separates
the
"mapping
name"
and
key
value
is
".”,
which
can
be
changed
opKon
in
the
"config_opKons"
system
table
with
name
of
"table_map_delimiter"
INSERT INTO containers VALUES ("setup_3", ”db1", ”tbl1",
"c1", "c2", "c3", "c4", "c5", "idx");
get @@setup_3.key_a
(OR
simply, get @@setup_3)
INSERT INTO config_options VALUES("table_map_delimiter", “|");

49. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
HTTP
Plugin
for
MySQL
• Server
Plugin
adds
HTTP(S)
endpoints
to
MySQL
• Results
are
serialized
to
JSON
format
encoded
as
UTF8
• Provides
3
choices
of
User
Endpoint
Types
– SQL
– CRUD
-­‐
Key-­‐Value
– JSON
-­‐
Document
59
labs.mysql.com

50. Copyright
©
2014,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• json_valid()
• json_search()
• json_extract()
• json_replace()
• json_append()
• json_remove()
• json_set()
• json_merge()
• json_safe_merge()
• json_deep_merge()
• json_contains_key()
• json_depth()
• json_count()
• json_version()
• json_test_parser()
JSON
SQL
FuncKons
labs.mysql.com