Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Cluster as Transactional NoSQL (KVS)

840 views

Published on

Presentation at FOSS ASIA 2015 in Singapore, on March 15th 2015. This presentation explains benefit of "SQL and NoSQL" hybrid data management solution MySQL Cluster. MySQL Cluster is designed as high available and high scalable database cluster. In the same time MySQL Cluster can work as ACID compliant transactional Key-Value Datastore aka KVS #FOSSASIA

Published in: Technology
  • Be the first to comment

  • Be the first to like this

MySQL Cluster as Transactional NoSQL (KVS)

  1. 1. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   NoSQL  and  SQL   The  Best  of  Both  Worlds         梶山隆輔  /  Ryusuke  KAJIYAMA   MySQL  Sales  ConsulKng  Senior  Manager   MySQL  Global  Business  Unit  
  2. 2. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Safe  Harbor  Statement   The  following  is  intended  to  outline  our  general  product  direcKon.  It  is  intended  for   informaKon  purposes  only,  and  may  not  be  incorporated  into  any  contract.  It  is  not  a   commitment  to  deliver  any  material,  code,  or  funcKonality,  and  should  not  be  relied  upon   in  making  purchasing  decisions.  The  development,  release,  and  Kming  of  any  features  or   funcKonality  described  for  Oracle’s  products  remains  at  the  sole  discreKon  of  Oracle.  
  3. 3. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   The  Data  Management  Landscape Source: 451Group
  4. 4. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Session  Agenda   •  NoSQL  –  What  are  people  looking  for?   •  RDBMS  –  What  advantages  do  they  sKll  have?   •  How  MySQL  Delivers  the  Best  of  Both  Worlds   – MySQL  Cluster   •  NoSQL  aZributes:  Scale-­‐out,  performance,  ease-­‐of-­‐use,  schema  flexibility,  on-­‐line  operaKons   •  NoSQL  APIs   – Key-­‐Value  store  access  to  InnoDB  (Memcached)  
  5. 5. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Key-­‐Value  Store  
  6. 6. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Document  Store  
  7. 7. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Graph  Database  
  8. 8. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   •  Massive  scalability   – No  applicaKon-­‐level  sharding   •  Performance   •  High  Availability/Fault  Tolerance   •  Ease  of  use   – Simple  operaKons/administraKon   – Simple  APIs   – Quickly  evolve  applicaKon  &  schema   Scalability   Performance   HA   Ease  of  use   What  NoSQL  must  deliver  
  9. 9. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  10. 10. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   NoSQL   Simple  access  paZerns   Compromise  on  consistency   for  performance   Ad-­‐hoc  data  format   Simple  operaKon   SQL   Complex  queries  with  joins   ACID  transacKons   Well  defined  schemas   Rich  set  of  tools   • No  best  single  soluKon  fits  all   • Mix  and  match   SKll  a  role  for  SQL  (RDBMS)?   Scalability   Performance   HA   Ease  of  use   SQL/Joins   ACID  TransacBons  
  11. 11. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  Overview   •  Auto-­‐Sharding,  MulK-­‐Master   •  ACID  Compliant,  OLTP  +  Real-­‐Time  AnalyKcs   HIGH  SCALE,  READS  +   WRITES   •  Shared  nothing,  no  Single  Point  of  Failure   •  Self  Healing  +  On-­‐Line  OperaKons   99.999%  AVAILABILITY   •  In-­‐Memory  OpKmizaKon  +  Disk-­‐Data   •  Predictable  Low-­‐Latency,  Bounded  Access  Time   REAL-­‐TIME   •  Key/Value  +  Complex,  RelaKonal  Queries   •  SQL  +  Memcached  +  JavaScript  +  Java  +  HTTP/REST  &  C++   SQL  +  NoSQL   •  Open  Source  +  Commercial  EdiKons   •  Commodity  hardware  +  Management,  Monitoring  Tools   LOW  TCO  
  12. 12. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Who’s  Using  MySQL  Cluster?  
  13. 13. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  Architecture     MySQL  Cluster  Data  Nodes   Clients   ApplicaKon  Layer   Data  Layer  
  14. 14. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  Scaling     MySQL  Cluster  Data  Nodes   Clients   ApplicaKon  Layer   Data  Layer  
  15. 15. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  16. 16. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  HA     MySQL  Cluster  Data  Nodes   Clients   ApplicaKon  Layer   Data  Layer  
  17. 17. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   On-­‐line  OperaKons   •  Scale  the  cluster  (add  &  remove  nodes  on-­‐line)   •  ReparKKon  tables   •  Upgrade  /  patch  servers  &  OS   •  Upgrade  /  patch  MySQL  Cluster   •  Back-­‐Up   •  Evolve  the  schema  on-­‐line,  in  real-­‐Kme  
  18. 18. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  19. 19. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   1.2  Billion  UPDATEs  per  Minute   •  NoSQL  C++  API,   flexaSynch  benchmark   •  30  x  Intel  E5-­‐2600  Intel   Servers,  2  socket,  64GB   •  ACID  TransacKons,  with   Synchronous  ReplicaKon   0 5 10 15 20 25 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 MillionsofUPDATEsperSecond MySQL Cluster Data Nodes
  20. 20. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   •  Memory  opKmized  tables   – Durable   – Mix  with  disk-­‐based  tables   •  Massively  concurrent  OLTP   •  Distributed  Joins  for  analyKcs   •  Parallel  table  scans  for  non-­‐indexed   searches   •  MySQL  Cluster  7.4  FlexAsych   – 200M  NoSQL  Reads/Second   3/15/15   30   MySQL  Cluster  7.4  NoSQL  Performance   200  Million  NoSQL  Reads/Second   Copyright  2015,  oracle  and/or  its  affiliates.  All  rights  reserved   !"!!!! !50,000,000!! !100,000,000!! !150,000,000!! !200,000,000!! !250,000,000!! 2! 4! 6! 8! 10! 12! 14! 16! 18! 20! 22! 24! 26! 28! 30! 32! Reads&per&second& Data&Nodes& FlexAsync&Reads&
  21. 21. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   •  Memory  opKmized  tables   – Durable   – Mix  with  disk-­‐based  tables   •  Massively  concurrent  OLTP   •  Distributed  Joins  for  analyKcs   •  Parallel  table  scans  for  non-­‐indexed   searches   •  MySQL  Cluster  7.4  DBT2  BM   – 2.5M  SQL  Statements/Second   3/15/15   31   MySQL  Cluster  7.4  SQL  Performance   2.5M  SQL  Statements/Second   Copyright  2015,  oracle  and/or  its  affiliates.  All  rights  reserved   !"!!!! !500,000!! !1,000,000!! !1,500,000!! !2,000,000!! !2,500,000!! !3,000,000!! 2! 4! 6! 8! 10! 12! 14! 16! SQL$Statements/sec$ Data$Nodes$ DBT2$SQL$Statements$per$Second$
  22. 22. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  23. 23. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Scalability   !" Performance   !" HA   !" Ease  of  use   !" SQL/Joins   !" ACID  TransacBons   !"
  24. 24. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  25. 25. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  Auto-­‐Installer   • Fast  configuraKon   • Auto-­‐discovery   • Workload  opKmized   • Repeatable  best  pracKces   Specify   Workload   Auto-­‐ Discover   Define   Topology  Deploy  
  26. 26. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  27. 27. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Scalability   !" Performance   !" HA   !" Ease  of  use   !" SQL/Joins   !" ACID  TransacBons   !"
  28. 28. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  29. 29. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |  
  30. 30. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   NoSQL  Access  to  MySQL  Cluster  data   Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps JPA Cluster  JPA PHP Perl Python Ruby JDBC Cluster  J JS Apache Memcached MySQL JNI Node.JS mod_ndb ndb_eng NDB  API  (C++) MySQL  Cluster  Data  Nodes  
  31. 31. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  7.1:  ClusterJ/JPA   •  Domain  Object  Model  Persistence  API  (ClusterJ):   •  Java  API   •  High  performance,  low  latency   •  Feature  rich   •  JPA  interface  built  upon  this  new  Java  layer:   •  Java  Persistence  API  compliant   •  Implemented  as  an  OpenJPA  plugin   •  Uses  ClusterJ  where  possible,  reverts  to  JDBC  for  some   operaKons   •  Higher  performance  than  JDBC   •  More  natural  for  most  Java  designers   •  Easier  Cluster  adopKon  for  web  applicaKons   Apps Apps Apps Apps JPA Cluster  JPA JDBC Cluster  J MySQL JNI NDB  API  (C++) MySQL  Cluster  Data  Nodes  
  32. 32. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  7.2:  Memcached   •  Memcached  is  a  distributed  memory  based     hash-­‐key/value  store  with  no  persistence  to  disk   •  NoSQL,  simple  API,  popular  with  developers   •  MySQL  Cluster  already  provides  scalable,  in-­‐ memory  performance  with  NoSQL  (hashed)   access  as  well  as  persistence   •  Provide  the  Memcached  API  but  map  to  NDB  API  calls   •  Writes-­‐in-­‐place,  so  no  need  to  invalidate  cache   •  Simplifies  architecture  as  caching  &  database   integrated  into  1  Ker   •  Access  data  from  exisKng  relaKonal  tables   Apps Memcached ndb_eng NDB  API  (C++) MySQL  Cluster  Data  Nodes  
  33. 33. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Schema-­‐Free  apps   •  Rapid  applicaKon  evoluKon   – New  types  of  data  constantly   added   – No  Kme  to  get  schema  extended   – Missing  skills  to  extend  schema   – IniKally  roll  out  to  just  a  few  users   – Constantly  adding  to  live  system  
  34. 34. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  &  Memcached  –  Schema-­‐Free   <town:maidenhead,SL6> key   value   <town:maidenhead,SL6> key   value   Key   Value   town:maidenhead   SL6   generic  table   ApplicaKon  view   SQL  view  
  35. 35. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Cluster  &  Memcached  -­‐  Configured  Schema   <town:maidenhead,SL6> prefix   key   value   <town:maidenhead,SL6> key   value   Prefix   Table   Key-­‐col   Val-­‐col   policy   town:   mapzip   town   code   cluster   Config  tables   town   ...   code   ...   maidenhead   ...   SL6   ...   mapzip   ApplicaKon  view   SQL  view  
  36. 36. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   Node.js  NoSQL  API   •  NaKve  JavaScript  access  to  MySQL  Cluster   – End-­‐to-­‐End  JavaScript:  browser  to  app  &  DB   – Storing  and  retrieving  JavaScript  objects   directly  in  MySQL  Cluster     – Eliminate  SQL  transformaKon   •  Implemented  as  a  module  for  node.js   – Integrates  Cluster  API  library  within  web  app   •  Couple  high  performance,  distributed  apps,   with  high  performance  distributed  database   •  OpKonally  routes  through  MySQL  Server   V8 JavaScript Engine MySQL Cluster Node.js Module MySQL Cluster Data Nodes Clients
  37. 37. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   What  does  an  asynchronous  API  mean?   do_stuff1(data, callback1) callback1(results) do_stuff2(data, callback2) callback2(results) Get  on  with  other  stuff   Get  on  with  other  stuff  
  38. 38. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  7.3:  JavaScript  Connector  for  node   var nosql = require('mysql-js'); var annotations = new nosql.TableMapping('towns').applyT oClass(Town); var dbProperties = nosql.ConnectionProperties('ndb'); nosql.openSession(dbProperties, annotations, onSession); Modular connector with various back-end adapters: •  ndb: low-level native access to MySQL Cluster •  mysql: access to any MySQL server (translates operations to SQL statements) Tutorial https://blogs.oracle.com/MySQL/entry/mysql_cluster_tutorial_nosql_api
  39. 39. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   var onSession = function(err, session) { if (err) {console.log(err); process.exit(0);} else { var data = new Town('Maidenhead', 'Berkshire'); session.persist(data, onInsert, data, session); } }; MySQL  Cluster  NoSQL  API  for  Node.js  
  40. 40. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  NoSQL  API  for  Node.js   var onInsert = function(err, object, session) { if (err) {console.log(err);} else { console.log('Inserted: ' + JSON.stringify(object)); session.find(Town, 'Maidenhead', onFind); } };
  41. 41. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  Cluster  NoSQL  API  for  Node.js   var onFind = function(err, result) { if (err) {console.log(err);} else { console.log('Found: ' + JSON.stringify(result)); } process.exit(0); };
  42. 42. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   NoSQL  API  for  Node.js  &  FKs   FKs  enforced  on  all  APIs:   { message: 'Error', sqlstate: '23000', ndb_error: null, cause: {message: 'Foreign key constraint violated: No parent row found [255]', sqlstate: '23000', ndb_error: { message: 'Foreign key constraint violated: No parent row found', code: 255, classification: 'ConstraintViolation', handler_error_code: 151, status: 'PermanentError' }, cause: null } }  
  43. 43. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   MySQL  5.6  Memcached  with  InnoDB   0 10000 20000 30000 40000 50000 60000 70000 80000 8 32 128 512 TPS Client Connections Memcached API SQL Clients  and  ApplicaBons   MySQL  Server   Memcached  Plug-­‐in   innodb_   memcached   local  cache   (opKonal)   Handler  API   InnoDB  API   InnoDB  Storage  Engine   mysqld  process   SQL   Memcached  Protocol   Up  to  9x  Higher  “SET  /  INSERT”  Throughput  
  44. 44. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   InnoDB  as  a  NoSQL  Database •  A  NoSQL  product  such  as  Memcached  directly  accesses  InnoDB   •  Simple  commands,  much  smaller  network  transmit  packages   •  Persistent  storage  from  InnoDB   •  Index  on  the  key  column     •  Full  ACID  compliance   •  Bypass  OpKmizer  and  QP  layer  of  MySQL   •  Dual  access  of  data  (SQL  and  Memcached)   •  Enabling  binlog  with  innodb_api_enable_binlog   •  ConfiguraKon  Parameters   – daemon_memcached_r_batch_size   – daemon_memcached_w_batch_size  
  45. 45. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   InnoDB  memcached  Plugin •  Supported  Plaqorm   – Linux,  Solaris,  and  Mac  OS  X   •  Sosware  Prerequisites   – You  must  have  libevent  installed,  since  it  is  required  by  memcached   •  Setng  Up  Required  Tables   – run  the  configuraKon  script  scripts/innodb_memcached_config.sql to   install  the  necessary  tables   •  Installing  the  Daemon  Plugin   mysql> install plugin daemon_memcached soname "libmemcached.so”;
  46. 46. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   InnoDB  memcached  Plugin •  ConfiguraKon  tables  created  by  install  script   mysql> use innodb_memcache; Database changed mysql> show tables; +---------------------------+ | Tables_in_innodb_memcache | +---------------------------+ | cache_policies | | config_options | | containers | +---------------------------+ 3 rows in set (0.01 sec)
  47. 47. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   InnoDB  memcached  Plugin •  The  mapping  is  done  through  specifying  corresponding  column  values  in   containers  table:   mysql> desc containers; +------------------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +------------------------+--------------+------+-----+---------+-------+ | name | varchar(50) | NO | PRI | NULL | | | db_schema | varchar(250) | NO | | NULL | | | db_table | varchar(250) | NO | | NULL | | | key_columns | varchar(250) | NO | | NULL | | | value_columns | varchar(250) | YES | | NULL | | | flags | varchar(250) | NO | | 0 | | | cas_column | varchar(250) | YES | | NULL | | | expire_time_column | varchar(250) | YES | | NULL | | | unique_idx_name_on_key | varchar(250) | NO | | NULL | | +------------------------+--------------+------+-----+---------+-------+ 9 rows in set (0.02 sec)
  48. 48. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   InnoDB  memcached  Plugin •  Mapping  to  table  can  be  added  via  SQL   •  To  switch  to  table  ”db1/tbl1",  and  get  the  value  corresponding  to  “key_a”  in   "get  @@new_mapping_name.key”  format;       prefix  "@@"  is  required  for  signaling  a  mapped  table  change   •  The  default  delimiter  that  separates  the  "mapping  name"  and  key  value  is    ".”,   which  can  be  changed  opKon  in  the  "config_opKons"  system  table  with  name   of  "table_map_delimiter"   INSERT INTO containers VALUES ("setup_3", ”db1", ”tbl1", "c1", "c2", "c3", "c4", "c5", "idx"); get @@setup_3.key_a (OR  simply, get @@setup_3) INSERT INTO config_options VALUES("table_map_delimiter", “|");
  49. 49. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   HTTP  Plugin  for  MySQL   •  Server  Plugin  adds  HTTP(S)  endpoints  to  MySQL   •  Results  are  serialized  to  JSON  format  encoded  as  UTF8   •  Provides  3  choices  of  User  Endpoint  Types   – SQL   – CRUD  -­‐  Key-­‐Value   – JSON  -­‐  Document   59   labs.mysql.com  
  50. 50. Copyright  ©  2014,  Oracle  and/or  its  affiliates.  All  rights  reserved.    |   •  json_valid()   •  json_search()   •  json_extract()   •  json_replace()   •  json_append()   •  json_remove()   •  json_set()   •  json_merge()   •  json_safe_merge()   •  json_deep_merge()   •  json_contains_key()   •  json_depth()   •  json_count()   •  json_version()   •  json_test_parser()   JSON  SQL  FuncKons labs.mysql.com  

×