NoSQL and MySQL webinar - best of both worlds

NoSQL Access to MySQL: The Best of Both Worlds Andrew Morgan – andrew.morgan@oracle.com – www.clusterdb.com MySQL Product Management1 29th March 2012 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Session Agenda •  NoSQL – What are people looking for? •  RDBMS – What advantages do they still have? •  How MySQL Delivers the Best of Both Worlds –  MySQL Cluster •  NoSQL attributes: Scale-out, performance, ease-of-use, schema flexibility, on-line operations •  NoSQL APIs –  Key-Value store access to InnoDB (Memcached)3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

NoSQL – Why is it needed? 2.1BN USERS 8X DATA GROWTH IN 5 YRS 850M USERS •  Web applications 70+ NEW DOMAINS EVERY 60 SECONDS demanding: 20M APPS PER DAY 40% DATA GROWTH PER YEAR 1 TR VIDEO PLAYBACKS –  Development velocity: $1TR BY 2014 Simplicity & flexibility of data250m TWEETS PER DAY $700BN IN 2011 model & APIs –  Scalability & performance: 5.9BN MOBILE SUBS IN 2010 (78% PENETRATION) high write throughput and 1 BILLION Ios & ANDROIDAPPS DOWNLOADED PER WEEK 370K CALL MINUTES EVERY 60 SECONDS Key/Value access –  Support for “Big Data” 4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

What NoSQL must deliver•  Massive scalability –  No application-level sharding•  Performance Scalability •  High Availability/Fault Tolerance Performance •  Ease of use HA –  Simple operations/administration Ease of use –  Simple APIs –  Quickly evolve application & schema 5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Types of NoSQL stores Key-‐Value Document Graph • Cassandra • MongoDB • Neo4J • Memcached • CouchDB • FlockDB • BigTable • Hadoop • Voldermort 6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Still a role for the RDBMS? •  No best single solution fits all •  Mix and matchNoSQL RDBMS Simple access paGerns Complex queries with joins Scalability Performance Compromise on consistency ACID transacJons HA for performance Ease of use Ad-‐hoc data format Well deﬁned schemas SQL/Joins Simple operaJon Rich set of tools ACID Transac>ons 7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MySQL Cluster Architecture REST Scalability Application Nodes Performance HA Node Group 1 Node Group 2 Ease of use Node 1 F1 F2 Node 3Cluster Cluster SQL/Joins "Mgr F3 F4 Mgr ACID Transac>ons " Node 2 Node 4 F3 F4 F1 F2 Data Nodes 9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MySQL Cluster Architecture http://clusterdb.com/u/demo REST Scalability Application Nodes Performance HA " Node Group 1 Node Group 2 Ease of use Node 1 F1 F2 Node 3Cluster Cluster SQL/Joins "Mgr F3 F4 Mgr ACID Transac>ons " Node 2 Node 4 F3 F4 F1 F2 Data Nodes 10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Scale-Out: Auto-Partitioning Table T1 Data Node 1 F1 F3 P1 Node Group 1 Data Node 2 P2 F3 F1 P3 Data Node 3 F2 F4 P4 Node Group 2 Data Node 4 F4 F216 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Scale-Out: Auto-Partitioning Table T1 Data Node 1 F1 F3 P1 Node Group 1 Data Node 2 P2 F3 F1 P3 Data Node 3 F2 F4 P4 Node Group 2 Data Node 4 F4 F217 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Scale-Out Reads & Writes on Commodity Hardware SELECT Queries per Minute 1,200 1,000 800Millions 600 400 200 0 2 4 8 Number of Data Nodes •  8 x Commodity Intel Servers UPDATE Queries per Minute –  2 x 6-core processors 2.93GHz 150 –  x5670 processors (24 threads) 100 –  48GB RAM Millions 50 •  Infiniband networking 0 •  flexAsynch benchmark (NDB API) 4 8 Number of Data Nodes 19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JOIN Performance 70x More performance Query Execution Time Seconds100 •  33K rows over 11 tables 90 80 •  Must Analyze tables for best 70 results 60 50 mysql> ANALYZE TABLE <tab-name>; 40 30 Scalability " 20 10 Performance " 0 HA " MySQL Cluster 7.1 MySQL Cluster 7.2 Ease of use SQL/Joins " ACID Transac>ons " 20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Creating & running your first Cluster The traditional way (pre-MCM) – Up and running in 15 mins Download & Conﬁgure Start processes Extract •  Cluster-‐wide •  Management Nodes •  edelivery.oracle.com “conﬁg.ini” •  Data Nodes •  www.mysql.com •  Per-‐mysqld “my.cnf” •  MySQL Servers •  dev.mysql.com •  Up & running in 10-15 minutes using Quick Start guides from http://dev.mysql.com/downloads/cluster/ –  Versions for Linux, Windows & Solaris 21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MySQL Cluster Manager Bootstrap single host Cluster 1.  Download MCM/Cluster package from edelivery.oracle.com: 2.  Unzip 3.  Run agent, define, create & start Cluster! $> binmcmd –bootstrap MySQL Cluster Manager 1.1.2 started Connect to MySQL Cluster Manager by running "D:AndrewDocumentsMySQLmcmbinmcm" -a NOVA:1862 Configuring default cluster mycluster... Starting default cluster mycluster... Cluster mycluster started successfully ndb_mgmd NOVA:1186 ndbd NOVA ndbd NOVA mysqld NOVA:3306 mysqld NOVA:3307 ndbapi * Connect to the database by running "D:AndrewDocumentsMySQLmcmclusterbinmysql" -h NOVA -P 3306 -u root23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MCM: On-Line Add-Node Scale out with no loss of service mysql> add hosts --hosts=192.168.0.35,192.168.0.36 mysite; mysql> add package --basedir=/usr/local/mysql_7_1_9a –hosts=192.168.0.35,192.168.0.36 7.1; mysql> add process --processhosts=mysqld@192.168.0.33,mysqld@192.168.0.34,ndbd@192.168.0.35,ndbd@192.168.0.36 mycluster; mysql> start process --added mycluster; 25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

On-line Schema changes On-Line Operations •  Scale the cluster (add & remove nodes on-line) •  Repartition tables •  Upgrade / patch servers & OS •  Upgrade / patch MySQL Cluster •  Back-Up •  Evolve the schema on-line, in real-time26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MySQL Cluster 7.1: ClusterJ/JPA •  New Domain Object Model Persistence API (ClusterJ) : –  Java API –  High performance, low latency –  Feature rich •  JPA interface built upon this new Java layer: –  Java Persistence API compliant •  Implemented as an OpenJPA plugin –  Uses ClusterJ where possible, reverts to JDBC for some operations –  Higher performance than JDBC –  More natural for most Java designers –  Easier Cluster adoption for web applications29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

ClusterJ•  High Performance, Easy to Use•  In the style of Hibernate / JPA / JDO•  Domain Object Model DataMapper pattern –  Data is represented as domain objects –  Domain objects are separate from business logic –  Domain objects are mapped to database tables•  Built on ndbjtie –  JNI adapter –  integral part of MySQL Cluster –  Straight mapping of MySQL Cluster API (a.k.a NDB API) to Java•  Does not support relationships –  Look at JDO / JPA for these modelling patterns 30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

NoSQL with Memcached (MySQL Cluster 7.2) •  Memcached is a distributed memory based hash-key/ value store with no persistence to disk •  NoSQL, simple API, popular with developers •  MySQL Cluster already provides scalable, in-memory performance with NoSQL (hashed) access as well as persistence –  Provide the Memcached API but map to NDB API calls •  Writes-in-place, so no need to invalidate cache •  Simplifies architecture as caching & database integrated into 1 tier •  Access data from existing relational tables31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Traditional Memcached Architecture Two levels of hashing httpd memcached hash keyPHP/Perl ver memcached pick ser to find data key toMemcache hash friends:12389! memcached memcache key 32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Cache hit httpd hash key PHP/Perl se rver memcached to find data ke yto pickMemcache hash friends:12389! VALUE friends:12389 0 31rn! 101, 11009, 11150, 55881, 77798 rn! 33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Cache miss (1): fetch from DB httpd hash keyPHP/Perl ver memcached memcache pick ser to find data key toMemcache hash FOUN D NOT mysql SELECT fri FROM u end_id ! MySQL ser_frien WHERE ds ! user_id = Slave ?!34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Cache miss (2): manage cache httpd hash keyPHP/Perl memcached to find dataMemcache ! ! 31 rn 798 rn s:12389 5881, 77 riend , 11150, 5 set f 009 11 101,35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Data change (1): Write to DB httpdPHP/Perl mysql INSERT INTO user_friends ! (user_id, friend_id) ! VALUES ( 12389, 999101);! MySQL Master36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

NoSQL with Memcached•  Flexible: set maidenhead 0 0 3 SL6 –  Deployment options STORED –  Multiple Clusters –  Simultaneous SQL Access get maidenhead –  Can still cache in Memcached server VALUE maidenhead 0 3 –  Flat key-value store or map to multiple tables/ SL6 columns END 38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Schema-Free apps •  Rapid application evolution –  New types of data constantly added –  No time to get schema extended –  Missing skills to extend schema –  Initially roll out to just a few users –  Constantly adding to live system39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Cluster & Memcached – Schema-Free key value <town:maidenhead,SL6> Application view SQL view key value <town:maidenhead,SL6> Key Value town:maidenhead SL6 generic table40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Cluster & Memcached - Configured Schema key value <town:maidenhead,SL6> Application view SQL view prefix key value <town:maidenhead,SL6>Preﬁx Table Key-‐col Val-‐col policy town ... code ... town: map.zip town code cluster maidenhead ... SL6 ... Config tables map.zip41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Memcached NoSQL Access with InnoDB •  Memcached as a plugin of MySQL Server; same process space, with very low latency Application access to data •  Memcapable: supports both memcached SQL Memcached protocol ascii protocol and binary protocolmysqld •  Support multiple columns: users can map multiple columns into “value” memcached plugin MySQL Server •  Optional local caching: “innodb-only”, local cache “cache-only”, and “caching” innodb_memcache (optional) •  Batch operations for performance Handler API InnoDB API •  Available from labs.mysql.com. Is not GA. InnoDB Storage Engine 43 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Next Steps •  Guide to MySQL and NoSQL - Delivering the Best of Both Worlds –  http://mysql.com/why-mysql/white-papers/mysql-wp-guide-to- nosql.php •  Evaluate MySQL Cluster 7.2 –  http://www.mysql.com/downloads/cluster/ •  Bootstrap a Cluster –  https://edelivery.oracle.com/ •  Try Memcached API for InnoDB –  http://labs.mysql.com/46 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Summary Today’s web workloads demand more from databases Performance, scale-out, simples access patterns & APIs MySQL meets these needs while still delivering benefits of an ACID RDBMS47 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

MySQL Cluster 7.2 GA Enabling Next GeneraJon Web Services •  70x Higher Complex Query Performance: AdapJve Query LocalizaJon •  NaJve Memcached API •  MySQL 5.5 Server IntegraJon •  MulJ-‐Threaded Data Node Extensions •  VM Support Enhancing Cross Data Center Scalability •  MulJ-‐Site Clustering •  Simpliﬁed AcJve / AcJve ReplicaJon Ease of Use •  MySQL Cluster Manager 1.1.4 •  Consolidated Privileges 48 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

NoSQL and MySQL webinar - best of both worlds

by Mat Keep

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

NoSQL and MySQL webinar - best of both worlds Presentation Transcript