• Like
  • Save
Developing high-throughput services with no sql ap-is to innodb and mysql cluster
Upcoming SlideShare
Loading in...5
×
 

Developing high-throughput services with no sql ap-is to innodb and mysql cluster

on

  • 1,315 views

Ever-increasing performance demands of Web-based services have generated significant interest in providing NoSQL access methods to MySQL (MySQL Cluster from Oracle and the InnoDB storage engine of ...

Ever-increasing performance demands of Web-based services have generated significant interest in providing NoSQL access methods to MySQL (MySQL Cluster from Oracle and the InnoDB storage engine of MySQL), enabling users to maintain all the advantages of their existing relational databases while providing blazing-fast performance for simple queries. Get the best of both worlds: persistence; consistency; rich SQL queries; high availability; scalability; and simple, flexible APIs and schemas for agile development. This session describes the memcached connectors and examines some use cases for how MySQL and memcached fit together in application architectures. It does the same for the newest MySQL Cluster native connector, an easy-to-use, fully asynchronous connector for Node.js.

Statistics

Views

Total Views
1,315
Views on SlideShare
1,315
Embed Views
0

Actions

Likes
1
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Developing high-throughput services with no sql ap-is to innodb and mysql cluster Developing high-throughput services with no sql ap-is to innodb and mysql cluster Presentation Transcript

    • Developing High-ThroughputServices with NoSQL APIs toInnoDB and MySQL ClusterAndrew Morgan – MySQL Product ManagementJohn Duncan – MySQL Cluster Engineering1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Program Agenda  Why NoSQL  MySQL Cluster Attributes  NoSQL APIs for MySQL Cluster  Memcached API for InnoDB  Node.js – Native javascript access  Next Steps2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Disclaimer The f ll i is intended to tli Th following i i t d d t outline our general product di ti l d t direction. It i is intended for information purposes only, and may not be incorporated into i t any contract. It is not a commitment to deliver any material, t t i t it t t d li t i l code, or functionality, and should not be relied upon in making purchasing d i i h i decisions. Th d The development, release, and ti i of any l t l d timing f features or functionality described for Oracle’s products remains at the l discretion of O l th sole di ti f Oracle.3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • NoSQL – Why the demand? 2.1BN USERS Web W b applications li ti 8X DATA GROWTH IN 5 YRS 900M USERS demanding: 40% DATA 70+ NEW DOMAINS EVERY 60 SECONDS 20M APPS PER DAY  Development velocity: GROWTH PER YEAR 60 HOURS Simplicity & flexibility of UPLOADED EVERY MINUTE data model & APIs  Scalability & $1TR BY 2014 350m TWEETS performance: high write PER DAY $700BN IN 2011 throughput and Key/Value access  Support for “Big Data” 5.9BN MOBILE SUBS IN 2011 1 BILLION iOS & ANDROID 85% HANDSETS SHIPPED WITH A APPS DOWNLOADED PER WEEK BROWSER 4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • What NoSQL must deliver  Massive scalability Scalability  No application-level sharding o app cat o e e s a d g Performance  Performance  High Availability/Fault Tolerance HA  Ease of use Ease of use  Simple operations/administration  Simple APIs  Quickly evolve application & schema5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Still a role for the RDBMS? No best single solution fits all NoSQL RDBMS Mix and match Simple Si l access patterns tt Complex queries with j i C l i ith joins Scalability Compromise on consistency Performance ACID transactions for performance HA Ad-hoc data format Well defined schemas Ease o use ase of SQL/Joins Simple operation Rich set of tools ACID Transactions 6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster introduction Carrier Grade • Shared-nothing in-memory parallel database server Database • ACID compliant relational database • Five nines (99.999%) availability Highly Available • Self-healing, sub-second failover Real-time • High load, real-time performance Performance • Predictable low latency, bounded access times • Incrementally grow out with application demands Linearly Scalable • Linearly scale with distribution awareness y • Open Source, multiple data access Open Development • High performance APIs (C++, Java, SQL, LDAP) (C ,7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Who’s Using MySQL Cluster?8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability MySQL Cluster Architecture Performance HA Ease of use SQL/Joins SQL/J i  Clients ACID Transactions  Application Layer Data LayerManagement MySQL Cluster Data Nodes 9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability  MySQL Cluster Architecture Performance HA Ease of use SQL/Joins SQL/J i  Clients ACID Transactions  Application Layer Data LayerManagement Management MySQL Cluster Data Nodes 10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability  MySQL Cluster Architecture Performance HA  Ease of use SQL/Joins SQL/J i  Clients ACID Transactions  Application Layer Data LayerManagement Management MySQL Cluster Data Nodes 11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability  On-line Scheduled Maintenance Performance HA  Ease of use SQL/Joins SQL/J i  ACID Transactions  Scale Backup Evolve Upgrade pg12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability  Extreme performance Performance  HA  Ease of use SQL/Joins SQL/J i  ACID Transactions  writes13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Creating & running your first Cluster The traditional way (pre-MCM) – Up and running in 15 mins Download & Configure Start processes Extract •Cluster-wide •Cluster wide •Management Nodes •edelivery.oracle.com “config.ini” •Data Nodes •www.mysql.com •Per-mysqld “my.cnf” •MySQL Servers •dev.mysql.com  Up & running in 10-15 minutes using Quick Start guides from http://dev mysql com/downloads/cluster/ http://dev.mysql.com/downloads/cluster/ – Versions for Linux, Windows & Solaris14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability  Performance  HA  Ease of use  SQL/Joins  ACID Transactions 15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster Manager Bootstrap single h t Cl t B t t i l host Cluster 1. Download MCM/Cluster package from edelivery.oracle.com: 2. Unzip 3. Run agent, define, create & start Cluster! $ binmcmd bootst ap $> b c d –bootstrap MySQL Cluster Manager 1.1.2 started Connect to MySQL Cluster Manager by running "D:AndrewDocumentsMySQLmcmbinmcm" -a NOVA:1862 Configuring default cluster mycluster... Starting default cluster mycluster... Cluster mycluster started successfully ndb_mgmd NOVA 1186 db d NOVA:1186 ndbd NOVA ndbd NOVA mysqld NOVA:3306 mysqld NOVA:3307 ndbapi * Connect to the database by running "D:AndrewDocumentsMySQLmcmclusterbinmysql" -h NOVA -P 3306 -u root16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster 7.3 labs: Auto-Installer • Fast configuration Specify p y Auto- Workload Discover • Auto-discovery • Workload ti i d W kl d optimized • Repeatable best practices ti • For MySQL Cluster Confirm 7.2 7 3 7 2 + 7.3 Deploy Topology17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster 7.3 labs: Auto-Installer • Fast configuration • Auto-discovery • Workload ti i d W kl d optimized • Repeatable best practices ti • For MySQL Cluster 7.2 7 3 7 2 + 7.318 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Scalability  Performance  HA  Ease of use  SQL/Joins  ACID Transactions 19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • NoSQL Access to MySQL Cluster data Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps Apps JPA ClusterJPA PHP PERL Python Ruby JDBC ClusterJ JSON Apache Memcached Node.js mod-ndb ndb-eng MySQL JNI NDB API (C++) MySQL Cluster Data Nodes20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster 7.1: ClusterJ/JPA • Domain Object Model Persistence API (ClusterJ): • Java API • High performance, low latency • Feature rich • JPA interface built upon this new Java layer: • Java Persistence API compliant • Implemented as an OpenJPA plugin • Uses ClusterJ where possible, reverts to JDBC for some operations • Higher performance than JDBC • More natural for most Java designers • Easier Cluster adoption for web applications Cl ster eb21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster 7.2: Memcached • Memcached is a distributed memory based hash- key/value t k / l store with no persistence to disk ith i t t di k • NoSQL, simple API, popular with developers • MySQL Cluster already provides scalable, in- y yp memory performance with NoSQL (hashed) access as well as persistence • Provide the Memcached API but map to NDB API calls • Writes-in-place, so no need to invalidate cache • Simplifies Si lifi architecture as caching & d t b hit t hi database integrated into 1 tier • Access data from existing relational tables22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Traditional Memcached Architecture Two l T levels of hashing l f h hi httpd memcached hash key PHP/Perl memcached to find data Memcache friends:12389 memcached memcache key23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Memcache Protocol Elements• Key Up to 250 characters• Expire Time A number in seconds, up to 30 days seconds• Flags 32-bit number for application use• CAS ID 64-bit incrementing version number on value 64 bit i ti i b l• Value Value stored for key; traditionally up to 1 MB.24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Memcache Protocol Commands (1)• get k t key Fetch value for this key al e ke• set key value Store this data• add key value Store this data (strict insert)• replace key value Store this data (strict update)• cas key value cas-id Update with version number check 25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Memcache Protocol Commands (2)• append key value Append text to current value• prepend key value Insert text before current value• incr key amount Add to current stored integer value• decr key amount Subtract from current stored value• delete key Delete a key and value• flush_all Delete all stored values on the server• stats stat-name Fetch server statistics26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • NDB & Memcache Architecture: Memcache protocol + NDB storage M h t l t MySQL y QApplication Cluster memcached Data Node Memcache NDB Engine MySQL Client Cluster Cl Data Node27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Goals • Access stored data directly from memcache client • Memcached perspective: • MySQL Cluster is a write-scalable, replicated data store • with reliable in-memory storage, • plus on-disk storage when data is too big for memory. • MySQL Cluster perspective: • memcache is a high performance API • providing easy access to in-memory data, • plus an extra layer of caching when data is on disk.28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Memcached/MySQL Cluster latency memcachetest -t 2 -M 7000 -c 2500029 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Cluster & Memcached – Schema-Free key value <town:maidenhead,SL6> Application i A li ti view SQL view key value <town:maidenhead,SL6> Key Value town:maidenhead SL6 generic table30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Cluster & Memcached – Configured Schema key value <town:maidenhead,SL6> Application i A li ti view SQL view prefix key value <town:maidenhead,SL6> Prefix Table Key-col Val-col policy town ... code ... town: map.zip town code cluster maidenhead ... SL6 ... Config tables C fi t bl map.zip i31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Memcached NoSQL Access with InnoDB• Memcached as a plugin of Application MySQL Server; same process space, with very low latency access to data SQL Memcached protocol• Memcapable: supports both mysqld memcached ascii protocol MySQL Server and binary protocol memcached plugin• Support multiple columns: local cache users can map multiple innodb_memcache (optional) columns into “value” Handler API InnoDB API• Optional local caching: “innodb-only”, “cache-only”, InnoDB Storage Engine and “caching”• Batch operations for performance 32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster 7.3 EA: Node.js NoSQL API  Native JavaScript access to MySQL Cluster S SQ C – End-to-End JavaScript: browser to the app and Clients database – Storing and retrieving JavaScript objects directly in MySQL Cluster y – Eliminate SQL transformation V8 JavaScript Engine  Implemented as a module for node.js p j MySQL Cluster Node.js Module – Integrates full Cluster API library within the web app  Couple high performance, distributed apps, with high performance distributed database MySQL Cluster Data Nodes33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster NoSQL API for Node.js Application Code A li ti C d // Constructor function Tweet(user, message) { this.id = UUID.generate(); this.timestamp = Date.now(); p this.user = user; this.message = message; message }34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster NoSQL API for Node.js • Modular connector with// Server Startup various back-end adapters: • ndb: low-level native access to MySQL Clustervar nosql = require(mysql-js); • mysql: access to any MySQL server (translates y (var sessionFactory = operations to SQL nosql.connectSync(ndb); statements)nosql.mapClass(Tweet, tweets); 35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster NoSQL API for Node.js // Create a tweet function newTweet(err, dbSession, httpReq) { var tweet = new Tweet( httpReq.user, httpReq.message); dbSession.persist(tweet); }36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster NoSQL API for Node.js function onNewTweetRequest(err, httpReq) { sessionFactory.openSession(null, newTweet, httpReq); }37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • MySQL Cluster NoSQL API for Node.js Sync s Async Execution S nc vs. As nc E ec tion  If you had all synchronous operations y y p – Single execution context blocked for I/O. – Low performance p  If you had all async operations – Complex code – Each individual operation nested within its own callback function.  The right balance – Define a database operation as an immediate call (no I/O) – Execute database operations in async calls with callbacks38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Next Steps Learn More • www.mysql.com/cluster • A th ti M SQL C i l Authentic MySQL Curriculum: http://oracle.com/education/mysql Try O t T it Out • dev.mysql.com/cluster • labs.mysql.com y Let us know what you think • clusterdb.com • @clusterdb • forums mysql com/list php?25 forums.mysql.com/list.php?2539 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012
    • Graphic Section Divider40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. September 30th 2012