1   Copyright © 2011 Oracle Corp.
<Insert Picture Here>    MySQL Cluster with and without SQL    John David Duncan    Senior Software Engineer, Oracle Corp.2
Program Agenda                                  <Insert Picture Here>    • Overview of MySQL Cluster    • SQL interface   ...
The following is intended to outline our general    product direction. It is intended for information    purposes only, an...
MySQL Cluster Overview    ACID Compliant Relational Database     SQL & NoSQL interfaces    Write-Scalable & Real-Time     ...
The Basics                         API Nodes       Node Group 1                Node Group 2                      NDB Data ...
Geographic Replication                                                               Synchronous    Cluster 1             ...
MySQL Cluster – Users & Applications      HA, Transactional Services: Web & Telecoms    • Web       •   User profile manag...
SQL interface                                                     <Insert Picture Here>        Node Group 1               ...
The long road to SQL JOIN performance     • Initial integration (2004)       – Move the data to the query       – ... one ...
Results: Adaptive Query Localization         in MySQL Cluster 7.2     mysql> SELECT COUNT(*)            FROM residents, po...
NoSQL interfaces                                                  <Insert Picture Here>       mod_ndb                     ...
SQL and NoSQL Together         Schema creation         Reporting         Analytics                                     Nod...
mod_ndb                                   <Insert Picture Here>14             Copyright © 2011 Oracle Corp.
mod_ndb     http://mod-ndb.googlecode.com/ • First released March 2007 • Use REST and JSON to manage data   stored in MySQ...
mod_ndb     httpd.conf     <Location /ndb/app/car>       SELECT * from cars                                    Apache     ...
mod_ndb     Client   GET /ndb/app/car?id=371 HTTP/1.1                                                  Apache             ...
mod_ndb     Client           GET /ndb/app/car?id=371 HTTP/1.1HTTP/1.1 200 OKETag: 700847d8b5b0901a2f2451efc4c4Content-type...
The view from Javascript     Browser            reply = XMLHttpRequest.get();                                             ...
mod_ndb     http://mod-ndb.googlecode.com/•    Very small user community•    Not officially supported by a vendor•    Not ...
ClusterJ                                   <Insert Picture Here>21              Copyright © 2011 Oracle Corp.
ClusterJ     • High Performance, Light Weight, Easy to Use Direct       Connection        – In the style of Hibernate / JP...
Domain Object Model Mapping     • Tables map to Persistent Interfaces / Classes     • Columns map to Persistent Properties...
ClusterJ – Generated Class     @PersistenceCapable(table="employee")     public interface Employee {         long getId();...
ClusterJ – Dynamic Object     public class Employee         extends DynamicObject {         public String table() {       ...
Numeric Column Mapping     • Java boolean, Boolean                  • Java long, Long        – BIT(1)                     ...
Date Column Mapping     • Java util Date        – DATETIME        – TIMESTAMP        – TIME        – DATE     • Java sql D...
Variable Size Column Mapping     • Java String        – CHAR        – VARCHAR        – TEXT     • Java byte[ ]        – BI...
ClusterJ Features     • Character Set Translation (all MySQL charsets)     • Automatic detection of primary keys, indexes ...
ClusterJ Limitations     • No Relationships        – primitive types only     • No Multi-table inheritance        – single...
ClusterJ Interfaces     • SessionFactory        – Instance per connection to cluster     • Session        – Instance per "...
ClusterJ User View                                Domain                                Object   Domain                   ...
Example     Session session;     void getSession() {          session = sessionFactory.getSession();     }     Employee cr...
Example     Transaction transaction;     void getTransaction() {          transaction = session.getTransaction();     }   ...
Query     • Builder pattern     • Similar to JPA criteria query     • Compare column values to parameters:        – equal,...
Query ExampleQueryDomainType qemp =   builder.createQueryDefinition(Employee.class) ;Predicate geAge = qemp.get("age")  .g...
Performance37                 Copyright © 2011 Oracle Corp.
Memcache API                                   <Insert Picture Here>38                  Copyright © 2011 Oracle Corp.
Memcached Overview:        Two levels of hashing                                                                    memcac...
Cache hit       httpd                                                                                hash key     PHP/Perl...
Cache miss: fetch from DB       httpd                                                                            hash key ...
Expected Latency & Throughput       httpd                               c.                                  t ions/se     ...
Goals• Access stored data directly from memcache clients     – Memcached perspective:          • MySQL Cluster is a write-...
Goals• Support existing schemas and all MySQL data types• Cache MySQL Cluster data inside memcached when    desired     – ...
A Key Prefix                      user:1248         the prefix                                     the database key45     ...
Standard Tables in ndbmemcache     • meta            – stores configuration schema version (for upgrade                 co...
A key-prefix mapping     Memcache                                                            Cache        key            C...
A memcache server role                                key                               Con-    Cache                     ...
Measured Latency                  memcachetest -t 2 -M 7000 -c 2500049                Copyright © 2011 Oracle Corp.
Measured Throughput                         memslap50               Copyright © 2011 Oracle Corp.
Limitations of Memcache API     • The size of stored values is limited to the MySQL         Cluster row size (without BLOB...
Program Review                                                           <Insert Picture Here>     Overview of MySQL Clust...
53   Copyright © 2011 Oracle Corp.
Upcoming SlideShare
Loading in …5
×

Thu 1100 duncan_john_color

730 views
625 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
730
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Thu 1100 duncan_john_color

  1. 1. 1 Copyright © 2011 Oracle Corp.
  2. 2. <Insert Picture Here> MySQL Cluster with and without SQL John David Duncan Senior Software Engineer, Oracle Corp.2
  3. 3. Program Agenda <Insert Picture Here> • Overview of MySQL Cluster • SQL interface • NoSQL interfaces – mod_ndb – ClusterJ – Memcache3
  4. 4. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.4 Copyright © 2011 Oracle Corp.
  5. 5. MySQL Cluster Overview ACID Compliant Relational Database SQL & NoSQL interfaces Write-Scalable & Real-Time Distributed, auto-partitioning (sharding), multi-master 99.999% Availability Shared-nothing, integrated clustering & sub-second recovery, local & geographic replication, on-line operations Low TCO Open-source, management & monitoring tools, scale-out on commodity hardware5 Copyright © 2011 Oracle Corp.
  6. 6. The Basics API Nodes Node Group 1 Node Group 2 NDB Data Nodes6 Copyright © 2011 Oracle Corp.
  7. 7. Geographic Replication Synchronous Cluster 1 Cluster 2 replication Asynchronous replication MyISAM MyISAM InnoDB7 Copyright © 2011 Oracle Corp.
  8. 8. MySQL Cluster – Users & Applications HA, Transactional Services: Web & Telecoms • Web • User profile management • Session stores • eCommerce • On-Line Gaming • Application Servers • Telecoms • Subscriber Databases (HLR/HSS) • Service Delivery Platforms • VoIP, IPTV & VoD • Mobile Content Delivery • On-Line app stores and portals • IP Management • Payment Gateways http://www.mysql.com/customers/cluster/8 Copyright © 2011 Oracle Corp.
  9. 9. SQL interface <Insert Picture Here> Node Group 1 Node Group 29 Copyright © 2011 Oracle Corp.
  10. 10. The long road to SQL JOIN performance • Initial integration (2004) – Move the data to the query – ... one row at a time – ... one loop iteration in a nested-loop join = 1 network trip • 2005 - 2009 – Improve efficiency ... – within the existing framework of the MySQL optimizer – e.g. Batch Key Access, push-down filters • MySQL Cluster 7.2 (2011) – Adaptive Query Localization – Algorithmic changes to optimizer – Push the majority of the query to the data nodes10
  11. 11. Results: Adaptive Query Localization in MySQL Cluster 7.2 mysql> SELECT COUNT(*) FROM residents, postcodes, towns WHERE residents.postcode=postcodes.postcode AND postcodes.town=towns.town AND towns.county="Berkshire"; +----------+ | COUNT(*) | +----------+ | 40001 | +----------+ Before: 48.68 sec • After: 2.02 sec11 Copyright © 2011 Oracle Corp.
  12. 12. NoSQL interfaces <Insert Picture Here> mod_ndb Memcache ClusterJ Node Group 1 Node Group 212 Copyright © 2011 Oracle Corp.
  13. 13. SQL and NoSQL Together Schema creation Reporting Analytics Node Group 1 Real-Time Operations Node Group 213 Copyright © 2011 Oracle Corp.
  14. 14. mod_ndb <Insert Picture Here>14 Copyright © 2011 Oracle Corp.
  15. 15. mod_ndb http://mod-ndb.googlecode.com/ • First released March 2007 • Use REST and JSON to manage data stored in MySQL Cluster Apache • Open source project based on C++ NDBAPI NDB API Data Data Node Node15 Copyright © 2011 Oracle Corp.
  16. 16. mod_ndb httpd.conf <Location /ndb/app/car> SELECT * from cars Apache WHERE PRIMARY KEY = $id; Format JSON NDB API </Location> Data Data Node Node16 Copyright © 2011 Oracle Corp.
  17. 17. mod_ndb Client GET /ndb/app/car?id=371 HTTP/1.1 Apache NDB API Data Data Node Node17 Copyright © 2011 Oracle Corp.
  18. 18. mod_ndb Client GET /ndb/app/car?id=371 HTTP/1.1HTTP/1.1 200 OKETag: 700847d8b5b0901a2f2451efc4c4Content-type: application/json ApacheContent-length: 90 NDB API{ "car_id" : 371, "tag" : "807AHC", "state" : "OR", Data Data "make" : "Dodge", Node Node "year" : 1984}18 Copyright © 2011 Oracle Corp.
  19. 19. The view from Javascript Browser reply = XMLHttpRequest.get(); Apache NDB API my_car = JSON.parse(reply); Data Data Node Node19 Copyright © 2011 Oracle Corp.
  20. 20. mod_ndb http://mod-ndb.googlecode.com/• Very small user community• Not officially supported by a vendor• Not as widely functional as Cluster/J• Not as fast as NDB+Memcache• Moves an unusually large portion of application logic into httpd.conf• Could use a major upgrade (maybe later this year?)20
  21. 21. ClusterJ <Insert Picture Here>21 Copyright © 2011 Oracle Corp.
  22. 22. ClusterJ • High Performance, Light Weight, Easy to Use Direct Connection – In the style of Hibernate / JPA / JDO • Shared Data storage with: – MySQL server – Native C++ applications – Other ClusterJ applications • Domain Object Model DataMapper pattern – Data is represented as domain objects – Domain objects are separate from business logic – Domain objects are mapped to database tables • Does not support relationships – Look at JDO / JPA for these modeling patterns22
  23. 23. Domain Object Model Mapping • Tables map to Persistent Interfaces / Classes • Columns map to Persistent Properties – column names default to property name • Rows map to Persistent Instances • Annotations on Interfaces / Classes customize mappings • User chooses to write: – User interface (ClusterJ then generates implementation class) – Persistent class (ClusterJ provides base implementation class)23 Copyright © 2011 Oracle Corp.
  24. 24. ClusterJ – Generated Class @PersistenceCapable(table="employee") public interface Employee { long getId(); void setId(int id); @Column(name="full_name") String getName(); void setName(String value); int getSalary(); void setSalary(int value); Integer getAge(); void setAge(Integer value); }24 Copyright © 2011 Oracle Corp.
  25. 25. ClusterJ – Dynamic Object public class Employee extends DynamicObject { public String table() { return "employee"; long getId() { return (Long)get(0); } void setId(long value) { set(0, value); } ... // other fields and behavior }25 Copyright © 2011 Oracle Corp.
  26. 26. Numeric Column Mapping • Java boolean, Boolean • Java long, Long – BIT(1) – BIT(1) to BIT(64) • Java byte, Byte – BIGINT – BIT(1) to BIT(8) – BIGUNSIGNED – TINYINT • Java float, Float • Java short, Short – FLOAT – BIT(1) to BIT(16) • Java double, Double – SMALLINT – DOUBLE – YEAR • Java BigDecimal • Java int, Integer – NUMERIC – BIT(1) to BIT(32) – DECIMAL – INT • Java BigInteger – NUMERIC – DECIMAL26 Copyright © 2011 Oracle Corp.
  27. 27. Date Column Mapping • Java util Date – DATETIME – TIMESTAMP – TIME – DATE • Java sql Date – DATE • Java sql Time – TIME • Java sql Timestamp – DATETIME – TIMESTAMP27 Copyright © 2011 Oracle Corp.
  28. 28. Variable Size Column Mapping • Java String – CHAR – VARCHAR – TEXT • Java byte[ ] – BINARY – VARBINARY – BLOB28 Copyright © 2011 Oracle Corp.
  29. 29. ClusterJ Features • Character Set Translation (all MySQL charsets) • Automatic detection of primary keys, indexes • Compound Primary Keys • Ordered (btree) indexes • Unique (hash) indexes • Automatic use of partition key • Multi-threaded applications29 Copyright © 2011 Oracle Corp.
  30. 30. ClusterJ Limitations • No Relationships – primitive types only • No Multi-table inheritance – single table per persistent interface or class • No joins in queries – column comparisons and boolean operators • No Table creation – user needs to create tables and indexes • No Lazy Loading – entire record is loaded at one time, including LOBs30 Copyright © 2011 Oracle Corp.
  31. 31. ClusterJ Interfaces • SessionFactory – Instance per connection to cluster • Session – Instance per "user" – persist(), remove(), update(), write() – Find by ID • Transaction – Instance per Session – begin(), commit(), rollback() • Query – Multiple instances per Session31 Copyright © 2011 Oracle Corp.
  32. 32. ClusterJ User View Domain Object Domain Domain Object Domain Object Domain Object Domain Object Domain Domain Object Object Domain Object Domain Domain Object Object Object Domain Object Session and Transaction Session and Session and Transaction Transaction SessionFactory Configuration Properties32 Copyright © 2011 Oracle Corp.
  33. 33. Example Session session; void getSession() { session = sessionFactory.getSession(); } Employee createEmployee(long id, String name, int salary, int age) { Employee employee = session.newInstance(Employee.class); employee.setId(id); employee.setName(name); employee.setSalary(salary); employee.setAge(age); session.persist(employee); return employee; }33 Copyright © 2011 Oracle Corp.
  34. 34. Example Transaction transaction; void getTransaction() { transaction = session.getTransaction(); } void createEmployees() { getTransaction(); transaction.begin(); createEmployee(1, "Amos", 10000, 44); createEmployee(2, "Barbara", 14000, 48); createEmployee(3, "Chuck", 78000, 61); createEmployee(4, "Dave", 3000, 22); transaction.commit(); }34 Copyright © 2011 Oracle Corp.
  35. 35. Query • Builder pattern • Similar to JPA criteria query • Compare column values to parameters: – equal, lessEqual, greaterEqual, lessThan, greaterThan, in – comparison with null • Combine terms using boolean operators: – or, and, not • Execution is optimized to use indexes – primary or unique key lookup – ordered scan for complete or partial keys – table scan if no index can be used35
  36. 36. Query ExampleQueryDomainType qemp = builder.createQueryDefinition(Employee.class) ;Predicate geAge = qemp.get("age") .greaterEqual(qemp.param("ageFloor"));Predicate leSalary = qemp.get("salary") .lessEqual(qemp.param("salaryCap"));qemp.where(geAge.and(leSalary));Query query = session.createQuery(qemp) ;query.setParameter("ageFloor",33);query.setParameter("salaryCap", 44000);List<Employee> results = query.getResultList() ;36 Copyright © 2011 Oracle Corp.
  37. 37. Performance37 Copyright © 2011 Oracle Corp.
  38. 38. Memcache API <Insert Picture Here>38 Copyright © 2011 Oracle Corp.
  39. 39. Memcached Overview: Two levels of hashing memcached httpd hash key PHP/Perl ve r memcached ic k ser to find data ke y to p Memcache hash friends:12389 memcached memcache key39 Copyright © 2011 Oracle Corp.
  40. 40. Cache hit httpd hash key PHP/Perl ve r memcached ic k ser to find data ke y to p Memcache hash friends:12389 VALUE friends:12389 0 31rn 101, 11009, 11150, 55881, 77798 rn40 Copyright © 2011 Oracle Corp.
  41. 41. Cache miss: fetch from DB httpd hash key PHP/Perl ve r memcached memcache ic k ser to find data ke y to p hash Memcache F OUND NOT mysql SELECT friend_id FROM u MySQL ser_frien WHERE ds user_id = Slave ?41 Copyright © 2011 Oracle Corp.
  42. 42. Expected Latency & Throughput httpd c. t ions/se o f opera 10,000s nd trip u PHP/Perl ~2 00 µs ro memcached memcache mysql 1,000s of operatio ~ 2 ms ro ns/sec. MySQL und trip Slave42 Copyright © 2011 Oracle Corp.
  43. 43. Goals• Access stored data directly from memcache clients – Memcached perspective: • MySQL Cluster is a write-scalable, replicated data store – with reliable in-memory storage, – plus on-disk storage when data is too big for memory. – MySQL Cluster perspective: • memcache is a high performance API – providing easy access to in-memory data, – plus an extra layer of caching when data is on disk.43 Copyright © 2011 Oracle Corp.
  44. 44. Goals• Support existing schemas and all MySQL data types• Cache MySQL Cluster data inside memcached when desired – with automatic cache management – and flexibility to fine-tune (or disable) the cache policies• Support the whole memcache protocol• Achieve superior performance – latency as expected from memcached – throughput as expected from memcached44 Copyright © 2011 Oracle Corp.
  45. 45. A Key Prefix user:1248 the prefix the database key45 Copyright © 2011 Oracle Corp.
  46. 46. Standard Tables in ndbmemcache • meta – stores configuration schema version (for upgrade compatibility); consider it to be read-only • ndb_clusters • containers – existing tables where data is stored • cache_policies – rules describing how it can be accessed • key_prefixes • memcache_server_roles • last_memcached_signon • demo_table46 Copyright © 2011 Oracle Corp.
  47. 47. A key-prefix mapping Memcache Cache key Cluster Container Policy prefix47 Copyright © 2011 Oracle Corp.
  48. 48. A memcache server role key Con- Cache Cluster prefix tainer Policy Server Role key Con- Cache Cluster ID prefix tainer Policy key Con- Cache Cluster prefix tainer Policy key Con- Cache Cluster prefix tainer Policy48 Copyright © 2011 Oracle Corp.
  49. 49. Measured Latency memcachetest -t 2 -M 7000 -c 2500049 Copyright © 2011 Oracle Corp.
  50. 50. Measured Throughput memslap50 Copyright © 2011 Oracle Corp.
  51. 51. Limitations of Memcache API • The size of stored values is limited to the MySQL Cluster row size (without BLOBs) – This is about 13KB (up from 8KB in 7.1)51 Copyright © 2011 Oracle Corp.
  52. 52. Program Review <Insert Picture Here> Overview of MySQL Cluster Highly available Write-scalable SQL interface Redesigned in 7.2 NoSQL interfaces mod_ndb, ClusterJ, Memcache plus: C++ NDBAPI, LDAP, etc.52 Copyright © 2011 Oracle Corp.
  53. 53. 53 Copyright © 2011 Oracle Corp.

×