• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
NYC Meetup November 15, 2012
 

NYC Meetup November 15, 2012

on

  • 371 views

Wiqar Chaudry and Trek Palmer present NuoDB and how to overcome administrative acrobatics with your database.

Wiqar Chaudry and Trek Palmer present NuoDB and how to overcome administrative acrobatics with your database.

Statistics

Views

Total Views
371
Views on SlideShare
371
Embed Views
0

Actions

Likes
0
Downloads
18
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    NYC Meetup November 15, 2012 NYC Meetup November 15, 2012 Presentation Transcript

    • The Elastically Scalable Database™ 1  
    • 20th Century Database Powerful Query 9% 3% 4% Language 44% 19% Industry StandardsORACLEIBM Data GuaranteesMicrosoftSybase 21% ToolsTeradataOthers Employee Skills Existing Data 2  
    • 21st Century Problem Commodity Datacenters ✗ Big Data ✗ Powerful Query Language Modern Workloads ✗ Industry Standards 24x7 Operation ✗ Data Guarantees Geo-distribution ✗ ToolsDeveloperEmpowerment ✗ Employee Skills Existing Data 3  
    • Database Crisis Amazon Flickr WikipediaFacebook Google Source: Marc Bojoly 4  
    • Jim Starkey“Elastically Scalable Transactions represent the biggest breakthroughin database technology in 25 years” ‣  DEC RDB/ELN ‣  InterBase ‣  Firebird ‣  Falcon ‣  BLOBS ‣  MVCC 5  
    • Emergent Database Architecture"  “An emergent behavior can appear when a number of simple entities operate in an environment, forming more complex behaviors as a collective.” "   - Wikipedia 6  
    • NuoDB Plus One"  Second machinetypically doubles TPS"  Second machine isadded to live databasewhile it is running at1,000’s of TPS"  Performance increase isimmediate"  BTW - you can takeeither machine away and Second  Machine    the database keeps Instant  Performance  Increase  running without data loss 7  
    • Adding a Third Machine"  Third machinetypically triples singlemachine TPS"  Third machine isadded to live databasewhile it is running at1,000’s of TPS"  Performance increase Second  &  Third  Machine    is immediate Instant  Performance  Increase  "  BTW - you can takeany machine away andthe database keepsrunning without dataloss 8  
    • More Machines? Bring ‘em On Nodes TPSMySQL 1 3,000NuoDB 1 4,500 27,00NuoDB 9 0"  Technical Details: TPS  ‣  2-9 Tx engines‣  1 storage manager‣  Best sustained TPS and # clients Number  of  Nodes   combination‣  50% updates NuoDB running on 9 nodes was approx. 9x faster than MySQL running on 1 node. 9  
    • Or Scale-out on IAAS‣  Nuodb scales linearly on EC2‣  Per-node performance on m1.large nodes TPS   approx 50% of our commodity servers‣  Just started on optimizing‣  RDS runs on 1 node, and gets overloaded Number  of  EC2  Nodes   with 10+ connections 10  
    • Trek Palmer
    • Building An Elastically Scalable Database The Easy Way
    • Tonight’s Agenda"  A Bit About Me"   Introduction to NuoDB"   Architecture Overview"   Some Unnatural Acts "  Quick Demo"  Beer 1 3  
    • A Bit About Me"  A Refugee from Academia "  -Researched Programming Languages "  -Transactional Memory impl. and semantics"  Worked on distributed metadata database for HDS HCP "  -Clustered appliance 1 4  
    • What is NuoDB?"  Elastically scalable"  Multi-tenant"  Transactionally Consistent"  Easy to Manage 1 5  
    • 娜 graceful, like a cloud 1 6  
    • Architecture"  Three tiers "  Each is independent Management   "  Single model for all environments "  Extensible at TransacEon   various points Handling   Storage  
    • Agents"  Management tier"  Provision hosts for use"  Expose XML messaging for management"  Make scripting and automation easy 1 8  
    • Brokers"  Agent with additional special knowledge"  At least one per domain"  Redirects clients to TE "  -Clients need no knowledge of topology "  -Brokers are responsible for any load- balancing 1 9  
    • Transaction Engines"  Peer-to-peer"  In-memory"  Multi-Version Concurrency Control"  Asynchronous messaging (replication)"  Atoms 2 0  
    • Storage Managers"  Persistence points for atoms"  Key-value backing stores"  -Local FS, S3, HDFS"  Independent archives 2 1  
    • And Now, Some Unnatural Acts 2 2  
    • Sharding, an Unnatural Act"  The ideal DB application Scales  up  to  the  capacity     of  a  single  node   client   DB   What  if  you  need  more  read     client   and/or  write  throughput?   2 3  
    • Sharding"  Shard the DB among several nodes New  Client  Layer   Now  you  need  to  implement   consistency  in  your  applicaEon   Client1   Client2   TransacEonal  consistency  is   DB0   DB1   very  very  hard  to  get  right   2 4  
    • Other Sharding Bugbears"  Global operations (searches, scans)"  -Doing joins in the application"  -Implementing Cursors"  -Chunking and memory management"  And, of course, adding or removing shards 2 5  
    • Scaling Shards"  A recipe for changing the number of shards"  1) Ask boss for permission"  2) Provision hardware"  3) Rewrite the app over 6 months"  4) Hope / Pray 2 6  
    • NuoDB SolutionManagement  Client  Domain   Host A Host B Host C   Broker     Agent     Agent   2 7  
    • NuoDB on a single node Management   Client   Client   Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Database A Storage   Manager   2 8  
    • NuoDB Scaling outManagement   Client   Client   Client  Client  Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Txn  Engine   Database A Storage   Storage   Manager   Manager   2 9  
    • NuoDB ‘Adding a Shard’ Management   Client   Client   Client   Client   Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Txn  Engine   Txn  Engine   Database A Storage   Storage   Storage   Manager   Manager   Manager   3 0  
    • NuoDB ‘Sharding’"  Literally as simple as just adding nodes"  No client code had to be harmed in the making of this distributed database"  -Brokers hide topology changes"  -NuoDB is transactionally consistent 3 1  
    • Eventual Consistency"  Eventual consistency is latent inconsistency"  -Not transactionally consistent"  -Application porting is non-trivial"  -Performance/Correctness tradeoff icky 3 2  
    • NuoDB Consistency"  NuoDB is transactionally consistent"  All the time, everywhere"  When a transaction is committed, it’s guaranteed consistent"  Tradeoff is between Performance and Availability 3 3  
    • Multi-Tenancy"  Traditional databases monopolize a node"  NuoDB supports many databases in a single pool of machines (a domain)"  Each DB can be scaled as needed, independantly 3 4  
    • Multi-Tenancy ExampleManagement   JDBC  Client  Client  Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Storage   Manager   Database A 3 5  
    • Multi-Tenancy ExampleManagement   JDBC  Client  Client   JDBC  Client  Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Storage   Txn  Engine   Manager   Database A 3 6  
    • Multi-Tenancy ExampleManagement   JDBC  Client  Client   JDBC  Client  Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Storage   Txn  Engine   Manager   Database A Storage   Txn  Engine   Manager   Database 1 3 7  
    • Multi-Tenancy ExampleManagement   JDBC  Client   SQL  Client   ...  Client   JDBC  Client  Domain   Host A Host B Host C   Broker     Agent     Agent   Txn  Engine   Storage   Txn  Engine   Manager   Database A Storage   Txn  Engine   Manager   Database 1 3 8  
    • And now, a demo…
    • The Elastically Scalable Database™ 40