• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ScimoreDB @ CommunityDays 2011
 

ScimoreDB @ CommunityDays 2011

on

  • 1,365 views

Scimore presentation @ CommunitDays 2011 Copenhagen....

Scimore presentation @ CommunitDays 2011 Copenhagen.

Quick introduction to motivation for ScimoreDB.

Explanation of cluster topology - sharding/grouping and replication.


Statistics

Views

Total Views
1,365
Views on SlideShare
1,361
Embed Views
4

Actions

Likes
0
Downloads
7
Comments
0

1 Embed 4

http://www.techgig.com 4

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    ScimoreDB @ CommunityDays 2011 ScimoreDB @ CommunityDays 2011 Presentation Transcript

    • YesSQL
    • Evolution of database – birth of NoSQL
      15 year ago :
      Availability requirements was different from today (ATM shutdown 2 AM, services maintenance windows).
      Small amount of data.
      Database loads was small.
      Today internet has changed the game:
      24x7 availability.
      Large data.
      Insane database loads
      Tomorrow
      Switching to hosted apps and thin clients. Even larger load
    • NoSQL – mosaic of options
      key‐value‐caches
      memcached, repcached, coherence, infinispan, eXtreme scale, jboss cache, velocity, terracoqa
      key‐value‐store
      keyspace, flare, schema‐free, RAMCloud
      eventually‐consistent key‐value‐store
      dynamo, voldemort, Dynomite, SubRecord, Mo8onDb, Dovetaildb
      ordered‐key‐value‐store
      tokyo tyrant, lightcloud, NMDB, luxio, memcachedb, actord
      data‐structures server
      redis
      tuple‐store
      gigaspaces, coord, apache river
      object database
      ZopeDB, db4o, Shoal
      document store
      CouchDB, Mongo, Jackrabbit, XML Databases, ThruDB, CloudKit, Perservere, Riak Basho, Scalaris
      wide columnar store
      BigTable, Hbase, Cassandra, Hypertable, KAI, OpenNeptune, Qbase, KDI
    • Best of two worlds
      SQL
      Transactions
      Consistency
      Ad-hoc query language
      Common language
      No-SQL
      Scales horizontally
      Super fast
      Always available
      Comodity hardware
    • History of ScimoreDB
      Driven by demand:
      1999 Jubii - memory based / COM interface
      2003 transaction/disc enabled
      2004 distributed and DQL
      2005 Scimore founded
      2007 sql
      2009 embedded
      2010 replication, merge/bi-directional
      2011 new distributied version for massive scale, fault tolerant.
    • ScimoreDB v.4
      Native SQL Database for Windows
      Distributed
      Elastic
      Fault tolerant
      Transactional / consistent
      Scale on commodity hardware
    • Going distributed is easy
      Used to select primary key and indexes pr. Table. Now you additionally need to select distribution pr. table.
      All existing sql queries continue to run.
      There is no magic – its just doing it how you would program your own sharding and map-reduce layer!
    • Partition Groups (shard)
      Group1
      Group2
      Group3
      Node #6
      Node #4
      Node #1
      Node #2
      Node #5
      Node #3
    • Distributed data over large amount of partition groups
      • Scale horizontally for writes
      • Scale for large data sets
      Group
      Group
      Group
      Group
      Group
      Group
      Group
      Group
      Group
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
    • Many nodes in each group – higher replication
      • Safety – decide how safe do you want to be?
      • Slower on insert/update – more machines needs to be updated
      • Fast reads – more machines with same data
      Group
      Group
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
      Node
    • Partitioning
      Distribute to all – replicated on all groups.
      Partition by column hash value.
      Round-robin.
      Relation.
    • Partitioning by column hash value
      Column [col1]>hash(100)MOD 1024
      Select * from table where col1 = 100
      0
      1024
      512
      Group2
      Group1
      Node #3
      Node #1
      Node #4
      Node #2
    • Demo on Amazon EC2
      Customer
      c_id bigint
      c_name varchar
      c_zip varchar
      Products
      p_id bigint
      p_Name varchar
      p_price money
      Orders
      o_id autobigint
      o_c_id bigint
      o_p_id bigint
      o_amount int
      o_date datetime
      o_price money
    • Performance
      Single machine 8 core Simple select:
      75.000 queries/s (10 client threads)
      Vodafone cluster of 6 machines:
      21.000 transactions inserting 1 row
      DTU cluster of 31 small machines TPC-C :
      140.000 transactions/s (35% insert, 35% update, 30% select)
    • ACID transactions
      Crash safe recovery
      Row & tabel level locking
      Dynamic phase commit (D2PC)
      Dynamic group commit
      Transactions isolation levels (read commit, read repeatable)
      In-Doubt transaction state
      Multiversioning Concurrency control MVCC
      Local and distributed deadlock detection
      Write ahead logging
      Fuzzy checkpoint - non blocking checkpoints
      B+-Tree
      Page compression
      TEXT/NTEXT field compression
      System tables: performance, monitoring, schema
      T-SQL
      Recursive queries/CTE
      Security – users & roles
      Free text (lucene)
      ScimoreOS, fiber based tasks scheduling
      100% asynchronious, io-completion based
      NUMA aware
      Distributed query optimizer
      Distributed tree execution
      Query prioritization and throttling
    • Questions