NoSQL Data Stores
         Introduzione alle basi di dati non relazionali




DiTeDi, Udine - Italia 15-12-2012
Architettura Web ‘Classica’
Architettura Web ‘Classica’
4
5
6
Termine Usurpato!




                       Carlo Strozzi
         http://www.strozzi.it/users/carlo/vitae.html
Concetti totalmente nuovi?!?
Concetti totalmente nuovi?!?




                         DBM
Le origini di Tokyo Cabinet




                              DBM
Chi conosce questo signore?
E questo?
E questo?




                           Ken Thompson
            http://en.wikipedia.org/wiki/Ken_Thompson
Eppure...
PERCHÈ QUESTA HYPE?
MOLTI PIÙ DATI
Big Data




http://blog.thomsonreuters.com/index.php/big-data-graphic-of-the-day/
VARIABILITÀ DEI DATI
Progresso tecnologico


                                           Costo 1MB DRAM




   http://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/12_MassStorage.html
CONNESSIONI
MOLTI PIÙ ACCESSI
Dynamo   BigTable
Implementazione di Dynamo
Cenni su Riak
•   Database key-value
•   Interfaccia HTTP
•   Usando i link si possono collegare chiavi
    fra di loro
•   CAP «configurabile» (si può decidere fra
    coerenza e disponibilità per ciascuna
    richiesta)
•   Supporta ricerca full-text
Clustering




                  Fonte: http://wiki.basho.com/

28
Persistenza Configurabile


     N            numero nodi del
     R            cluster in cui
                  replicare il dato

     W

29
Replicazione




                    Fonte: http://wiki.basho.com/

30
Persistenza Configurabile


     N            numero nodi da cui
     R            il dato deve essere
                  letto affinchè sia

     W            considerato valido




31
Persistenza Configurabile


     N            numero nodi in cui il
                  dato deve essere
     R            scritto affinchè la
                  scrittura sia

     W            considerata valida




32
Coerenza eventuale




          W + R <= N


33
Quorum




          W+R>N


34
Recap su Riak:
     •   Interfaccia HTTP
     •   Adatto alle situazioni in cui c’è bisogno di
         HA
     •   Senza rinunciare ad alcune comode
         operazioni sui dati (usando links, pre/post
         commit hooks)
     •   Non adatto se c’è necessità di eseguire
         query complesse


35
Implementazione di BigTable




   http://hadoop-hbase.blogspot.it/2011/12/introduction-to-hbase.html
Salvataggio dei dati
Salvataggio dei dati




        Es. Data Location: Stefano Valle/Profilo:Paese
Cenni su HBase

•   Controllo di versione incorporato
•   Compressione
•   Garbage collection
•   Tabelle in memoria
•   Coerenza (no availability)
•   Write Ahead Logging
•   Basato su Hadoop
•   Pensato per cluster grandi (5-20 nodi)
SCALABILITÀ ORIZZONTALE
ACID         BASE
Atomic       Basic
Consistent   Available
Isolated     Soft State
Durable      Eventually Consistent
Concentriamoci sull’Atomicità
                              Aggiornamento ordine


         Ordine


                  Oggetto 1

                      Oggetto 2


                                  Cliente




43
DA ENTITÀ RELAZIONI
AD AGGREGATI
Aggregate data model
        order_id = 1001
        date = 2012-11-10
        total_amount = 10.00€

        name = Johnny
        surname = Appleseed

        product_name: Pear
        quantity: 2
        item_price: 2.50€
        total_price: 5.00€

        product_name: Mango
        quantity: 1
        item_price: 5.00€
        total_price: 5.00€
ACID con gli aggregati
                          Aggiornamento ordine


     Ordine

                                             order_id = 1001
                                             date = 2012-11-10
              Oggetto 1                      total_amount = 10.00€
                                                 name = Johnny
                  Oggetto 2                      surname = Appleseed

                                                 product_name: Pear
                                                 quantity: 2
                              Cliente            item_price: 2.50€
                                                 total_price: 5.00€


                                                 product_name: Mango
                                                 quantity: 1
                                                 item_price: 5.00€
                                                 total_price: 5.00€



47
COME SONO I DATI SU RDBMS?
COME LI VORREMMO UTILIZZARE?
Impedence Mismatch
Film Genere Drammatico:
Titolo                            Anno   Visioni
Schindler’s List                  1993    3454
Pulp Fiction                      1994    2114
Il Padrino                        1972    3873

Film Genere Western:
Titolo                            Anno   Visioni
Il buono, il brutto, il cattivo   1966    7834


Film Genere Commedia:
Titolo                            Anno   Visioni
Mamma ho perso l’aereo            1990     3555
Impedence Mismatch
Impedence Mismatch




      JOIN per evitare molteplici query
Dati in forma aggregata:
    {
        "genere":"Drammatico",
        "film":[
           {
              "titolo":"Schindler's List",
              "visioni":3454,
              "anno":1993
           },
           {
              "titolo":"Pulp Fiction",
              "visioni":2114,
              "anno":1994
           },
           {
              "titolo":"Il Padrino",
              "visioni":3873,
              "anno":1972
           }
        ]
    }
SCHEMALESS IS A LIE!
Da...
                  Pannello      Analisi
        Room TV
                  Controllo   Statistiche




                    Dati



55
A...
                      Pannello      Analisi
            Room TV
                      Controllo   Statistiche




                        API




                        Dati


56
O meglio...
                    Pannello       Analisi
        Room TV
                    Controllo    Statistiche




           API         API           API




         Metadati   Dati hotel   Statistiche


57
Domain
     Driven
     Design
58
Fonte: http://www.domainlanguage.com/ddd/patterns/

59
+ CQRS
     Fonte: http://www.domainlanguage.com/ddd/patterns/

60
POLYGLOT PERSISTANCE
Come scegliere?
TRADEOFFS
PERSISTENZA VS SPEED
DISPONIBILITÀ VS COERENZA
TCO, CONOSCENZE INTERNE
TEMPO, RISORSE, FATICA
SOLIDITÀ
Hosting




69
MODELLO DATI
Modello dati
Dimensione


        Key Value


                            Colonne

                                          Documentale

                                                                            A grafo




                                > 90% dei casi d’uso


                                                                                      Complessità
    Fonte: http://www.slideshare.net/emileifrem/an-overview-of-nosql-jfokus-2011
KEY/VALUE
Esempi




73
Confronto
                 replication   sharding          multi-master
                               support           support
     Memcached   No (use       Yes (through      No (use
                 repcached     client library)   repcached
                 instead)                        instaed)
     Project     Async         Yes               Yes
     Voldemort
     Redis       Async         No                No


     Riak        Async         Yes               Yes
                 Peer-based




74
Confronto (2)
                 data        concurrency data types   durability
                 storage     control

     Memcached   RAM         Lock-free   Blob         n.a.


     Project     RAM, BDB, MVCC          Typed        Data
     Voldemort   MySql,                               storage
                 others                               dependant
     Redis       RAM + disk Locks        Semi-typed   Append-
                                                      only log
     Riak        RAM, disk   MVCC        Semi-typed   Durable
                             (Vector-                 write
                             clocks)                  quorum



75
Confronto (3)
                 requires       transactions   secondary
                 compaction                    indexex
     Memcached   No             No             No


     Project     Data storage   No             No
     Voldemort   dependant
     Redis       Snapshot       Yes (multi     No
                                operations
                                queues)
     Riak        No             No             Yes




76
Confronto (4)
                 security   interface       query      language
                            protocol        support
     Memcached   Via SASL   TCP, UDP        Commands   C


     Project     None       Protocol        Commands   Java
     Voldemort              Buffers,
                            Thrift, Avro
                            and Java
                            Serialization
     Redis       Passwords Simple text      Commands   C/C++
                           over TCP
     Riak        None       HTTP,           Map reduce Erlang
                            protobuf        (weak
                                            support)

77
DOCUMENTALE
Esempi




79
Confronto
                replication      sharding         multi-master
                                 support          support
     CouchDB    Master-master    Yes (with filters Yes
                                 in BigCouch)
     MongoDB    Master-slave     Yes              No
                (replica sets)
     OrientDB   Master-master    Dec. 2012 /      Yes
                                 Jan. 2013
     RavenDB    Master-master    Yes              Yes




80
Confronto (2)
                data        concurrency    data types   durability
                storage     control

     CouchDB    Disk        Lock-free      JSON         Crash-only
                            MVCC

     MongoDB    Disk        Write lock     BSON         Write-ahead
                                                        log, safe
                                                        mode
     OrientDB   Disk        MVCC           JSON         Write-ahead
                                                        log

     RavenDB    RAM, Disk   Configurable   JSON         Write-ahead
                            optimistic                  log, snapshot
                            concurrency                 (via ESE)




81
Confronto (3)
                requires         transactions   secondary
                compaction                      indexex
     CouchDB    Yes, file rewrite No            Yes


     MongoDB    No               No             Yes


     OrientDB   No               Yes            No


     RavenDB    No               Yes            Yes




82
Confronto (4)
                security   interface     query       language
                           protocol      support
     CouchDB    Users      HTTP          Map reduce Erlang
                           (REST)        (weak
                                         support)
     MongoDB    Users      Binary over   Commands, C++
                           TCP           map reduce
     OrientDB   Users,     HTTP          SQL(+),     Java
                ACL        (REST),       Tinkerpop
                           Binary over
                           TCP
     RavenDB    Users      .NET API,     LINQ        C#
                           HTTP
                           (REST)

83
COLONNE
Esempi




85
GRAFO
Esempi




              Flock

87
COSA PORTARE A CASA
ONE SIZE DOES NOT FIT ALL
DATABASE RELAZIONALI
DATASTORE NOSQL
GRAZIE
DOMANDE?
http://www.mvassociati.it/
http://www.hubme.in/
Per Approfondire:




96
Per Approfondire:




97
Per Approfondire:




98
Per Approfondire:
     •   Links:
         –   http://nosql.findthebest.com/
         –   http://www.dzone.com/mz/nosql
         –   newsletter Nosql Weekly
         –   http://www.nosql-database.org/
         –   http://couchdb.apache.org/
         –   http://www.mongodb.org/
         –   http://redis.io/
         –   http://www.orientdb.org/
         –   http://www.draftq.com/ebooks/DraftQ/the-little-mongodb-book




99
Photo Credits
      •   http://www.flickr.com/photos/kunel/246544081/
      •   http://www.flickr.com/photos/lordcolus/5977615749/
      •   http://www.flickr.com/photos/gewitterhexer/5540504147/
      •   http://blog.thomsonreuters.com/index.php/big -data-graphic-of-the-day/
      •   http://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/12_MassStorage.html
      •   http://www.nathaninfocus.com/2011/01/social-networking-phenomenon/facebook-growth-chart/
      •   http://www.flickr.com/photos/phuonglovejesus2782010/5055970759/
      •   http://www.flickr.com/photos/kalexanderson/5944050340/
      •   http://www.flickr.com/photos/arichards-gallery/3578048793/
      •   http://www.flickr.com/photos/jabb/6956142046/
      •   http://www.flickr.com/photos/druclimb/47310529/
      •   http://www.flickr.com/photos/miggslives/5351504116/
      •   http://www.flickr.com/photos/polkadotcreations/2480587587/
      •   http://www.flickr.com/photos/wien/418840561/
      •   http://www.flickr.com/photos/heartlover1717/2605609803/
      •   http://www.flickr.com/photos/sepehrehsani/5766453552/
      •   http://www.flickr.com/photos/47108884@N07/6949078701/
      •   http://www.flickr.com/photos/ricephotos/354939144/
      •   http://www.flickr.com/photos/toptechwriter/3069396941/
      •   http://www.flickr.com/photos/djnordic/167433120/
      •   http://www.flickr.com/photos/jpstanley/69523927/
      •   http://www.flickr.com/photos/lodigs/2833648828/
      •   http://www.flickr.com/photos/ppym1/387781444/
      •   http://www.flickr.com/photos/freefoto/3844247553/
      •   http://www.flickr.com/photos/31492524@N00/3801200094/
      •   http://www.flickr.com/photos/sararah/2119107500/
      •   http://www.flickr.com/photos/wlef70/7507679760/
      •   http://www.flickr.com/photos/wold/369244351/
      •   http://www.flickr.com/photos/ilri/7839428936/
      •   http://www.flickr.com/photos/dhwright/8012651441/
      •   http://www.flickr.com/photos/birminghammag/7979485144/
      •   http://www.flickr.com/photos/capcase/4970062870/
      •   http://www.flickr.com/photos/aidanwojtas/5879866927/


100
Stefano Maraspin            Stefano Valle
@maraspin                   @stefanovalle
s.maraspin@mvassociati.it   s.valle@mvassociati.it

NoSQL Data Stores: Introduzione alle Basi di Dati Non Relazionali

  • 1.
    NoSQL Data Stores Introduzione alle basi di dati non relazionali DiTeDi, Udine - Italia 15-12-2012
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
    Termine Usurpato! Carlo Strozzi http://www.strozzi.it/users/carlo/vitae.html
  • 8.
  • 9.
  • 10.
    Le origini diTokyo Cabinet DBM
  • 11.
  • 12.
  • 13.
    E questo? Ken Thompson http://en.wikipedia.org/wiki/Ken_Thompson
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
    Progresso tecnologico Costo 1MB DRAM http://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/12_MassStorage.html
  • 20.
  • 22.
  • 25.
    Dynamo BigTable
  • 26.
  • 27.
    Cenni su Riak • Database key-value • Interfaccia HTTP • Usando i link si possono collegare chiavi fra di loro • CAP «configurabile» (si può decidere fra coerenza e disponibilità per ciascuna richiesta) • Supporta ricerca full-text
  • 28.
    Clustering Fonte: http://wiki.basho.com/ 28
  • 29.
    Persistenza Configurabile N numero nodi del R cluster in cui replicare il dato W 29
  • 30.
    Replicazione Fonte: http://wiki.basho.com/ 30
  • 31.
    Persistenza Configurabile N numero nodi da cui R il dato deve essere letto affinchè sia W considerato valido 31
  • 32.
    Persistenza Configurabile N numero nodi in cui il dato deve essere R scritto affinchè la scrittura sia W considerata valida 32
  • 33.
    Coerenza eventuale W + R <= N 33
  • 34.
    Quorum W+R>N 34
  • 35.
    Recap su Riak: • Interfaccia HTTP • Adatto alle situazioni in cui c’è bisogno di HA • Senza rinunciare ad alcune comode operazioni sui dati (usando links, pre/post commit hooks) • Non adatto se c’è necessità di eseguire query complesse 35
  • 36.
    Implementazione di BigTable http://hadoop-hbase.blogspot.it/2011/12/introduction-to-hbase.html
  • 37.
  • 38.
    Salvataggio dei dati Es. Data Location: Stefano Valle/Profilo:Paese
  • 39.
    Cenni su HBase • Controllo di versione incorporato • Compressione • Garbage collection • Tabelle in memoria • Coerenza (no availability) • Write Ahead Logging • Basato su Hadoop • Pensato per cluster grandi (5-20 nodi)
  • 40.
  • 42.
    ACID BASE Atomic Basic Consistent Available Isolated Soft State Durable Eventually Consistent
  • 43.
    Concentriamoci sull’Atomicità Aggiornamento ordine Ordine Oggetto 1 Oggetto 2 Cliente 43
  • 44.
  • 45.
  • 46.
    Aggregate data model order_id = 1001 date = 2012-11-10 total_amount = 10.00€ name = Johnny surname = Appleseed product_name: Pear quantity: 2 item_price: 2.50€ total_price: 5.00€ product_name: Mango quantity: 1 item_price: 5.00€ total_price: 5.00€
  • 47.
    ACID con gliaggregati Aggiornamento ordine Ordine order_id = 1001 date = 2012-11-10 Oggetto 1 total_amount = 10.00€ name = Johnny Oggetto 2 surname = Appleseed product_name: Pear quantity: 2 Cliente item_price: 2.50€ total_price: 5.00€ product_name: Mango quantity: 1 item_price: 5.00€ total_price: 5.00€ 47
  • 48.
    COME SONO IDATI SU RDBMS?
  • 49.
    COME LI VORREMMOUTILIZZARE?
  • 50.
    Impedence Mismatch Film GenereDrammatico: Titolo Anno Visioni Schindler’s List 1993 3454 Pulp Fiction 1994 2114 Il Padrino 1972 3873 Film Genere Western: Titolo Anno Visioni Il buono, il brutto, il cattivo 1966 7834 Film Genere Commedia: Titolo Anno Visioni Mamma ho perso l’aereo 1990 3555
  • 51.
  • 52.
    Impedence Mismatch JOIN per evitare molteplici query
  • 53.
    Dati in formaaggregata: { "genere":"Drammatico", "film":[ { "titolo":"Schindler's List", "visioni":3454, "anno":1993 }, { "titolo":"Pulp Fiction", "visioni":2114, "anno":1994 }, { "titolo":"Il Padrino", "visioni":3873, "anno":1972 } ] }
  • 54.
  • 55.
    Da... Pannello Analisi Room TV Controllo Statistiche Dati 55
  • 56.
    A... Pannello Analisi Room TV Controllo Statistiche API Dati 56
  • 57.
    O meglio... Pannello Analisi Room TV Controllo Statistiche API API API Metadati Dati hotel Statistiche 57
  • 58.
    Domain Driven Design 58
  • 59.
  • 60.
    + CQRS Fonte: http://www.domainlanguage.com/ddd/patterns/ 60
  • 61.
  • 62.
  • 63.
  • 64.
  • 65.
  • 66.
  • 67.
  • 68.
  • 69.
  • 70.
  • 71.
    Modello dati Dimensione Key Value Colonne Documentale A grafo > 90% dei casi d’uso Complessità Fonte: http://www.slideshare.net/emileifrem/an-overview-of-nosql-jfokus-2011
  • 72.
  • 73.
  • 74.
    Confronto replication sharding multi-master support support Memcached No (use Yes (through No (use repcached client library) repcached instead) instaed) Project Async Yes Yes Voldemort Redis Async No No Riak Async Yes Yes Peer-based 74
  • 75.
    Confronto (2) data concurrency data types durability storage control Memcached RAM Lock-free Blob n.a. Project RAM, BDB, MVCC Typed Data Voldemort MySql, storage others dependant Redis RAM + disk Locks Semi-typed Append- only log Riak RAM, disk MVCC Semi-typed Durable (Vector- write clocks) quorum 75
  • 76.
    Confronto (3) requires transactions secondary compaction indexex Memcached No No No Project Data storage No No Voldemort dependant Redis Snapshot Yes (multi No operations queues) Riak No No Yes 76
  • 77.
    Confronto (4) security interface query language protocol support Memcached Via SASL TCP, UDP Commands C Project None Protocol Commands Java Voldemort Buffers, Thrift, Avro and Java Serialization Redis Passwords Simple text Commands C/C++ over TCP Riak None HTTP, Map reduce Erlang protobuf (weak support) 77
  • 78.
  • 79.
  • 80.
    Confronto replication sharding multi-master support support CouchDB Master-master Yes (with filters Yes in BigCouch) MongoDB Master-slave Yes No (replica sets) OrientDB Master-master Dec. 2012 / Yes Jan. 2013 RavenDB Master-master Yes Yes 80
  • 81.
    Confronto (2) data concurrency data types durability storage control CouchDB Disk Lock-free JSON Crash-only MVCC MongoDB Disk Write lock BSON Write-ahead log, safe mode OrientDB Disk MVCC JSON Write-ahead log RavenDB RAM, Disk Configurable JSON Write-ahead optimistic log, snapshot concurrency (via ESE) 81
  • 82.
    Confronto (3) requires transactions secondary compaction indexex CouchDB Yes, file rewrite No Yes MongoDB No No Yes OrientDB No Yes No RavenDB No Yes Yes 82
  • 83.
    Confronto (4) security interface query language protocol support CouchDB Users HTTP Map reduce Erlang (REST) (weak support) MongoDB Users Binary over Commands, C++ TCP map reduce OrientDB Users, HTTP SQL(+), Java ACL (REST), Tinkerpop Binary over TCP RavenDB Users .NET API, LINQ C# HTTP (REST) 83
  • 84.
  • 85.
  • 86.
  • 87.
    Esempi Flock 87
  • 88.
  • 89.
    ONE SIZE DOESNOT FIT ALL
  • 90.
  • 91.
  • 92.
  • 93.
  • 94.
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
    Per Approfondire: • Links: – http://nosql.findthebest.com/ – http://www.dzone.com/mz/nosql – newsletter Nosql Weekly – http://www.nosql-database.org/ – http://couchdb.apache.org/ – http://www.mongodb.org/ – http://redis.io/ – http://www.orientdb.org/ – http://www.draftq.com/ebooks/DraftQ/the-little-mongodb-book 99
  • 100.
    Photo Credits • http://www.flickr.com/photos/kunel/246544081/ • http://www.flickr.com/photos/lordcolus/5977615749/ • http://www.flickr.com/photos/gewitterhexer/5540504147/ • http://blog.thomsonreuters.com/index.php/big -data-graphic-of-the-day/ • http://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/12_MassStorage.html • http://www.nathaninfocus.com/2011/01/social-networking-phenomenon/facebook-growth-chart/ • http://www.flickr.com/photos/phuonglovejesus2782010/5055970759/ • http://www.flickr.com/photos/kalexanderson/5944050340/ • http://www.flickr.com/photos/arichards-gallery/3578048793/ • http://www.flickr.com/photos/jabb/6956142046/ • http://www.flickr.com/photos/druclimb/47310529/ • http://www.flickr.com/photos/miggslives/5351504116/ • http://www.flickr.com/photos/polkadotcreations/2480587587/ • http://www.flickr.com/photos/wien/418840561/ • http://www.flickr.com/photos/heartlover1717/2605609803/ • http://www.flickr.com/photos/sepehrehsani/5766453552/ • http://www.flickr.com/photos/47108884@N07/6949078701/ • http://www.flickr.com/photos/ricephotos/354939144/ • http://www.flickr.com/photos/toptechwriter/3069396941/ • http://www.flickr.com/photos/djnordic/167433120/ • http://www.flickr.com/photos/jpstanley/69523927/ • http://www.flickr.com/photos/lodigs/2833648828/ • http://www.flickr.com/photos/ppym1/387781444/ • http://www.flickr.com/photos/freefoto/3844247553/ • http://www.flickr.com/photos/31492524@N00/3801200094/ • http://www.flickr.com/photos/sararah/2119107500/ • http://www.flickr.com/photos/wlef70/7507679760/ • http://www.flickr.com/photos/wold/369244351/ • http://www.flickr.com/photos/ilri/7839428936/ • http://www.flickr.com/photos/dhwright/8012651441/ • http://www.flickr.com/photos/birminghammag/7979485144/ • http://www.flickr.com/photos/capcase/4970062870/ • http://www.flickr.com/photos/aidanwojtas/5879866927/ 100
  • 101.
    Stefano Maraspin Stefano Valle @maraspin @stefanovalle s.maraspin@mvassociati.it s.valle@mvassociati.it