Published on

Traditionally, relational database have been
the number one choice for storing structured data in enterprise business
applications. Relational data stores have been widely adopted and are often
thought of as the only alternative for data storage accessible by multiple
clients. There have been other approaches over the years, such as Object
XML databases, but these technologies never came close to the market share
by RDBMS. Instead, these types of innovations simply became absorbed by
generations of relational database management systems.
With the rise of cloud computing, a "one-size fits all" mentality has
emerged concerning data store
architectures, leading to a new type of data store commonly labeled with
the term of "NoSQL" (or "Not only SQL").
NoSQL data stores can be categorised as tabular/columnar, document, graph
and key/value store databases, each optimised
to handle certain kinds of data processing requirements.
The main driver for the creation of NoSQL data stores has been the
popularity of
"Web-scale" data at large, global Web sites and services, such as Amazon,
Google, Yahoo!, and Facebook. Recently, predictive analytics,
voice-of-the-customer, fraud, and other BigData use cases have surfaced to
further accelerate the demand for NoSQL.
But are NoSQL databases only useful in BigData scenarios? Or should they
be positioned as an alternative to relational
databases for persistence in an N-Tier architecture?
This session presents the most popular NoSQL data storage engines,
discuses the factors to consider with the
potential tradeoffs of imposed by NoSQL, and demonstrates how the concept
of data virtualization can help create an abstraction layer that hides the
complexities of the underlying data sources and by that provides a unified
view of enterprise data which can also be used directly for providing Data
in a Service-Oriented Architecture.

Published in: Technology
1 Comment
  • Great presentation! All in one greatly explained. Too bad this presentation can not be saved locally
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  2. 2. Guido Schmutz• Working for Trivadis for more than 15 years• Oracle ACE Director for Fusion Middleware and SOA• Co-Author of different books• Consultant, Trainer Software Architect for Java, Oracle, SOA and EDA• Member of Trivadis Architecture Board• Technology Manager @ Trivadis• More than 20 years of software development experience• Contact:• Blog:• Twitter: gschmutz 2012 © Trivadis2 NoSQL and SOA 10.10.2012
  3. 3. Agenda1. Why NoSQL and what is it?2. NoSQL Database Types3. Polyglot Persistence4. NoSQL and Service-Oriented Architecture5. Summary 2012 © Trivadis3 NoSQL and SOA 10.10.2012
  4. 4. History of Database1960s File-based, Network (CODASYL) and HierarchicalDatabases1970s Relational Database1980 SQL became the standard query languageEarly 1990 Object-DatabasesLate 1990 XML Databases2004 NoSQL Databases 2012 © Trivadis4 NoSQL and SOA 10.10.2012
  5. 5. What„s wrong with Relational Databases ? They are great ….• SQL provides a rich, declarative query language• Database enforce referential integrity• ACID semantics• Well understood by developers, database administrators• Well supported by different languages, frameworks and tools • Hibernate, JPA, JDBC, iBATIS, Entity Framework• Well understood and accepted by operations people (DBAs) • Configuration • Monitoring • Backup and Recovery • Tuning • Design 2012 © Trivadis5 NoSQL and SOA 10.10.2012
  6. 6. Relational Databases are great ... But! ORDER Order ID: 1001Problem: Complex Object graphs Order Date: 15.9.2012 Customer  Object/Relational impedance mismatch CUSTOMER First Name: Peter Last Name: Sample  Complicated to map rich domain model Billing Address Street: Somestreet 10 to relational schema City: Somewhere Postal Code: 55901 ADDRESS  Performance issues Line Items  Many rows in many tables Name Ipod Touch Quantity 1 Price 220.95 ORDER_LINES  Many joins Monster Beat 2 190.00  Eager vs. lazy loading Apple Mouse 1 69.90Problem: Schema evolution  Adding attributes to an object => have to add columns to table  Expensive, if lots of data in that table - Holding locks on the tables for long time - Application downtime … 2012 © Trivadis6 NoSQL and SOA 10.10.2012
  7. 7. ORDER OrderRelational Databases are great ... But! ID: 1001 Order Date: 15.9.2012 Customer CUSTOMER First Name: Peter Last Name: Sample Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901 ADDRESS Line Items Name Quantity Price ORDER_LINES Ipod Touch 1 220.95 Monster Beat 2 190.00 Apple Mouse 1 69.90 Consumer REST/SOAP Service Repository/DAO O/R Mapping SQL RDBMS 2012 © Trivadis7 NoSQL and SOA 10.10.2012
  8. 8. Relational Databases are great ... But!Problem: Semi-structured data  Relational schema doesn„t easily handle semi-structured data  Common solutions - Name/Value table - Poor performance - Lack of constraint - Serialize as Blob - Fewer joins, but no query capabilitiesProblem: Scaling  Scaling writes difficult/expensive/impossible => BigData  Vertical scaling is limited and is expensive  Horizontal scaling is limited and is expensive 2012 © Trivadis8 NoSQL and SOA 10.10.2012
  9. 9. Solution: NoSQL ?No standard definition of what NoSQL means• Not Only SQLTerm began in a workshop organized in 2009but some common characteristics of NoSQL databases• They don„t use the relational data model and thus don„t use SQL• Tend to be designed to run on cluster RDBMS NoSQL• Tend to be Open Source Presentation Tier User Interface User Interface• Schema-Less - Don„t have a fixed Key Value Stores schema, allowing to store any Services Caching Search Middle Tier Object-Relational Relational-Object Lucene Transactions Batch data in any record MapReduce• Different APIs Search Blobs Database Tier Transactions Batch Data Caching Triggers 2012 © Trivadis9 NoSQL and SOA 10.10.2012
  10. 10. Central vs. Application DatabasesCentral Database Application Database• Using SQL as the integration mechanism • Only accessed by a single application between applications • Only the application using the database• applications store data in common DB needs to know about the structure• Improves communication, all applications • Easier to maintain and evolve the schema operate on consistent set of data • More freedom to choose the database• Structure ends up to be more complex • Applicable to SOA (i.e. Data Service/Entity• Changes need to be coordinated with all Service) with good Service Autonomy other applications using the database • Ready for the cloud• Side-effects (i.e. adding database index) Application 1 Application 2 Application 3 Application 1 Application 2 Application 3 DB DB DB DB 2012 © Trivadis10 NoSQL and SOA 10.10.2012
  11. 11. Relational vs. Aggregate Data Models The relational model takes the  Aggregate is a term that comes information and divides it into from Domain-Driven Design tuples (rows) (Evans) A tuple is a limited data structure  An aggregate is a collection of  no nesting of tuples related objects, that should be  no list of values treated as a unit  Unit for data manipulation and management of consistency 2012 © Trivadis11 NoSQL and SOA 10.10.2012
  12. 12. Relational vs. Aggregate Data ModelRelational Instance Aggregate InstanceCUSTOMER PRODUCT ID NAME ID NAME { 1 Guido 1000 IPod Touch „id“:1, 1020 Monster Beat „name“:“Guido“, BILLING_ADDRESS „billingAddress“:[{„street“:“Chaumontweg“,“city“:“Spiegel“,“postCode“:“3095“}] } ID CUSTOMER_ID ADDRESS_ID 1 1 55 { „id“:90, ADDRESS „customerId“:1, ID STREET CITY POST_CODE „orderItems“:[ { 55 Chaumontweg Spiegel 3095 „productId“:1000,“price“: 250.55, „produtName“: „iPod Touch“ }, ORDER { ID CUSTOMER_ID SHIPPING_ADDRESS_ID „productId“:1020,“price“: 199.55, „produtName“: „Monster Beat“ 90 1 55 }], „sippingAddress“:[{„street“:“Chaumontweg“,“city“:“Spiegel“,“postCode“:“3095“}] }ORDER_ITEM ID ORDER_ID PRODUCT_ID PRICE 1 90 1000 250.55 1 90 1020 199.55 2012 © Trivadis12 NoSQL and SOA 10.10.2012
  13. 13. Agenda1. Why NoSQL and what is it?2. NoSQL Database Types3. Polyglot Persistence4. NoSQL and Service-Oriented Architecture5. Summary 2012 © Trivadis15 NoSQL and SOA 10.10.2012
  14. 14. NoSQL Database Types Key/Value Column Family Document Graph Key/Value StoresDesign Collections of Columns and Key/Value pairs Focus on the Ordered Key-Value Stores Colum Families. Key/Value Pairs but value is connections Accesses directly interpreted by between data and Big Table Stores (map-of-maps-of-maps) the column values. the database the fast navigation Document StoresScalability/ +++ +++ ++ ++Performance Graph DatabasesAggregate- Yes Yes Yes NoorientedComplexity + ++ ++ +++Inspiration and Berkley DB, SAP Sybase IQ, Lotus Notes Graph TheoryRelation Memcached, BigTable Distributed HashmapsNoSQL Voldemort Hbase CouchDB SonesProducts Redis Cassandra MongoDB Neo4J Riak Hypertable OrientDB InfoGrid Amazon SimpleDB RavenDB FlockDB 2012 © Trivadis16 NoSQL and SOA 10.10.2012
  15. 15. NoSQL Database TypesSize Key-value stores Column Family Document Graph Relational Complexity 2012 © Trivadis17 NoSQL and SOA 10.10.2012
  16. 16. Key Value Databases A key-value store is a simple hash table Primarily used when all access to the database is via primary key Simplest NoSQL data stores to use (from an API perspective)  PUT, GET, DELETE (matches REST) Value is a blob with the data store not caring or knowing what is inside Aggregate-OrientedSuitable Use Cases• Storing Session Information• User Profiles, Preferences• Shopping Cart Data 2012 © Trivadis 18 NoSQL and SOA 10.10.2012
  17. 17. Column-Family Stores Store data in column families as rows that have many columns associated with a row key Column families are groups of related data, often accessed together Aggregate-OrientedSuitable Use Cases• Event Logging• Content Management Systems• Counters Source: NoSQL Distilled• Expiring Usage 2012 © Trivadis 19 NoSQL and SOA 10.10.2012
  18. 18. Document Databases Documents are the main concept Stores and retrieves documents, which can be XML, JSON, BSON, … Documents are self-describing, hierarchical tree data structures which can consist of maps, collections and scalar values Documents stored are similar to each other but do not have to be exactly the same Aggregate-OrientedSuitable Use Cases• Event Logging• Content Management Systems• Web Analytics or Real-Time Analytics• Product Catalog 2012 © Trivadis 20 NoSQL and SOA 10.10.2012
  19. 19. Document Database - MongoDB 2012 © Trivadis21 NoSQL and SOA 10.10.2012
  20. 20. Graph Databases Allow to store entities and relationships between these entities Entities are known as nodes, which have properties Relations are known as edges, which also have properties A query on the graph is also known as traversing the graph Traversing the relationships is very fast Tag CustomerSuitable Use Cases Country RATED TAG• Connected Data ADDRESS COUNTRY Product• Routing, Dispatch and Location-Based BILLING_ LINE_ITEM Services ADDRESS Address Recommendation Engines Order• DELIVERY_ ADDRESS 2012 © Trivadis 22 NoSQL and SOA 10.10.2012
  21. 21. Graph Database – Neo4JQuery through Cypher START MATCH WHERE RETURN ORDER BY LIMIT customer=node:Customer(email = "") customer-[:ORDERED]->order-[item:LINEITEM]->product > 20120101, sum(item.amount) AS product products DESC 20 2012 © Trivadis 23 NoSQL and SOA 10.10.2012
  22. 22. Agenda1. Why NoSQL and what is it?2. NoSQL Database Types3. Polyglot Persistence4. NoSQL and Service-Oriented Architecture5. Summary 2012 © Trivadis24 NoSQL and SOA 10.10.2012
  23. 23. Polyglot PersistenceIn 2006, Neal Ford coined the term Polyglot Programming Applications should be written in a mix of languages to take advantage of the fact that different languages are suitable for tackling different problemsPolyglot Persistence defines a a hybrid approach to persistence Using multiple data storage technologies Selected based on the way data is being used by individual applications  Why store binary images in relational databases, when there are better storage systems? Can occur both over the enterprise as well as within a single application 2012 © Trivadis25 NoSQL and SOA 10.10.2012
  24. 24. „Traditional“ Persistence ModelPolyglot Persistence E-commerce ApplicationToday we use the samedatabase for all kind of data Shopping cart data User Sessions Completed Order Product Catalog Recomendations• Business transactions, session management RDBMS data, reporting, logging information, content information, ... Polygot Persistence ModelNo need for same properties ofavailability, consistency or E-commerce Applicationbackup requirementsPolyglot Data Storage Usage Shopping cart data User Sessions Completed Order Product Catalog Recomendationsallows to mix and matchRelational and NoSQL datastores Key-Value RDMBS Document Graph 2012 © Trivadis26 NoSQL and SOA 10.10.2012
  25. 25. Polyglot Persistence – Challenges• Decisions • Have to decide what data storage technology to use • Today it„s easier to go with relational• New Data Access APIs • Each data store has its own mechanisms for accessing the data • Different API‟s Service-Oriented Polygot Persistence Model E-commerce Application • Solution: Wrap the data access code into services (Data/Entity Service) exposed to applications • Will enforce a contract/schema Shopping cart data User Sessions Completed Order Product Catalog Recomendations to a schemaless database Key-Value Graph RDMBS Document Shopping Cart User Session Product Catalog Recomendation Service Service Order Service Service Service 2012 © Trivadis27 NoSQL and SOA 10.10.2012
  26. 26. Polyglot Persistence – Challenges• Immaturity • NoSQL tools are still young, full of rough edges that new tools have • Not much experience, we don„t know how to use them well • No patterns and best practices exist yet• Organizational Change • How will the different data groups in an enterprise react to this new technology• Dealing with eventual consistency paradigm • Reaction of different stakeholders to the fact that data could be stale • How to enforce rules to sync data across systems 2012 © Trivadis28 NoSQL and SOA 10.10.2012
  27. 27. Agenda1. What is NoSQL and Big Data2. NoSQL Database Types3. Polyglot Persistence4. NoSQL and Service-Oriented Architecture5. Summary 2012 © Trivadis29 NoSQL and SOA 10.10.2012
  28. 28. Data Access Architecture for Polyglot Persistencewell known design patterns are still valid!some best practices we know in data access are still valid! Consumer Consumer Consumer REST/SOAP REST/SOAP Service Service REST Repository/DAO Repository/DAO O/R Mapping ??? SQL REST API RDBMS NoSQL NoSQL 2012 © Trivadis30 NoSQL and SOA 10.10.2012
  29. 29. Middle Tier Architecture for Polyglot Persistence Resource Tier Middle Tier Consumer Integration Service Application Domain Integration Domain Service Bean Web Service Exporter Application Service Bean REST Composite Application Factory Bean SOAP O/R Mapping Domain Objects NoSQL API Repository Bean Aggregate SQL API DAO Bean Data Transfer Objects (DTO) 2012 © Trivadis31 NoSQL and SOA 10.10.2012
  30. 30. Polyglot Persistence with Spring Datamakes it easier to build Spring-powered applications that use new dataaccess technologiesprovide improved support for relational database technologiesCommons project supports Polyglot PersistenceCurrently support for:• JPA and JDBC (relational) Consumer• Apache Hadoop REST/SOAP• GemFire Service• REST Repository/DAO• Redis• MongoDB ???• Neo4J• Hbase NoSQL 2012 © Trivadis32 NoSQL and SOA 10.10.2012
  31. 31. Spring Data – Mapping to Relational Database (using JPA) Annotations define the mapping: @Entity, @Id, @Column, @OneToOne, @OneToMany, @JoinCol umn, Consumer REST/SOAP Service Repository/DAO ??? NoSQL 2012 © Trivadis33 NoSQL and SOA 10.10.2012
  32. 32. Spring Data – Mapping to Relational Database Consumer REST/SOAP Servicepublic interface CustomerRepository extends Repository<Customer, Long> { Repository/DAO Customer findByEmailAddress(EmailAddress emailAddress);} ???@Repository NoSQL@Profile(“jpa")class JpaCustomerRepository implements CustomerRepository { @Override public Customer findByEmailAddress(EmailAddress emailAddress) { TypedQuery<Customer> query = em.createQuery( "select c from Customer c where c.emailAddress = :email“, Customer.class); query.setParameter("email", emailAddress); Customer guido= repository.findByEmailAddress(new return query.getSingleResult(); EmailAddress(“"));} Customer anotherCust= new Customer(“Peter", “Sample");<jpa:repositories base-package="com.oreilly.springdata.jpa" /> anotherCust.setEmailAddress(guido.getEmailAddress());;<bean class="org.springframework.orm.jpa.JpaTransactionManager"> <property name="entityManagerFactory" ref="entityManagerFactory" /></bean><bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"> <property name="dataSource" ref="dataSource" /> <property name="packagesToScan" value="com.oreilly.springdata.jpa" /></bean> 2012 © Trivadis34 NoSQL and SOA 10.10.2012
  33. 33. Spring Data – Mapping to MongoDB Annotations define the mapping: @Document, @Id, @Indexed, @PersistenceConstructor, @Compound Index, @DBRef, @GeoSpatialIndex, @V alue Consumer REST/SOAP Service Repository/DAO ??? NoSQL 2012 © Trivadis35 NoSQL and SOA 10.10.2012
  34. 34. Spring Data – Generic Repositories for MongoDB Consumer REST/SOAP Servicepublic interface CustomerRepository extends Repository<Customer, Long> { Repository/DAO Customer findByEmailAddress(EmailAddress emailAddress);} ???@Repository NoSQL@Profile("mongodb")class MongoDbCustomerRepository implements CustomerRepository { @Override public Customer findByEmailAddress(EmailAddress emailAddress) { Query query = query(where("emailAddress").is(emailAddress)); return operations.findOne(query, Customer.class); }<mongo:db-factory id="mongoDbFactory" dbname="e-store" /><mongo:mapping-converter id="mongoConverter" base-package="com.oreilly.springdata.mongodb"><mongo:custom-converters base-package="com.oreilly.springdata.mongodb" /></mongo:mapping-converter><bean id="mongoTemplate" class=""> Customer guido= repository.findByEmailAddress(new <constructor-arg ref="mongoDbFactory" /> EmailAddress(“")); <constructor-arg ref="mongoConverter" /> <property name="writeConcern" value="SAFE" /> Customer anotherCust= new Customer(“Peter", “Sample");</bean> anotherCust.setEmailAddress(guido.getEmailAddress());<mongo:repositories base-package="com.oreilly.springdata.mongodb" />; 2012 © Trivadis36 NoSQL and SOA 10.10.2012
  35. 35. Spring Data – Mapping to Neo4J Annotations define the mapping: @NodeEntity, RelationShipEntity, @Gra phId, @RelatedTo, @RelatedToVia, @E ndNode, @Fetch, Tag Customer Country RATED TAG ADDRESS COUNTRY Product BILLING_ ADDRESS LINE_ITEM Address Order DELIVERY_ ADDRESS 2012 © Trivadis37 NoSQL and SOA 10.10.2012
  36. 36. Spring Data – Generic Repositories for MongoDB Consumer REST/SOAP Service Repository/DAOpublic interface CustomerRepository extends GraphRepository<Customer> { ??? Customer findByEmailAddress(EmailAddress emailAddress);} NoSQL<neo4j:config graphDatabaseService="graphDatabaseService" /><neo4j:repositories base-package="com.oreilly.springdata.neo4j" /><bean id="graphDatabaseService" class="org.neo4j.kernel.EmbeddedGraphDatabase" destroy-method="shutdown"> <constructor-arg value="target/graph.db" /></bean>Customer guido= repository.findByEmailAddress(newEmailAddress(“"));Customer anotherCust= new Customer(“Peter", “Sample");anotherCust.setEmailAddress(guido.getEmailAddress());; 2012 © Trivadis38 NoSQL and SOA 10.10.2012
  37. 37. Expose contract-first Web service Consumer REST/SOAP Service Repository/DAOUse any Java Web Service Framework which supportsContract-First approach ???Can be SOAP or can be REST NoSQLMaps the data contract (WSDL or WADL) to the schemalessdatabaseUses the different Repository implementationsMust handle data migration issues together with theRepository 2012 © Trivadis39 NoSQL and SOA 10.10.2012
  38. 38. Using SOA for Integrating “old” with “new” world 2012 © Trivadis Five Cool Use Cases for the Spring Component of Oracle SOA Suite 1.10.2012
  39. 39. NoSQL databases asCaching in a SOAResults are returned from cache rather than invoking alwaysthe external service Product data is rather static, so ideal candidate for caching Proxy Business 1 Product DB Service Service 2 3 Result CacheOracle ServiceBus 2012 © Trivadis Effective Fault Handling in Oracle SOA Suite 11g 12.10.2012
  40. 40. NoSQL databases asCaching in a SOAUsing Mongo DB to cache results in a SOA 2012 © Trivadis42 NoSQL and SOA 10.10.2012
  41. 41. Schemaless – We still have to migrate the data!With RDMBS we are used to keep DDL scripts together Customerwith DML scripts for each single data model change Name: Peter Sample First Name: Peter Last Name: Sample BillingAddress Billing Address • Has to be in sync with the data access code Street: Somestreet 10 Version 1.0 City: Somewhere Postal Code:55901 PostalCode: 55901RDBMS has to be changed before the applicationis changed => possible application downtime Customer • This is what the schemaless approach of most NoSQL Name: Peter Sample FirstName: Peter DB tries to avoid LastName: Sample Transition Billing Address Version 1.0 => 2.0Schemaless DBs still need careful migration, due to Street: Somestreet 10 City: Somewhereimplicit schema in any data access code PostalCode: 55901But a more “on-demand” approach is possible Customer • Code can read data in a way that it tolerant to First Name: Peter Last Name: Sample changes in the data‟s implicit schema and migrate Billing Address Version 2.0 the data on the next update Street: Somestreet 10 City: Somewhere PostalCode: 55901 • Similar to service versioning => gradual change 2012 © Trivadis43 NoSQL and SOA 10.10.2012
  42. 42. Agenda1. What is NoSQL and Big Data2. NoSQL Database Types3. Polyglot Persistence4. NoSQL and Service-Oriented Architecture5. Summary 2012 © Trivadis44 NoSQL and SOA 10.10.2012
  43. 43. Pros & Cons of NoSQL compared to RDBMSPros Cons• No O/R impedance mismatch • Lacks in tool and framework support• Can easily evolve schemas • Few other implementations =>• Can represent semi-structured potential lock in info • No support for ad-hoc queries• Can represent graphs/networks (with performance) • Another/A new database in production to take care of 2012 © Trivadis45 NoSQL and SOA 10.10.2012
  44. 44. SummaryRelational databases are here to stay but NoSQL offers new persistencemodelPolyglot Persistence will be the futureSchemaless does not mean there is no data migration! => but a more on-demand model might be possibleEncapsulate data access code to be able to switch databasesService-orientation provides the data contract to a NoSQL database => tomake information reusableDon„t commit to a NoSQL until you have done a significant PoCMake sure that Operations people (DBAs) are on board early enoughNon-relational is not new in an enterprise (OLTP vs. OLAP) 2012 © Trivadis46 NoSQL and SOA 10.10.2012
  45. 45. Further Information 2012 © Trivadis47 NoSQL and SOA 10.10.2012
  47. 47. Possible Use Cases NoSQL for parallel ETL? NoSQL for modern BI NoSQL for stateful Middletier (i.e. shopping cart) NoSQL for aggregated master data (i.e. through REST for Web apps) NoSQL for a CMS-Store, directly accessible through REST NoSQL as a local Store for Mobile applications NoSQL for Event Sourcing and CQRS architectures 2012 © Trivadis49 NoSQL and SOA 10.10.2012