NVISIA’s data architects, Michael Vogt, Director Data Management, leads a discussion where he explores emerging technologies often referred to as Big Data 2.0. What started as a research for a specific engagement for a specific large manufacturing and distribution customer has yielded surprising results.

1. Big Data 2.0
Milwaukee Big Data Users Group
12.1.2014

2. DBMS Technology Overview
Goal
• Provide a technology recommendation for
serving reporting needs for the next 3 – 5
years
• Explore different technologies for their
suitability for a strategic reporting data
platform

3. DBMS Technology Overview
Vendor agnostic approach
• Vendor agnostic DBMS technologies evaluated
– Categories
• RDBMS
– Row based vs column based
• In-memory data base
– Row based vs column based
• NoSQL
– Document based (Disk)
– Key value based (IMDG)
– Graph based (IMDG)
– Criteria
• Overall design
• Pros/cons

4. DBMS Technology Overview
Representative vendor evaluation criteria
• …followed by quick evaluation of two vendors
representing each technology
– Thought leadership
– Market share / # of production customers
– Capacity / scalability
– Functionality
– Expertise availability
– Resilience
– Cost (license, infrastructure & expertise)
– Interface compatibility (drop-in-ability)

5. DBMS Technology Overview
Open Discussion on drop-in-ability
• Re-tooling interfaces is expensive
– Focus is on query/reporting tools (in my evaluation)
– List of possible solutions drastically reduced by this
criterion
– SQL compatibility (very important syntactic sugar)
– ACID compliance (dual use technology for OLTP needs)
• A cost-effective, performant, resilient solution
that requires interface re-tooling is DOA for my
client’s environment

6. Striking phrases
• Disk is the new tape, memory is the new disk
• IMDG’s are increasingly being referred to as
Big Data 2.0

7. RDBMS
row based
• OLAP needs typically serviced by partitioning (row &
column)
• 30 years old (proven technology)
• IMDB implementations typically have same pros/cons,
although cost and performance characteristics are different
• Pros
Row-based
Data Cols Time Location Product Vendor
Block 1
2/23 0900 IL023 Gown112 ML
Block 2
2/23 0423 OH12 Mask221 123
Block3
2/24 1543 CN881 Swab993 456
– Great OLTP performance
– Efficient at whole-row operations
• Cons
– Inefficient at data set operations
– Scalability is typically not linear

8. RDBMS
column based
• Optimized for OLAP needs as it’s optimized to answer questions on data
characteristics
• Great performance on aggregate functions (avg, count, sum, min, max)
• IMDB implementations typically have same pros/cons, although cost and
performance characteristics are different
• Pros
Column-based
Block 0 Time 2/23 0900 2/23 0423 2/24 1543
Block 1
Location IL023 OH12 CN881
Block 2
Product Gown112 Mask221 Swab993
Block3
Vendor ML 123 456
– Aggregate functions are very fast as entire column can be fetched quickly
– Efficient at data set operations
– Easily compressed, especially for data that is sparsely populated
• Cons
– Inefficient at retrieving many columns of a single row
– Row functions are slower

9. NoSQL
document/XML based
• Focus is typically on sharding strategy as opposed to up-front data modeling
(models typically evolve greatly during construction)
• Similar to key-value stores, where values are stored in a standardized structure
(although document stores keep metadata as well)
• An example of data in a document database:
– {officeName:”3Pillar Noida”,
{Street: “B-25, City:”Noida”, State:”UP”, Pincode:”201301”}
}
– {officeName:”3Pillar Timisoara”,
{Boulevard:”Coriolan Brediceanu No. 10”, Block:”B, Ist Floor”, City: “Timisoara”, Pincode:
300011”}
}
– {officeName:”3Pillar Cluj”,
{Latitude:”40.748328”, Longitude:”-73.985560”}
}
• Pros
– Not limited to querying by keys (can query inside documents using JSON/XML query
mechanisms)
– Maps well to semi-structured or variable structured data
• Cons
– Sharding strategy can be challenging
– Doesn’t support relations (no RI), as opposed to key value or graph stores

10. NoSQL
IMDG (common to key-value / graph)
• IMDG’s referred to as “Big Data 2.0”
• Host data in memory and distribute across cluster of commodity
servers
• Employ an object-oriented data model that provides read/write
times << 1 ms
• As data is stored in virtual memory pool, parallel data computations
are easily performed
• As in document databases, focus is on sharding strategy as opposed
to up-front physical data modeling
• Majority of implementations utilize JVM’s (although a handful of
.Net are out there)
• GC, specifically the unpredictability of GC, is a major concern
– Vendors utilize off-heap storage to alleviate this by moving LRU data to
off-heap JVM’s, relying on high-speed messaging for data transport

11. NoSQL
IMDG (key-value)
• Typically stored as a set of distributable maps
• Pros
– Data distribution is designed from the ground up
– Keys and values are Java (or .Net) objects
– No bias between OLTP and OLAP
• Cons
– Alternate lookup mechanisms require a map with an
alternate key (although main data payload can be
shared as values are objects that support multiple
pointers)
– Expertise is typically harder to find (on characteristics
of memory structure behavior at larger sizes)

12. NoSQL
IMDG (graph)
• Allow a set of nodes (object instances) with dynamic
properties (cols/attributes) to be arbitrarily linked to
other nodes through edges (associations)
• Each node only knows its adjacent nodes
• As the number of nodes increases, cost of a local hop
remains constant
• Whereas a RDBMS is optimized for aggregation, a
graph database is optimized for connections
• Fastest growth area in NoSQL in the last year – 250%

13. NoSQL
IMDG (graph cont’d)
• 60% of Facebook graph is hosted on one instance
of Neo4J
• Pros
– Powerful general purpose (reusable) data model
– Connected data locally indexed
– Easy to query
– Optimized for recursive structures (think BoM)
– Great at use cases with complex relationships (supply
chain management)
• Cons
– Sharding strategy is difficult
– Requires re-wiring your brain (think object model
instead of data model)

14. Particular vendor evaluations
<<Vendor evaluation.xls>>

15. Recap
• In addition to normal criteria (scalability,
functionality, cost, etc.), drop-in-ability should
be considered as well
• Niche-technologies are available for more
mainstream use cases, due to falling hardware
prices

16. Questions/Comments
?

17. Thank you
… for your time
Michael Vogt
Director, Data Management
NVISIA
mvogt@nvisia.com
(o) 414.347.1303 or 312.985.8100
(c) 312.772.4762