2. Who am I?
• Zohar Elkayam, CTO at Brillix
• DBA, team leader, and a senior consultant for over 17 years
• Oracle ACE Associate
• Involved with Big Data projects since 2011
• Blogger – www.realdbamagic.com
http://brillix.co.il2
3. About Brillix
• Brillix is a leading company that specialized in Data
Management
• We provide professional services and consulting for
Databases, Security and Big Data solutions
3
4. Agenda: Big Data
• Big Data
• Why
• What
• Where
• Who and How
• A Big Data Solution: Hadoop
• NoSQL vs. RDBMS
4 http://brillix.co.il
10. MORE stories..
• Crime Prevention in Los Angeles
• Diagnosis and treatment of genetic diseases
• Investments in the financial sector
• Generation of personalized advertising
• Astronomical discoveries
http://brillix.co.il10
11. Examples of Big Data Use Cases Today
MEDIA/
ENTERTAINMENT
Viewers / advertising
effectiveness
COMMUNICATIONS
Location-based advertising
EDUCATION &
RESEARCH
Experiment sensor
analysis
CONSUMER PACKAGED
GOODS
Sentiment analysis of what’s
hot, problems
HEALTH CARE
Patient sensors,
monitoring, EHRs
Quality of care
LIFE SCIENCES
Clinical trials
Genomics
HIGH TECHNOLOGY /
INDUSTRIAL MFG.
Mfg quality
Warranty analysis
OIL & GAS
Drilling exploration
sensor analysis
FINANCIAL
SERVICES
Risk & portfolio analysis
New products
AUTOMOTIVE
Auto sensors reporting
location, problems
RETAIL
Consumer sentiment
Optimized marketing
LAW ENFORCEMENT
& DEFENSE
Threat analysis - social
media monitoring, photo
analysis
TRAVEL &
TRANSPORTATION
Sensor analysis for optimal
traffic flows
Customer sentiment
UTILITIES
Smart Meter
analysis for
network capacity,
ON-LINE SERVICES /
SOCIAL MEDIA
People & career
matching
Web-site
optimization
http://brillix.co.il11
12. Most Requested Uses of Big Data
• Log Analytics & Storage
• Smart Grid / Smarter Utilities
• RFID Tracking & Analytics
• Fraud / Risk Management & Modeling
• 360° View of the Customer
• Warehouse Extension
• Email / Call Center Transcript Analysis
• Call Detail Record Analysis
12 http://brillix.co.il
15. Volume
• Big data come in one size: Big.
• Size is measured in Terabyte(1012), Petabyte(1015), Exabyte(1018),
Zettabyte (1021)
• The storing and handling of the data becomes an issue
• Producing value out of the data in a reasonable time is an issue
15 http://brillix.co.il
16. Some numbers
• How much data in the world?
• 800 Terabytes, 2000
• 160 Exabytes, 2006 (1EB = 1018B)
• 4.5 Zettabytes, 2012 (1ZB = 1021B)
• 44 Zettabytes by 2020
• How much is a zettabyte?
• 1,000,000,000,000,000,000,000 bytes
• A stack of 1TB hard disks that is 25,400 km high
http://brillix.co.il16
17. Growth Rate
• How much data
generated in a
day?
• 7 TB, Twitter
• 10 TB, Facebook
http://brillix.co.il17
19. Variety
• Big Data extends beyond structured data: including
semi-structured and unstructured information: logs,
text, audio and videos.
• Wide variety of rapidly evolving data types requires
highly flexible stores and handling.
19 http://brillix.co.il
21. Big Data is ANY data
• Some has fixed structure
• Some is “bring own structure”
• We want to find value in all of it
Unstructured, Semi-Structure and Structured
http://brillix.co.il21
23. Velocity
• The speed in which the data is being generated and collected
• Streaming data and large volume data movement
• High velocity of data capture – requires rapid ingestion
• Might cause the backlog problem
23 http://brillix.co.il
26. Veracity
• Quality of the data can vary greatly
• Data sources might be messy or corrupted
http://brillix.co.il26
27. So, What Defines Big Data?
• When we think that we can produce value from that data and
want to handle it
• When the data is too big or moves too fast to handle in a
sensible amount of time
• When the data doesn’t fit conventional database structure
• When the solution becomes part of the problem
27 http://brillix.co.il
29. Why Big Data Now?
• Because we have data:
• Data is born already in digital form
• 40% of data growth per year
• Because we can:
• 500$ for a drive in which to store all the music of the world
• 40 years of Moore's Law = large computational resources
• 64% of organizations have invested in big data in 2013
• 34 billion $ invested in big data in 2013
“Because we reached dead end with logic”
http://brillix.co.il29
32. Big Data in Practice
• Big data is big: technological infrastructure solutions needed
• Big data is messy: data sources must be cleaned before use
• Big data is complicated: need developers and system admins
to manage intake of data
http://brillix.co.il32
33. Big Data in Practice (cont.)
• Data must be broken out of silos in order to be mined, analyzed
and transformed into value
• The organization must learn how to communicate and interpret
the results of analysis
http://brillix.co.il33
34. Infrastructure Challenges
• Infrastructure that is built for:
• Large-scale
• Distributed
• Data-intensive jobs that spread the problem across clusters of server
nodes
34 http://brillix.co.il
35. Infrastructure Challenges (cont.)
• Storage:
• Efficient and cost-effective enough to capture and store terabytes, if
not petabytes, of data
• With intelligent capabilities to reduce your data footprint such as:
• Data compression
• Automatic data tiering
• Data deduplication
35 http://brillix.co.il
36. Infrastructure Challenges (cont.)
• Network infrastructure that can quickly import large data sets
and then replicate it to various nodes for processing
• Security capabilities that protect highly-distributed
infrastructure and data
36 http://brillix.co.il
38. Positions in Big Data management
• DevOps are handling the infrastructure – sys admins and
cluster manager
• Data scientists are in charge of producing value from the data
http://brillix.co.il38
41. Apache Hadoop
• Open source project run by Apache (2006)
• Hadoop brings the ability to cheaply process large amounts of
data, regardless of its structure
• It Is has been the driving force behind the growth of the big
data Industry
• Get the public release from:
• http://hadoop.apache.org/core/
41 http://brillix.co.il
43. Key points
• An open-source framework that uses a simple programming model to
enable distributed processing of large data sets on clusters of
computers.
• The complete technology stack includes
• common utilities
• a distributed file system
• analytics and data storage platforms
• an application layer that manages distributed processing, parallel
computation, workflow, and configuration management
• Cost-effective for handling large unstructured data sets than
conventional approaches, and it offers massive scalability and speed
43
44. Why use Hadoop?
Cost Flexibility
Near linear
performance up
to 1000s of
nodes
Leverages
commodity HW &
open source SW
Versatility with
data, analytics &
operation
Scalability
http://brillix.co.il44
45. What Hadoop Is Not?
• Hadoop does not replace DW or relational databases
• Hadoop is not for OLTP or real-time systems
• Very good for large amount, not so much for smaller sets
• Designed for clusters – there is Hadoop monster server (single
server)
http://brillix.co.il45
46. Hadoop Cluster in Yahoo
46
Cluster of machine running Hadoop at Yahoo! (credit: Yahoo!)
http://brillix.co.il
48. Hadoop Main Components
• HDFS: Hadoop Distributed File System – distributed file
system that runs in a clustered environment.
• MapReduce – programming paradigm for running processes
over a clustered environments.
48 http://brillix.co.il
49. HDFS is...
• A distributed file system
• Redundant storage
• Designed to reliably store data using commodity hardware
• Designed to expect hardware failures
• Intended for large files
• Designed for batch inserts
• The Hadoop Distributed File System
49 http://brillix.co.il
50. MapReduce is...
• A programming model for expressing distributed
computations at a massive scale
• An execution framework for organizing and performing such
computations
• An open-source implementation called Hadoop
50 http://brillix.co.il
51. MapReduce is good for...
• Embarrassingly parallel algorithms
• Summing, grouping, filtering, joining
• Off-line batch jobs on massive data sets
• Analyzing an entire large dataset
51 http://brillix.co.il
52. MapReduce is OK for...
• Iterative jobs (i.e., graph algorithms)
• Each iteration must read/write data to disk
• IO and latency cost of an iteration is high
52 http://brillix.co.il
53. MapReduce is NOT good for...
• Jobs that need shared state/coordination
• Tasks are shared-nothing
• Shared-state requires scalable state store
• Low-latency jobs
• Jobs on small datasets
• Finding individual records
53 http://brillix.co.il
54. Spark
• Fast and general MapReduce-like engine for large-scale data
processing
• Fast
• In memory data storage for very fast interactive queries Up to 100 times
faster then Hadoop
• General
• Unified platform that can combine: SQL, Machine Learning , Streaming ,
Graph & Complex analytics
• Ease of use
• Can be developed in Java, Scala or Python
• Integrated with Hadoop
• Can read from HDFS, HBase, Cassandra, and any Hadoop data source.
54
55. Key Concepts
55
Resilient Distributed Datasets
• Collections of objects spread
across a cluster, stored in RAM
or on Disk
• Built through parallel
transformations
• Automatically rebuilt on failure
Operations
• Transformations
(e.g. map, filter, groupBy)
• Actions
(e.g. count, collect, save)
Write programs in terms of transformations on
distributed datasets
56. Unified Platform
• Continued innovation bringing new functionality, e.g.:
• Java 8 (Closures, LambaExpressions)
• Spark SQL (SQL on Spark, not just Hive)
• BlinkDB(Approximate Queries)
• SparkR(R wrapper for Spark)
56
58. The Challenge
• We want scalable, durable, high volume, high velocity,
distributed data storage that can handle non-structured data
and that will fit our specific need
• RDBMS is too generic and doesn’t cut it any more – it can do
the job but it is not cost effective to our usages
58 http://brillix.co.il
59. The Solution: NoSQL
• Let’s take some parts of the standard RDBMS out to and
design the solution to our specific uses
• NoSQL databases have been around for ages under different
names/solutions
59 http://brillix.co.il
60. Example Comparison: RDBMS vs. Hadoop
60
Typical Traditional RDBMS Hadoop
Data Size Gigabytes Petabytes
Access Interactive and Batch Batch – NOT Interactive
Updates Read / Write many times Write once, Read many times
Structure Static Schema Dynamic Schema
Scaling Nonlinear Linear
Query Response
Time
Can be near immediate Has latency (due to batch processing)
http://brillix.co.il
61. Best Used For:
Structured or Not (Flexibility)
Scalability of Storage/Compute
Complex Data Processing
Cheaper compared to RDBMS
Relational Database
Best Used For:
Interactive OLAP Analytics
(<1sec)
Multistep Transactions
100% SQL Compliance
Best when used together
Hadoop And Relational Database
61 http://brillix.co.il
62. The NOSQL Movement
• NOSQL is not a technology – it’s a concept
• We need high performance, scale out abilities or agile structure
• We are willing to sacrifice our sacred database cows:
consistency, transactions, durability
• Over 150 different brands and solutions
(http://nosql-database.org/).
62 http://brillix.co.il
63. Is NoSQL a RDMS Replacement?
NO
63
Well... Sometimes it does…
http://brillix.co.il
65. Key Value Store
• Distributed hash tables
• Very fast to get a single value
• Examples:
• Amazon DynamoDB
• Berkeley DB
• Redis
• Riak
• Cassandra
65 http://brillix.co.il
66. Document Store
• Similar to Key/Value, but value is a document
• JSON or something similar, flexible schema
• Agile technology
• Examples:
• MongoDB
• CouchDB
• CouchBase
66 http://brillix.co.il
67. What is a Column Store Database?
• Column Store databases are management systems that uses
data managed in a columnar structure format for better
analysis of single column data (i.e. aggregation). Data is saved
and handled as columns instead of rows.
• Examples:
• HP Vertica
• Pivotal (EMC) GreenPlum
• Hadoop Hbase
• Amazon’s SimpleDB
• Cassandra
http://brillix.co.il67
68. Query Data
• When we query data, records are read at the
order they are organized in the physical structure
• Even when we query a single
column, we still need to read the
entire table and extract the column
Row 1
Row 2
Row 3
Row 4
Col 1 Col 2 Col 3 Col 4
Select Col2
From MyTable
Select *
From MyTable
http://brillix.co.il68
69. How Does Column Stores Keep Data
Organization in row store Organization in column store
http://brillix.co.il69
Select Col2
From MyTable
71. Graph Store
• Inspired by the graph theory
• Data model: nodes, relationships, properties on both sides
• Relational database have a hard time to represent a graph in
the Database
• Example:
• Neo4j
• InfiniteGraph
• RDF
72 http://brillix.co.il
73. Conclusion
• We do Big Data to gain Value. Without value, there is no Big Data
• Handling Big Data is a challenge – we talked about who uses it, when
and where
• Hadoop is a solution for Big Data usages but it’s not a magical solution
• NoSQL, NewSQL and RDBMS are all solutions we can integrate for
different usages
• New organizational positions: cluster devops and data scientist.
http://brillix.co.il74