2. Agenda
The Next Wave of Computing
Information, Data & Knowledge
Data warehouse vs. Database
Row-oriented vs. Column-oriented DB
Big Data Era
HP-Vertica
HP-Vertica Bulk Upload Example
3. The next wave of computing
Cloud + Big Data Analytics, Mobile & Social leads to what IBM calls “The Era of Smart.”
4. Data, Information & Knowledge (1/3)
Data are raw facts, and have no meaning on their own
Example
Yes, Yes, No, Yes, No, Yes, No, Yes
5. Data, Information & Knowledge (2/3)
Information is data that have meaning in a context
Example
Yes, Yes, No, Yes, No, Yes, No,
Yes, No, Yes, YesRaw Data
Context
Responses to the market
research question – “Would
you buy brand x at price y?”
Information ???
Processing
6. Data, Information & Knowledge (3/3)
Knowledge
understanding of someone or something
acquired through experience or learning
Example
Based on last collected information, A Marketing Manager could use this information to
decide whether or not to raise or lower price.
7. Data Warehouse vs. Database (1/2)
Feature DWH DB
Data Stored It usually stores the Historical data
whose accuracy is maintained over
time.
It mainly stores the Current data
which always guaranteed to be up-
to-date.
Characteristic It is based on Informational
Processing.
It is based on Operational
Processing.
Function It is used for long-term informational
requirements and decision support.
It is used for day-to-day operations.
Focus The focus is on “Information OUT” The focus is on “Data IN”
Number of
records
accessed
A bunch of millions of records. A few tens of records.
8. Data Warehouse vs. Database (2/2)
Feature DWH DB
Access It mostly use the read access for the
stored data.
The most frequent type of access
type is read/write.
Orientation Based on Analysis. Based on Transaction.
Common users Analysts DBAs
KPI Query throughput Transaction throughput
Unit of work Complex queries Short and simple transactions
9. Row-based vs. Column-based DB (1/2)
Feature Row-oriented DB Column-oriented DB
Mechanism stores table records in sequence or
rows.
stores table records in sequence or
columns.
Retrieval of a few
columns
Slower Faster
Retrieval of a single
record
Faster Slower
10. Row-based vs. Column-based DB (2/2)
Feature Row-oriented DB Column-oriented DB
Insertion/Updating of single new
record
Faster Slower
Aggregation of Single Column, e.g.
sum(price)
Slower Faster
Compression Higher. As stores similar data
together
It depends!
Examples Oracle OLTP
MS SQL Server up to 2008
HP-Vertica
Sybase IQ
MS SQL Server 2012
11. Big Data Era – Famous quote
From the dawn of civilization until
2003, humankind generated five
exabytes of data. Now we produce
five exabytes every two days…and
the pace is accelerating.
Eric Schmidt,
Executive Chairman, Google
12. Big Data – Definition
The basic idea behind the phrase 'Big Data' is that everything we do is
increasingly leaving a digital trace (or data), which we (and others) can
use and analyze.
Big Data therefore refers to our ability to make use of the ever-increasing
volumes of data.
14. Big Data – What is Hadoop ?
Apache™ Hadoop® is an open source platform that enables distributed
processing of large data sets across clusters of servers.
It is designed to scale up from a single server to thousands of machines,
with very high degree of fault tolerance.
15. Big Data – HL Archit. Of Hadoop
A programming model for large scale
data processing.
A distributed file-system that stores data
on Clustered machines.
Schedules map or reduce jobs to task
trackers with an awareness of the data
location.
keeps the directory tree of all files in the file
system, and tracks where across the cluster
the file data is kept.
16. Big Data – MapReduce, How it works ?
Map: Filter & sort of data Reduce: Summarize & aggregate of data
17. HP-Vertica
An analytic database management platform. Founded in 2005.
Helps you monetize all of your data in real-time and at massive scale.
Queries run 50-1,000x faster.
Store 10-30x more data per server.
Openness and simplicity (use any BI/ETL tools, Hadoop, etc.)
Suited for “Structured” data only.
HP Vertica Flex Zone: built on the HP Vertica core, enables load and
analyze structured and semi-structured data, such as social media, sensor,
log files, and machine data.
18. HP-Vertica Features (1/6)
Column Orientation
Vertica organizes data for each column
Each column is stored separately on disk
Only reads the columns needed to answer
the query
Significant reduction of disk I/O
19. HP-Vertica Features (2/6)
Advanced Compression
Vertica replaces slower disk I/O with faster CPU cycles
to encode data elements into a more compact form
and query them.
Vertica’s innovative query engine operates directly
on compressed data, meaning that it can actually
require fewer CPU operations to process
the compressed version of a table.
20. HP-Vertica Features (3/6)
High Availability
RAID-like functionality within database
If a node fails, a copy is available on one of the surviving nodes
Always-on Queries and Loads
System continues to load and query when nodes are down
Automatically recovers missing data by querying other nodes
21. HP-Vertica Features (4/6)
Automatic Database Designer (DBD)
Recommends a physical DB design that provides the best performance for the user's workload
Analyzes your logical schema, sample data, and sample queries
Minimizes DBA tuning
Run anytime for additional optimization, without stopping the database
22. HP-Vertica Features (5/6)
Massively Parallel Processing (MPP)
Parallel design leverages data projections to
enable distributed storage and workload
Active redundancy
Automatic replication, failover and recovery
23. HP-Vertica Features (6/6)
Native SQL and Application Integration
Standard SQL Interface
Simple integration with Hadoop and existing BI and ETL tools
Supports SQL, ODBC, JDBC and majority ETL and BI reporting products
24. HP-Vertica Bulk Upload Example (1/3)
HP Vertica 7.0, Single instance on VMware workstation v.11
Installed on Windows 8 machine HP-EliteBook 8440p, 8GB RAM
HP Vertica virtual machine has below specs:
Memory: 4GB
Disk: 16 GB
Processor: Single Processor Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz
25. HP-Vertica Bulk Upload Example (2/3)
Scenario Details
Upload a table with “|” delimited columns from a local text file
Table file size is: 1.30099 GB
No. of rows is: 10252864
No. of columns Is: 21
Table is partitioned into 10 partitions by using “customer_age“ column
Only one projection (the default “<table_name>_Super” )
26. HP-Vertica Bulk Upload Example (3/3)
Scenario Actions and Results
Upload by using “DIRECT” option to load directly to disk (ROS: Read Optimized
Storage)
COPY public.myTable from '/opt/vertica/examples/VMart_Schema/myTable.tbl'
DIRECT;
Time taken is more than 4 minutes
Upload without using “DIRECT” option to load to memory(WOS: Write Optimized
Storage)
COPY public.myTable from '/opt/vertica/examples/VMart_Schema/myTable.tbl';
Time taken is about 2.3 minutes
Source: http://www.slideshare.net/ibmcanada/ibm-canada-cloud-and-softlayer
SmarterBiz 1980 2000 Mainframe 2020 PC Internet Web 2.0 1 2 3 Computing Platform Generations Web 3.0Eras of IT Deployment (Illustrative) Cloud + Analytics, Mobile & Social leads to what IBM calls “The Era of Smart.” Mobile & Wireless “Things” Sensors, Phones, Tablets, etc. Cloud Social Analytics Big Data Data Collected and Stored Mobile The Era Of “Smart” IBM HorizonWatch: Technology Trends To Watch In 2014 Cognitive Computing “Watson”