4. TODAY A NEW SET OF QUESTIONS ARE BEING ASKED OF
THE BUSINESS:
What’s the social How do I better
sentiment for my predict future
brand or products outcomes?
How do I optimize
my fleet based on
weather and traffic
patterns?
10. NEW HARDWARE APPROACH
Traditional Big Data
Exotic HW Commodity HW
• Big central servers • racks of pizza boxes
• SAN • Ethernet
• RAID • JBOD
Hardware reliability Unreliable HW
Limited scalability Scales further
Cost effective
11. NEW SOFTWARE APPROACH
Traditional Big Data
Monolotic Distributed
• Centralized - storage & compute nodes
• RDBMS Raw data
Schema first
Proprietary
21. HIVE
A data warehouse infrastructure built on top of
Hadoop for providing data summarization, query, and
analysis.
– Ideal for ad hoc querying
– Query execution via MapReduce.
Key Building Principles:
– SQL
– Extensibility
– Types
– Functions
– Scripts
22. HIVE
It supports many SQL features like:
– Data partitioning
– Aggregations
– Grouping
– Joins
23. HIVE
And it’s extendable using UDFs.
package com.example.hive.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
public final class Lower extends UDF {
public Text evaluate(final Text s) {
if (s == null) { return null; }
return new Text(s.toString().toLowerCase());
}
}
There are many UDFs published by external parties, for:
- Loading / Saving (SerDe)
- Field Transformations
27. HADOOP PIG
data = LOAD 'employee.csv' USING PigStorage() AS (
first_name:chararray,
last_name:chararray,
age:int,
wage:float,
department:chararray
);
28. HADOOP PIG
grouped_by_department = GROUP data BY department;
total_wage_by_department =
FOREACH grouped_by_department
GENERATE
group AS department,
COUNT(data) as employee_count,
SUM(data::wage) AS total_wage;
total_ordered = ORDER total_wage_by_department BY total_wage;
total_limited = LIMIT total_ordered 10;
30. UDF
● Custom Load and Store classes.
● Hbase
● ProtocolBuffers
● CombinedLog
● Custom extraction
eg. date, ...
● Take a look at the PiggyBank.
31.
32. HBASE
A distributed, versioned, column-oriented
database.
• Main features:
• Horizontal scalability
• Machine failure tolerance
• Row-level atomic operations including compare-and-swap ops like
incrementing counters
• Augmented key-value schemas, the user can group columns into families which
are configured independently
• Multiple clients like its native Java library, Thrift, and REST
• Upcoming Security
42. DATA IS MORE THAN INFORMATION
Not all information is equal.
Some information is derived from other pieces of information.
43. DATA IS MORE THAN INFORMATION
Eventually you will reach the most ‘raw’
form of information.
This is the information you hold true, simple because it exists.
Let’s call this ‘data’, very similar to ‘event’.
44. EVENTS
Everything we do generates events:
• Pay with Credit Card
• Commit to Git
• Click on a webpage
• Tweet
50. CAPTURING CHANGE TRADITIONALLY
Person Location Person Location
Nathan Antwerp Nathan Ghent
Geert Dendermonde Geert Dendermonde
John Ghent John Ghent
51. CAPTURING CHANGE
Person Location Timestamp Person Location Time
Nathan Antwerp 2005-01-01
Nathan Antwerp 2005-01-01
Geert Dendermonde 2011-10-08
Geert Dendermonde 2011-10-08
John Ghent 2010-05-02
John Ghent 2010-05-02
Nathan Ghent 2013-02-03
52. QUERY
The data you query is often
transformed, aggregated, ...
Rarely used in it’s original form.
54. NUMBER OF PEOPLE LIVING IN EACH CITY.
Person Location Time Location Count
Nathan Antwerp 2005-01-01 Ghent 2
Dendermonde 1
Geert Dendermonde 2011-10-08
John Ghent 2010-05-02
Nathan Ghent 2013-02-03
67. MAPREDUCE
1. Take a large problem and divide it into sub-problems
…
MAP
2. Perform the same function on all sub-problems
…
DoWork() DoWork() DoWork()
3. Combine the output from all sub-problems
REDUCE
…
Output
75. SPEED LAYER
Storing a limited window of data.
Compensating for the last few hours of data.
76. SPEED LAYER
All the complexity is isolated in the
Speed layer. If anything goes wrong,
it’s auto-corrected.
77. CAP
You have a choice between:
• Availability
• Queries are eventual consistent.
• Consistency
• Queries are consistent.
78. EVENTUAL ACCURACY
Some algorithms are hard to
implement in real time. For those
cases we could estimate the results.
79. SPEED LAYER
Real
Time
View 1
Incoming Data
Real
Time
View 2
80. SPEED LAYER VIEWS
• The views are stored in Read & Write database.
• MS SQL Server
• Column Store
• Cassandra
• …
• Much more complex than a read only view.
87. OVERVIEW
SQL
Query
Incoming Data
HD Insight
Column
Store
88. LAMBDA ARCHITECTURE
• Can discard any view, batch and real time, and just recreate
everything from the master data.
• Mistakes are corrected via recomputation.
• Write bad data? Remove the data & recompute.
• Bug in view generation? Just recompute the view.
• Data storage is highly optimized.
91. INSIGHTS FROM ANY DATA, ANY SIZE, ANYWHERE
010101010101010101
1010101010101010
01010101010101
101010101010
92. WE DELIVER INSIGHTS TO EVERYONE BY ENABLING BIG DATA
ANALYSIS WITH FAMILIAR END USER TOOLS
Benefits
Interaction and analysis of
unstructured data in Hadoop
Key Features
Hive add-in for Excel
93. UNLOCKING IMMERSIVE INSIGHTS FROM ALL DATA
WITH MICROSOFT BI TOOLS
Benefits
Familiar self service BI tools
Key Features
Hive ODBC Driver integrates Hadoop
to SQL Server Analysis Services,
PowerPivot, and Power View
94. WHILE DRAMATICALLY SIMPLIFYING PROGRAMMING
ON HADOOP
MapReduce
programs
Benefits
in JavaScript
Simplified Simplified Deployment of
Programming MapReduce jobs
Key Features
JS
Deploy JavaScript Hadoop
Integration with .NET and jobs from a simple web
new JavaScript libraries for browser on any supported
Hadoop device
95. WE MANAGE STREAMING DATA WITH STREAMINSIGHT
Benefits
Key Features
StreamInsight SQL StreamInsight
97. WE MANAGE RELATIONAL DATA WITH MICROSOFT
ENTERPRISE DATA WAREHOUSE SOLUTIONS
Reference Architectures Appliances
Dell Parallel HP Enterprise
Data Data
Fast Track for Warehouse Warehouse
Dell
HP Business
Quickstart
Data
Data
Warehouse
Warehouse
98. INTRODUCING POLYBASE
Fundamental Breakthrough in Data Processing
Single Query; Structured and Unstructured
SQL
• Query and join Hadoop tables with Relational Tables
SQL Server 2012 • Use Standard SQL language
PDW Powered • Select, From Where
by PolyBase
Existing SQL No IT Save Time Analyze All
Skillset Intervention and Costs Data Types
99. AND SUPPORT UNSTRUCTURED DATA WITH ENTERPRISE
CLASS HADOOP ON PREMISE AND IN THE CLOUD
Benefits
Key Features
100. MICROSOFT BRINGS THE SIMPLICITY AND MANAGEABILITY
OF WINDOWS AND SQL SERVER TO HADOOP
Benefits
Key Features
101. MICROSOFT DELIVERS BIG DATA THROUGH OPEN
PLATFORM AND A RICH PARTNER ECOSYSTEM
Benefits
Key Features
102. BIG DATA DEMO:
FROM DATA TO INSIGHTS!
Analysis with familiar Collaboration on
Simplicity tools insights
104. RESOURCES
• Microsoft Big Data Solution: www.microsoft.com/bigdata
• Windows Azure: www.windowsazure.com/en-us/home/scenarios/big-data
• Try Now: https://www.hadooponazure.com
• HDInsight For Windows Beta Download: http://hortonworks.com/download/
• HDInsight Services For Windows:
http://social.technet.microsoft.com/wiki/contents/articles/6204.hdinsight-services-for-
windows.aspx#videos
• Hadoop in PowerPivot: http://social.technet.microsoft.com/wiki/contents/articles/6294.how-to-
connect-excel-powerpivot-to-hive-on-azure-via-hiveodbc.aspx
• Hadoop in SSIS: http://msdn.microsoft.com/en-us/library/jj720569.aspx
• Hurricane Sandy: http://sqlcat.com/sqlcat/b/msdnmirror/archive/2013/02/01/hurricane-sandy-
mash-up-hive-sql-server-powerpivot-amp-power-view.aspx
• Hadoop PowerShell: http://blogs.msdn.com/b/cindygross/archive/2012/08/23/how-to-install-the-
powershell-cmdlets-for-apache-hadoop-based-services-for-windows.aspx
• SQL Server BCP to Hive: http://blogs.msdn.com/b/cindygross/archive/2012/09/28/load-sql-server-
bcp-data-to-hive.aspx
• Internal vs External Table Hive: http://blogs.msdn.com/b/cindygross/archive/2013/02/06/hdinsight-
hive-internal-and-external-tables-intro.aspx
• Microsoft.NET SDK for Hadoop: http://hadoopsdk.codeplex.com/
• Twitter Analytics Example: http://twitterbigdata.codeplex.com/
105. DATACRUNCHERS
We enable companies in envisioning, defining and implementing a data
strategy.
A one-stop-shop for all your Big Data needs.
The first Big Data Consultancy agency in Belgium.
