The document discusses the rise of big data and how companies are leveraging it. It covers key aspects of big data including the volume, velocity, and variety of data. Hadoop and MapReduce are presented as important technologies for processing and analyzing large datasets in a distributed manner. Case studies of Google and how it uses big data are provided. Related technologies like HDFS, Hive, Pig, and HBase are also overviewed along with how they fit into the Hadoop ecosystem. The value of extracting insights from big data through machine learning, collective intelligence, and predictive analytics is highlighted.

1. Hadoop, Oracle and the industrial
revolution of data
Guy Harrison
VP R&D, Database Management
© 2012 Quest Software Inc. All rights reserved.

2. Hadoop, Oracle and the
industrial revolution of data
Guy Harrison
Executive Director,
R&D Business
Intelligence
Software

3. Introductions
www.guyharrison.net
guy.harrison@quest.com
http://twitter.com/guyharrison
© 2012 Quest Software Inc. All rights reserved. Pg. 3

4. Quest
© 2012 Quest Software Inc. All rights reserved. Pg. 4

5. © 2012 Quest Software Inc. All rights reserved. Pg. 5

6. © 2012 Quest Software Inc. All rights reserved. Pg. 6

7. © 2012 Quest Software Inc. All rights reserved. Pg. 7

9. © 2012 Quest Software Inc. All rights reserved. Pg. 9

10. © 2012 Quest Software Inc. All rights reserved. Pg. 10

11. Star trek shirt fatality analysis
Red
Yellow
Blue
0 10 20 30 40 50 60 70 80
Pct
© 2012 Quest Software Inc. All rights reserved. Pg. 11

12. © 2012 Quest Software Inc. All rights reserved. Pg. 12

13. © 2012 Quest Software Inc. All rights reserved. Pg. 13

14. What is Big Data?
© 2012 Quest Software Inc. All rights reserved. Pg. 14

15. Value
The 3-4 V’s Competitive or Community
advantage
Volume Variety
Terabytes Structured
Petabytes Unstructured
Exabytes Human Generated
Zetabytes Machine Generated
Velocity
User populations x
Transaction rates x
Machine data
© 2012 Quest Software Inc. All rights reserved. Pg. 15

16. Volume
 Data volumes have always been increasing
2006 Perspective
© 2012 Quest Software Inc. All rights reserved. Pg. 16

17. But the vastness is becoming mind boggling
Digital information created 2011
2.13E+21
Total Digital capacity
1.18E+21
Digital information 2008
4.87E+18
Living Human Genomes
5.48E+18
Google
1.10E+17
Human Brain
2.81E+15
1.00E+09 1.00E+10 1.00E+11 1.00E+12 1.00E+13 1.00E+14 1.00E+15 1.00E+16 1.00E+17 1.00E+18 1.00E+19 1.00E+20 1.00E+21 1.00E+22
Gigabyte Terabyte Petabyte Exabyte zettabyte
© 2012 Quest Software Inc. All rights reserved. Pg. 17

18. Velocity
© 2012 Quest Software Inc. All rights reserved. Pg. 18

19. Fail whales
© 2012 Quest Software Inc. All rights reserved. Pg. 19

20. Variety
The Industrial Revolution of Data
© 2012 Quest Software Inc. All rights reserved. Pg. 20

21. © 2012 Quest Software Inc. All rights reserved. Pg. 21

22. © 2012 Quest Software Inc. All rights reserved. Pg. 22

23. Big Data is driven by the smallest devices
© 2012 Quest Software Inc. All rights reserved. Pg. 23

24. Samsung Galaxy S IIII specifications
 Quad-core 1.4 GHz CPU
 1GB RAM
 64GB Storage
 1080p display
 GSM/Bluetooth/WiFi Network
 8MP Camera
 GPS & Compass
© 2012 Quest Software Inc. All rights reserved. Pg. 24

25. © 2012 Quest Software Inc. All rights reserved. Pg. 25

26. © 2012 Quest Software Inc. All rights reserved. Pg. 26

27. © 2012 Quest Software Inc. All rights reserved. Pg. 27

28. © 2012 Quest Software Inc. All rights reserved. Pg. 28

29. © 2012 Quest Software Inc. All rights reserved. Pg. 29

30. © 2012 Quest Software Inc. All rights reserved. Pg. 30

31. © 2012 Quest Software Inc. All rights reserved. Pg. 31

32. © 2012 Quest Software Inc. All rights reserved. Pg. 32

33. © 2012 Quest Software Inc. All rights reserved. Pg. 33

34. © 2012 Quest Software Inc. All rights reserved. Pg. 34

35. Name: Willy
Bowman
Nationality: German
DON‟T MENTION
THE WAR
35

36. Data Input
© 2012 Quest Software Inc. All rights reserved. Pg. 36

37. © 2012 Quest Software Inc. All rights reserved. Pg. 37

38. Siri
“Siri call me an “I want to jump off a
ambulance” bridge”
From now on, I‟ll call you
„An Ambulance‟. OK? I found 14 bridges nearby:

39. © 2012 Quest Software Inc. All rights reserved. Pg. 39

40. © 2012 Quest Software Inc. All rights reserved. Pg. 40

41. Brain Control
© 2012 Quest Software Inc. All rights reserved. Pg. 41

42. © 2012 Quest Software Inc. All rights reserved. Pg. 42

43. © 2012 Quest Software Inc. All rights reserved. Pg. 43

44. © 2012 Quest Software Inc. All rights reserved. Pg. 44

45. © 2012 Quest Software Inc. All rights reserved. Pg. 45

46. © 2012 Quest Software Inc. All rights reserved. Pg. 46

47. All of this requires and Generates Big Datasets
But what are they good for?
© 2012 Quest Software Inc. All rights reserved. Pg. 47

48. Value?
Achieve competitive advantage
From Big Data using
Collective Intelligence,
Machine Learning
and Predictive Analytics
© 2012 Quest Software Inc. All rights reserved. Pg. 48

49. Big Data Analytics
How do we derive
value from the data?
Machine Collective
Learning Intelligence
Programs that Programs that use
evolve with inputs from “crowds‟
“experience” to seem intelligent
Predictive
Analytics
Programs that
extrapolate from
existing data into
the future

50. © 2012 Quest Software Inc. All rights reserved. Pg. 50

51. © 2012 Quest Software Inc. All rights reserved. Pg. 51

52. © 2012 Quest Software Inc. All rights reserved. Pg. 52

53. © 2012 Quest Software Inc. All rights reserved. Pg. 53

54. © 2012 Quest Software Inc. All rights reserved. Pg. 54

55. © 2012 Quest Software Inc. All rights reserved. Pg. 55

56. © 2012 Quest Software Inc. All rights reserved. Pg. 56

57. © 2012 Quest Software Inc. All rights reserved. Pg. 57

58. © 2012 Quest Software Inc. All rights reserved. Pg. 58

59. © 2012 Quest Software Inc. All rights reserved. Pg. 59

60. © 2012 Quest Software Inc. All rights reserved. Pg. 60

61. Applications Search Optimization
Advertising
Recommendation
• Targeting Systems
• Tailoring
Security
Game optimization
Collective
• Vulnerability
Intelligence • Penetration Detection
Medical
• Risk analysis Fraud Detection
• Diagnosis
• Prognosis
Predictive Analytics
• Churn
• Defaults
© 2012 Quest Software Inc. All rights reserved. Pg. 61

62. Collective Intelligence beats Artificial Intelligence
?
© 2012 Quest Software Inc. All rights reserved. Pg. 62

63. © 2012 Quest Software Inc. All rights reserved. Pg. 63

64. © 2012 Quest Software Inc. All rights reserved. Pg. 64

65. © 2012 Quest Software Inc. All rights reserved. Pg. 65

66. © 2012 Quest Software Inc. All rights reserved. Pg. 66

67. © 2012 Quest Software Inc. All rights reserved. Pg. 67

68. For the past 40 years, AI has been consistently
disappointing
© 2012 Quest Software Inc. All rights reserved. Pg. 68

69. © 2012 Quest Software Inc. All rights reserved. Pg. 69

70. © 2012 Quest Software Inc. All rights reserved. Pg. 70

71. © 2012 Quest Software Inc. All rights reserved. Pg. 71

72. © 2012 Quest Software Inc. All rights reserved. Pg. 72

73. © 2012 Quest Software Inc. All rights reserved. Pg. 73

74. © 2012 Quest Software Inc. All rights reserved. Pg. 74

75. © 2012 Quest Software Inc. All rights reserved. Pg. 75

76. © 2012 Quest Software Inc. All rights reserved. Pg. 76

77. © 2012 Quest Software Inc. All rights reserved. Pg. 77

78. Google: pioneers of big data
© 2012 Quest Software Inc. All rights reserved. Pg. 78

79. © 2012 Quest Software Inc. All rights reserved. Pg. 79

80. © 2012 Quest Software Inc. All rights reserved. Pg. 80

81. © 2012 Quest Software Inc. All rights reserved. Pg. 81

82. © 2012 Quest Software Inc. All rights reserved. Pg. 82

83. Google Software Architecture
Google Applications
Map Reduce Chubby BigTable
Google File System (GFS)
© 2012 Quest Software Inc. All rights reserved. Pg. 83

84. Map Reduce
MAP
MAP
MAP
MAP
MAP
MAP
MAP
MAP
MAP
MAP
START MAP REDUCE
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP MAP
MAP
MAP
© 2012 Quest Software Inc. All rights reserved. Pg. 84

85. Multi-stage Map-Reduce
SORT AGGREGATE SCAN
MAPPER MAPPER MAPPER
MAPPER MAPPER MAPPER
CLIENT
REDUCE
HDFS
MAPPER MAPPER MAPPER
MAPPER MAPPER MAPPER
© 2012 Quest Software Inc. All rights reserved. Pg. 85

86. Hadoop: Open Source Map-Reduce Stack
© 2012 Quest Software Inc. All rights reserved. Pg. 86

87. Hadoop at Yahoo!
 Yahoo! Hadoop cluster:
− 4000 nodes
− 16PB disk
− 64 TB of RAM
− 32,000 Cores
© 2012 Quest Software Inc. All rights reserved. Pg. 87

88. © 2012 Quest Software Inc. All rights reserved. Pg. 88

89. Hadoop MAP REDUCE
(DISTRIBUTED
HADOOP CLIENT
(JAVA, PIG, HIVE)
Architecture PROCESSING)
(1.0) HDFS (DISTRIBUTED
STORAGE)
JOB TRACKER NAME NODE SECONDARY NAME
NODE
DATA NODE TASK DATA NODE TASK DATA NODE TASK
TRACKER TRACKER TRACKER
DATA NODE TASK DATA NODE TASK DATA NODE TASK
TRACKER TRACKER TRACKER
DATA NODE TASK DATA NODE TASK DATA NODE TASK
TRACKER TRACKER TRACKER
DATA NODE TASK DATA NODE TASK DATA NODE TASK
TRACKER TRACKER TRACKER
© 2012 Quest Software Inc. All rights reserved. Pg. 89

90. Schema on Read vs Schema on Write
© 2012 Quest Software Inc. All rights reserved. Pg. 90

91. Schema on Write
Code Analyse
Data Extract Transform Load Utilize
Cleanse Aggregate Data Warehouse
Normalize
Schema on Read
Code Analyse
Data Load Utilize
Hadoop
Cleanse
© 2012 Quest Software Inc. All rights reserved. Pg. 91

92. Hadoop Oozie (Workflow manager)
Ecosystem
Hive Pig SQOOP Flume
(Query) (Scripting) (RDBMS loader) (Log Loader)
ZooKeeper Hbase
Hadoop Map Reduce
(Locking) (Database)
Hadoop File System (HDFS)
© 2012 Quest Software Inc. All rights reserved. Pg. 92

93. HBase
© 2012 Quest Software Inc. All rights reserved. Pg. 93

94. HBase
HBase is a real-time database built on Hadoop
Log
Buffer Cache MemStore
Buffer
Table Table Table Table
Datafiles Redo HFile HFile WA Log
ASM HDFS
Disks Disks
© 2012 Quest Software Inc. All rights reserved. Pg. 94

95. Hbase Data Model
Name Site Counter NameId Name SiteId SiteName
Dick Ebay 507,018 1 Dick 1 Ebay
Dick Google 690,414 2 Jane 2 Google
Jane Google 716,426 3 Facebook
Dick Facebook 723,649 4 ILoveLarry.com
Jane Facebook 643,261 5 MadBillFans.com
Jane ILoveLarry.com 856,767
Dick MadBillFans.com 675,230
NameId SiteId Counter
1 1 507,018
1 3 690,414
2 3 716,426
1 3 723,649
2 3 643,261
2 4 856,767
1 5 675,230
Id Name Ebay Google Facebook (other columns) MadBillFans.com
1 Dick 507,018 690,414 723,649 . . . . . . . . . . . . . . 675,230
Id Name Google Facebook (other columns) ILoveLarry.com
2 Jane 716,426 643,261 . . . . . . . . . . . . . . 856,767

96. Hive
© 2012 Quest Software Inc. All rights reserved. Pg. 96

97. © 2012 Quest Software Inc. All rights reserved. Pg. 97

98. SQL
JAVA
Results
© 2012 Quest Software Inc. All rights reserved. Pg. 98

99. Pig
© 2012 Quest Software Inc. All rights reserved. Pg. 99

100. Pig Latin
SQL or Hive QL
© 2012 Quest Software Inc. All rights reserved. Pg. 100

101. Meanwhile, back at the Death Star….
© 2012 Quest Software Inc. All rights reserved. Pg. 101

103. © 2012 Quest Software Inc. All rights reserved. Pg. 103

104. Oracle Exadata
Database servers Storage Servers
64 cores, 576 GB RAM 112 cores,
100 TB SAS or
336 TB SATA plus
5 TB SSD
© 2012 Quest Software Inc. All rights reserved. Pg. 104

105. © 2012 Quest Software Inc. All rights reserved.

106. © 2012 Quest Software Inc. All rights reserved. Pg. 106

107. © 2012 Quest Software Inc. All rights reserved. Pg. 107

108. Oracle Big Data Appliance
 18 Sun X4270 M2 servers
− 48GB RAM per node (864GB total)
− 2x6 Core CPU per node (216 total)
− 12x2TB HDD per node (216 spindles, 864 TB)
− 40Gb/s Infiniband between nodes
− 10Gb/s Ethernet to datacentre
 Competitive Pricing
www.oracle.com/us/bigdata/index.html
© 2012 Quest Software Inc. All rights reserved. Pg. 108

109. Big Data Appliance Software
 Cloudera Enterprise
 Oracle Enterprise R
 Oracle NoSQL
 Oracle Big Data
Connectors
© 2012 Quest Software Inc. All rights reserved. Pg. 109

110. Latency
Oracle’s ORACLE
BIG DATA
ORACLE
EXALOGIC
ORACLE
EXALYTICS
Storage APPLIANCE
Hierarchy
ORACLE
WEBLOGIC
ORACLE ORACLE
NOSQL ESSBASE
ORACLE
ORACLE
EXADATA
LOADER
FOR
HADOOP
APACHE
ORACLE
HADOOP ORACLE
RDBMS
TIMES TEN
Storage Costs
© 2012 Quest Software Inc. All rights reserved. Pg. 110

111. 111
© 2012 Quest Software Inc. All rights reserved. Pg. 111

112. © 2012 Quest Software Inc. All rights reserved. Pg. 112

113. Hadoop and RDBMS integration
© 2012 Quest Software Inc. All rights reserved. Pg. 113

114. Scenario #1: Reference data in RDBMS
PRODUCTS
CUSTOMERS
HDFS
WEBlOGS
© 2012 Quest Software Inc. All rights reserved.
RDBMS Pg. 114

115. Scenario #2: Hadoop for off-line analytics
PRODUCTS
CUSTOMERS
HDFS SALES
HISTORY
© 2012 Quest Software Inc. All rights reserved. RDBMS Pg. 115

116. Scenario #3: MapReduce output to
RDBMS DB QUERY
TOOL
WEBLOGS
SUMMARY
HDFS
WEBLOGS
© 2012 Quest Software Inc. All rights reserved. RDBMS Pg. 116

117. Scenario #4: Hadoop as RDBMS “active archive”
QUERY
TOOL
SALES 2011
SALES 2010
SALES 2009 SALES 2009
SALES 2008 SALES 2008
HDFS
© 2012 Quest Software Inc. All rights reserved. RDBMS Pg. 117

118. The Big Data Stack
© 2012 Quest Software Inc. All rights reserved. Pg. 118

119. The Big Data Stack
DATA SCIENTIST
CASCADING R (ET AL)
PIG MAHOUT
JAVA API JAVA API
HIVE
MAP-REDUCE HBASE
HDFS

121. The Big Data Stack
BIG DATA ANALAYTIC PLATFORM
DATA SCIENTIST
CASCADING R (ET AL)
PIG MAHOUT
JAVA API JAVA API
HIVE
MAP-REDUCE HBASE
HDFS

122. Big Data Analytics Platform
INDEXING AND
SEARCH
SENTIMENT
ANALYSIS VISUALIZATION
BASKET
ANALYSIS
RECOMMENDERS
BIG DATA
ANALYTICS
ADVERTISING
CLUSTERING
OPTIMIZATION
CLASSIFICATION
EXPERT SYSTEMS (LIKE
WATSON)

123. In Summary
© 2012 Quest Software Inc. All rights reserved. Pg. 123

124. Hadoop is….
© 2012 Quest Software Inc. All rights reserved. Pg. 124

125. © 2012 Quest Software Inc. All rights reserved.

126. Scalable
• 4000 nodes at Yahoo!
• >100 PB at Facebook
• 10,000 node design
goal for Hadoop 2.0
© 2012 Quest Software Inc. All rights reserved. Pg. 126

127. A platform for AI, CI & analytics
© 2012 Quest Software Inc. All rights reserved. Pg. 127

128. ETL “Free”
Schema on Write
Code Analyse
Data Extract Transform Load Utilize
Cleanse Aggregate Data Warehouse
Normalize
Schema on Read
Code Analyse
Data Load Utilize
Hadoop
Cleanse
© 2012 Quest Software Inc. All rights reserved. Pg. 128

129. The most concrete technology enabling the Big Data
revolution
© 2012 Quest Software Inc. All rights reserved. Pg. 129

130. Hadoop is not….
© 2012 Quest Software Inc. All rights reserved. Pg. 130

131. A replacement for RDBMS
But future Enterprise Data Architectures will likely incorporate Hadoop side by
side with RDBMS
© 2012 Quest Software Inc. All rights reserved. Pg. 131

132. Suitable for OLTP
Though OLTP systems can be built with Hadoop-compatible NoSQL systems such
as HBase and Cassandra
© 2012 Quest Software Inc. All rights reserved. Pg. 132

133. A complete solution
Hadoop alone only solves the storage challenge of Big Data
© 2012 Quest Software Inc. All rights reserved. Pg. 133

134. Shameless plugs
© 2012 Quest Software Inc. All rights reserved. Pg. 134

136. Toad for Cloud
Databases
Work with
Hive, Hbase, Oracle, SQ
L
Server, Cassandra, MyS
QL, MongoDB, BI
servers and other NoSQL
© 2012 Quest Software Inc. All rights reserved. Pg. 136
and SQL datastores

137. Toad for Cloud
Databases
Toad for Cloud Databases
• Federated SQL queries across
Hive, Hbase, NoSQL, RDBMS
© 2012 Quest Software Inc. All rights reserved. Pg. 137

138. © 2012 Quest Software Inc. All rights reserved.

139. Toad BI Suite
Business Intelligence solutions
with first class support for
Hadoop, Oracle and many other
platforms
© 2012 Quest Software Inc. All rights reserved. Pg. 139

140. SharePlex® for Hadoop
Hadoop
JMS Queue
Poster
Change Data
Capture
Audit / Change
Redo-logs
Data
Batched
HDFS HBase Real
File Copy Time replication
© 2012 Quest Software Inc. All rights reserved. Pg. 140

141. Toad for Hadoop
• Hive Query IDE
• Oracle <-> Hadoop data management
• Basic Hadoop administration
• ETA beta H1 2013
© 2012 Quest Software Inc. All rights reserved. Pg. 141

143. © 2012 Quest Software Inc. All rights reserved. Pg. 143

144. Summary:
 The future belongs to those of us prepared to wear funny hats and
glasses
 The connected and mobile internet requires and produces “big
data” that is qualitatively different from the data we’ve had before
− Requiring different types of datastores
 Enterprise can leverage big data for competitive advantage
− Requiring different types of analytical engines
© 2012 Quest Software Inc. All rights reserved. Pg. 144

145. Thank You
guy.harrison@quest.com
www.guyharrison.net
@guyharrison
© 2012 Quest Software Inc. All rights reserved. Pg. 145