Notes available upon request

1. Anthony Nyström
Fellow, Managing Director of Engineering

3. What is Intridea?

4. What is Intridea?
We design and
develop apps:
Web, Mobile and Data

5. What is Intridea?
We design and
develop apps:
Web, Mobile and Data
Founded in
Washington, DC

6. What is Intridea?
We design and
We work with cool develop apps:
clients – really! Web, Mobile and Data
Founded in
Washington, DC

7. What is Intridea?
We design and
We work with cool develop apps:
clients – really! Web, Mobile and Data
Founded in
Washington, DC
40+ Intrideans:
Designers/Developers/Scientists
+ Smart biz folks

8. What is Intridea?
We design and
We work with cool develop apps:
clients – really! Web, Mobile and Data
We work from anywhere!
Founded in
Washington, DC
40+ Intrideans:
Designers/Developers/Scientists
+ Smart biz folks

9. What is Intridea?
We design and
We work with cool develop apps:
clients – really! Web, Mobile and Data
We work from anywhere!
Founded in
Washington, DC
40+ Intrideans:
Designers/Developers/Scientists
+ Smart biz folks
We are growing

10. What is Intridea?
We design and
We work with cool develop apps:
clients – really! Web, Mobile and Data
We work from anywhere!
Founded in
Washington, DC
40+ Intrideans:
Designers/Developers/Scientists
+ Smart biz folks We hire the best and
We are growing the smartest

11. Anthony Nyström
Fellow, Managing Director of Engineering

12. Intridean:
The guy on stage
Anthony Nyström
Fellow, Managing Director of Engineering

14. Data Science in the NOW!
It takes an army of TOOLS

16. An Army of Tools you say?

17. An Army of Tools you say?
• I am going to talk about what NOW means in Data Science
• Databases, Streaming Engines, Query Engines and Interfaces
• We are going to look at many of them and single out a few
• Each has a respected and in some cases competing set of
features

19. Then was NOW as NOW was Then

20. Then was NOW as NOW was Then
Now is indeed Then

21. Then was NOW as NOW was Then
Now is indeed Then

22. Then was NOW as NOW was Then
Now is indeed Then
Then is indeed Now

24. Why is NOW in data Special?

25. Why is NOW in data Special?
Actionable Intelligence & Knowledge

26. Why is NOW in data Special?
Actionable Intelligence & Knowledge
NOW has innate context

27. Why is NOW in data Special?
Actionable Intelligence & Knowledge
NOW has innate context
TIME is THE natural facet for our minds &
life!

29. Why is NOW in data Special?

30. Why is NOW in data Special?
Trends | Patterns | Extraction

31. Why is NOW in data Special?
Trends | Patterns | Extraction
Data Centric Trends

32. Why is NOW in data Special?
Trends | Patterns | Extraction
Data Centric Trends
Pattern Extraction (ML/NLP)

33. Why is NOW in data Special?
Trends | Patterns | Extraction
Data Centric Trends
Pattern Extraction (ML/NLP)
Signature Extraction (Binary, Encoded)

34. Why is NOW in data Special?
Trends | Patterns | Extraction
Data Centric Trends
Not user input data like Google, Yahoo etc.
Pattern Extraction (ML/NLP)
Signature Extraction (Binary, Encoded)

35. Why is NOW in data Special?
Trends | Patterns | Extraction
Data Centric Trends
Not user input data like Google, Yahoo etc.
Pattern Extraction (ML/NLP)
“I am looking for data that conforms to a learned or known pattern”
Signature Extraction (Binary, Encoded)

36. Why is NOW in data Special?
Trends | Patterns | Extraction
Data Centric Trends
Not user input data like Google, Yahoo etc.
Pattern Extraction (ML/NLP)
“I am looking for data that conforms to a learned or known pattern”
Signature Extraction (Binary, Encoded)
“I am looking for data that matches a predefined signature”

38. Why is NOW in data Special?

39. Why is NOW in data Special?
Routing | Transformation | Computation

40. Why is NOW in data Special?
Routing | Transformation | Computation
Intelligent Routing

41. Why is NOW in data Special?
Routing | Transformation | Computation
Intelligent Routing
Transformation & Computation

42. Why is NOW in data Special?
Routing | Transformation | Computation
Intelligent Routing
“I need to replicate/fork that of criteria x portions of this data
stream”
Transformation & Computation

43. Why is NOW in data Special?
Routing | Transformation | Computation
Intelligent Routing
“I need to replicate/fork that of criteria x portions of this data
stream”
Transformation & Computation
“I need to transform certain fields” or “I need to compute
a some value on certain fields”

45. Why is NOW in data Special?

46. Why is NOW in data Special?
Algorithmic Speciality

47. Why is NOW in data Special?
Algorithmic Speciality
Concepts

48. Why is NOW in data Special?
Algorithmic Speciality
Concepts
Regression

49. Why is NOW in data Special?
Algorithmic Speciality
Concepts
Regression
Relationships

50. Why is NOW in data Special?
Algorithmic Speciality
Concepts
What does a value represent or infer (NLP/ML/k-NN)
Regression
Relationships

51. Why is NOW in data Special?
Algorithmic Speciality
Concepts
What does a value represent or infer (NLP/ML/k-NN)
Regression
How is a value related to another value or
How can we predict such relations
Relationships

52. Why is NOW in data Special?
Algorithmic Speciality
Concepts
What does a value represent or infer (NLP/ML/k-NN)
Regression
How is a value related to another value or
How can we predict such relations
Relationships
Topological, Ontological, Forest
(Evolutionary/Random) (NLP)

54. When NOW matters!

55. When NOW matters!
Industry/Vertical

56. When NOW matters!
Industry/Vertical
Medical
Algorithms are the new medical tests
Scientific
Eco, Bio, Geo
Financial
Stocks, Actuary Science

60. Point of Sale System
• Terminal
• Admin
• Tablet

62. Merck
• RT Persona
• RT Data
• Browser

64. Where is NOW in data?

65. Where is NOW in data?
Data Creation Time | Data Consumption Time

67. Latency

68. Latency
Data Creation Time | Data Consumption Time

69. Latency
Data Creation Time | Data Consumption Time
Standard - NOPE!

70. Latency
Data Creation Time | Data Consumption Time
Standard - NOPE!
Depends upon the Medium - YEP!

71. Latency
Data Creation Time | Data Consumption Time
Standard - NOPE!
Depends upon the Medium - YEP!
Depends upon the Consumer - YEP!

72. Latency
Data Creation Time | Data Consumption Time
Standard - NOPE!
Depends upon the Medium - YEP!
Depends upon the Consumer - YEP!
Depends upon Technology - YEP!

74. NOW and Latency

75. NOW and Latency
Real-Time

76. NOW and Latency
Real-Time
Near Real-Time

77. NOW and Latency
Real-Time
Near Real-Time
Some-Time

78. NOW and Latency
Real-Time
Data that is consumed immediately after creation
Near Real-Time
Some-Time

79. NOW and Latency
Real-Time
Data that is consumed immediately after creation
Near Real-Time
Data is consumed within seconds/minutes
Some-Time

80. NOW and Latency
Real-Time
Data that is consumed immediately after creation
Near Real-Time
Data is consumed within seconds/minutes
Some-Time
Data is consumed when requested & is NOT RT nor NRT

82. Physiological Latency

83. Physiological Latency
Perception:
Research suggests that the human retina transmits data to the brain at the
rate of 10 million bits per second, which is close to that of 10 base Ethernet
connection!
We can perceive changes in reality at ~ 13-15 frames per second (fps, or
Hz), Our perception of reality fully refreshes itself ~ once every 77

84. Physiological Latency
Perception:
Research suggests that the human retina transmits data to the brain at the
rate of 10 million bits per second, which is close to that of 10 base Ethernet
connection!
We can perceive changes in reality at ~ 13-15 frames per second (fps, or
Hz), Our perception of reality fully refreshes itself ~ once every 77
Stock Exchange ~ 5-100 milliseconds (ms)

85. Physiological Latency
Perception:
Research suggests that the human retina transmits data to the brain at the
rate of 10 million bits per second, which is close to that of 10 base Ethernet
connection!
We can perceive changes in reality at ~ 13-15 frames per second (fps, or
Hz), Our perception of reality fully refreshes itself ~ once every 77
Stock Exchange ~ 5-100 milliseconds (ms)
Web Sites ~ 50-400 milliseconds (ms)

86. Physiological Latency
Perception:
Research suggests that the human retina transmits data to the brain at the
rate of 10 million bits per second, which is close to that of 10 base Ethernet
connection!
We can perceive changes in reality at ~ 13-15 frames per second (fps, or
Hz), Our perception of reality fully refreshes itself ~ once every 77
Stock Exchange ~ 5-100 milliseconds (ms)
Web Sites ~ 50-400 milliseconds (ms)
Games (FPS) ~ 10-150 milliseconds (ms)

87. Physiological Latency
Perception:
Research suggests that the human retina transmits data to the brain at the
rate of 10 million bits per second, which is close to that of 10 base Ethernet
connection!
We can perceive changes in reality at ~ 13-15 frames per second (fps, or
Hz), Our perception of reality fully refreshes itself ~ once every 77
Stock Exchange ~ 5-100 milliseconds (ms)
Web Sites ~ 50-400 milliseconds (ms)
Games (FPS) ~ 10-150 milliseconds (ms)
Social/Games ~ 200 ms -1 second

89. Let’s talk about TOOLS!

90. Let’s talk about TOOLS!
Real or Near Real-Time (DB’s, Index’s, FS’s)

91. Let’s talk about TOOLS!
Real or Near Real-Time (DB’s, Index’s, FS’s)
Real or Near Real-Time (nRTQE’s)

92. Let’s talk about TOOLS!
Real or Near Real-Time (DB’s, Index’s, FS’s)
Real or Near Real-Time (nRTQE’s)
Real or Near Real-Time (nRTSE’s)

94. Real-Time (DB’s, Index’s, FS’s)

95. Real-Time (DB’s, Index’s, FS’s)
No particular order

96. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL

97. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL
• SQL Server

98. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL
• SQL Server
• PostgreSQL

99. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL
• SQL Server
• PostgreSQL
• Neo4j (Graph)

100. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL
• SQL Server
• PostgreSQL
• Neo4j (Graph)
• Mongo

101. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL
• SQL Server
• PostgreSQL
• Neo4j (Graph)
• Mongo
• Elastic Search (Lucene)

102. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL
• SQL Server
• PostgreSQL
• Neo4j (Graph)
• Mongo
• Elastic Search (Lucene)
• Solr

103. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server
• PostgreSQL
• Neo4j (Graph)
• Mongo
• Elastic Search (Lucene)
• Solr

104. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server • HBase
• PostgreSQL
• Neo4j (Graph)
• Mongo
• Elastic Search (Lucene)
• Solr

105. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server • HBase
• PostgreSQL • Oracle
• Neo4j (Graph)
• Mongo
• Elastic Search (Lucene)
• Solr

106. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server • HBase
• PostgreSQL • Oracle
• Neo4j (Graph) • ERTFS
• Mongo
• Elastic Search (Lucene)
• Solr

107. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server • HBase
• PostgreSQL • Oracle
• Neo4j (Graph) • ERTFS
• Mongo • Redis
• Elastic Search (Lucene)
• Solr

108. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server • HBase
• PostgreSQL • Oracle
• Neo4j (Graph) • ERTFS
• Mongo • Redis
• Elastic Search (Lucene) • Cassandra
• Solr

109. Real-Time (DB’s, Index’s, FS’s)
No particular order
• MySQL • HDFS
• SQL Server • HBase
• PostgreSQL • Oracle
• Neo4j (Graph) • ERTFS
• Mongo • Redis
• Elastic Search (Lucene) • Cassandra
• Solr • Riak

111. HBase

112. HBase
Regions and HDFS

113. HBase
Regions and HDFS
Scaling

114. HBase
Regions and HDFS
Scaling
Hadoop

115. HBase
Regions and HDFS
“Regions” Data files for regions are stored in HDFS and replicated to
multiple nodes in the cluster. As well, allocation in to the cluster is
rather automatic
Scaling
Hadoop

116. HBase
Regions and HDFS
“Regions” Data files for regions are stored in HDFS and replicated to
multiple nodes in the cluster. As well, allocation in to the cluster is
rather automatic
Scaling
Fault Tolerance
Commodity Machines
Hadoop

117. HBase
Regions and HDFS
“Regions” Data files for regions are stored in HDFS and replicated to
multiple nodes in the cluster. As well, allocation in to the cluster is
rather automatic
Scaling
Fault Tolerance
Commodity Machines
Hadoop
Runs on top of Hadoop
MapReduce Integration

119. Cassandra

120. Cassandra
Always Writable

121. Cassandra
Always Writable
Scaling

122. Cassandra
Always Writable
Scaling
More...

123. Cassandra
Always Writable
Even when internally the write fails. However, the data will eventually
become consistent (Tunable)
Scaling
More...

124. Cassandra
Always Writable
Even when internally the write fails. However, the data will eventually
become consistent (Tunable)
Scaling
Can span data centers
Peer-to-Peer communication between nodes (Gossip)
More...

125. Cassandra
Always Writable
Even when internally the write fails. However, the data will eventually
become consistent (Tunable)
Scaling
Can span data centers
Peer-to-Peer communication between nodes (Gossip)
More...
Supports MapReduce
Supports Range Queries

127. Redis

128. Redis
Transactions

129. Redis
Transactions
An evolutionary Key-Value Store

130. Redis
Transactions
An evolutionary Key-Value Store
Pub-Sub

131. Redis
Transactions
Atomic operations (MULTI/EXEC/Discard) Queue your operations and
EXEC/Commit as transaction. Allows for Roll-back support.
An evolutionary Key-Value Store
Pub-Sub

132. Redis
Transactions
Atomic operations (MULTI/EXEC/Discard) Queue your operations and
EXEC/Commit as transaction. Allows for Roll-back support.
An evolutionary Key-Value Store
Supports complex types that are closely related to fundamental data
structures. No need for abstraction layer.
Pub-Sub

133. Redis
Transactions
Atomic operations (MULTI/EXEC/Discard) Queue your operations and
EXEC/Commit as transaction. Allows for Roll-back support.
An evolutionary Key-Value Store
Supports complex types that are closely related to fundamental data
structures. No need for abstraction layer.
Pub-Sub
Publish - Push messages to a channel
Subscribe - Listen to a channel

135. Near Real-Time & Real-Time

136. Near Real-Time & Real-Time
Queries and Streams

137. Near Real-Time & Real-Time
Queries and Streams
• Storm

138. Near Real-Time & Real-Time
Queries and Streams
• Storm
• Kafka

139. Near Real-Time & Real-Time
Queries and Streams
• Storm
• Kafka
• Drill/Dremel

140. Near Real-Time & Real-Time
Queries and Streams
• Storm
• Kafka
• Drill/Dremel
• Hadoop

141. Near Real-Time & Real-Time
Queries and Streams
• Storm
• Kafka
• Drill/Dremel
• Hadoop
• MapReduce

142. Near Real-Time & Real-Time
Queries and Streams
• Storm • MapReduce v2 (YARN)
• Kafka
• Drill/Dremel
• Hadoop
• MapReduce

143. Near Real-Time & Real-Time
Queries and Streams
• Storm • MapReduce v2 (YARN)
• Kafka • Pig
• Drill/Dremel
• Hadoop
• MapReduce

144. Near Real-Time & Real-Time
Queries and Streams
• Storm • MapReduce v2 (YARN)
• Kafka • Pig
• Drill/Dremel • Hive
• Hadoop
• MapReduce

145. Near Real-Time & Real-Time
Queries and Streams
• Storm • MapReduce v2 (YARN)
• Kafka • Pig
• Drill/Dremel • Hive
• Hadoop • Cascalog
• MapReduce

146. Near Real-Time & Real-Time
Queries and Streams
• Storm • MapReduce v2 (YARN)
• Kafka • Pig
• Drill/Dremel • Hive
• Hadoop • Cascalog
• MapReduce • DataTurbine

148. MapReduce/Hadoop

149. MapReduce/Hadoop
Scale

150. MapReduce/Hadoop
Scale
Development

151. MapReduce/Hadoop
Scale
Development
Batch

152. MapReduce/Hadoop
Scale
100’s to 1000’s of server nodes
Extreme and cheap
Simple programming model
Development
Batch

153. MapReduce/Hadoop
Scale
100’s to 1000’s of server nodes
Extreme and cheap
Simple programming model
Development
Java, Python, Grep & Others...
Batch

154. MapReduce/Hadoop
Scale
100’s to 1000’s of server nodes
Extreme and cheap
Simple programming model
Development
Java, Python, Grep & Others...
Batch
Complex Multi-Step Processing

156. Storm

157. Storm
FAST

158. Storm
FAST
Integration

159. Storm
FAST
Integration
Assurance

160. Storm
FAST
Over a million tuples processed per second per node
Integration
Assurance

161. Storm
FAST
Over a million tuples processed per second per node
Integration
Integrates with any queueing system and any database system
Handles the parallelization, partitioning, and retrying on
failures when necessary
Assurance

162. Storm
FAST
Over a million tuples processed per second per node
Integration
Integrates with any queueing system and any database system
Handles the parallelization, partitioning, and retrying on
failures when necessary
Assurance
Scalable, Fault-Tolerant, Guarantees your data will be processed!

164. CQL/StreamQL/SparQL/QL-RTDB/

165. CQL/StreamQL/SparQL/QL-RTDB/
Languages

166. CQL/StreamQL/SparQL/QL-RTDB/
Languages
Scalable

167. CQL/StreamQL/SparQL/QL-RTDB/
Languages
Scalable
SQL Idioms

168. CQL/StreamQL/SparQL/QL-RTDB/
Languages
Human Readable
Scalable
SQL Idioms

169. CQL/StreamQL/SparQL/QL-RTDB/
Languages
Human Readable
Scalable
Simultaneous n Queries upon both stream data and static
SQL Idioms

170. CQL/StreamQL/SparQL/QL-RTDB/
Languages
Human Readable
Scalable
Simultaneous n Queries upon both stream data and static
SQL Idioms
All support to a large degree what you would expect from SQL

172. PIG

173. PIG
Language

174. PIG
Language
Parallelization

175. PIG
Language
Parallelization
Underneath

176. PIG
Language
High Level and easy to understand (Pig Latin)
Parallelization
Underneath

177. PIG
Language
High Level and easy to understand (Pig Latin)
Parallelization
It is trivial to achieve parallel execution of simple, "embarrassingly
parallel" data analysis tasks
Underneath

178. PIG
Language
High Level and easy to understand (Pig Latin)
Parallelization
It is trivial to achieve parallel execution of simple, "embarrassingly
parallel" data analysis tasks
Underneath
Essentially a MapReduce sequence compiler

180. PIG

181. PIG
Example Pig Script

182. PIG
Example Pig Script

184. PIG

185. PIG
That same example using MR Java code

187. The perfect Army!

188. The perfect Army!
In Memory

189. The perfect Army!
In Memory
Identify and Plan

190. The perfect Army!
In Memory
Identify and Plan
Consumer

191. The perfect Army!
In Memory
Keep as much as you can IN MEMORY! Think Redis...
Identify and Plan
Consumer

192. The perfect Army!
In Memory
Keep as much as you can IN MEMORY! Think Redis...
Identify and Plan
What data can be batch processed and what can’t! Think
Hadoop and Storm (for stream) and HBase (for adhoc)
Consumer

193. The perfect Army!
In Memory
Keep as much as you can IN MEMORY! Think Redis...
Identify and Plan
What data can be batch processed and what can’t! Think
Hadoop and Storm (for stream) and HBase (for adhoc)
Consumer
Who is the data consumer? Person or Process? Think Pig or xQL’s for
both!

194. Anthony Nyström Thank You
Fellow, Managing Director Gracias
of Engineering
Merci
anthony@intridea.com Danke
@AnthonyNystrom
www.intridea.co