The document discusses the evolution of big data architecture, moving beyond the concept of a big data lake to embrace heterogeneity while addressing the complexity of data sources and architectures. It highlights the importance of speed in data processing and the role of data science as a key differentiator in various sectors, particularly in financial services. Additionally, the document invites participation in a Hadoop user group meetup in Germany for further discussions on these themes.

1. Hadoop &
Germany &
2016
uweseiler

2. /whoami
&
/disclaimer
Hadoop & Germany & 2016

3. We finally stopped talking
infrastructure!
Hadoop & Germany & 2016

4. We now talk
architectures and use cases!
Hadoop & Germany & 2016

5. #1
The Big Data Lake is an illusion!
Hadoop & Germany & 2016

6. DataSourcesDataSystemsApplications
Traditional Sources
RDBMS OLTP OLAP …
Traditional Systems
RDBMS EDW MPP …
Business
Intelligence
Business
Applications
Custom
Applications
Operation
Manage
&
Monitor
Dev Tools
Build
&
Test
New Sources
Logs Mails Sensor …Social
Media
Enterprise
Hadoop
Plattform
#1 The Vision of the Big Data Lake

7. Hadoop is not the one tool
to rule them all
#1 Vision & Reality

8. Embrace heterogeneity!
(and learn to deal with the complexity)
#1 After the reality shock…

9. #1 Real world architecture - Insurance
DataSourcesDataSystemsApplications
Traditional Sources
RDBMS OLTP OLAP …
Traditional Systems
DWH
Business
Intelligence
New Sources
Logs Sensor …Social
Media
Enterprise Hadoop Plattform
SAS LASR Server
Apache Zeppelin

10. #2
Speed is the new king!
Hadoop & Germany & 2016

11. #2 The “classic“ Lambda Architecture

12. Batch Layer
Speed Layer
Data Ingestion
Data Processing
Data Storage
Data Storage Data Analysis
Visualization
Visualization
…
Data
Channels
ms - s
min - h
#2 Lambda in Action - (e)Commerce

13. SMACK
Spark
Mesos
Akka
Cassandra
Kafka
#2 The lust for speed

14. Data Ingestion
Data Processing
Raw Data
#2 Cassandra & Hadoop - AdServing
Data Processing
User Journey
Aggregated Data
Web Frontend
Aggregated Data
< 120 days
Data Science

15. #3
Data Science to the help!
Hadoop & Germany & 2016

16. Hadoop is about to become
commodity
#3 Let’s face it..

17. Algorithms will be the new
differentiator
#3 We need new challenges…

18. Batch Layer
Speed Layer
Data Ingestion
Stream Processing
ms - s
min - h
#3 Fraud detection - Financial services
Data
Import
Data
Preparation
Model
Generation
Model
Validation
Feature &
Parameter
Selection
Manual or automatic
Iterations to tune
parameters
Use
Model
Refresh Model from
latest input data

19. Every major company is
building teams of unicorns
#3 The solution?

20. #4
Hadoop for good!
Hadoop & Germany & 2016

21. Hadoop User Group Rhein-Main
http://www.meetup.com/de-DE/HUG-Rhein-Main/
Next Meetup: 23.06.2016, Talks welcome