The document discusses various architectures for Big Data solutions, including traditional, streaming, lambda, kappa, and unified architectures. It emphasizes the importance of understanding specific use cases when selecting the appropriate architecture, as Big Data remains an evolving field without standardized practices. Apache Spark is highlighted as a leading technology that enhances performance, while the document details the integration of batch and real-time processing approaches to improve data analytics outcomes.

1. Guido Schmutz | Trivadis
Architektur von Big Data Lösungen

2. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Architektur von Big Data Lösungen
Guido Schmutz

3. Guido Schmutz
Working for Trivadis for more than 19 years
Oracle ACE Director for Fusion Middleware and SOA
Co-Author of different books
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Member of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 25 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Twitter: gschmutz

4. Agenda
1. Introduction
2. Traditional Architecture for Big Data
3. Streaming Analytics Architecture for Fast Data
4. Lambda/Kappa/Unifed Architecture for Big Data
5. Summary

5. Introduction

6. How to do Big Data?

7. Big Data is still “work in progress”
Choosing the right architecture is key for any (big data) project
Big Data is still quite a young field and therefore there are no standard architectures
available which have been used for years
In the past few years, a few architectures have evolved and have been discussed online
Know the use cases before choosing your architecture
To have one/a few reference architectures can help in choosing the right components

8. Building Blocks for (Big) Data Processing
Data
Acquisition
Format
File System
Stream Processing
Batch SQL
Graph DBMS
Document
DBMS
Relational
DBMS
Visualization
IoT
Messaging
Analytics
OLAP DBMS
Query
Federation
Table-Style
DBMS
Key Value
DBMS
Batch Processing
In-Memory

9. Big Data Ecosystem – many choices ….

10. Important Properties to choose a Big Data Architecture
Latency
Keep raw and un-interpreted data “forever” ?
Volume, Velocity, Variety, Veracity
Ad-Hoc Query Capabilities needed ?
Robustness & Fault Tolerance
Scalability
…

11. From Volume and Variety to Velocity
Big Data has evolved …
and the Hadoop Ecosystem as well ….
Past
Big Data = Volume & Variety
Present
Big Data = Volume & Variety & Velocity
Past
Batch Processing
Time to insight of Hours
Present
Batch & Stream Processing
Time to insight in Seconds
Adapted from Cloudera blog article

12. Traditional Architecture for Big
Data

13. “Traditional Architecture” for Big Data
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
RDBMS
Sensor
ERP
Logfiles
Mobile
Machine
Batch
compute
Stage
Result Store
Query
Engine
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest

14. Use Case 1) – Log File Analysis (Audit, Compliance,
Root Cause Analysis)
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Data
Consumer
Channel
Batch
compute
Computed
Information
Raw Data
(Reservoir)
Result Store
Query
Engine
Reports
Analytic
Tools
Logfiles

15. Use Case 1a) – Log File Analysis (Audit, Compliance,
Root Cause Analysis)
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Data
Consumer
Channel
Batch
compute
Computed
Information
Raw Data
(Reservoir)
Result Store
Query
Engine
Reports
Analytic
Tools
Logfiles

16. Use Case 2) – Customer Data Hub (360 degree view of
customer)
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Data
Consumer
RDBMS
Batch
compute
Computed
Information
Raw Data
(Reservoir)
Result Store
Query
Engine
Reports
Service
Analytic
Tools
Alerting
Tools

17. Use Case 2a) – Customer Data Hub (360 degree view
of customer)
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Data
Consumer
RDBMS
(CDC) Batch
compute
Computed
Information
Raw Data
(Reservoir)
Result Store
Query
Engine
Reports
Service
Analytic
Tools
Alerting
Tools
Channel

18. “Hadoop Ecosystem” Technology Mapping
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
RDBMS
Sensor
ERP
Logfiles
Mobile
Machine
Batch
compute
Staging
Result Store
Query
Engine
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest

19. Apache Spark – the new kid on the block
Apache Spark is a fast and general engine for large-scale data processing
• The hot trend in Big Data!
• Originally developed 2009 in UC Berkley’s AMPLab
• Can run programs up to 100x faster than Hadoop MapReduce in memory, or 10x
faster on disk
• One of the largest OSS communities in big data with over 200 contributors in 50+
organizations
• Open Sourced in 2010 – since 2014 part of Apache Software foundation
• Supported by many vendors

20. Motivation – Why Apache Spark?
Apache Hadoop MapReduce: Data Sharing on Disk
Apache Spark: Speed up processing by using Memory instead of Disks
map reduce . . .
Input
HDFS
read
HDFS
write
HDFS
read
HDFS
write
op1 op2
. . .
Input
Output
Output

21. Apache Spark “Ecosystem”
Spark SQL
(Batch Processing)
Blink DB
(Approximate
Querying)
Spark Streaming
(Real-Time)
MLlib, Spark R
(Machine Learning)
GraphX
(Graph Processing)
Spark Core API and Execution Model
Spark
Standalone
MESOS YARN HDFS
Elastic
Search
NoSQL S3
Libraries
Core Runtime
Cluster Resource Managers Data / Data Stores

22. “Spark Ecosystem” Technology Mapping
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
RDBMS
Sensor
ERP
Logfiles
Mobile
Machine
Batch
compute
Stage
Result Store
Query
Engine
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest

23. Use Case 3) – (Batch) Recommendations on Historic
Data
Data
Collection
(Analytical) Data Processing
Result StoreData
Sources
Data
Consumer
Machine
Batch
compute
Computed
Information
Raw Data
(Reservoir)
Result Store
Query
Engine
Reports
Service
Analytic
Tools
Alerting
Tools
DB
StagingFile
Channel

24. Traditional Architecture for Big Data
• Batch Processing
• Not for low latency use cases
• Spark can speed up, but if positioned as alternative to Hadoop
Map/Reduce, it’s still Batch Processing
• Spark Ecosystems offers a lot of additional advanced analytic capabilities
(machine learning, graph processing, …)

25. Streaming Analytics Architecture
for Big Data

26. Streaming Analytics Architecture for Big Data
aka. (Complex) Event Processing)
Data
Collection
Batch
compute
Data
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
Logfiles
Sensor
RDBMS
ERP
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store
= Data in Motion = Data at Rest

27. Use Case 4) Algorithmic Trading
Data
Collection
Batch
compute
Data
Sources
Channel
Data
Consumer
Analytic
Tools
Alerting
Tools
Sensor
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store

28. Use Case 5) Real-Time Analytics / Dashboard
Data
Collection
Batch
compute
Data
Sources
Channel
Data
Consumer
Analytic
Tools
Sensor
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store

29. Native Streaming vs.
Micro-Batching
Native Streaming
• Events processed as they
arrive
• + low-latency
• - throughput
• - fault tolerance is expensive
Micro-Batching
• Splits incoming stream in
small batches
• + high(er) throughput
• + easier fault tolerance
• - lower latency
Source: Distributed Real-Time Stream Processing:
Why and How by Petr Zapletal

30. Unified Log (Event) Processing
Stream processing allows for
computing feeds off of other feeds
Derived feeds are no
different than
original feeds
they are computed off
Single deployment of “Unified
Log” but logically different feeds
Meter
Readings
Collector
Enrich /
Transform
Aggregate by
Minute
RawMeter
Readings
Meter with
Customer
Meter by Customer by
Minute
Customer
Aggregate by
Minute
Meter by Minute
Persist
Meter by
Minute
Persist
Raw Meter
Readings

31. Streaming Analytics Technology Mapping
Data
Collection
Batch
compute
Data
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
Logfiles
Sensor
RDBMS
ERP
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store
= Data in Motion = Data at Rest

32. Streaming Analytics Architecture for Big Data
The solution for low latency use cases
Process each event separately => low latency
Process events in micro-batches => increases latency but offers better
reliability
Previously known as “Complex Event Processing”
Keep the data moving / Data in Motion instead of Data at Rest => raw events
were not stored

33. Keep raw event data
Data
Collection
Batch
compute
Data
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
Logfiles
Sensor
RDBMS
ERP
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store
= Data in Motion = Data at Rest
(Analytical) Batch Data Processing
Raw Data
(Reservoir)

34. Lambda Architecture for Big Data

35. “Lambda Architecture” for Big Data
Data
Collection
(Analytical) Batch Data Processing
Batch
compute
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
RDBMS
Sensor
ERP
Logfiles
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Batch
compute
Messaging
Result Store
Query
Engine
Result Store
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest

36. Use Case 6) Social Media Analysis
Data
Collection
(Analytical) Batch Data Processing
Batch
compute
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
(Analytical) Real-Time Data Processing
Stream/Event Processing
Batch
compute
Messaging
Result Store
Query
Engine
Result Store
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest

37. Use Case 7) Customer Event Hub
Data
Collection
(Analytical) Batch Data Processing
Batch
compute
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
(Analytical) Real-Time Data Processing
Stream/Event Processing
Batch
compute
Messaging
Result Store
Query
Engine
Result Store
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest
Social
RDBMS
(CDC)
Sensor
ERP
Logfiles

38. Lambda Architecture for Big Data
Combines (Big) Data at Rest with (Fast) Data in Motion
Closes the gap from high-latency batch processing
Keeps the raw information forever
Makes it possible to rerun analytics operations on whole data set if necessary
=> because the old run had an error or
=> because we have found a better algorithm we want to apply
Have to implement functionality twice
• Once for batch
• Once for real-time streaming

39. „Kappa“ Architecture for Big Data

40. “Kappa Architecture” for Big Data
Data
Collection
“Raw Data Reservoir”
Batch
compute
Data
Sources
Messaging
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
Logfiles
Sensor
RDBMS
ERP
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store
Raw Data
(Reservoir)
= Data in Motion = Data at Rest
Computed
Information

41. Use Case 8) Social Network Analysis
Data
Collection
“Raw Data Reservoir”
Batch
compute
Data
Sources
Messaging
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
Logfiles
Sensor
RDBMS
ERP
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store
Raw Data
(Reservoir)
= Data in Motion = Data at Rest
Computed
Information

42. Kappa Architecture for Big Data
Today the stream processing infrastructure are as scalable as Big Data
processing architectures
• Some using the same base infrastructure, i.e. Hadoop YARN
Only implement processing / analytics logic once
Can Replay historical events out of an historical (raw) event store
• Provided by either the Messaging or Raw Data (Reservoir) component
Updates of processing logic / Event replay are handled by deploying new
version of logic in parallel to old one
• New logic will reprocess events until it caught up with the current events and then
the old version can be de-commissioned.

43. „Unified“ Architecture for Big Data

44. “Unified Architecture” for Big Data
Data
Collection
(Analytical) Batch Data Processing (Calculate
Models of incoming data)
Batch
compute
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Social
RDBMS
Sensor
ERP
Logfiles
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Batch
compute
Messaging
Result Store
Query
Engine
Result Store
Computed
Information
Raw Data
(Reservoir)
= Data in Motion = Data at Rest
Prediction
Models

45. Use Case 8) Industry 4.0 => Predictive Maintainance
Data
Collection
(Analytical) Batch Data Processing (Calculate
Models of incoming data)
Batch
compute
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
RDBMS
Sensor
ERP
Logfiles
Mobile
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Batch
compute
Messaging
Result Store
Query
Engine
Result Store
Computed
Information
Raw Data
(Reservoir)
Prediction
Models
J-PMML

46. Summary

47. Summary
Know your use cases and then choose your architecture and the relevant
components/products/frameworks
You don’t have to use all the components of the Hadoop Ecosystem to be successful
Big Data is still quite a young field and therefore there are no standard architectures
available which have been used for years
Lambda, Kappa Architecture & Unified Architecture are best practices architectures
which you have to adapt to your environment
=> Combination of traditional Big Data Batch & Complex Event Processing (CEP)

49. Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com