Streaming data allows for faster reactions and more profits by enabling real-time processing compared to typical batch analysis workflows. Prediction is more valuable than rule-based categorization and clustering because it allows for pattern discovery, adaptation to changing data, and "smart thinking". Spark Streaming is presented as a tool for building streaming architectures with ingestion, processing, and serving layers. A demo of card transaction analysis with real-time classification, fraud detection, and alert generation is shown.

1. Mattia Bertorello

2. Who is Agile Lab?
GO BIG (data) or GO HOME

3. Summary
•Why streaming matters
•Why prediction?
•Streaming architecture
•Spark streaming
•Demo time

4. Why streaming
matters
Ⓒ2015 Agile Lab S.r.l.

5. Why streaming matters
Data Big Data
Business
Reaction
Batch Analysis
Typical BigData Workflow

6. Why streaming matters
Data
Real Time
Processing
FASTER REACTIONS  MORE PROFITS
Business
Reaction
Streaming BigData Workflow

7. Why streaming matters
• Fleet Management
• Insurance
• Recommendation
• Etc...

8. Why prediction?
Ⓒ2015 Agile Lab S.r.l.

9. Why prediction?
• Rule based categorization and clustering is
obsolete
• Pattern discovery
• Adaptation to fast changing data
• Smart thinking: no dummies
• Prediction is more valuable

10. Streaming
architecture
Ⓒ2015 Agile Lab S.r.l.

11. Streaming architecture
Ingestion Layer
Processing Layer
Serving Layer

12. Spark Streaming

13. Spark Notebook
Use Apache Spark straight from the Browser

14. Demo time...
Ⓒ2015 Agile Lab S.r.l.

15. Card transaction analysis
PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP
Classificazione delle
transazioni
online/offline
PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP | ISONLINE
fraud detection
algorithm
SQL aggregation
Generazione di allarmi in
tempo reale

16. We are
hiring...
Ⓒ2015 Agile Lab S.r.l.
paolo.platter@agilelab.it