Massive Streaming Analytics with Spark Streaming and Spark Notebook

1. Mattia Bertorello

2. Who is Agile Lab?
GO BIG (data) or GO HOME

3. Summary
•Why streaming matters
•Why prediction?
•Streaming architecture
•Spark streaming
•Demo time

4. Why streaming
matters
Ⓒ2015 Agile Lab S.r.l.

5. Why streaming matters
Data Big Data
Business
Reaction
Batch Analysis
Typical BigData Workflow

6. Why streaming matters
Data
Real Time
Processing
FASTER REACTIONS  MORE PROFITS
Business
Reaction
Streaming BigData Workflow

7. Why streaming matters
• Fleet Management
• Insurance
• Recommendation
• Etc...

8. Why prediction?
Ⓒ2015 Agile Lab S.r.l.

9. Why prediction?
• Rule based categorization and clustering is
obsolete
• Pattern discovery
• Adaptation to fast changing data
• Smart thinking: no dummies
• Prediction is more valuable

10. Streaming
architecture
Ⓒ2015 Agile Lab S.r.l.

11. Streaming architecture
Ingestion Layer
Processing Layer
Serving Layer

12. Spark Streaming

13. Spark Notebook
Use Apache Spark straight from the Browser

14. Demo time...
Ⓒ2015 Agile Lab S.r.l.

15. Card transaction analysis
PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP
Classificazione delle
transazioni
online/offline
PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP | ISONLINE
fraud detection
algorithm
SQL aggregation
Generazione di allarmi in
tempo reale

16. We are
hiring...
Ⓒ2015 Agile Lab S.r.l.
paolo.platter@agilelab.it