WASP is a framework to develop big data pipelines, working with streaming analytics, multi model storages and machine learning models. Everything is in real time.

1. www.agilelab.it

2. What we do
«I’m not a speed reader, I’m a speed understander»
Quote: Isaac Asimov
SCALABLE
TECHNOLOGIES
MACHINE LEARNING
FOR BIGDATA

3. Technologies
In memory and data exploration NoSQL Data Ingestion and Hi-scalable applications
Hadoop Frameworks and storage format Deployment and resource management Cloud deployment
Change Data Capture BI Tools Development and Data Visualization

4. About me
• Paolo Platter
• CTO @ Agile Lab
• Data & Tech addicted
• Certified Cassandra Architect
• Certified Spark Developer

5. Big Data is Hard
• Distributed computing is complex
• Technology Landscape is huge
• Technologies are not general purpose
• Without experience you fail
• Learning curve is flat

6. Big Data Streaming is harder
• Long running processes
• 24/7 requirements
• Anti-patterns are super common
• Software upgrade is hard
• Data delivery semantics are common pitfalls

7. WASP Platform
WASP is a pre-integrated solution: a big data infrastructre that allows to focus on business logic
instead of hadoop components integration and other side developments.

8. WASP Architecture

9. Out of the box features
• Telemetry
• Log collection
• Model Server
• Push Notification

10. Clustering at any level
Infrastructure
Orchestration
Application
Orchestration

11. Kappa + Lamda
Data Sources Ingestion & Decoupling Data ConsumersResult Stores
Event/Stream Processing
Streaming Data Trasformation
Batch Computing
Raw
Index
KeyValue
LOGS
RDMS
SENSOR
QUEUES
HTTP
TCP
MQTT
BI
File Export
Mobile
Web App
Dashboard
Alerting
Logs
Processing
Model
Server
Model
Server

12. Wasp Plugins
WASP CORE
Key Value Store Index Store Raw Store

13. Pipegraphs
• Pipegraph is an abstraction layer,
composed by a collection of Spark and
Akka transformers.
• Data routing and Storage can be
changed without coding
• Strategy classes are pluggable with class
loading to customize transformer
behaviour
• Anti-patterns are avoided by design:
only one single output for each
transformer

14. Pipegraphs
Topic 1 Enrichement With
Static Data HDFS
ELASTIC
Anti Pattern !

15. Pipegraphs
Topic 1 Enrichement With
Static Data
Topic 2
HDFS
Save to HDFS HDFS
Save to ElasticSearch ELASTIC
Without coding !!

16. Pipegraphs
Topic 1 Enrichement With
Static Data
Topic 2
Topic 3
Alert Generation
Save to Cassandra C*
HDFS
Save to HDFS HDFS
Save to ElasticSearch ELASTIC
Here we need a strategy

17. WASP Producers
WASP Core
Producer
Producer
Wasp
Master
Rest Proxy
- Enterprise Service Integration
- Lightweight Integration
- Local Connection
Edge Computing
for IoT and SoT
- Decoupled Ingestion
- Http Proxy
- Powered By Confluent
- Intelligent Streaming
- Edge Computing
- Site to Site communication
- Data in Motion with QoS

18. WASP IoT & SoT
WASP CoreWASP Fog
Controller
Enterprise Tier
Platform Tier
(Plant)
Gateway
Intelligence At Edge
Lightweight Process
Clustered
Acquisition
Devices in
Mobility
On Device
Bidirectional Protocol
Site to Site
Site to Site
IndustrialPhysical Store Edge Tier

19. Wasp Fog computing
WASP Core
WASP
Master
Enterprise Tier
Platform Tier
(Plant)
Sensor Layer
Fog Layer
Site to Site
Edge Tier
Fog Layer
Site to Site
Fog
Controller
• Controls a hierarchy of fog and sensor layers.
• Remoted and centralized monitoring of the
whole topology
• Leverages a transactional semantic delivery and
a reliable site to site communication protocol
• Data processing can be clustered to enhance the
throughput
• Provides separation between control plane and
data plane.
• Just one port opened on your firewalls
• Fog controller communicates only with the local
peer

20. Control Plane & Data Plane
Enterprise Tier
Platform Tier
(Plant)
Data
processor
Data
processor
Site to Site
Edge Tier
Data
Processor
Site to Site
Fog
Controller
Data Plane
Control layer
Control layer
Site to Site
Control Plane
Control Plane:
• Always active
• Start / Stop data processors
• Collects metrics and status from data
plane
• Provide inputs to Data Plane, like ML
models or customized BL
• Bidirectional communication
Site to Site
Data
Data Plane:
• Active or Inactive
• Process data in just one direction
• Can load a ML model or a custom BL

21. Thanks