Modern IoT operations can drive digital transformation by analyzing the unprecedented amounts of data generated from devices and sensors in real-time. Apache Spark is a widely used stream processing engine for real-time IoT applications. Spark streaming offers a rich set of APIs in the areas of ingestion, cloud integration, multi-source joins, blending streams with static data, time-window aggregations, transformations, data cleansing, and strong support for machine learning and predictive analytics. Join Anand Venugopal, AVP & Business Head, StreamAnalytix and Sameer Bhide, Senior Solutions Architect, StreamAnalytix to learn about the rapid development and operationalization of real-time IoT applications covering an end-to-end flow of ingest, insight, action, and feedback. The webinar will cover the following: Generic IoT application blueprint Case studies on IoT applications built on Apache Spark – connected car and industrial IoT Demonstration of an easy, visual approach to building IoT Spark apps

1. Streaming Analytics for IoT
with Apache Spark
WEBINAR | FEBRUARY 16, 2018
Sameer BhideAnand Venugopal
AVP & Business Head, STREAMANALYTIX Senior Solution Architect, STREAMANALYTIX

2. Agenda
| IoT Market Perspective
| The IoT Application Architecture
| Use Cases: Connected Car | Industrial IoT
| Demo – Building IoT Spark Apps Using
| Key Takeaways
| Q & A

3. Mission critical
technology solutions
since 1996
Fortune 500:
Big data clients
1700 people; US,
India, global reach
Unique mix of
big data products
and services
ABOUT

4. IOT Market: Gartner Prediction
7.6 Billion
Things that will ship in 2021
32% CAGR
End point growth rate 2016-2021
25.1 Billion
Units install base in 2021
$3.9 Trillion
Total spending: 2021
end points and services

5. Smart Home Smart Car Smart Building Smart City Smart Agriculture
Smart Factory Smart Healthcare Smart Data Center Smart Energy Smart Retail
IoT – Market Sub-Domains

6. IoT Solution Architecture and the Role of Spark
Field IoT
Gateway
Cloud IoT
Integration HUB
Field IoT Gateway
Connected
Things
Enterprise
Applications
Centralized IoT Data Mgmt.
& Analytics Platform

7. POLL

8. •Variation in Device Capabilities
• MEP: State-notifications,
Commands, Telemetry
• State Management & Security
•Agility
•Extensibility
•Time-to-market
•Automation
• RT, NRT Analytics
• Offline Analytics & Reporting
• Schematization
• Data Blending & Enrichment
• Multi-party Interaction
•Network & Connectivity
Dynamics
•Real Time
Data Ingestion Telemetry
Computation &
Analytics
Device
Management
Operationalize
IOT Application Architecture: Key Aspects

9. TELEMETRY
• Network & Connectivity Dynamics
• Real Time Data Ingestion
Data Ingestion
Abstractions
COMPUTATION
& ANALYTICS
• RT, NRT Analytics
• Offline Analytics & Reporting
• Schematization
• Data Blending & Enrichment
• Multi-party Interaction
Event & Micro-Batch
Computation Engines
(Spark, Storm etc.)
DEVICE MANAGEMENT
• Variation in Device Capabilities
• MEP - State-notifications, Commands, Telemetry
• State Management & Security
Metadata Registry &
Management Gateway
OPERATIONALIZATION
• Agility
• Extensibility
• Time-to-market
• Automation
Dataflow (Pipeline)
Management & DSL
Capabilities
1
2
3
4
IOT Application Architecture: Foundational Elements

10. Managed Devices
EDA / SEDA Sources
IOT Application Architecture: Conceptual Layers
Management LayerIoT Gateway Data Ingestion
Data Processing
& Storage Layer
Insights Layer Action Layer
Security Data
Sources
Compute
Engine
(Spark)
ML - Model
Updates Patterns–
(A-B, Champion
Challenger etc…)
Notification
Services
Fault Tolerance Protocol
Support
Data Filtering,
Blending &
Enrichment
Rule Engine Alerts
State
Management
Ontology &
Metadata
Management
Structured
Query
Feedback
Loop
External
Services
Integration
Device Proxy Data
Persistence
Custom Business
Flows (IFTTT,
Lambda etc)
INGEST ENRICH ANALYZE ACT
Configuration &
Connection Management
Performance
Management
Application Life Cycle
Management
Version Updates
PaaS Integration Computer Infrastructure
SPARK
Infrastructure Layer

11. Spark as the IoT Compute Engine
| Massively scalable
| Rich set of transformations
| Industry adoption
| Unified & simplified programming model
| Support for machine learning
| Micro-batch capable – tending to NRT

12. Recommendations
| Adopt an integrated approach to IoT development
| Design a platform layer that can adopt to business’ dynamic needs
| Create a vendor neutral & interoperable architecture
| Adopt software products to quickly operationalize IoT use cases

13. © 2017 Impetus Technologies
Real-time Stream Processing & Machine Learning Platform
+ Visual Spark Studio

14. Ingest Analyze
Define Rules
& Actions
Design Once –
Execute Anywhere
Manage
Key Features

15. Connected Cars Industrial IoT
Use Cases

16. Connected Car – Driver Risk Profiling
Brief Background
Leading insurance provider in the US
• Classify drivers based on current driving
pattern and historical data
• Raise alerts on behavior change
• Blend data from syndicated and open /
public data marts & services
• Derive additional analytics through
supplemental data flows
Business Need
To create an end-to-end analytics application
for driver profiling & RT risk assessment

17. Central
Aggregation
Server / Data
Flow
Manager
On-premise: Bare-Metal and/or VMs | Public / Hybrid Cloud
Data Center / Cloud
Storage and Offline Analytics
Device Provisioning and
Management (identity /
registration etc.)
Open Interfaces –
extensibility and
customizability in all
directions
Real Time Dash-boarding
Condition Monitoring
Predictive Maintenance
Smart Alerting
Root Cause Analytics
Closed-loop Feedback
Edge
Custom solution OR
3rd party IoT interface vendor
Data flow
Control flow
End Device 1
End Device 2
End Device 3
End Device 1
End Device 2
End Device 3
Smart Car 1
Smart Car 2
IoT / Connected Car Solution with StreamAnalytix
IOT data
interface
(MQTT / HTTP /
WebSockets)
IOT data
interface
(MQTT / HTTP /
WebSockets)
Gateway

18. AWS IoT
(Spark)
Ingestion Enrichment Analytics
Public Cloud Services /
Third-party Services
Dashboards
Handheld
Devices
Persistence
Automated Device
Installed in OBD-II Port
High Level Solution Overview
Alerts

19. Connected Car – Driver Risk Profiling

20. Connected Car – Driver Risk Profiling
Ingest events using
AWS IoT gateway
Mask PII & enrich data with
external & historical sources
Score a ‘Risk Assessment’ model that uses
• Weather conditions
• Time of trip
• Hard brakes and acceleration
• Duration over 70mph
• Previous number of risk instances
Raise alerts based on
risk scores
Create RT and historical
dashboards

21. Industrial IoT Use Case – Device Health Monitoring
Brief Background
Leader in industrial automation, information,
and engineering services
• Various machine health parameters
collected in different timelines from an
array of sensors
• Compute and store correlation between
sensor data when a process parameter is
altered
• Leverage existing investments in Azure
cloud infrastructure
Business Need
Measure impact to process dynamics by
calculating correlation between various
sensor data

22. End-to-end solution deployed
on StreamAnalytix
Data pipelines used pre-built
components:
• Data ingestion
• Statistical functions
• Data enrichment
• Visualization
Cloud: Microsoft Azure
Source: Event Hub
Compute: Spark jobs on HDInsight
Orchestration: StreamAnalytix
High Level Solution Approach
Reporting
Dashboard
Manufacturing
Units
HD-Insights

23. • Ingest data from different MS Azure Event Hub sources
• Enrich incoming data
• Outer-join on incoming datasets
• Aggregate result data and group by plantIDs
• Post streaming results on WebSockets
Data Pipeline View
Industrial Automation - Turbine Data Analytics

24. Key Takeaways
| IoT capabilities in StreamAnalytix
• Data Sources : Azure Event Hub, AWS IoT, MQTT, Kinesis, S3
• Data Sink : Redshift, Hadoop, MQTT, S3, Kinesis, WebSockets
• PaaS Service Integration : SQS, Lambda, SNS
| Integrated approach to IoT development
| IoT applications are dynamic
| Vendor neutral & interoperable architecture
| COTS & open source offerings to quickly operationalize IoT use cases

25. SanJose,CA|March5-8Grapevine,TX|March5-8
Booth#809 Booth#817
Data&AnalyticsSummit2018

26. Thank you.
Questions?
© 2018 Impetus Technologies
Email: inquiry@streamanalytix.com | @ImpetusTech / @StreamAnalytix
Download Visual Spark Studio™ - Zero cost, light weight development tool.
Full range of data processing and analytics functionality to build Spark applications.