This document discusses managing data collection at the edge using Cloudera Edge Management (CEM). CEM allows users to develop edge data collection flows using a NiFi-like interface, deploy those flows to thousands of edge devices grouped by class or manufacturer, and monitor all the edge agents and flows centrally. The document provides an example of using CEM to deploy a new edge flow to a class of Freightliner trucks to ingest data from a new speed sensor and filter events where speed is over 50.

1. © Cloudera, Inc. All rights reserved.
EDGE2AI DELIVERED BY
CLOUDERA EDGE MANAGEMENT (CEM)
George Vetticaden
VP of Product Management, Data In Motion (DIM) BU

2. © Cloudera, Inc. All rights reserved. 2
STREAMING ANALYTICS REFERENCE ARCHITECTURE
IOT Ingest
Gateway Powered
by Kafka
Kafka Topic
gateway-
freightliner-
sensors-
Kafka Topic
gateway-
mack-
sensors-
Kafka Topic
gateway-
tata-
sensors-
Kafka Cluster Geo 2
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Kafka Cluster Geo 1
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Data Flow
Apps Powered
by NiFi
Data Collection
at the Edge
Freightliner Fleet Truck
Mack Fleet Truck
Tata Fleet Truck
C++
Agent
C++
Agent
C++
Agent
Streaming
Analytics Apps
/ Data Lakes
Analytics App
Micro Services
DS / AI
Workbench
CDSW
Data Lake
CDH / HDP
Analytics App

3. © Cloudera, Inc. All rights reserved. 3
MANAGING EDGE DATA COLLECTION IS A HARD PROBLEM
Kafka Topic
gateway-
freightliner-
sensors-
Kafka Topic
gateway-
mack-
sensors-
Kafka Topic
gateway-
tata-
sensors-
Kafka Cluster Geo 2
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Kafka Cluster Geo 1
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Subscribing
Streaming
Analytics Apps
Subcribing
Streaming Analytics
Apps
Analytics App 4
Analytics App 1
Analytics App 2
Micro Services
Analytics App 3
Data Flow
Apps Powered
by NiFi
Freightliner Fleet Truck
Mack Fleet Truck
Tata Fleet Truck
C++
Agent
C++
Agent
C++
Agent
Data Collection
at the Edge
IOT Ingest
Gateway Powered
by Kafka
Challenge #1:
Developing Edge Flows
How do I develop flows meant to run on
the edge using NiFi-like flow based
programming with no code?

4. © Cloudera, Inc. All rights reserved. 4
MANAGING EDGE DATA COLLECTION IS A HARD PROBLEM
Kafka Topic
gateway-
freightliner-
sensors-
Kafka Topic
gateway-
mack-
sensors-
Kafka Topic
gateway-
tata-
sensors-
Kafka Cluster Geo 2
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Kafka Cluster Geo 1
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Subscribing
Streaming
Analytics Apps
Subcribing
Streaming Analytics
Apps
Analytics App 4
Analytics App 1
Analytics App 2
Micro Services
Analytics App 3
Data Flow
Apps Powered
by NiFi
Freightliner Fleet Truck
Mack Fleet Truck
Tata Fleet Truck
C++
Agent
C++
Agent
C++
Agent
Data Collection
at the Edge
IOT Ingest
Gateway Powered
by Kafka
Challenge #2:
Deploying Flows/ Over the Air Updates (OTA)
How do I deploy data collection flows to the edge?
How do I do updates to only the Freightliner class
of trucks?

5. © Cloudera, Inc. All rights reserved. 5
MANAGING EDGE DATA COLLECTION IS A HARD PROBLEM
Kafka Topic
gateway-
freightliner-
sensors-
Kafka Topic
gateway-
mack-
sensors-
Kafka Topic
gateway-
tata-
sensors-
Kafka Cluster Geo 2
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Kafka Cluster Geo 1
Data Syndicate Services
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
speed
Kafka Topic
syndicate
freightliner-
geo
Kafka Topic
syndicate
freightliner-
geo
Subscribing
Streaming
Analytics Apps
Subcribing
Streaming Analytics
Apps
Analytics App 4
Analytics App 1
Analytics App 2
Micro Services
Analytics App 3
Data Flow
Apps Powered
by NiFi
Freightliner Fleet Truck
Mack Fleet Truck
Tata Fleet Truck
C++
Agent
C++
Agent
C++
Agent
Data Collection
at the Edge
IOT Ingest
Gateway Powered
by Kafka
Challenge #3:
Monitoring Agents and Flows
How do I manage hundreds of thousands of edge
agents and the flows that they are running?

6. © Cloudera, Inc. All rights reserved. 6
INTRODUCING CLOUDERA EDGE MANAGEMENT (CEM)
What is CEM?
 New product offering is an edge management
solution consisting of edge agents + edge
management hub.
 What does CEM do? Manage, Control and Monitor
edge agents to collect data from edge devices and
push intelligence back to the edge.
 How does CEM do it? Develop, Deploy, Run &
Monitor edge flow apps on thousands of edge
devices.
N
ew

7. © Cloudera, Inc. All rights reserved. 7
INTRODUCING CLOUDERA EDGE MANAGEMENT (CEM)
Key Benefits
 Agility - Build edge data collections with the same
agility as they build flow apps with NiFi.
 Key Enabler for IOT Initiatives - Build successful
IoT initiatives by collecting, curating and analyzing
data from thousands of edge devices.
 Operational Confidence - Gain complete
operational confidence with your edge
implementation by gaining visibility into all of your
deployed agents.
N
ew

8. © Cloudera, Inc. All rights reserved. 8
The EDGE: Truck Fleet
COMMON EDGE DATAFLOW REQUIREMENTS
Use Case Setup
#1: Company X has 1000s of vehicles
in its fleet with a sensors streaming
various telemetric data
#2: The Fleet consists of 4 different truck
manufacturers each with its own slightly
different data and flow requirements.

9. © Cloudera, Inc. All rights reserved. 9
COMMON EDGE DATAFLOW REQUIREMENTS
Edge Flow Requirements
1. Edge collection agents need to be lightweight, resource
efficient (size, memory, processing) and be able to run in
diverse environments like a gateway on truck.
3. Each truck manufacturer has different sensors and
data ingest and processing requirements
2. Each agent should be able perform local collection,
filter/routing, & transformation functions, and send data to
the Cloud for further more complex processing/analytics
4. Development and deployment flows to 1000s of agents
grouped my manufacturer must be made easy.
5. The edge data processing needs to support intermittent
network connectivity.
The EDGE: Truck Fleet

10. © Cloudera, Inc. All rights reserved. 10
The EDGE: Truck Fleet
Agent Class: Mack Agent Class: Freightliner
Agent Class: Benz Agent Class: Tata
IMPLEMENTING REQUIREMENTS WITH CEM
Steps
1. Install MiNifi C++ agent on each truck
2. On each agent configure a class that
represent the Truck manufacturer type.
This allows users to to push out different
flows to the edge based on the truck type.
4. Develop the edge flow using NiFi like
flow based development paradigm
5. Publish/Update the Flow. This will push
out the flow to every agent that is
configured with the flow’s class.
6. Monitor the agents and the flows
running on each truck
3. In Edge Flow Manager (EFM), select
an class of Agents you are going to
develop an edge flow
N
ew

11. © Cloudera, Inc. All rights reserved. 11
OTA UPDATE TO ALL THE FREIGHTLINER TRUCKS MADE EASY BY CEM
Update the Edge Flow with a
new/improved predictive model
Capture/Ingest New Speed Sensor.
Filter events whose speed > 50

12. © Cloudera, Inc. All rights reserved.
DEMO CEM
DEVELOP, DEPLOY & MANAGE
AN EDGE “SMART” FLOW FOR THE FREIGHTLINER
CLASS OF MINIFI AGENTS

13. © Cloudera, Inc. All rights reserved.
THANK YOU