1. Time Series Analysis: DataFlow
Timothy Spann
Principal DataFlow Field Engineer
@PaasDev

2. 2© 2020 Cloudera, Inc. All rights reserved.

3. © 2020 Cloudera, Inc. All rights reserved. 3
Welcome to Future of Data - Princeton
@PaasDev
https://www.meetup.com/futureofdata-princeton/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

4. © 2020 Cloudera, Inc. All rights reserved. 4
Welcome to Future of Data - New York
https://www.meetup.com/futureofdata-newyork/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

5. © 2020 Cloudera, Inc. All rights reserved. 5
Welcome to Future of Data - Philadelphia
@futureofdataphl
https://www.meetup.com/futureofdata-philadelphia/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to K8 to ...

6. © 2020 Cloudera, Inc. All rights reserved. 6
Meetup Presenter 1
Who am I?
Principal DataFlow Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB;
Princeton NJ Future of Data Meetup;
ex-Pivotal Field Engineer;
Apache Kafka, Tensorflow, Apache Spark RefCards
https://github.com/tspannhw https://www.datainmotion.dev/
https://dzone.com/users/297029/bunkertor.html

7. © 2019 Cloudera, Inc. All rights reserved. 7
CLOUDERA DATA PLATFORM
World’s first enterprise data cloud

8. © 2020 Cloudera, Inc. All rights reserved. 8
THE ENTERPRISE DATA CLOUD
COMPONENTS
Traditional Platform Consumption:
• Data Hub Clusters
New analytic experiences:
• Data Warehouse
• Machine Learning
• More to come
Control Plane services:
• Workload Manager
• Replication Manager
• Data Catalog
• Management Console

9. © 2020 Cloudera, Inc. All rights reserved. 9
CLOUDERA - THE ENTERPRISE DATA COMPANY
STREAMING &
DATA FLOW
DATA
WAREHOUSE
VISUAL
APPLICATIONS
Collect Report Enable
SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION
DATA
ENGINEERING
Enrich
MACHINE
LEARNING & AI
Predict

10. © 2020 Cloudera, Inc. All rights reserved. 10
CSA

11. © 2020 Cloudera, Inc. All rights reserved. 11
Streaming Analytics Powered by Apache Flink

12. 12© 2020 Cloudera, Inc. All rights reserved.
Buffer Analyze Connect
Modern AppsApache Kafka
gateway-west-
raw-sensors
gateway-central-
raw-sensors
gateway-east-
raw-sensors
C++ agent
US-Central Plants
C++ agent
US-East Plants
Data-at-Rest
opDB
CDP-DC
S3 Azure
Streaming
Analytics Powered
by FLINK
Microservices by
KAFKA Streams
Ingest Gateway
Powered by Kafka
US-West Plants
C++ agent
Model Scoring
Powered by CML
Data-at-Rest
Cloud Storage
Data Lake
Operational Stores
Data Collection
at the Edge
Collect
Apache NiFi & MiNiFi
Distribute
Apache NiFi
Data Flow Apps
Powered by NIFI
A DATA-IN-MOTION REFERENCE ARCHITECTURE

13. © 2020 Cloudera, Inc. All rights reserved. 13
SQL & Table API
● Unified APIs for streaming data and data at rest
○ Run the same query on batch and streaming data
○ ANSI SQL: No stream-specific syntax or semantics!
○ Many common stream analytics use cases supported
SELECT
userId,
COUNT(*) AS cnt
SESSION_START(clicktime, INTERVAL '30' MINUTE)
FROM clicks
GROUP BY
SESSION(clicktime, INTERVAL '30' MINUTE),
userId
Count clicks per user and session (defined
by 30 min. gap of inactivity).

14. Quick Flink SQL Demo Preview

15. FLaNK Stack https://github.com/tspannhw/MmFLaNK
https://www.datainmotion.dev/2019/11/introducing-mm-flank-apache-flink-stack.html
SELECT * FROM sensors;

16. © 2020 Cloudera, Inc. All rights reserved. 16
CFM
CSM

17. © 2020 Cloudera, Inc. All rights reserved. 17
CDF
The Active Data Warehouse with Apache Kudu
IOT Devices
Applications
Metrics
Logs & Files
HDFS/
Object Storage
Hot Storage
Cold Storage
SQL Real-Time
Analytics
Alerting
Event Driven
Applications
Dashboards
Authorization Audit & LineageAuthentication
Kerberos
Encryption
NavEncrypt

18. Sensor Data
https://www.datainmotion.dev/2020/04/predicting-sensor-readings-with-time.html

19. Sensors
● BME280 temperature, pressure, humidity sensor
● LTR-559 light and proximity sensor
● MICS6814 analog gas sensor
● ADS1015 ADC
● MEMS microphone
● 0.96-inch, 160 x 80 color LCD

20. Sensor Data - Edge

21. Sensor Data - Hydrate Date Lakes

22. Sensor Data - Example Row
{"uuid": "rpi4_uuid_omi_20200417211935", "amplitude100": 0.3, "amplitude500": 0.1, "amplitude1000": 0.1,
"lownoise": 0.1, "midnoise": 0.1, "highnoise": 0.1, "amps": 0.3, "ipaddress": "192.168.1.243", "host": "rp4",
"host_name": "rp4", "macaddress": "dc:a6:32:03:a6:e9", "systemtime": "04/17/2020 17:19:36", "endtime":
"1587158376.22", "runtime": "36.47", "starttime": "04/17/2020 17:18:58", "cpu": 0.0, "cpu_temp": "59.0",
"diskusage": "46651.6 MB", "memory": 6.3, "id": "20200417211935_7b7ae5da-905b-418b-94f1-270a15dbc1df",
"temperature": "38.7", "adjtemp": "29.7", "adjtempf": "65.5", "temperaturef": "81.7", "pressure": 1015.6,
"humidity": 6.8, "lux": 1.2, "proximity": 0, "oxidising": 8.3, "reducing": 306.4, "nh3": 129.5, "gasKO": "Oxidising:
8300.63 OhmsnReducing: 306352.94 OhmsnNH3: 129542.17 Ohms"}

23. Sensor Ingest Demo

24. Data Science Up Next

26. © 2020 Cloudera, Inc. All rights reserved. 26
LINKS

27. LINKS
● https://www.datainmotion.dev/2019/12/iot-series-minifi-agent-on-raspberry-pi.html
● https://learn.pimoroni.com/tutorial/sandyj/getting-started-with-enviro-plus
● https://github.com/tspannhw/meetup-sensors/
● https://github.com/tspannhw/ClouderaFlowManagementWorkshop
● https://github.com/tspannhw/minifi-enviroplus
● https://github.com/tspannhw/minifi-movidius-electric
● https://github.com/tspannhw/table-ddl

28. TH N Y U