In this presentation we discuss IoT and Edge use cases with a focus on how MiNiFi C++ has been able to tackle both while clearly living within the constraints dictated by these use cases.

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi MiNiFi C++ and the tale of
edgey stuff
Marc Parisi
phrocker@apache.org
@MrTheSegFault

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
BIO
marc(int*):
push rbp
mov rbp, rsp
mov QWORD PTR [rbp-8], rdi
mov QWORD PTR [rbp-8], 0
mov rax, QWORD PTR [rbp-8]
mov eax, DWORD PTR [rax]
pop rbp
ret
https://www.s.mrsegfault.com/jailbreak

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
⚫ IoT / Edge Breakdown
⚫ MiNIFI C++ deep dive
⚫ Use Cases
⚫ Demo if we're lucky

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT Versus Edge
⚫ IoT
− Devices that share and collect data
⚫ Edge
− Data processing at edge of one’s network ( or the
internet ?)
⚫ One in the same?

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Explosion of Devices

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT use cases
⚫ Connected car
⚫ Cell phone collection
⚫ Smart watches
⚫ Sensor readers
⚫ ...oh my
https://pixabay.com/en/network-iot-internet-of-things-782707/

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
From the device to the warehouse...
⚫ Devices → Warehouse/Regional
− Minimal grooming
− Apache NiFi regional and core
⚫ May not have compute capacity
− ‘Cause a toaster’s gonna toast, toast, toast, toast
[a]

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Edge use cases
⚫ Connected car
⚫ Cell phone collection
⚫ Smart watches
⚫ Sensor readers
⚫ ...OR
⚫ Custom devices
⚫ Robotics
⚫ …oh my?
https://pixabay.com/en/network-iot-internet-of-things-782707/

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Living on the Edge
⚫ Edge processing increases capacity
⚫ Edge devices can be ..
− Raspberry Pi
− Computer
− Smart Phone/Watch

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT/Edge Constraints [1]
• Limited computing capability
• Limited power/network
• Restricted software library/platform availability
• Not frequently updated
• Competing standards/protocols
• Scalability
• Privacy & Security

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi MiNiFi C++
⚫ “...[C]omplimentary data collection approach
that supplements the core tenets of NiFi in
dataflow management, focusing on the
collection of data at the source of its creation”
⚫ Deliver data across a common platform of
extensions and components
⚫ NiFi empowers infrastructure

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The NiFi Ecosystem

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Not just another NiFi
• Bi-Directional (Guaranteed) data transfer
• Data Provenance
• Resource Awareneness
• Small memory footprint
• Custom set of processors
• State Management/Command and Control
• Flow Updates
• Flow Metrics *No JVM

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Not just another NiFi
• Processors
• GetUSBCamera
• Tensorflow
• GetGPS
• Full list [4]

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Versace-ability
⚫ Agent/Library
⚫ Data collector/Sender
⚫ Edge/Analytic Processor

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Resource Awareness
• Battery Aware resource manager [3]
• Monitor consumption
• Adjust as needed
• Network manager (coming soon...)

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Command and Control
⚫ Command and control design [6,7]
⚫ Command line controller [8]
⚫ Update flow dynamically
⚫ Start/Stop components
⚫ Clear Queues/Repos
⚫ No UI
⚫ Remote/Local Admin

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT/Collector Use Cases
⚫ Connected car -- Data collector

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car
• Simple collection
• Tail File or gather GPS information
• Minimal Processing

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Edge use case
• Robotic controller
• Not typical use case
• Ferdinand
• OpenCV detects red in image *
• ListenHTTP receives movement commands
• Computation determines motor speed
through GPIO
• Computation for servo control

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Edge use case

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bibliography
[1] https://www.statista.com/statistics/471264/iot-number-of-connected-devices-
worldwide/
[2] https://www.slideshare.net/Hadoop_Summit/apache-nifi-crash-course
[3] https://www.youtube.com/watch?v=1TMArQS7w40
[4] https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md
[5] https://blog.sasken.com/the-connected-car/
[6] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[7] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal
[8] https://github.com/apache/nifi-minifi-cpp/#managing-minfi-c-through-the-minifi-
controller
[9] https://youtu.be/a9EoCRYu39o
[a] https://www.youtube.com/watch?v=nfWlot6h_JM