The document discusses the complexities and idiosyncrasies involved in IoT and edge device development, focusing on the use of Apache NiFi MiNiFi C++. It outlines various use cases and challenges such as limited computing capacity, power management, and dependency issues while emphasizing the importance of resource awareness and command/control design. Lessons learned highlight the need for simple, controlled approaches to managing data and resources in IoT environments.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Navigating Idiosyncrasies
of IoT Development
Marc Parisi
Principal Software Engineer @ Hortonworks

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
BIO
marc(int*):
push rbp
mov rbp, rsp
mov QWORD PTR [rbp-8], rdi
mov QWORD PTR [rbp-8], 0
mov rax, QWORD PTR [rbp-8]
mov eax, DWORD PTR [rax]
pop rbp
ret

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• What is MiNiFI C++ and its use cases
• Idiosyncracies ( Use case derived)
• Lessons learned

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
Apache NiFi MiNiFi C++
• “...[C]omplimentary data collection approach that supplements the core tenets of NiFi in
dataflow management, focusing on the collection of data at the source of its creation”
• Deliver data across a common platform of extensions and components
• NiFi empowers infrastructure

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
The NiFi Ecosystem

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
How is MiNiFi used
• Agent
• Full integration with NiFI
• API Library
• Developers choose pieces

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
IoT Versus Edge
• IoT
• Devices that share and collect data
• Edge
• Data processing at edge of one’s network ( or the internet ?)
• One in the same?

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Explosion of Devices

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
https://pixabay.com/en/network-iot-internet-of-things-782707/
IoT Use Cases
• Connected car
• Cell phone collection
• Smart watches
• Sensor readers
• ...oh my

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
From the Device to the Warehouse...
• Devices → Warehouse/Regional
• Minimal grooming
• Apache NiFi regional and core
• May not have compute capacity
• ‘Cause a toaster’s gonna toast, toast, toast, toast [a]

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
https://pixabay.com/en/network-iot-internet-of-things-782707/
Edge Use Cases
• Connected car
• Cell phone collection
• Smart watches
• Sensor readers
• ...OR
• Custom devices
• Robotics
• …oh my?

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Living on the Edge
• Edge processing increases capacity
• Edge devices can be ..
• Raspberry Pi
• Computer
• Phone/Watch
• Have different idiosyncrasies

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Myriad of use cases
• Agent/Library
• Data collector/Sender
• Edge/Analytic Processor

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
Idiosyncrasies of building
• Many caveats and ways to…
• Run
• Build
• Use…
• Must account for all of this
• Custom builds

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
The struggle is real
• Software Versions
• Operating system variances
• Dependency management
• Custom hardware
• Custom software

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
IoT/Edge Runtime Nuances [1]
• Limited computing capability
• Limited power/network
• Restricted software library/platform availability
• Not frequently updated
• Competing standards/protocols
• Scalability
• Privacy & Security

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Resource Awareness
• Battery Aware resource manager [3]
• Monitor consumption
• Adjust as needed
• Network manager
• Durable repositories

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Command and Control
• Command and control design [6,7]
• Provide updates
• Need full control
• Full disks
• Disk failure

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Connected car -- Data collector
IoT/Collector Examples

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Connected Car concerns
• Limited Processing
• Power is transient
• May not have ability to shutdown gracefully

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Edge Example
• Robotic controller
• Not typical use case
• OpenCV detects red in image *
• ListenHTTP receives movement commands
• Computation determines motor speed through GPIO
• Computation for servo control

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Concerns faced here
• Device has processing, but must be battery aware
• Throttle if necessary
• Storage must be durable
• Power loss
• Device Failure
• Persistence can’t be a bottleneck

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
What we’ve learned
• Not all use cases created equally

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved
That boy sure is a runnin’ fool
• Failure is always an option
• Upgrades to flow/binary must be small
• Mo’ features, Mo’ problems

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Break it down simple for me
• Build a bear is the best approach
• Control data and resources
• Expect failure

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you
phrocker@apache.org // @MrTheSegfault // https://www.parisi.io

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved
[1] https://www.statista.com/statistics/471264/iot-number-of-connected-devices-
worldwide/
[2] https://www.slideshare.net/Hadoop_Summit/apache-nifi-crash-course
[3] https://www.youtube.com/watch?v=1TMArQS7w40
[4] https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md
[5] https://blog.sasken.com/the-connected-car/
[6] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[7] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal
[8] https://github.com/apache/nifi-minifi-cpp/#managing-minfi-c-through-the-minifi-
controller
[9] https://youtu.be/a9EoCRYu39o
[a] https://www.youtube.com/watch?v=nfWlot6h_JM
Bibliography