Apache NiFi MiNiFi C++ and the tale of edgey stuff

•

2 likes•451 views

In this presentation we discuss IoT and Edge use cases with a focus on how MiNiFi C++ has been able to tackle both while clearly living within the constraints dictated by these use cases.

Engineering

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi MiNiFi C++ and the tale of
edgey stuff
Marc Parisi
phrocker@apache.org
@MrTheSegFault

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
BIO
marc(int*):
push rbp
mov rbp, rsp
mov QWORD PTR [rbp-8], rdi
mov QWORD PTR [rbp-8], 0
mov rax, QWORD PTR [rbp-8]
mov eax, DWORD PTR [rax]
pop rbp
ret
https://www.s.mrsegfault.com/jailbreak

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
⚫ IoT / Edge Breakdown
⚫ MiNIFI C++ deep dive
⚫ Use Cases
⚫ Demo if we're lucky

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT Versus Edge
⚫ IoT
− Devices that share and collect data
⚫ Edge
− Data processing at edge of one’s network ( or the
internet ?)
⚫ One in the same?

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Explosion of Devices

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT use cases
⚫ Connected car
⚫ Cell phone collection
⚫ Smart watches
⚫ Sensor readers
⚫ ...oh my
https://pixabay.com/en/network-iot-internet-of-things-782707/

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
From the device to the warehouse...
⚫ Devices → Warehouse/Regional
− Minimal grooming
− Apache NiFi regional and core
⚫ May not have compute capacity
− ‘Cause a toaster’s gonna toast, toast, toast, toast
[a]

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Edge use cases
⚫ Connected car
⚫ Cell phone collection
⚫ Smart watches
⚫ Sensor readers
⚫ ...OR
⚫ Custom devices
⚫ Robotics
⚫ …oh my?
https://pixabay.com/en/network-iot-internet-of-things-782707/

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Living on the Edge
⚫ Edge processing increases capacity
⚫ Edge devices can be ..
− Raspberry Pi
− Computer
− Smart Phone/Watch

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT/Edge Constraints [1]
• Limited computing capability
• Limited power/network
• Restricted software library/platform availability
• Not frequently updated
• Competing standards/protocols
• Scalability
• Privacy & Security

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi MiNiFi C++
⚫ “...[C]omplimentary data collection approach
that supplements the core tenets of NiFi in
dataflow management, focusing on the
collection of data at the source of its creation”
⚫ Deliver data across a common platform of
extensions and components
⚫ NiFi empowers infrastructure

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The NiFi Ecosystem

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Not just another NiFi
• Bi-Directional (Guaranteed) data transfer
• Data Provenance
• Resource Awareneness
• Small memory footprint
• Custom set of processors
• State Management/Command and Control
• Flow Updates
• Flow Metrics *No JVM

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Not just another NiFi
• Processors
• GetUSBCamera
• Tensorflow
• GetGPS
• Full list [4]

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Versace-ability
⚫ Agent/Library
⚫ Data collector/Sender
⚫ Edge/Analytic Processor

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Resource Awareness
• Battery Aware resource manager [3]
• Monitor consumption
• Adjust as needed
• Network manager (coming soon...)

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Command and Control
⚫ Command and control design [6,7]
⚫ Command line controller [8]
⚫ Update flow dynamically
⚫ Start/Stop components
⚫ Clear Queues/Repos
⚫ No UI
⚫ Remote/Local Admin

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT/Collector Use Cases
⚫ Connected car -- Data collector

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car
• Simple collection
• Tail File or gather GPS information
• Minimal Processing

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Edge use case
• Robotic controller
• Not typical use case
• Ferdinand
• OpenCV detects red in image *
• ListenHTTP receives movement commands
• Computation determines motor speed
through GPIO
• Computation for servo control

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Edge use case

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bibliography
[1] https://www.statista.com/statistics/471264/iot-number-of-connected-devices-
worldwide/
[2] https://www.slideshare.net/Hadoop_Summit/apache-nifi-crash-course
[3] https://www.youtube.com/watch?v=1TMArQS7w40
[4] https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md
[5] https://blog.sasken.com/the-connected-car/
[6] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[7] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal
[8] https://github.com/apache/nifi-minifi-cpp/#managing-minfi-c-through-the-minifi-
controller
[9] https://youtu.be/a9EoCRYu39o
[a] https://www.youtube.com/watch?v=nfWlot6h_JM

Similar to Apache NiFi MiNiFi C++ and the tale of edgey stuff

Building a Smarter Home with Apache NiFi and Spark

DataWorks Summit/Hadoop Summit

Apache NiFi + Tensorflow + Hadoop：Big Data AI サンドイッチの作り方

HortonworksJapan

In this talk, Edgar Orendain walks through a modern real-time streaming application serving as a reference framework for developing a big data pipeline, complete with a broad range of use cases and powerful reusable core components. Modern applications can ingest data and leverage analytics in real-time. These analytics are based on machine learning models typically built using historical big data. This reference application provides examples of connecting data-in-motion analytics to your application based on Big Data. We review code, best practices and considerations involved when integrating different components into a complete data platform. From IoT sensor data collection, to flow management, real-time stream processing and analytics, through to machine learning and prediction, this reference project aims to help developers seed their own open source solutions – fast.

Building a modern end-to-end open source Big Data reference application

DataWorks Summit

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python For processing images from IoT devices like Raspberry Pis, NVidia Jetson TX1, NanoPi Duos, and more that are equipped with attached cameras or external USB webcams, we use Python to interface via OpenCV and PiCamera. From there we run image processing at the edge on these IoT device using OpenCV and TensorFlow to determine attributes and image analytics. Apache MiniFi coordinates running these Python scripts and decides when and what to send from that analysis and the image to a remote Apache NiFi server for additional processing. At the Apache NiFi cluster, in the cluster it routes the images to one processing path and the JSON encoded metadata to another flow. The JSON data (with its schema referenced from a central Schema Registry) is routed using Record Processing and SQL. This data in enriched and augmented before conversion to AVRO to be sent via Apache Kafka to SAM. Streaming Analytics Manager then does deeper processing on this stream and others including weather and twitter to determine what should be done on this data. https://github.com/tspannhw/OpenSourceComputerVision http://dataworkssummit.com/ San Jose 2018 DWS18

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

Timothy Spann

[Hortonworks] Future Of Data: Madrid - HDF & Data in motion

Raúl Marín

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Aldrin Piri

Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi

Aldrin Piri

Description: MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love. Expected prior knowledge / intended audience: developers and data flow managers should have passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi. Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance.

Intelligently Collecting Data at the Edge – Intro to Apache MiNiFi

DataWorks Summit

Introduction to Apache NiFi - Seattle Scalability Meetup

Saptak Sen

IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018

Timothy Spann

Enterprise IIoT Edge Processing with Apache NiFi

Timothy Spann

Apache NiFi Crash Course - San Jose Hadoop Summit

Aldrin Piri

Originally created for Hadoop Summit 2016: Melbourne. http://www.hadoopsummit.org/melbourne/ Apache NiFi is becoming a defacto tool for handling orchestration, routing and mediation of data in the highly complex and heterogeneous world of Big Data, connecting many components (in-motion and at-rest) of its ecosystem into one homogenous and secure data flow. And while features such as security, provenance, dynamic prioritization and extensibility have long captured the attention of the enterprises, the innovation in NiFi land continues. This hands-on talk consisting of live demos and code will concentrate on what’s new an exciting in the world of NiFi. It will cover the newest and most advanced features of NiFi as well as demonstrate some of the "work in progress" essentially giving you a preview into the future.

The Avant-garde of Apache NiFi

Joe Percivall

The Avant-garde of Apache NiFi

DataWorks Summit/Hadoop Summit

Presentation from Future of Data Boston Meetup on Oct 24, 2017. Streaming data is rich with insights but these insights can be difficult to find due to the difficulty of developing and deploying streaming applications. During this presentation we will show how to build and deploy a complex streaming application in a few minutes using open source tools. First we will build an application using Streaming Analytics Manager and Schema Registry that ingests data into Apache Druid. Then we will use Apache Superset to build beautiful, informative dashboards.

Unlocking insights in streaming data

Carolyn Duby

HDF Powered by Apache NiFi Introduction

Milind Pandit

Apache NiFi MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love. Expected prior knowledge / intended audience: developers and data flow managers should have a passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi. Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance. Speaker: Andy LoPresto, Sr. Member of Technical Staff, Hortonworks

The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi

DataWorks Summit

Make Streaming IoT Analytics Work for You

Hortonworks

Connecting enterprise systems has always been a tough task. Modern IoT applications have exacerbated the issue by the need to integrate legacy systems with novel high velocity data streams. Various patterns like messaging, REST, etc. have been proposed, but they necessitate rearchitecting the integration layer which is extremely arduous. In this talk we will show you how to use Apache NiFi to solve your data integration, movement and ingestion problems. Next, we will examine how Apache NiFi can be used to construct durable, scalable and responsive IoT apps in conjunction with other stream processing and messaging frameworks.

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...

Data Con LA

In recent years, big data has moved from batch processing to stream-based processing since no one wants to wait hours or days to gain insights. Dozens of stream processing frameworks exist today and the same trend that occurred in the batch-based big data processing realm has taken place in the streaming world so that nearly every streaming framework now supports higher level relational operations. On paper, combining Apache NiFi, Kafka, and Spark Streaming provides a compelling architecture option for building your next generation ETL data pipeline in near real time. What does this look like in an enterprise production environment to deploy and operationalized? The newer Spark Structured Streaming provides fast, scalable, fault-tolerant, end-to-end exactly-once stream processing with elegant code samples, but is that the whole story? We discuss the drivers and expected benefits of changing the existing event processing systems. In presenting the integrated solution, we will explore the key components of using NiFi, Kafka, and Spark, then share the good, the bad, and the ugly when trying to adopt these technologies into the enterprise. This session is targeted toward architects and other senior IT staff looking to continue their adoption of open source technology and modernize ingest/ETL processing. Attendees will take away lessons learned and experience in deploying these technologies to make their journey easier. Speaker: Andrew Psaltis, Principal Solution Engineer, Hortonworks

Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise

DataWorks Summit

Similar to Apache NiFi MiNiFi C++ and the tale of edgey stuff (20)

Building a Smarter Home with Apache NiFi and Spark

Apache NiFi + Tensorflow + Hadoop：Big Data AI サンドイッチの作り方

Building a modern end-to-end open source Big Data reference application

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

[Hortonworks] Future Of Data: Madrid - HDF & Data in motion

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi

Intelligently Collecting Data at the Edge – Intro to Apache MiNiFi

Introduction to Apache NiFi - Seattle Scalability Meetup

IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018

Enterprise IIoT Edge Processing with Apache NiFi

Apache NiFi Crash Course - San Jose Hadoop Summit

The Avant-garde of Apache NiFi

Unlocking insights in streaming data

HDF Powered by Apache NiFi Introduction

The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi

Make Streaming IoT Analytics Work for You

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...

Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise

Recently uploaded

BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING

KOUSTAV SARKAR

The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...

Roi Lipman

Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf

JNTUA

Object Oriented Programming OOP Lab Manual.docx

RashidFaridChishti

E-Commerce Shopping using MERN Stack where different modules are present

jatinraor66

Electronic Shop Management software helps Electronic showrooms owners and management staff by producing different kind of financial and stock tracking reports, etc. This software is able to manage all electronic stocks. In this software shop owner can manage the data of customer and buyers. Also tax information and other government charges including recycle charges. Electrical shop management system is workable application for retail store inventory and account management. It keeps a list STOCKS and products at a store and can do operation on them. The most important operation on them. The most important operation is a PURCHASE, all the transactions and the billing details and stock purchasing details involved on it

Electrical shop management system project report.pdf

Kamal Acharya

The project work entitled “Paint Shop Management system”. This Project helps us to maintain the details of all the goods in the sports shop in computerized way. Computers are one of the most important tasks between people in everyday life. One would find it very difficult to live without computers. If computer all over the world come to a half or stand still, it would be like the earth has stopped revolving. Computer is must in today’s life. No matter is what application it is a computer assisted system much more beneficial than the manual system. Computers help not only in aspects for faster. Computations but also lies in the most important factor of accuracy. One might be fast at calculation at the fact lies to if he is accurate. Only here the computer plays a vital role and also the amount of data it can store, using related software helps in storing huge quantities of data. Technology has developed at a very faster rate but the most important aspect of any system is the human know-how and the use of ideas to gear the computer. So that it perform required task. This process is essentially what system development is all about. The main objective of this project is to Computerized all the manual function of the concern. The entire manual work is converted into computerized through different modules of this System. Using Visual Basic 6.0 as Front End and MS-Access as back-End and platform as Windows-98 developed the system.

Paint shop management system project report.pdf

Kamal Acharya

How to Design and spec harmonic filter.pdf

tawat puangthong

The Teachers Record Management System is a comprehensive solution designed to simplify the process of managing teacher records. In today's fast-paced world, it is imperative for schools and individuals to have access to accurate and up-to-date information about teachers. This system addresses the needs of both individuals and schools, offering an efficient and effective way to manage teacher records. This web-based technology was developed with the aim of providing an easy-to-use platform that allows users to search and manage teacher records with ease. The system is designed to be user-friendly, making it accessible to individuals with varying levels of computer literacy. The interface is intuitive and easy to navigate, making it possible for users to quickly find the information they need. For individuals who are searching for good teachers, the Teachers Record Management System offers a comprehensive database of teacher records that can be searched using various parameters such as subject, qualification, experience, and location. This allows users to find the best teachers that match their requirements in a matter of minutes. The system also offers the ability to rate and review teachers, providing valuable feedback to other users who are looking for good teachers. For schools, the Teachers Record Management System offers a robust and secure platform for managing teacher records. The system allows schools to maintain accurate and up-to-date records of their teachers, including their personal information, academic qualifications, work experience, and other relevant details. This makes it easy for schools to keep track of their teachers and ensure that they have the right teachers for the job. The system also offers a range of features that help schools manage their teachers more efficiently. For example, the system allows schools to schedule classes and assign teachers to specific classes based on their qualifications and experience. It also offers a range of reporting and analytics tools that enable schools to track their teacher's performance and identify areas where improvements can be made. In conclusion, the Teachers Record Management System is a powerful tool that offers a range of benefits to both individuals and schools. It provides an easy and efficient way to search and manage teacher records, allowing users to find the best teachers that match their requirements. The system is also designed to be user-friendly and offers a range of features that help schools manage their teachers more efficiently. Overall, the Teachers Record Management System is an essential tool for anyone who wants to find good teachers or manage their teacher records.

Teachers record management system project report..pdf

Kamal Acharya

RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5

T.D. Shashikala

Lect_Z_Transform_Main_digital_image_processing.pptx

MonirHossain707319

Diploma Engineering Drawing Qp-2024 Ece .pdf

JNTUA

Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...

Lovely Professional University

Multivibrator and its types defination and usges.pptx

alijaker017

The Project deals with the development of the computerized system for maintaining the regular records and services that are undertaken in the furniture business. This project titled "Web Based Integrated Furniture Showroom Management System" has been aimed to design and computerized system that can handle various activities are been carried out at the Furniture Showroom. This application has been developed using PHP Programming Language as its front end and the back end is MYSQL Server In the existing system all the activities and record maintenance of the furniture showroom are done manually by the manager. The Project deals with the development of the computerized system for maintaining the regular records and services that are undertaken in this most important and large business oriented furniture business. This Project also enables the users to perform all the day to day business operations in the furniture showroom business most efficiently.

Furniture showroom management system project.pdf

Kamal Acharya

Research Methodolgy & Intellectual Property Rights Series 1

T.D. Shashikala

Software Engineering - Part 2 which describes the following topics: Introduction, Modelling Concepts and Class Modelling: What is Object orientation? What is OO development? OO Themes; Evidence for usefulness of OO development; OO Modelling history. Modelling as Design technique: Modelling, abstraction, The Three models. Class Modelling: Object and Class Concept, Link and associations concepts, Generalization and Inheritance, A sample class model, Navigation of class models, and UML diagrams. Building the Analysis Models: Requirement Analysis, Analysis Model Approaches, Data Modelling Concepts.

Software Engineering - Modelling Concepts + Class Modelling + Building the An...

Prakhyath Rai

This topic aims to describe the project background, problem statement, objectives, scopes, project significance and expected output of the system. • The system is Burger Ordering System. This is online Customer Ordering System of Restaurant, which in most cases; the company has problem with order and disordered order. • This project intends to computerize Burger Ordering System to provide better customer service. Because of that, the restaurant can provide the easier way of travelling to the customer. • Burger Oder system aims to accelerate customer orders. • Burger Ordering System used by servers and kitchen employees to accept customer orders. • An online ordering system is not a new concept to many as this has been running successfully all over the world for quite some time now. • The whole concept behind having a professional website along with a system shows how you present yourself to the online world.

BURGER ORDERING SYSYTEM PROJECT REPORT..pdf

Kamal Acharya

ANSI(ST)-III_Manufacturing-I_05052020.pdf

BertinKamsipa1

The project developers created a system entitled Resort Management and Reservation System; it will provide better management and monitoring of the services in every resort business, especially D’ Rock Resort. To accommodate those out-of-town guests who want to remain and utilize the resort's services, the proponents planned to automate the business procedures of the resort and implement the system. As a result, it aims to improve business profitability, lower expenses, and speed up the resort's transaction processing. The resort will now be able to serve those potential guests, especially during the high season. Using websites for faster transactions to reserve on your desired time and date is another step toward technological advancement. Customers don’t need to walk in and hold in line for several hours. There is no problem in converting a paper-based transaction online; it's just the system that will be used that will help the resort expand. Moreover, Gerard (2012) stated that “The flexible online information structure was developed as a tool for the reservation theory's two primary applications. Computer use is more efficient, accurate, and faster than a manual or present lifestyle of operation. Using a computer has a vital role in our daily life and the advantages of the devices we use.

RESORT MANAGEMENT AND RESERVATION SYSTEM PROJECT REPORT.pdf

Kamal Acharya

Recently uploaded (20)

BRAKING SYSTEM IN INDIAN RAILWAY AutoCAD DRAWING

The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...

Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf

Object Oriented Programming OOP Lab Manual.docx

E-Commerce Shopping using MERN Stack where different modules are present

Electrical shop management system project report.pdf

Paint shop management system project report.pdf

How to Design and spec harmonic filter.pdf

Teachers record management system project report..pdf

RM&IPR M5 notes.pdfResearch Methodolgy & Intellectual Property Rights Series 5

Lect_Z_Transform_Main_digital_image_processing.pptx

Diploma Engineering Drawing Qp-2024 Ece .pdf

Activity Planning: Objectives, Project Schedule, Network Planning Model. Time...

Multivibrator and its types defination and usges.pptx

Furniture showroom management system project.pdf

Research Methodolgy & Intellectual Property Rights Series 1

Software Engineering - Modelling Concepts + Class Modelling + Building the An...

BURGER ORDERING SYSYTEM PROJECT REPORT..pdf

ANSI(ST)-III_Manufacturing-I_05052020.pdf

RESORT MANAGEMENT AND RESERVATION SYSTEM PROJECT REPORT.pdf

Apache NiFi MiNiFi C++ and the tale of edgey stuff

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved BIO marc(int*): push rbp mov rbp, rsp mov QWORD PTR [rbp-8], rdi mov QWORD PTR [rbp-8], 0 mov rax, QWORD PTR [rbp-8] mov eax, DWORD PTR [rax] pop rbp ret https://www.s.mrsegfault.com/jailbreak

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT Versus Edge ⚫ IoT − Devices that share and collect data ⚫ Edge − Data processing at edge of one’s network ( or the internet ?) ⚫ One in the same?

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT use cases ⚫ Connected car ⚫ Cell phone collection ⚫ Smart watches ⚫ Sensor readers ⚫ ...oh my https://pixabay.com/en/network-iot-internet-of-things-782707/

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved From the device to the warehouse... ⚫ Devices → Warehouse/Regional − Minimal grooming − Apache NiFi regional and core ⚫ May not have compute capacity − ‘Cause a toaster’s gonna toast, toast, toast, toast [a]

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Edge use cases ⚫ Connected car ⚫ Cell phone collection ⚫ Smart watches ⚫ Sensor readers ⚫ ...OR ⚫ Custom devices ⚫ Robotics ⚫ …oh my? https://pixabay.com/en/network-iot-internet-of-things-782707/

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT/Edge Constraints [1] • Limited computing capability • Limited power/network • Restricted software library/platform availability • Not frequently updated • Competing standards/protocols • Scalability • Privacy & Security

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi MiNiFi C++ ⚫ “...[C]omplimentary data collection approach that supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation” ⚫ Deliver data across a common platform of extensions and components ⚫ NiFi empowers infrastructure

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Not just another NiFi • Bi-Directional (Guaranteed) data transfer • Data Provenance • Resource Awareneness • Small memory footprint • Custom set of processors • State Management/Command and Control • Flow Updates • Flow Metrics *No JVM

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Command and Control ⚫ Command and control design [6,7] ⚫ Command line controller [8] ⚫ Update flow dynamically ⚫ Start/Stop components ⚫ Clear Queues/Repos ⚫ No UI ⚫ Remote/Local Admin

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Edge use case • Robotic controller • Not typical use case • Ferdinand • OpenCV detects red in image * • ListenHTTP receives movement commands • Computation determines motor speed through GPIO • Computation for servo control

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bibliography [1] https://www.statista.com/statistics/471264/iot-number-of-connected-devices- worldwide/ [2] https://www.slideshare.net/Hadoop_Summit/apache-nifi-crash-course [3] https://www.youtube.com/watch?v=1TMArQS7w40 [4] https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md [5] https://blog.sasken.com/the-connected-car/ [6] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control [7] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal [8] https://github.com/apache/nifi-minifi-cpp/#managing-minfi-c-through-the-minifi- controller [9] https://youtu.be/a9EoCRYu39o [a] https://www.youtube.com/watch?v=nfWlot6h_JM

Apache NiFi MiNiFi C++ and the tale of edgey stuff

Recommended

Recommended

More Related Content

Similar to Apache NiFi MiNiFi C++ and the tale of edgey stuff

Similar to Apache NiFi MiNiFi C++ and the tale of edgey stuff (20)

Recently uploaded

Recently uploaded (20)

Apache NiFi MiNiFi C++ and the tale of edgey stuff