TALK TRACK The emergence and explosion from the Internet of Anything data puts tremendous pressure on the existing platforms.
Exponential Growth. As of 2014 there was an estimated 4ZB of data across the cybersphere, and that is expected to grow to 44ZB by 2020, with 85% of this data growth coming from newer types of data from sources like sensors and machines, geo-location tracking devices, server logs, clickstreams, social media or emails and shared files. Variable structures. The incoming data is often unstructured, or its structure changes too frequently for reliable schema creation at time of ingest. Low Value Per Unit, but High in Aggregate. The incoming data can have little or no value as individual, or small groups of, records. But at high volumes and with longer retention horizons, the enterprise can find previously unknown patterns. Advanced analytic applications turn these new insights into business value.
This insight is transforming business outcomes in every major industry, but to participate in that transformation, companies must first ingest that new data into an analytic platform.
TALK TRACK The IoAT data edges created specific data flow requirements that Hortonworks DataFlow satisfies:
Edges with small footprints operate with very little power Limited bandwidth and high latency are commonplace Data availability often exceeds transmission bandwidth Data must be secured throughout its journey
What is the announcement? Hortonworks has signed a definitive agreement to acquire Onyara, including the Onyara products and team of engineers developing and supporting their products. The new Hortonworks DataFlow powered by Apache NiFi, an open source project based on technology that has been in development at the NSA as “Niagara Files” for the last 8 years, is complementary to the Hortonworks Data Platform. With this acquisition, customers will be able to securely and easily collect, conduct and curate any type of data from any origin with the new Hortonworks DataFlow offering. Traditional Data at rest as well as real time data in motion can now be blended to provide historical and perishable insights for predictive analytic.
What is the rationale behind the acquisition? As more and more data is generated from every possible source (machines, sensors, IoT, streaming, social, etc) Hortonworks capitalized on the opportunity to acquire key technology to augment and complement the Hortonworks Data Platform. Onyara, a spin out of the NSA Technology Transfer Program, has contributed and developed Apache NiFi over the last 8 years and have created a compelling set of tools to collect, conduct, and curate data. The new Hortonworks DataFlow powered by Apache NiFi provides the ability for more data to be delivered into the Hortonworks Data Platform and delivers full fidelity analytics on all data for every Hortonworks customer. Onyara’s employees, technology and products are complementary to Hortonworks’. With this acquisition, Hortonworks will be positioned as a leader in IoAT and Big Data with the Hortonworks DataFlow and Hortonworks Data Platform.
Focus on predictive analytics case – use the uptake/cat/etc.. Case but generified.
Introduce the architecture of NiFi, describe major system components, and describe the single node and clustering models. For each component describe its available (and potential)deployment models (relate it to Hadoop).
Focus on the two deployment models (single node & cluster) roughly think of this as ‘edge’ vs ‘data center’
Introduction to Apache NiFi - Seattle Scalability Meetup