Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly

•

19 likes•8,392 views

This document summarizes a presentation about generating real-time streaming recommendations using NiFi, Kafka, and Spark ML. The presentation demonstrates using NiFi to ingest data from HTTP requests, enrich it with geo data, and write it to a Kafka topic. It then shows how to create a Spark Streaming application that reads from Kafka to perform incremental matrix factorization recommendations in real-time and handles failures using circuit breakers. The presentation also provides an overview of Netflix's large-scale real-time recommendation pipeline.

Software

Generating Real-time,
Streaming Recommendations
[NiFi + Kafka + Spark ML]
Kafka Summit SF
April 26, 2016

Who am I?
Chris Fregly, Principal Data Solutions Engineer
@ IBM Spark Technology Center
Previously, Data Engineer @ Netflix and Databricks
Contributor @ Apache Spark, Committer @ Netflix OSS
Founder @ Advanced Spark and TensorFlow
Author @ Advanced Spark (advancedspark.com)

Relevant Spark Contribution
SPARK-1981: Add Kinesis support for Spark Streaming
Me

Fun Workshop!!
San Jose: May 14th (full details @ advancedspark.com)

Agenda
Live, Interactive Demo!
NiFi
Spark Streaming
Streaming Recommendations
Netflix Pipeline (Bonus!)

Live, Interactive Demo
http://demo2.advancedspark.com

Agenda
Live, Interactive Demo!
NiFi
Spark Streaming
Streaming Recommendations
Netflix Recommendations (Bonus!)

NiFi
NiFi = “Niagra Files”
Maintainers @ Hortonworks since 2015
Developed @ NSA over last 8+ years
Integrates with EVERYTHING!
Provides Data Provenance
Data Flow Management
Me,
Normal Guy
Joe Witt,
NiFi Co-Creator
Buffalo
Wild Wings
Hat

NiFi Provenance Event Types
ATTRIBUTES_MODIFIED (ie. Extract Topic Name)
CONTENT_MODIFIED (ie. Enrich with Geo)
RECEIVE (ie. Handle Http Request)
ROUTE (ie. Check Http Method)
SEND (ie. PutKafka)
DROP (Handle Http Response)

Spark Streaming
Submits Time-Based Micro Batches of Data as Spark Jobs
Supports Kinesis, Flume, MQTT, ZeroMQ, Sockets, KAFKA!
Framework for Custom Streaming Receivers
Flexible Window Operations, Optimized State Management
Basic Back Pressure and Throttling Support
At Least Once Guarantees through Write Ahead Log (WAL)

Spark Streaming KafkaRDD
Kafka “Direct” Streaming Implementation (Spark 1.4+)
Recover/Replay from Kafka using File System-like Offsets
Removes need for Write Ahead Log (WAL)
Uses Kafka, itself, as the WAL!
KafkaRDD

Streaming Recommendations
Incremental Matrix Factorization!!
(Based on github.com/brkyvz/streaming-matrix-factorization)

Recommendation Serving Layer
Use Case: Recommendation Service Depends on Redis Cache
Problem: Redis Cache Goes Down!?
Answer: github.com/Netflix/Hystrix Circuit Breaker!
Circuit States:
Closed: Service OK
Open: Service DOWN
Fallback to Non-Personalized Recommendations from Disk

Netflix Data Pipeline
9 million events, 22 GB per second @ peak!
EC2 D2XL
Disk: 6 TB, 475 MB/s
RAM: 30 G
Network: 700 Mbps
Auto-scaling,
Fault tolerance
A/B Tests,
Trending Now
SAMZA
Splits high and
normal priority

Recommendations Pipeline
Batch Matrix Factorization
Keep Batch Video (V) Matrix
Calculate Newer User (U) Matrix
Compute U x V Dot Product
Save Model to Disk and EVCache
https://github.com/Netflix/EVCache
Throw away
batch user factors (U)
Keep video
factors (V)

Thank You, Kafka Summit SF!
Chris Fregly
@cfregly
All Source Code, Demos, and Docker Images Available
@ advancedspark.com,
github.com/fluxcapacitor/pipeline
Join the Global Meetup for Slides, Videos, Book
@ advancedspark.com

Summarizes new capabilities added to Apache NiFi 1.2.0 (soon to be released). Disclaimer: - The contents in this slide deck are derived from Apache NiFi JIRA issues which is labeled with next release target 1.2.0 and source code available at Github (already merged into master branch), however it does NOT mean these are guaranteed to be released and still are subjects to change. - The motivation of this presentation is share what have been introduced into the project since the latest Apache NiFi 1.1.2 release. - The contents are created from information available under Apache NiFi project, however, the way summarize it is solely done with my personal thoughts and not a consensus built among Apache NiFi community.

The Avant-garde of Apache NiFi

DataWorks Summit/Hadoop Summit

Extending the Yahoo Streaming Benchmark + MapR Benchmarks

Jamie Grier

NiFi Developer Guide

Deon Huang

Machine Learning in the IoT with Apache NiFi

DataWorks Summit/Hadoop Summit

Across the globe energy systems are changing, creating unprecedented challenges for the organisations tasked with ensuring the lights stay on. In the UK, National Grid is facing shrinking margins, looming capacity shortages and unpredictable peaks and troughs in energy supply caused by increasing levels of renewable penetration. Open Energi uses its IoT technology to unlock demand-side capacity - from industrial equipment, co-generation and batery storage systems - creating a smarter grid; one that is cleaner, cheaper, more secure and more efficient. I'll talk about how we use Apache Nifi to orchestrate and coordinate Machine Learning microservices that operate on streams of data coming from IoT devices, providing a layer of fault-tolerance and traceability. With built-in retry logic, backpressure and clustering, Nifi helps us keep hard problems away from our code. It comes with processors that integrate with our cloud provider of choice (Microsoft Azure), fitting seamlessly into our processing pipeline.Finally, its straightforward graphical interface makes it easy enough to use that any team member can step in and troubleshoot a flow with little training.

No real-time insight without real-time data ingestion. No real-time data ingestion without NiFi ! Apache NiFi is an integrated platform for data flow management at entreprise level, enabling companies to securely acquire, process and analyze disparate sources of information (sensors, logs, files, etc) in real-time. NiFi helps data engineers accelerate the development of data flows thanks to its UI and a large number of powerful off-the-shelf processors. However, with great power comes great responsibilities. Behind the simplicity of NiFi, best practices must absolutely be respected in order to scale data flows in production & prevent sneaky situations. In this joint presentation, Hortonworks and Renault, a French car manufacturer, will present lessons learnt from real world projects using Apache NiFi. We will present NiFi design patterns to achieve high level performance and reliability at scale as well as the process to put in place around the technology for data flow governance. We will also show how these best practices can be implemented in practical use cases and scenarios. Speakers Kamelia Benchekroun, Data Lake Squad Lead, Renault Group Abdelkrim Hadjidj, Solution Engineer, Hortonworks

Architecting for Scale

Pooyan Jamshidi

Building Streaming Applications with Apache Storm 1.1

Hugo Louro

Seattle spark-meetup-032317

Nan Zhu

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Apache Kafka 0.8 basic training (120 slides) covering: 1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka 2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers 3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning 4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps 5. Playing with Kafka using Wirbelsturm Audience: developers, operations, architects Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/ Verisign is a global leader in domain names and internet security. Tools mentioned: - Wirbelsturm (https://github.com/miguno/wirbelsturm) - kafka-storm-starter (https://github.com/miguno/kafka-storm-starter) Blog post at: http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/ Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!

Music city data Hail Hydrate! from stream to lake

Timothy Spann

Running Apache Spark & Apache Zeppelin in Production

DataWorks Summit/Hadoop Summit

ApacheCon 2021 - Apache NiFi Deep Dive 300

Timothy Spann

21-September-2021 - ApacheCon - Tuesday 17:10 UTC Apache NIFi Deep Dive 300 * https://github.com/tspannhw/EverythingApacheNiFi * https://github.com/tspannhw/FLiP-ApacheCon2021 * https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html * https://github.com/tspannhw/FLiP-IoT * https://github.com/tspannhw/FLiP-Energy * https://github.com/tspannhw/FLiP-SOLR * https://github.com/tspannhw/FLiP-EdgeAI * https://github.com/tspannhw/FLiP-CloudQueries * https://github.com/tspannhw/FLiP-Jetson * https://www.linkedin.com/pulse/2021-schedule-tim-spann/ Tuesday 17:10 UTC Apache NIFi Deep Dive 300 Timothy Spann For Data Engineers who have flows already in production, I will dive deep into best practices, advanced use cases, performance optimizations, tips, tricks, edge cases, and interesting examples. This is a master class for those looking to learn quickly things I have picked up after years in the field with Apache NiFi in production. This will be interactive and I encourage questions and discussions. You will take away examples and tips in slides, github, and articles. This talk will cover: Load Balancing Parameters and Parameter Contexts Stateless vs Stateful NiFi Reporting Tasks NiFi CLI NiFi REST Interface DevOps Advanced Record Processing Schemas RetryFlowFile Lookup Services RecordPath Expression Language Advanced Error Handling Techniques Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

Using FLiP with influxdb for edgeai iot at scale 2022

Timothy Spann

DBCC 2021 - FLiP Stack for Cloud Data Lakes

Timothy Spann

[March sn meetup] apache pulsar + apache nifi for cloud data lake

Timothy Spann

https://www.meetup.com/new-york-city-apache-pulsar-meetup/events/283837865/ Learn how to use Apache Pulsar and Apache NiFi to Stream to your Data Lake Discover how to stream data to and from your data lake or data mart using Apache Pulsar™ and Apache NiFi®. Learn how these cloud-native, scalable open-source projects built for streaming data pipelines work together to enable you to quickly build applications with minimal coding. |WHAT THE SESSION WILL COVER| Best Practices for using Pulsar and NiFi A deep dive on Apache NiFi's Pulsar connector and demos Building an End-to-End Application in the Hybrid Cloud Attend for a chance to win a We <3 Pulsar t-shirt! The first 50 registrants who register through here [https://hubs.ly/Q013LTpn0] will be entered in a drawing! —------------------------ |AGENDA| 6:00 - 7:00 PM EST: Presentation - Tim Spann, StreamNative Developer Advocate 7:00 - 8:00 PM EST: Presentation - John Kuchmek, Cloudera Principal Solutions Engineer 8:00 - 8:30 PM EST: Q&A + Networking —------------------------ |ABOUT THE SPEAKERS| John Kuchmek is a Principal Solutions Engineer for Cloudera. Before joining Cloudera, John transitioned to the Autonomous Intelligence team where he was in charge of integrating the platforms to allow data scientists to work with various types of data. Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar™, Apache Flink®, Flink® SQL, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. He is currently working on a book about the FLiP Stack.

The Power of Intelligent Flows: Real-Time IoT Botnet Classification with Apac...

DataWorks Summit

The last 5 years have been marked by an explosion of Internet-connected devices. From cars to solar power, from TVs to juice makers, modern life is filled with interconnected smart devices. But while those ubiquitous devices enhance the interaction with the technology that surrounds us, the lifecycle management of IoT firmware and poor security design choices still present a significant threat to our daily lives. Despite the ascent of threats like the Mirai botnet, the amount of published research around how to programmatically detect new IoTs in the wild has been somewhat limited. In this presentation we introduce Data Engineering in the context of cyber security, discuss why it is important to move away from the view that security log pipelines are enrichment and indicator matching tools, and push the boundaries of “Simple Event Processing” to demonstrate how Apache NiFi and Apache MiNiFi’s feature rich dataflows can be used to dynamically identify new IoT botnet activities in the wild. Speakers Andre Fucs De Miranda, Independent Consultant, Fluenda Andy LoPresto, Sr. Member of Technical Staff, Hortonworks

fluentd -- the missing log collector

Muga Nishizawa

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Timothy Spann

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-ramp 2022 As the Pulsar communities grows, more and more connectors will be added. To enhance the availability of sources and sinks and to make use of the greater Apache Streaming community, joining forces between Apache NiFi and Apache Pulsar is a perfect fit. Apache NiFi also adds the benefits of ELT, ETL, data crunching, transformation, validation and batch data processing. Once data is ready to be an event, NiFi can launch it into Pulsar at light speed. I will walk through how to get started, some use cases and demos and answer questions. https://www.devfest-uki.com/schedule https://linktr.ee/tspannhw

Machine Learning Preliminaries and Math Refresherbutest

Boston Spark Meetup May 24, 2016

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly

Similar to Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly (20)

More from Chris Fregly

More from Chris Fregly (20)

Recently uploaded

Recently uploaded (20)

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly