Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly

Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly

Software

Generating Real-time,
Streaming Recommendations
[NiFi + Kafka + Spark ML]
Kafka Summit SF
April 26, 2016

Who am I?
Chris Fregly, Principal Data Solutions Engineer
@ IBM Spark Technology Center
Previously, Data Engineer @ Netflix and Databricks
Contributor @ Apache Spark, Committer @ Netflix OSS
Founder @ Advanced Spark and TensorFlow
Author @ Advanced Spark (advancedspark.com)

Relevant Spark Contribution
SPARK-1981: Add Kinesis support for Spark Streaming
Me

Fun Workshop!!
San Jose: May 14th (full details @ advancedspark.com)

Agenda
Live, Interactive Demo!
NiFi
Spark Streaming
Streaming Recommendations
Netflix Pipeline (Bonus!)

Live, Interactive Demo
http://demo2.advancedspark.com

Agenda
Live, Interactive Demo!
NiFi
Spark Streaming
Streaming Recommendations
Netflix Recommendations (Bonus!)

NiFi
NiFi = “Niagra Files”
Maintainers @ Hortonworks since 2015
Developed @ NSA over last 8+ years
Integrates with EVERYTHING!
Provides Data Provenance
Data Flow Management
Me,
Normal Guy
Joe Witt,
NiFi Co-Creator
Buffalo
Wild Wings
Hat

NiFi Provenance Event Types
ATTRIBUTES_MODIFIED (ie. Extract Topic Name)
CONTENT_MODIFIED (ie. Enrich with Geo)
RECEIVE (ie. Handle Http Request)
ROUTE (ie. Check Http Method)
SEND (ie. PutKafka)
DROP (Handle Http Response)

Spark Streaming
Submits Time-Based Micro Batches of Data as Spark Jobs
Supports Kinesis, Flume, MQTT, ZeroMQ, Sockets, KAFKA!
Framework for Custom Streaming Receivers
Flexible Window Operations, Optimized State Management
Basic Back Pressure and Throttling Support
At Least Once Guarantees through Write Ahead Log (WAL)

Spark Streaming KafkaRDD
Kafka “Direct” Streaming Implementation (Spark 1.4+)
Recover/Replay from Kafka using File System-like Offsets
Removes need for Write Ahead Log (WAL)
Uses Kafka, itself, as the WAL!
KafkaRDD

Streaming Recommendations
Incremental Matrix Factorization!!
(Based on github.com/brkyvz/streaming-matrix-factorization)

Recommendation Serving Layer
Use Case: Recommendation Service Depends on Redis Cache
Problem: Redis Cache Goes Down!?
Answer: github.com/Netflix/Hystrix Circuit Breaker!
Circuit States:
Closed: Service OK
Open: Service DOWN
Fallback to Non-Personalized Recommendations from Disk

Netflix Data Pipeline
9 million events, 22 GB per second @ peak!
EC2 D2XL
Disk: 6 TB, 475 MB/s
RAM: 30 G
Network: 700 Mbps
Auto-scaling,
Fault tolerance
A/B Tests,
Trending Now
SAMZA
Splits high and
normal priority

Recommendations Pipeline
Batch Matrix Factorization
Keep Batch Video (V) Matrix
Calculate Newer User (U) Matrix
Compute U x V Dot Product
Save Model to Disk and EVCache
https://github.com/Netflix/EVCache
Throw away
batch user factors (U)
Keep video
factors (V)

Thank You, Kafka Summit SF!
Chris Fregly
@cfregly
All Source Code, Demos, and Docker Images Available
@ advancedspark.com,
github.com/fluxcapacitor/pipeline
Join the Global Meetup for Slides, Videos, Book
@ advancedspark.com

What's hot

Local Apache NiFi Processor Debug

Deon Huang

Apache NiFi User Guide

Deon Huang

StreamNative FLiP into scylladb - scylla summit 2022

Data science online camp using the flipn stack for edge ai (flink, nifi, pu...

Real time stock processing with apache nifi, apache flink and apache kafka

Pulsar summit asia 2021 apache pulsar with mqtt for edge computing

No real-time insight without real-time data ingestion. No real-time data ingestion without NiFi ! Apache NiFi is an integrated platform for data flow management at entreprise level, enabling companies to securely acquire, process and analyze disparate sources of information (sensors, logs, files, etc) in real-time. NiFi helps data engineers accelerate the development of data flows thanks to its UI and a large number of powerful off-the-shelf processors. However, with great power comes great responsibilities. Behind the simplicity of NiFi, best practices must absolutely be respected in order to scale data flows in production & prevent sneaky situations. In this joint presentation, Hortonworks and Renault, a French car manufacturer, will present lessons learnt from real world projects using Apache NiFi. We will present NiFi design patterns to achieve high level performance and reliability at scale as well as the process to put in place around the technology for data flow governance. We will also show how these best practices can be implemented in practical use cases and scenarios. Speakers Kamelia Benchekroun, Data Lake Squad Lead, Renault Group Abdelkrim Hadjidj, Solution Engineer, Hortonworks

Best practices and lessons learnt from Running Apache NiFi at Renault

DataWorks Summit

Architecting for Scale

Pooyan Jamshidi

Building Streaming Applications with Apache Storm 1.1

Hugo Louro

Seattle spark-meetup-032317

Nan Zhu

Apache Kafka 0.8 basic training (120 slides) covering: 1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka 2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers 3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning 4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps 5. Playing with Kafka using Wirbelsturm Audience: developers, operations, architects Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/ Verisign is a global leader in domain names and internet security. Tools mentioned: - Wirbelsturm (https://github.com/miguno/wirbelsturm) - kafka-storm-starter (https://github.com/miguno/kafka-storm-starter) Blog post at: http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/ Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Music city data Hail Hydrate! from stream to lake

DataWorks Summit/Hadoop Summit

Running Apache Spark & Apache Zeppelin in Production

ApacheCon 2021 - Apache NiFi Deep Dive 300

Using FLiP with influxdb for edgeai iot at scale 2022

DBCC 2021 - FLiP Stack for Cloud Data Lakes

https://www.meetup.com/new-york-city-apache-pulsar-meetup/events/283837865/ Learn how to use Apache Pulsar and Apache NiFi to Stream to your Data Lake Discover how to stream data to and from your data lake or data mart using Apache Pulsar™ and Apache NiFi®. Learn how these cloud-native, scalable open-source projects built for streaming data pipelines work together to enable you to quickly build applications with minimal coding. |WHAT THE SESSION WILL COVER| Best Practices for using Pulsar and NiFi A deep dive on Apache NiFi's Pulsar connector and demos Building an End-to-End Application in the Hybrid Cloud Attend for a chance to win a We <3 Pulsar t-shirt! The first 50 registrants who register through here [https://hubs.ly/Q013LTpn0] will be entered in a drawing! —------------------------ |AGENDA| 6:00 - 7:00 PM EST: Presentation - Tim Spann, StreamNative Developer Advocate 7:00 - 8:00 PM EST: Presentation - John Kuchmek, Cloudera Principal Solutions Engineer 8:00 - 8:30 PM EST: Q&A + Networking —------------------------ |ABOUT THE SPEAKERS| John Kuchmek is a Principal Solutions Engineer for Cloudera. Before joining Cloudera, John transitioned to the Autonomous Intelligence team where he was in charge of integrating the platforms to allow data scientists to work with various types of data. Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar™, Apache Flink®, Flink® SQL, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. He is currently working on a book about the FLiP Stack.

[March sn meetup] apache pulsar + apache nifi for cloud data lake

The last 5 years have been marked by an explosion of Internet-connected devices. From cars to solar power, from TVs to juice makers, modern life is filled with interconnected smart devices. But while those ubiquitous devices enhance the interaction with the technology that surrounds us, the lifecycle management of IoT firmware and poor security design choices still present a significant threat to our daily lives. Despite the ascent of threats like the Mirai botnet, the amount of published research around how to programmatically detect new IoTs in the wild has been somewhat limited. In this presentation we introduce Data Engineering in the context of cyber security, discuss why it is important to move away from the view that security log pipelines are enrichment and indicator matching tools, and push the boundaries of “Simple Event Processing” to demonstrate how Apache NiFi and Apache MiNiFi’s feature rich dataflows can be used to dynamically identify new IoT botnet activities in the wild. Speakers Andre Fucs De Miranda, Independent Consultant, Fluenda Andy LoPresto, Sr. Member of Technical Staff, Hortonworks

The Power of Intelligent Flows: Real-Time IoT Botnet Classification with Apac...

DataWorks Summit

fluentd -- the missing log collector

Muga Nishizawa

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-ramp 2022 As the Pulsar communities grows, more and more connectors will be added. To enhance the availability of sources and sinks and to make use of the greater Apache Streaming community, joining forces between Apache NiFi and Apache Pulsar is a perfect fit. Apache NiFi also adds the benefits of ELT, ETL, data crunching, transformation, validation and batch data processing. Once data is ready to be an event, NiFi can launch it into Pulsar at light speed. I will walk through how to get started, some use cases and demos and answer questions. https://www.devfest-uki.com/schedule https://linktr.ee/tspannhw

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Feature Talk: Real-time Aggregations, Approximations, Similarities, and Recommendations at Scale using Spark Streaming, ML, GraphX, Kafka, Cassandra, Docker, CoreNLP, Word2Vec, LDA, and Twitter Algebird Talk Abstract: Starting with a live, interactive demo generating audience-specific recommendations, we'll dive deep into each of the key components including NiFi, Kafka, Stanford CoreNLP, Docker, Word2Vec, LDA, Twitter Algebird, Spark Streaming, SQL, ML, GraphX. As a bonus, we'll discuss the latest Netflix Recommendations Pipeline and related open source projects. Talk Agenda: • Intro • Live, Interactive Recommendations Demo • Spark Streaming, ML, GraphX, Kafka, Cassandra, Docker, CoreNLP, Word2Vec, LDA, and Twitter Algebird (advancedspark.com) • Types of Similarity • Euclidean vs. Non-Euclidean Similarity • Jaccard Similarity • Cosine Similarity • LogLikelihood Similarity • Edit Distance • Text-based Similarities and Analytics • Word2Vec • LDA Topic Extraction • TextRank • Similarity-based Recommendations • User-to-User • Content-based, Item-to-Item (Amazon) • Collaborative-based, User-to-Item (Netflix) • Graph-based, Item-to-Item "Pathways" (Spotify) • Aggregations, Approximations, and Similarities at Scale • Twitter Algebird • MinHash and Bucketing • Locality Sensitive Hashing (LSH) • BloomFilters • CountMin Sketch • HyperLogLog • Q & A Speaker Bio: Chris Fregly is a Research Engineer @ Flux Capacitor AI in SF, an Apache Spark Contributor, and a Netflix Open Source Committer. Chris is also the founder of the global Advanced Apache Spark Meetup and author of the upcoming book, Advanced Spark @ advancedspark.com. Previously, Chris was a Data Solutions Engineer at Databricks and a Streaming Data Engineer at Netflix.

What's hot (20)

Local Apache NiFi Processor Debug

Apache NiFi User Guide

StreamNative FLiP into scylladb - scylla summit 2022

Data science online camp using the flipn stack for edge ai (flink, nifi, pu...

Real time stock processing with apache nifi, apache flink and apache kafka

Pulsar summit asia 2021 apache pulsar with mqtt for edge computing

Best practices and lessons learnt from Running Apache NiFi at Renault

Architecting for Scale

Building Streaming Applications with Apache Storm 1.1

Seattle spark-meetup-032317

Apache Kafka 0.8 basic training - Verisign

Music city data Hail Hydrate! from stream to lake

Running Apache Spark & Apache Zeppelin in Production

ApacheCon 2021 - Apache NiFi Deep Dive 300

Using FLiP with influxdb for edgeai iot at scale 2022

DBCC 2021 - FLiP Stack for Cloud Data Lakes

[March sn meetup] apache pulsar + apache nifi for cloud data lake

The Power of Intelligent Flows: Real-Time IoT Botnet Classification with Apac...

fluentd -- the missing log collector

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Viewers also liked

Machine Learning Preliminaries and Math Refresher

butest

Boston Spark Meetup May 24, 2016

Top 10 Performance Gotchas in scaling in-memory Algorithms Abstract: Math Algorithms have primarily been the domain of desktop data science. With the success of scalable algorithms at Google, Amazon, and Netflix, there is an ever growing demand for sophisticated algorithms over big data. In this talk, we get a ringside view in the making of the world's most scalable and fastest machine learning framework, H2O, and the performance lessons learnt scaling it over EC2 for Netflix and over commodity hardware for other power users. Top 10 Performance Gotchas is about the white hot stories of i/o wars, S3 resets, and muxers, as well as the power of primitive byte arrays, non-blocking structures, and fork/join queues. Of good data distribution & fine-grain decomposition of Algorithms to fine-grain blocks of parallel computation. It's a 10-point story of the rage of a network of machines against the tyranny of Amdahl while keeping the statistical properties of the data and accuracy of the algorithm. Track: Scalability, Availability, and Performance: Putting It All Together Time: Wednesday, 11:45am - 12:35pm

qconsf 2013: Top 10 Performance Gotchas for scaling in-memory Algorithms - Sr...

Sri Ambati

02 math essentials

Poongodi Mano

The Genome Assembly Problem

In this talk, I describe some recent advancements in Streaming ML and AI Pipelines to enable data scientists to rapidly train and test on streaming data - and ultimately deploy models directly into production on their own with low friction and high impact. With proper tooling and monitoring, data scientist have the freedom and responsibility to experiment rapidly on live, streaming data - and deploy directly into production as often as necessary. I’ll describe this tooling - and demonstrate a real production pipeline using Jupyter Notebook, Docker, Kubernetes, Spark ML, Kafka, TensorFlow, Jenkins, and Netflix Open Source.

Big Data Spain - Nov 17 2016 - Madrid Continuously Deploy Spark ML and Tensor...

Empowering the Data Scientist with "1-Click" Production Deployment and Canary Testing of High-Performance and Highly-Scalable Spark ML and TensorFlow Models directly from Jupyter/iPython Notebooks using Docker, Kubernetes, Netflix OSS, Microservices, and Spinnaker. With proper tooling and metrics, Data Scientists can directly deploy, analyze, A/B test, rollback, and scale out their Spark ML and TensorFlow model into live production serving with zero friction. We will show you the open source tools that we've built based on Docker, Kubernetes, Netflix Open Source, Microservices, Spinnaker - and even Chaos Monkey! Speaker: Chris Fregly @ PipelineIO, formerly Databricks and Netflix

Advanced Spark and TensorFlow Meetup 08-04-2016 One Click Spark ML Pipeline D...

Using the latest advancements from TensorFlow including the Accelerated Linear Algebra (XLA) Framework, JIT/AOT Compiler, and Graph Transform Tool, I’ll demonstrate how to optimize, profile, and deploy TensorFlow Models in GPU-based production environment. This talk is contains many Spark ML and TensorFlow AI demos using PipelineIO's 100% Open Source Community Edition. All code and Docker images are available to reproduce on your own CPU or GPU-based cluster. Chris Fregly is Founder and Research Engineer at PipelineIO, a Streaming Machine Learning and Artificial Intelligence Startup based in San Francisco. He is also an Apache Spark Contributor, a Netflix Open Source Committer, founder of the Global Advanced Spark and TensorFlow Meetup, author of the O’Reilly Training and Video Series titled, "High Performance TensorFlow in Production." Previously, Chris was a Distributed Systems Engineer at Netflix, a Data Solutions Engineer at Databricks, and a Founding Member and Principal Engineer at the IBM Spark Technology Center in San Francisco. https://www.meetup.com/TensorFlow-Chicago/events/240267321/ https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/events/240587698/ http://pipeline.io https://github.com/fluxcapacitor/pipeline

Machine Learning Essentials (dsth Meetup#3)

Data Science Thailand

High Performance Distributed TensorFlow with GPUs - TensorFlow Chicago Meetup...

資料探勘是資料科學中一個基礎的修習科目，這個學問結合了機器學習、人工智慧、資料庫、訊號處理、與統計等不同領域的技術，期待能從雜亂、巨大的資料中抽取出有意義的知識。理論上，透過這個技術，資料科學家可以作出各種應用。然而實際上，由於資料未經處理前，往往混亂、難以著手，如果沒有正確處理資料，往往無法得到有價值的知識。本課程的目的，在於帶領初學者了解如何從整理混亂的資料、並找到最適合的技術來解決問題，除了會深入淺出的教授一般教科書有的技術外，並會給與實際應用的例子，讓初學者能練習面對問題的方法，也能運用技巧來分析成品並同時教導如何衡量分析結果。

[系列活動] 資料探勘速遊

陸永祥/全球網路攝影機帶來的機會與挑戰

高嘉良/Open Innovation as Strategic Plan

Gradient Descent, Back Propagation, and Auto Differentiation - Advanced Spark...

We could all have predicted this with our magical Big Data analytics platforms, but it seems that Machine Learning is the new hotness in Information Security. A great number of startups with ‘cy’ and ‘threat’ in their names that claim that their product will defend or detect more effectively than their neighbour's product "because math". And it should be easy to fool people without a PhD or two that math just works. Indeed, math is powerful and large scale machine learning is an important cornerstone of much of the systems that we use today. However, not all algorithms and techniques are born equal. Machine Learning is a most powerful tool box, but not every tool can be applied to every problem and that’s where the pitfalls lie. This presentation will describe the different techniques available for data analysis and machine learning for information security, and discuss their strengths and caveats. The Ghost of Marketing Past will also show how similar the unfulfilled promises of deterministic and exploratory analysis were, and how to avoid making the same mistakes again. Finally, the presentation will describe the techniques and feature sets that were developed by the presenter on the past year as a part of his ongoing research project on the subject, in particular present some interesting results obtained since the last presentation on DefCon 21, and some ideas that could improve the application of machine learning for use in information security, especially in its use as a helper for security analysts in incident detection and response.

Secure Because Math: A Deep-Dive on Machine Learning-Based Monitoring (#Secur...

Alex Pinto

TensorFlow 深度學習快速上手班--電腦視覺應用

In this completely 100% Open Source demo-based talk, Chris Fregly from PipelineIO will be addressing an area of machine learning and artificial intelligence that is often overlooked: the real-time, end-user-facing "serving” layer in a hybrid-cloud and on-premise deployment environment using Jupyter, NetflixOSS, Docker, and Kubernetes. Serving models to end-users in real-time in a highly-scalable, fault-tolerant manner requires not only an understanding of machine learning fundamentals, but also an understanding of distributed systems and scalable microservices. Chris will combine his work experience from both Databricks and Netflix to present a 100% open source, real-world, hybrid-cloud, on-premise, and NetflixOSS-based production-ready environment to serve your notebook-based Spark ML and TensorFlow AI models with highly-scalable and highly-available robustness. Speaker Bio Chris Fregly is a Research Scientist at PipelineIO - a Streaming Analytics and Machine Learning Startup in San Francisco. Chris is an Apache Spark Contributor, Netflix Open Source Committer, Founder of the Global Advanced Spark and TensorFlow Meetup, and Author of the upcoming book, Advanced Spark, and Creator of the upcoming O'Reilly video series, Scaling TensorFlow Distributed in Production. Previously, Chris was an engineer at Databricks and Netflix - as well as a Founding Member of the IBM Spark Technology Center in San Francisco.

Deploy Spark ML and Tensorflow AI Models from Notebooks to Microservices - No...

此課程專為 Spark 入門者設計，在六小時帶您從無到有建置 Spark 開發環境，並以實作方式帶領您了解 Spark 機器學習函式庫 (MLlib) 的應用及開發。課程實作將以 Spark 核心之實作語言 - Scala 為主，搭配 Scala IDE eclipse 及相關 Library 建置本機開發環境，透過 IDE 強大的開發及偵錯功能加速開發流程；並介紹如何佈置至 Spark 平台，透過 Spark-submit 執行資料分析工作。本課程涵蓋機器學習中最常使用之分類、迴歸及分群方法，歡迎對 Spark 感興趣，卻不知從何下手；或想快速的對 Spark 機器學習有初步的了解的您參與！

Machine Learning without the Math: An overview of Machine Learning

Arshad Ahmed

[DSC 2016] 系列活動：李泳泉 / 星火燎原 - Spark 機器學習初探

Generative Adversarial Networks

NTHU AI Reading Group: Improved Training of Wasserstein GANs

https://arcade.sqlbits.com/sessions/ Sql bits apache nifi 101 Introduction and best practices 11-March-2022 UK https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html https://github.com/tspannhw/EverythingApacheNiFi https://www.datainmotion.dev/2020/12/basic-understanding-of-cloudera-flow.html https://www.datainmotion.dev/2020/10/top-25-use-cases-of-cloudera-flow.html In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud. I will cover: Terminology Flow Files Version Control Repositories Basic Record Processing Provenance Backpressure Prioritizers System Diagnostics Processors Process Groups Scheduling and Cron Bulletin Board Relationships Routing Tasks Networking Basic Cluster Architecture Listeners Controller Services Remote Ports Handling Errors Funnels Feedback LInk - https://sqlb.it/?7108 ROOM 04 Fri 12:00 - 12:50

Viewers also liked (20)

Machine Learning Preliminaries and Math Refresher

Boston Spark Meetup May 24, 2016

qconsf 2013: Top 10 Performance Gotchas for scaling in-memory Algorithms - Sr...

02 math essentials

The Genome Assembly Problem

Big Data Spain - Nov 17 2016 - Madrid Continuously Deploy Spark ML and Tensor...

Advanced Spark and TensorFlow Meetup 08-04-2016 One Click Spark ML Pipeline D...

Machine Learning Essentials (dsth Meetup#3)

High Performance Distributed TensorFlow with GPUs - TensorFlow Chicago Meetup...

[系列活動] 資料探勘速遊

陸永祥/全球網路攝影機帶來的機會與挑戰

高嘉良/Open Innovation as Strategic Plan

Gradient Descent, Back Propagation, and Auto Differentiation - Advanced Spark...

Secure Because Math: A Deep-Dive on Machine Learning-Based Monitoring (#Secur...

TensorFlow 深度學習快速上手班--電腦視覺應用

Deploy Spark ML and Tensorflow AI Models from Notebooks to Microservices - No...

Machine Learning without the Math: An overview of Machine Learning

[DSC 2016] 系列活動：李泳泉 / 星火燎原 - Spark 機器學習初探

Generative Adversarial Networks

NTHU AI Reading Group: Improved Training of Wasserstein GANs

Similar to Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly

Sql bits apache nifi 101 Introduction and best practices

Introducing the FLiPN stack which combines Apache Flink, Apache NiFi, Apache Pulsar and other Apache tools to build fast applications for IoT, AI, rapid ingest. FLiPN provides a quick set of tools to build applications at any scale for any streaming and IoT use cases. Tools Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, Apache MXNet, DJL.AI References https://www.datainmotion.dev/2019/08/... https://www.datainmotion.dev/2019/09/... https://www.datainmotion.dev/2019/05/... https://www.datainmotion.dev/2019/03/... Get the presentation slides: https://www.slideshare.net/streamnati... Subscribe to the StreamNative Newsletter for Apache Pulsar for more Pulsar content: https://share.hsforms.com/1IS56E-RvSV... Get started with the on-demand Pulsar training by StreamNative Academy: https://www.academy.streamnative.io/

Apache Kafka 0.11 の Exactly Once Semantics

Yoshiyasu SAEKI

Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Summit Asia ...

StreamNative

Using the flipn stack for edge ai (flink, nifi, pulsar)

Since mid-2016, Spark-as-a-Service has been available to researchers in Sweden from the Rise SICS ICE Data Center at www.hops.site. In this session, Dowling will discuss the challenges in building multi-tenant Spark structured streaming applications on YARN that are metered and easy-to-debug. The platform, called Hopsworks, is in an entirely UI-driven environment built with only open-source software. Learn how they use the ELK stack (Elasticsearch, Logstash and Kibana) for logging and debugging running Spark streaming applications; how they use Grafana and InfluxDB for monitoring Spark streaming applications; and, finally, how Apache Zeppelin can provide interactive visualizations and charts to end-users. This session will also show how Spark applications are run within a ‘project’ on a YARN cluster with the novel property that Spark applications are metered and charged to projects. Projects are securely isolated from each other and include support for project-specific Kafka topics. That is, Kafka topics are protected from access by users that are not members of the project. In addition, hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.afka topics are protected from access by users that are not members of the project. We will also discuss the experiences of our users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and our novel solutions for helping researchers debug and optimize Spark applications.

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Databricks

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

What is Apache Kafka and What is an Event Streaming Platform?

confluent

Sink Your Teeth into Streaming at Any Scale

Using the low-latency Apache Pulsar we can build up millions of streams of concurrent data and join them in real time with Apache Flink. We need an ultra-low latency database that can support these workloads to build next-generation IoT, financial and instant analytical transit applications By sinking data into ScyllaDB we enable amazingly fast applications that can grow to any size and join with existing data sources. The next generation of apps is being built now, you must choose the right low-latency scalable platform for these massively data-intensive applications. ScyllaDB + Pulsar + Flink is that platform. Choose Open, Choose Fast, and Make the right choice.

Sink Your Teeth into Streaming at Any Scale

ScyllaDB

"Implementing real-time data pipelines is still a challenge. Even more so for data scientists who have often been brought up with batch processing and files and typically have only heard of Kafka, but never really used it. Now if your team consists only of data scientists and you want them to implement a real-time data pipeline, your fate seems to be sealed. You need to hire real-time and streaming experts first - and you are guaranteed to lose months before you can even start implementing. But is there really no other way? At Forecasty.AI, developer of a SaaS platform for commodity price forecasting called Commodity Desk, we thought again. And came up with a way how to bridge the gap between the batch and file-based world of most data scientists and the world of real-time and streaming. Our solution is a new Open Source library allowing any Python programmer, including the aforementioned data scientists, to access the Kafka API in an easier way than ever before. kash.py (""Kafka Shell for Python"") offers a large number of easy-to-use abstractions on top of the Kafka API, including bash-inspired commands like ""ls"" or ""l"" for listing Kafka topics and ""cat"", ""head"" or ""tail"" for displaying the content of topics. kash.py bridges the gap between the file and the streaming worlds with commands like ""upload"" to upload a file to a topic, or ""download"" to download a topic to a file, and offers commands inspired by functional programming to do Kafka-Kafka/File-Kafka or Kafka-File stream processing (""map"", ""flatMap"", ""filter"" etc.) in one-liners - of course with full support for JSON Schema, Avro and Protobuf. In this session, we show you how kash.py can be used to bring the two disparate worlds of files and streaming together - and thus not only save a lot of time and money hiring real-time and streaming experts, but also make your data scientists start loving real-time."

kash.py - How to Make Your Data Scientists Love Real-time with Ralph M. Debus...

HostedbyConfluent

What is Apache Kafka®?

Eventador

What is apache Kafka?

Kenny Gorman

データの民主化のために StackStorm を活用した事例

Yoshiyasu SAEKI

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

In a recent survey, 90% of over 2400 developers reported having at least some real-time functionality in their systems. Enterprises are realizing that the ability to extract value from streaming data in near real-time is the new competitive advantage. Two technologies–Akka Streams and Kafka Streams–have emerged as popular tools to use with Apache Kafka for addressing the shared requirements of availability, scalability, and resilience for both streaming microservices and Fast Data. So which one should you use for specific use cases?

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Lightbend

Codeless pipelines with pulsar and flink

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Kafka Streams for Java enthusiasts

Slim Baltagi

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Trivadis

Akka-chan's Survival Guide for the Streaming World

Konrad Malawski

Akka streams kafka kinesis

Peter Vandenabeele

My talk at Google DevFest Switzerland, Fribourg, Oct 2017. https://devfest.ch/schedule/day1?sessionId=118 Abstract: Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of information in real-time? The answer is stream processing, and the technology that has since become the core platform for streaming data is Apache Kafka. Among the thousands of companies that use Kafka to transform and reshape their industries are the likes of Netflix, Uber, PayPal, and AirBnB, but also established players such as Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: there are many technologies that need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we, as engineers, would like to work vs. how we actually end up working in practice. In this session we talk about how Apache Kafka helps you to radically simplify your data architectures. We cover how you can now build normal applications to serve your real-time processing needs — rather than building clusters or similar special-purpose infrastructure — and still benefit from properties such as high scalability, distributed computing, and fault-tolerance, which are typically associated exclusively with cluster technologies. We discuss common use cases to realize that stream processing in practice often requires database-like functionality, and how Kafka allows you to bridge the worlds of streams and databases when implementing your own core business applications (inventory management for large retailers, patient monitoring in healthcare, fleet tracking in logistics, etc), for example in the form of event-driven, containerized microservices. We will also give a brief shout-out to the recently launched KSQL, a streaming SQL engine for Apache Kafka.

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Michael Noll

Similar to Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi, Kafka, and Spark - Chris Fregly (20)

Sql bits apache nifi 101 Introduction and best practices

Apache Kafka 0.11 の Exactly Once Semantics

Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Summit Asia ...

Using the flipn stack for edge ai (flink, nifi, pulsar)

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

What is Apache Kafka and What is an Event Streaming Platform?

Sink Your Teeth into Streaming at Any Scale

kash.py - How to Make Your Data Scientists Love Real-time with Ralph M. Debus...

What is Apache Kafka®?

What is apache Kafka?

データの民主化のために StackStorm を活用した事例

KafkaとAWS Kinesisの比較

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Codeless pipelines with pulsar and flink

Kafka Streams for Java enthusiasts

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Akka-chan's Survival Guide for the Streaming World

Akka streams kafka kinesis

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

More from Chris Fregly

AWS reInvent 2022 reCap AI/ML and Data

Chris Fregly (Principal Solution Architect, AI and machine learning at AWS) will give a brief presentation on the various ways to perform scalable Pandas, Modin, and Ray on AWS. He will then answer questions from the audience and moderator, Alejandro Herrera (whatever he is) at Ponder. Chris Fregly is a Principal Solution Architect for AI and Machine Learning at Amazon Web Services (AWS) based in San Francisco, California. He is the organizer of the Global Data Science on AWS meetup. He is co-author of the O'Reilly Book, "Data Science on AWS." Related Links O'Reilly Book: https://www.amazon.com/dp/1492079391/ Website: https://datascienceonaws.com Meetup: https://meetup.datascienceonaws.com GitHub Repo: https://github.com/data-science-on-aws/ YouTube: https://youtube.datascienceonaws.com Slideshare: https://slideshare.datascienceonaws.com

Pandas on AWS - Let me count the ways.pdf

RSVP Webinar: https://www.eventbrite.com/e/webinarkubeflow-tensorflow-tfx-pytorch-gpu-spark-ml-amazonsagemaker-tickets-45852865154 Talk #0: Introductions and Meetup Announcements By Chris Fregly and Antje Barth Talk #1: Ray Overview, Ray AI Runtime on AWS using Amazon SageMaker, EC2, EMR, EKS by Chris Fregly, Principal Specialist Solution Architect, AI and Machine Learning @ AWS Talk #2: Deep-dive Blueprints for Amazon Elastic Kubernetes Service (EKS) including Ray and Spark by Apoorva Kulkarni, Sr. Specialist Solution Architect, Containers and Kubernetes @ AWS RSVP Webinar: https://www.eventbrite.com/e/webinarkubeflow-tensorflow-tfx-pytorch-gpu-spark-ml-amazonsagemaker-tickets-45852865154 Zoom link: https://us02web.zoom.us/j/82308186562 Related Links O'Reilly Book: https://www.amazon.com/dp/1492079391/ Website: https://datascienceonaws.com Meetup: https://meetup.datascienceonaws.com GitHub Repo: https://github.com/data-science-on-aws/ YouTube: https://youtube.datascienceonaws.com Slideshare: https://slideshare.datascienceonaws.com

Ray AI Runtime (AIR) on AWS - Data Science On AWS Meetup

Smokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated

Amazon reInvent 2020 Recap: AI and Machine Learning Video here: https://youtu.be/YSXe02Y5pHM NEW RELEASE! Build, Automate, Manage, and Scale ML Workflows with the NEW Amazon SageMaker Pipelines by Hallie Crosby Weishahn. Description of Talk and Demo AWS recently announced Amazon SageMaker Pipelines (https://aws.amazon.com/sagemaker/pipelines/), the first purpose-built, easy-to-use Continuous Integration and Continuous Delivery (CI/CD) service for machine learning. SageMaker Pipelines has three main components which improve the operational resilience and reproducibility of your workflows: 1) pipelines, 2) model registry, and 3) projects. In this talk and demo, Hallie will walk us through the new Amazon SageMaker Pipelines feature including MLOps support. Date/Time 9-10am US Pacific Time (Third Monday of Every Month) RSVP: https://www.eventbrite.com/e/1-hr-free-workshop-pipelineai-gpu-tpu-spark-ml-tensorflow-ai-kubernetes-kafka-scikit-tickets-45852865154 Meetup: https://www.meetup.com/Data-Science-on-AWS/ Zoom: https://zoom.us/j/690414331 Webinar ID: 690 414 331 Phone: +1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll) Related Links Meetup: https://meetup.datascienceonaws.com GitHub Repo: https://github.com/data-science-on-aws/ O'Reilly Book: https://datascienceonaws.com YouTube: https://youtube.datascienceonaws.com Slideshare: https://slideshare.datascienceonaws.com Support: https://support.pipeline.ai Monthly Workshop: https://www.eventbrite.com/e/full-day-workshop-kubeflow-gpu-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-tickets-63362929227 RSVP: https://www.eventbrite.com/e/1-hr-free-workshop-pipelineai-gpu-tpu-spark-ml-tensorflow-ai-kubernetes-kafka-scikit-tickets-45852865154

Amazon reInvent 2020 Recap: AI and Machine Learning

Waking the Data Scientist at 2am: Detect Model Degradation on Production Models with Amazon SageMaker Endpoints & Model Monitor In this talk, I describe how to deploy a model into production and monitor its performance using SageMaker Model Monitor. With Model Monitor, I can detect if a model's predictive performance has degraded - and alert an on-call data scientist to take action and improve the model at 2am while the DevOps folks sleep soundly through the night. Topics: AI and Machine Learning, Model Deployment, Anomaly Detection, Amazon SageMaker Endpoints, and Model Monitor

Waking the Data Scientist at 2am: Detect Model Degradation on Production Mod...

Quantum Computing with Amazon Braket

15 Tips to Scale a Large AI/ML Workshop - Both Online and In-Person

AWS Re:Invent 2019 Re:Cap

Video: https://youtu.be/T0L0JxDaPkc RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227 Description In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, Airflow, and MLflow. Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google. KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking. Airflow is the most-widely used pipeline orchestration framework in machine learning and data engineering. MLflow is a lightweight experiment-tracking system recently open-sourced by Databricks, the creators of Apache Spark. MLflow supports Python, Java/Scala, and R - and offers native support for TensorFlow, Keras, and Scikit-Learn. Pre-requisites Modern browser - and that's it! Every attendee will receive a cloud instance Nothing will be installed on your local laptop Everything can be downloaded at the end of the workshop Location Online Workshop The link will be sent a few hours before the start of the workshop. Only registered users will receive the link. If you do not receive the link a few hours before the start of the workshop, please send your Eventbrite registration confirmation to support@pipeline.ai for help. Agenda 1. Create a Kubernetes cluster 2. Install KubeFlow, Airflow, TFX, and Jupyter 3. Setup ML Training Pipelines with KubeFlow and Airflow 4. Transform Data with TFX Transform 5. Validate Training Data with TFX Data Validation 6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow 7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow 8. Analyze Models using TFX Model Analysis and Jupyter 9. Perform Hyper-Parameter Tuning with KubeFlow 10. Select the Best Model using KubeFlow Experiment Tracking 11. Run Multiple Experiments with MLflow Experiment Tracking 12. Reproduce Model Training with TFX Metadata Store 13. Deploy the Model to Production with TensorFlow Serving and Istio 14. Save and Download your Workspace Key Takeaways Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools. RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227 https://youtu.be/T0L0JxDaPkc

KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTo...

Title TensorFlow + Swift + OpenAI's Unsupervised Sentiment Neuron, KubeFlow, TFX, Kubernetes, Kafka, Airflow, Jupyter, Scikit Agenda 1. TensorFlow + Swift + OpenAI's Unsupervised Sentiment Neuron (45 mins) Speaker: Tanmay Bakshi (https://www.linkedin.com/in/tanmay-bakshi-b15012a1) (More details coming soon...) 2. KubeFlow, TFX, Kubernetes, Kafka, Airflow, Jupyter, Scikit, GPU/TPU, Kafka, Scikit-Learn and JupyterLab (15 mins) (More details coming soon...) ** RSVP & LOGIN HERE ** Eventbrite: https://www.eventbrite.com/e/1-hr-free-pipelineai-gpu-tpu-spark-tensorflow-kubernetes-kafka-scikit-tickets-45852865154 Meetup: https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/events/kczhrpyxhbcc/ Zoom: https://zoom.us/j/690414331 Webinar ID: 690 414 331 Phone: +1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll) Related Links PipelineAI Home: https://pipeline.ai PipelineAI Community Edition: http://community.pipeline.ai PipelineAI GitHub: https://github.com/PipelineAI/pipeline Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup YouTube Videos: https://youtube.pipeline.ai SlideShare Presentations: https://slideshare.pipeline.ai Slack Support: https://joinslack.pipeline.ai Web Support and Knowledge Base: https://support.pipeline.ai Email Support: help@pipeline.ai

Swift for TensorFlow - Tanmay Bakshi - Advanced Spark and TensorFlow Meetup -...

Title Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark + Jupyter + TPU Video https://youtu.be/vaB4IM6ySD0 Description In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, and Airflow. Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google. KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking. Airflow is the most-widely used pipeline orchestration framework in machine learning. Pre-requisites Modern browser - and that's it! Every attendee will receive a cloud instance Nothing will be installed on your local laptop Everything can be downloaded at the end of the workshop Location Online Workshop Agenda 1. Create a Kubernetes cluster 2. Install KubeFlow, Airflow, TFX, and Jupyter 3. Setup ML Training Pipelines with KubeFlow and Airflow 4. Transform Data with TFX Transform 5. Validate Training Data with TFX Data Validation 6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow 7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow 8. Analyze Models using TFX Model Analysis and Jupyter 9. Perform Hyper-Parameter Tuning with KubeFlow 10. Select the Best Model using KubeFlow Experiment Tracking 11. Reproduce Model Training with TFX Metadata Store and Pachyderm 12. Deploy the Model to Production with TensorFlow Serving and Istio 13. Save and Download your Workspace Key Takeaways Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools. Related Links 1. PipelineAI Home: https://pipeline.ai 2. PipelineAI Community Edition: http://community.pipeline.ai 3. PipelineAI GitHub: https://github.com/PipelineAI/pipeline 4. Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup 5. YouTube Videos: https://youtube.pipeline.ai 6. SlideShare Presentations: https://slideshare.pipeline.ai 7. Slack Support: https://joinslack.pipeline.ai 8. Web Support and Knowledge Base: https://support.pipeline.ai 9. Email Support: support@pipeline.ai

Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...

Speaker: Umayah Abdennabi Agenda * Intro Grammarly (Umayah Abdennabi, 5 mins) * Meetup Updates and Announcements (Chris, 5 mins) * Custom Functions in Spark SQL (30 mins) Speaker: Umayah Abdennabi Spark comes with a rich Expression library that can be extended to make custom expressions. We will look into custom expressions and why you would want to use them. * TF 2.0 + Keras (30 mins) Speaker: Francesco Mosconi Tensorflow 2.0 was announced at the March TF Dev Summit, and it brings many changes and upgrades. The most significant change is the inclusion of Keras as the default model building API. In this talk, we'll review the main changes introduced in TF 2.0 and highlight the differences between open source Keras and tf.keras * SQUAD Deep-Dive: Question & Answer with Context (45 mins) Speaker: Brett Koonce (https://quarkworks.co) SQuAD (Stanford Question Answer Dataset) is an NLP challenge based around answering questions by reading Wikipedia articles, designed to be a real-world machine learning benchmark. We will look at several different ways to tackle the SQuAD problem, building up to state of the art approaches in terms of time, complexity, and accuracy. https://rajpurkar.github.io/SQuAD-explorer/ https://dawn.cs.stanford.edu/benchmark/#squad Food and drinks will be provided. The event will be held at Grammarly's office at One Embarcadero Center on the 9th floor. When you arrive at One Embarcadero, take the escalator to the second floor where you will find the lobby and elevators to the office suites. Come on up to the 9th floor (no need to check in at security), and ring the Grammarly doorbell.

Spark SQL Catalyst Optimizer, Custom Expressions, UDFs - Advanced Spark and T...

Traditional machine learning pipelines end with life-less models sitting on disk in the research lab. These traditional models are typically trained on stale, offline, historical batch data. Static models and stale data are not sufficient to power today's modern, AI-first Enterprises that require continuous model training, continuous model optimizations, and lightning-fast model experiments directly in production. Through a series of open source, hands-on demos and exercises, we will use PipelineAI to breathe life into these models using 4 new techniques that we’ve pioneered: * Continuous Validation (V) * Continuous Optimizing (O) * Continuous Training (T) * Continuous Explainability (E). The Continuous "VOTE" techniques has proven to maximize pipeline efficiency, minimize pipeline costs, and increase pipeline insight at every stage from continuous model training (offline) to live model serving (online.) Attendees will learn to create continuous machine learning pipelines in production with PipelineAI, TensorFlow, and Kafka.

PipelineAI Continuous Machine Learning and AI - Rework Deep Learning Summit -...

Perform Online Predictions using Slack A/B and multi-armed bandit model compare Train Online Models with Kafka Streams Create new models quickly Deploy to production safely Mirror traffic to validate online performance Any Framework, Any Hardware, Any Cloud Dashboard to manage the lifecycle of models from local development to live production Generates optimized runtimes for the models Custom targeting rules, shadow mode, and percentage-based rollouts to safely test features in live production Continuous model training, model validation, and pipeline optimization https://youtu.be/zpkH9oiIovU https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/events/258276286/ Related Links PipelineAI Home: https://pipeline.ai PipelineAI Community Edition: https://community.pipeline.ai PipelineAI GitHub: https://github.com/PipelineAI/pipeline PipelineAI Quick Start: https://quickstart.pipeline.ai Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup YouTube Videos: https://youtube.pipeline.ai SlideShare Presentations: https://slideshare.pipeline.ai Slack Support: https://joinslack.pipeline.ai Web Support and Knowledge Base: https://support.pipeline.ai Email Support: help@pipeline.ai

PipelineAI Real-Time Machine Learning - Global Artificial Intelligence Confer...

Chris Fregly, Founder @ PipelineAI, will walk you through a real-world, complete end-to-end Pipeline-optimization example. We highlight hyper-parameters - and model pipeline phases - that have never been exposed until now. While most Hyperparameter Optimizers stop at the training phase (ie. learning rate, tree depth, ec2 instance type, etc), we extend model validation and tuning into a new post-training optimization phase including 8-bit reduced precision weight quantization and neural network layer fusing - among many other framework and hardware-specific optimizations. Next, we introduce hyperparameters at the prediction phase including request-batch sizing and chipset (CPU v. GPU v. TPU). Lastly, we determine a PipelineAI Efficiency Score of our overall Pipeline including Cost, Accuracy, and Time. We show techniques to maximize this PipelineAI Efficiency Score using our massive PipelineDB along with the Pipeline-wide hyper-parameter tuning techniques mentioned in this talk. Bio Chris Fregly is Founder and Applied AI Engineer at PipelineAI, a Real-Time Machine Learning and Artificial Intelligence Startup based in San Francisco. He is also an Apache Spark Contributor, a Netflix Open Source Committer, founder of the Global Advanced Spark and TensorFlow Meetup, author of the O’Reilly Training and Video Series titled, "High Performance TensorFlow in Production with Kubernetes and GPUs." Previously, Chris was a Distributed Systems Engineer at Netflix, a Data Solutions Engineer at Databricks, and a Founding Member and Principal Engineer at the IBM Spark Technology Center in San Francisco.

Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ...

PipelineAI Optimizes Your Enterprise AI Pipeline from Distributed Training to...

Advanced Spark and TensorFlow Meetup - Dec 12 2017 - Dong Meng, MapR + Kubern...

Online Workshop Note: A GPU-based cloud instance will be provided to each attendee for the duration of this event!! At 8am PT on the morning of this workshop, we will email the Webinar details to your email address registered with Eventbrite. If this email address is not up to date - or you do not get the email by 8am PT - please email your Eventbrite confirmation to help@pipeline.ai and we'll send you the details. http://pipeline.ai Title PipelineAI Distributed Spark ML + Tensorflow AI + GPU Workshop Time Start: 9am PT Time End: 1pm PT Time Highlights We will each build an end-to-end, continuous Tensorflow AI model training and deployment pipeline on our own GPU-based cloud instance. At the end, we will combine our cloud instances to create the LARGEST Distributed Tensorflow AI Training and Serving Cluster in the WORLD! Pre-requisites Just a modern browser, internet connection, and a good night's sleep! We'll provide the rest. Agenda Spark ML TensorFlow AI Storing and Serving Models with HDFS Trade-offs of CPU vs. *GPU, Scale Up vs. Scale Out CUDA + cuDNN GPU Development Overview TensorFlow Model Checkpointing, Saving, Exporting, and Importing Distributed TensorFlow AI Model Training (Distributed Tensorflow) TensorFlow's Accelerated Linear Algebra Framework (XLA) TensorFlow's Just-in-Time (JIT) Compiler, Ahead of Time (AOT) Compiler Centralized Logging and Visualizing of Distributed TensorFlow Training (Tensorboard) Distributed Tensorflow AI Model Serving/Predicting (TensorFlow Serving) Centralized Logging and Metrics Collection (Prometheus, Grafana) Continuous TensorFlow AI Model Deployment (TensorFlow, Airflow) Hybrid Cross-Cloud and On-Premise Deployments (Kubernetes) High-Performance and Fault-Tolerant Micro-services (NetflixOSS) More Info including GitHub and Docker Repos http://pipeline.ai

High Performance Distributed TensorFlow in Production with GPUs - NIPS 2017 -...

PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...