Enterprise Data Science at Scale @ Princeton, NJ 14-Nov-2017

Dataflow with Apache NiFi - Crash Course - HS16SJ

Apache Hadoop Crash Course - HS16SJ

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It provides an end-to-end platform that can collect, curate, analyze, and act on data in real-time, on-premises, or in the cloud with a drag-and-drop visual interface. It’s being used across industries on large amounts of data that had stored in isolation which made collaboration and analysis difficult. Join industry experts from Hortonworks and Attunity as they explain how Apache NiFi and streaming CDC technology provides a distributed, resilient platform for unlocking the value of data in new ways.

Automatic Detection, Classification and Authorization of Sensitive Personal D...

The newly enacted GDPR regulations which become effective in 2018 require comprehensive protection of personal information of EU subjects. In this paper, we outline a solution that discovers and classifies personal data that is subject to GDPR in Hadoop ecosystem and uses such precise classification to automatically create a robust set of policies for authorization. The solution consists of using Dataguise’s DgSecure sensitive data detection to automatically classify sensitive data assets in Apache Atlas and author comprehensive and robust authorization policies via Apache Ranger. DgSecure is used to detect sensitive data in Hive databases and continuously update the classification in Apache Atlas via tags. Apache Atlas tags are used to create Apache Ranger policies that protect access to sensitive HDFS files, Hive tables, and Hive columns. We demonstrate a workflow where the components of the solution are automated requiring little or no manual intervention to provide protection of such sensitive data in Hadoop clusters.

IoT Crash Course Hadoop Summit SJ

Daniel Madrigal

SparkR Best Practices for R Data Scientists

Spark plays an important role on data scientists to solve all kinds of problems, especially the release of SparkR which provide very friendly APIs for traditional data scientists. However, processing various data size, data format and models will lead to different application patterns compared with traditional R. In this talk, we will illustrate the practical experience that using SparkR to solve some typical data science problems, such as the performance improvement for SparkR and native R interoperation, how to load data from HBase which is a very common data source efficiently, how to schedule a large scale machine learning job with multiple single R machine learning jobs, how to tuning performance for jobs triggered by many different users, how to use SparkR in the cloud-based environment, etc. At last, we will shortly introduce the community efforts in progress on SparkR in the coming releases. Speakers: Yanbo Liang, Software Engineer, Hortonworks Casey Stella, Principal Software Engineer/Data Scientist, Hortonworks

Data Science with Apache Spark - Crash Course - HS16SJ

Flink and NiFi, Two Stars in the Apache Big Data Constellation

Matthew Ring

How Hadoop Makes the Natixis Pack More Efficient

Improving business performance is never easy! The Natixis Pack is like Rugby. Working together is key to scrum success. Our data journey would undoubtedly have been so much more difficult if we had not made the move together. This session is the story of how ‘The Natixis Pack’ has driven change in its current IT architecture so that legacy systems can leverage some of the many components in Hortonworks Data Platform in order to improve the performance of business applications. During this session, you will hear: • How and why the business and IT requirements originated • How we leverage the platform to fulfill security and production requirements • How we organize a community to: o Guard all the players, no one gets left on the ground! o Us the platform appropriately (Not every problem is eligible for Big Data and standard databases are not dead) • What are the most usable, the most interesting and the most promising technologies in the Apache Hadoop community We will finish the story of a successful rugby team with insight into the special skills needed from each player to win the match! DETAILS This session is part business, part technical. We will talk about infrastructure, security and project management as well as the industrial usage of Hive, HBase, Kafka, and Spark within an industrial Corporate and Investment Bank environment, framed by regulatory constraints.

HDF Powered by Apache NiFi Introduction

Milind Pandit

Intro to Spark & Zeppelin - Crash Course - HS16SJ

Apache Hadoop Crash Course

Integrating Apache Spark and NiFi for Data Lakes

Modernise your EDW - Data Lake

Introduction to Streaming Analytics Manager

Yifeng Jiang

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...

Companies in every industry look for ways to explore new data types and large data sets that were previously too big to capture, store and process. They need to unlock insights from data such as clickstream, geo-location, sensor, server log, social, text and video data. However, becoming a data-first enterprise comes with many challenges. Join this webinar organized by three leaders in their respective fields and learn from our experts how you can accelerate the implementation of a scalable, cost-efficient and robust Big Data solution. Cisco, Hortonworks and Red Hat will explore how new data sets can enrich existing analytic applications with new perspectives and insights and how they can help you drive the creation of innovative new apps that provide new value to your business.

Running Apache NiFi with Apache Spark : Integration Options

Enterprise Data Science at Scale

Artem Ervits

Enterprise Data Science at Scale Meetup - IBM and Hortonworks - Oct 2017

View the recording of the meet up, including the live demos, here: https://www.youtube.com/watch?v=uaJWB3K8lkg Data science holds tremendous potential for organizations to uncover new insights and drivers of revenue and profitability. Big Data has brought the promise of doing data science at scale to enterprises, however this promise also comes with challenges for data scientists to continuously learn and collaborate. Data Scientists have many tools at their disposal such as notebooks like Juypter and Apache Zeppelin & IDEs such as RStudio with languages like R, Python, Scala and frameworks like Apache Spark. Given all the choices how do you best collaborate to build your model and then work through the development lifecycle to deploy it from test into production? Why Data Science on Big Data? In this meetup you will cover the attributes of a modern data science platform that empowers data scientists to build models using all the data in their data lake and foster continuous learning and collaboration. We will show a demo of Apache Zeppelin, Apache Spark, Apache Livy and Apache Hadoop with the focus on integration, security and model deployment and management. Data Science at Scale DEMO The demo will cover the Data Science life cycle: develop model in team environment, train the model with all the data on a Hadoop cluster, deploy model into production. The model will be a Spark ML model Practical ML with Apache Spark To deliver machine learning solutions data scientists not only need to fit models but also do familiar tasks data collection & wrangling, labelling, feature extraction and transformation, model tuning and evaluation, etc. Apache Spark provide provides a unified solution for all this under the same framework. For example, one can use Spark SQL to generate training data from different sources and then pass it directly to MLlib for feature engineering and model tuning, instead of using Hive/Pig for the first half and then downloading the data to a single machine to train models in R. The latter is actually very common in practice but painful to maintain. Spark MLlib makes life easier for data scientists and machine learning engineers so that they can focus on building better ML models and applications. We will discuss the underlying principles required to develop practical machine learning and data science pipelines and show some hands-on experience using Apache Spark to solve typical machine learning and data science problem. We will also have a short discussion about how Spark MLlib faces challenges from other machine learning libraries such as TensorFlow and XGBoost.

What's hot

Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi

Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi

Apache NiFi Meetup - Princeton NJ 2016

Dataflow with Apache NiFi - Crash Course - HS16SJ

Apache Hadoop Crash Course - HS16SJ

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Automatic Detection, Classification and Authorization of Sensitive Personal D...

IoT Crash Course Hadoop Summit SJ

Daniel Madrigal

SparkR Best Practices for R Data Scientists

Data Science with Apache Spark - Crash Course - HS16SJ

Flink and NiFi, Two Stars in the Apache Big Data Constellation

Matthew Ring

How Hadoop Makes the Natixis Pack More Efficient

HDF Powered by Apache NiFi Introduction

Milind Pandit

Intro to Spark & Zeppelin - Crash Course - HS16SJ

Apache Hadoop Crash Course

Integrating Apache Spark and NiFi for Data Lakes

Modernise your EDW - Data Lake

Introduction to Streaming Analytics Manager

Yifeng Jiang

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...

Running Apache NiFi with Apache Spark : Integration Options

What's hot (20)

Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi

Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi

Apache NiFi Meetup - Princeton NJ 2016

Dataflow with Apache NiFi - Crash Course - HS16SJ

Apache Hadoop Crash Course - HS16SJ

Unlock Value from Big Data with Apache NiFi and Streaming CDC

Automatic Detection, Classification and Authorization of Sensitive Personal D...

IoT Crash Course Hadoop Summit SJ

SparkR Best Practices for R Data Scientists

Data Science with Apache Spark - Crash Course - HS16SJ

Flink and NiFi, Two Stars in the Apache Big Data Constellation

How Hadoop Makes the Natixis Pack More Efficient

HDF Powered by Apache NiFi Introduction

Intro to Spark & Zeppelin - Crash Course - HS16SJ

Apache Hadoop Crash Course

Integrating Apache Spark and NiFi for Data Lakes

Modernise your EDW - Data Lake

Introduction to Streaming Analytics Manager

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...

Running Apache NiFi with Apache Spark : Integration Options

Similar to Enterprise Data Science at Scale @ Princeton, NJ 14-Nov-2017

Enterprise Data Science at Scale

Artem Ervits

Enterprise Data Science at Scale Meetup - IBM and Hortonworks - Oct 2017

Enterprise data science at scale

Carolyn Duby

Zementis hortonworks-webinar-2014-09

Hortonworks and Red Hat Webinar_Sept.3rd_Part 1

As the enterprise's big data program matures and Apache Hadoop becomes more deeply embedded in critical operations, the ability to support and operate it efficiently and reliably becomes increasingly important. To aid enterprise in operating modern data architecture at scale, Red hat and Hortonworks have collaborated to integrate Hortonworks Data Platform with Red Hat's proven platform technologies. Join us in this interactive 3-part webinar series, as we'll demonstrate how Red Hat JBoss Data Virtualization can integrate with Hadoop through Hive and provide users easy access to data.

Splunk-hortonworks-risk-management-oct-2014

Innovative Management Services

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Mr. Slim Baltagi is a Systems Architect at Hortonworks, with over 4 years of Hadoop experience working on 9 Big Data projects: Advanced Customer Analytics, Supply Chain Analytics, Medical Coverage Discovery, Payment Plan Recommender, Research Driven Call List for Sales, Prime Reporting Platform, Customer Hub, Telematics, Historical Data Platform; with Fortune 100 clients and global companies from Financial Services, Insurance, Healthcare and Retail. Mr. Slim Baltagi has worked in various architecture, design, development and consulting roles at. Accenture, CME Group, TransUnion, Syntel, Allstate, TransAmerica, Credit Suisse, Chicago Board Options Exchange, Federal Reserve Bank of Chicago, CNA, Sears, USG, ACNielsen, Deutshe Bahn. Mr. Baltagi has also over 14 years of IT experience with an emphasis on full life cycle development of Enterprise Web applications using Java and Open-Source software. He holds a master’s degree in mathematics and is an ABD in computer science from Université Laval, Québec, Canada. Languages: Java, Python, JRuby, JEE , PHP, SQL, HTML, XML, XSLT, XQuery, JavaScript, UML, JSON Databases: Oracle, MS SQL Server, MYSQL, PostreSQL Software: Eclipse, IBM RAD, JUnit, JMeter, YourKit, PVCS, CVS, UltraEdit, Toad, ClearCase, Maven, iText, Visio, Japser Reports, Alfresco, Yslow, Terracotta, Toad, SoapUI, Dozer, Sonar, Git Frameworks: Spring, Struts, AppFuse, SiteMesh, Tiles, Hibernate, Axis, Selenium RC, DWR Ajax , Xstream Distributed Computing/Big Data: Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, HBase, R, RHadoop, Cloudera CDH4, MapR M7, Hortonworks HDP 2.1

The Big Data Gusher: Big Data Analytics, the Internet of Things and the Oil B...

Platfora

The proliferation of machine sensors, interaction, and transaction data is driving a significant transformation within the oil and gas industry. Some industry analysts estimate that correctly implementing big data analytics can provide a 4-8% improvement in operational efficiency for oil companies. Other research shows that nearly 90% of oil industry executives rate big data analytics as a top priority, while fewer than a third have implemented solutions.

Introduction to Hadoop

POSSCON

Bridging the Big Data Gap in the Software-Driven World

CA Technologies

Implementing and managing a Big Data environment effectively requires essential efficiencies such as automation, performance monitoring and flexible infrastructure management. Discover new innovations that enable you to manage entire Big Data environments with unparalleled ease of use and clear enterprise visibility across a variety of data repositories. To learn more about Mainframe solutions from CA Technologies, visit: http://bit.ly/1wbiPkl

Building a Modern Data Architecture with Enterprise Hadoop

Slim Baltagi

Create a Smarter Data Lake with HP Haven and Apache Hadoop

An organization’s information is spread across multiple repositories, on-premise and in the cloud, with limited ability to correlate information and derive insights. The Smart Content Hub solution from HP and Hortonworks enables a shared content infrastructure that transparently synchronizes information with existing systems and offers an open standards-based platform for deep analysis and data monetization. - Leverage 100% of your data: Text, images, audio, video, and many more data types can be automatically consumed and enriched using HP Haven (powered by HP IDOL and HP Vertica), making it possible to integrate this valuable content and insights into various line of business applications. - Democratize and enable multi-dimensional content analysis: - Empower your analysts, business users, and data scientists to search and analyze Hadoop data with ease, using the 100% open source Hortonworks Data Platform. - Extend the enterprise data warehouse: Synchronize and manage content from content management systems, and crack open the files in whatever format they happen to be in. - Dramatically reduce complexity with enterprise-ready SQL engine: Tap into the richest analytics that support JOINs, complex data types, and other capabilities only available with HP Vertica SQL on the Hortonworks Data Platform. Speakers: - Ajay Singh, Director, Technical Channels, Hortonworks - Will Gardella, Product Management, HP Big Data

Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycl...

Data Science Salon: Applying Machine Learning to Modernize Business Processes

Formulatedby

Next DSS MIA Event - https://datascience.salon/miami/ For most data scientist building models is hard work, but deploying them into production and impacting business processes can be even harder. In fact, research shows that only about 10% of data science models get deployed into production, and those that do can take between 6 to 9 months to be deployed. This session will highlight the challenges that data scientist and organizations alike face when trying to deploy machine learning models and how to overcome these challenges. It will examine several use cases where models built in R and Python have been able to deliver impactful results across several industries.

Edw Optimization Solution

Innovation in the Enterprise Rent-A-Car Data Warehouse

Big Data adoption is a journey. Depending on the business the process can take weeks, months, or even years. With any transformative technology the challenges have less to do with the technology and more to do with how a company adapts itself to a new way of thinking about data. Building a Center of Excellence is one way for IT to help drive success. This talk will explore Enterprise Holdings Inc. (which operates the Enterprise Rent-A-Car, National Car Rental and Alamo Rent A Car) and their experience with Big Data. EHI’s journey started in 2013 with Hadoop as a POC and today are working to create the next generation data warehouse in Microsoft’s Azure cloud utilizing a lambda architecture. We’ll discuss the Center of Excellence, the roles in the new world, share the things which worked well, and rant about those which didn’t. No deep Hadoop knowledge is necessary, architect or executive level.

Apache Hadoop on the Open Cloud

Eliminating the Challenges of Big Data Management Inside Hadoop

Eliminating the Challenges of Big Data Management Inside Hadoop

Your Big Data strategy is only as good as the quality of your data. Today, deriving business value from data depends on how well your company can capture, cleanse, integrate and manage data. During this webinar, we discussed how to eliminate the challenges to Big Data management inside Hadoop. Go over these slides to learn: · How to use the scalability and flexibility of Hadoop to drive faster access to usable information across the enterprise. · Why a pure-YARN implementation for data integration, quality and management delivers competitive advantage. · How to use the flexibility of RedPoint and Hortonworks to create an enterprise data lake where data is captured, cleansed, linked and structured in a consistent way.

Verizon Centralizes Data into a Data Lake in Real Time for Analytics

Verizon – Global Technology Services (GTS) was challenged by a multi-tier, labor-intensive process when trying to migrate data from disparate sources into a data lake to create financial reports and business insights. Join this session to learn more about how Verizon: • Easily accessed data from multiple sources including SAP data • Ingested data into major targets including Hadoop • Achieved real-time insights from data leveraging change data capture (CDC) technology • Reduced costs and labor

Similar to Enterprise Data Science at Scale @ Princeton, NJ 14-Nov-2017 (20)

Enterprise Data Science at Scale

Enterprise Data Science at Scale Meetup - IBM and Hortonworks - Oct 2017

Enterprise data science at scale

Zementis hortonworks-webinar-2014-09

Hortonworks and Red Hat Webinar_Sept.3rd_Part 1

Splunk-hortonworks-risk-management-oct-2014

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

The Big Data Gusher: Big Data Analytics, the Internet of Things and the Oil B...

Introduction to Hadoop

Bridging the Big Data Gap in the Software-Driven World

Building a Modern Data Architecture with Enterprise Hadoop

Create a Smarter Data Lake with HP Haven and Apache Hadoop

Getting to What Matters: Accelerating Your Path Through the Big Data Lifecycl...

Data Science Salon: Applying Machine Learning to Modernize Business Processes

Edw Optimization Solution

Innovation in the Enterprise Rent-A-Car Data Warehouse

Apache Hadoop on the Open Cloud

Eliminating the Challenges of Big Data Management Inside Hadoop

Verizon Centralizes Data into a Data Lake in Real Time for Analytics

More from Timothy Spann

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...

06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI Discussion on Vector Databases, Unstructured Data and AI https://www.meetup.com/unstructured-data-meetup-new-york/ This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

Generative AI on Enterprise Cloud with NiFi and Milvus

Gen AI on Enterprise Cloud Apache NiFi Milvus Apache Kafka Apache Flink Cloudera Machine Learning Cloudera DataFlow https://medium.com/@tspann/building-a-milvus-connector-for-nifi-34372cb3c7fa https://www.meetup.com/futureofdata-princeton/events/300737266/ https://lu.ma/q7pcfyjn?source=post_page-----34372cb3c7fa--------------------------------&tk=TTyakY If you're interested in working with Generative AI on the cloud, this virtual workshop is for you. Tim Spann from Cloudera and Yujian Tang from Zilliz will cover how you can implement your own GenAI workflows on the cloud at enterprise scale. 9:00 - 9:05: Intro 9:05 - 9:15: What is Milvus 9:15 - 9:25: Cloudera Development Platform 9:25 - 10:00: Demo Location https://www.youtube.com/watch?v=IfWIzKsoHnA https://github.com/tspannhw/SpeakerProfile https://www.linkedin.com/in/yujiantang/

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

Real-Time AI Streaming - AI Max Princeton

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines

Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines https://www.youtube.com/watch?v=Yeua8NlzQ3Y https://www.conf42.com/Large_Language_Models_LLMs_2024_Tim_Spann_generative_ai_streaming Adding Generative AI to Real-Time Streaming Pipelines Abstract Let’s build streaming pipelines that convert streaming events into prompts, call LLMs, and process the results. Summary Tim Spann: My talk is adding generative AI to real time streaming pipelines. I'm going to discuss a couple of different open source technologies. We'll touch on Kafka, Nifi, Flink, Python, Iceberg. All the slides, all the code and GitHub are out there. Llm, if you didn't know, is rapidly evolving. There's a lot of different ways to interact with models. That enrichment, transformation, processing really needs tools. The amount of models and projects and software that are available is massive. Nifi supports hundreds of different inputs and can convert them on the fly. Great way to distribute your data quickly to whoever needs it without duplication, without tight coupling. Fun to find new things to integrate into. So what we can do is, well, I want to get a meetup chat going. I have a processor here that just listens for events as they come from slack. And then I'm going to clean it up, add a couple fields and push that out to slack. Every model is a little bit of different tweaking. Nifi acts as a whole website. And as you see here, it can be get, post, put, whatever you want. We send that response back to flink and it shows up here. Thank you for attending this talk. I'm going to be speaking at some other events very shortly. Transcript This transcript was autogenerated. To make changes, submit a PR. Hi, Tim Spann here. My talk is adding generative AI to real time streaming pipelines, and we're here for the large language model conference at Comp 42, which is always a nice one, great place to be. I'm going to discuss a couple of different open source technologies that work together to enable you to build real time pipelines using large language models. So we'll touch on Kafka, Nifi, Flink, Python, Iceberg, and I'll show you a little bit of each one in the demos. I've been working with data machine learning, streaming IoT, some other things for a number of years, and you could contact me at any of these places, whether Twitter or whatever it's called, some different blogs, or in person at my meetups and at different conferences around the world. I do a weekly newsletter, cover streaming ML, a lot of LLM, open source, Python, Java, all kinds of fun stuff, as I mentioned, do a bunch of different meetups. They are not just in the east coast of the US, they are available virtually live, and I also put them on YouTube, and if you need them somewhere else, let me know. We publish all the slides, all the code and GitHub. Everything you need is out there. Let's get into the talk. Llm, if you didn't know, is rapidly evolving. While you're typing down the things that you use, it

2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...

28March2024-Codeless-Generative-AI-Pipelines

28March2024-Codeless-Generative-AI-Pipelines https://www.meetup.com/futureofdata-princeton/events/299440871/ https://www.meetup.com/real-time-analytics-meetup-ny/events/299290822/ ******Note***** The event is seat-limited, therefore please complete your registration here. Only people completing the form will be able to attend. ----------------------- We're excited to invite you to join us in-person, for a Real-Time Analytics exploration! Join us for an evening of insights, networking as we delve into the OSS technologies shaping the field! Agenda: 05:30-06:00: Pizza and friends 06:00- 06:40: Codeless GenAI Pipelines with Flink, Kafka, NiFi 06:40- 07:20 Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders 07:20-07:30 QNA Codeless GenAI Pipelines with Flink, Kafka, NiFi | Tim Spann, Cloudera Explore the power of real-time streaming with GenAI using Apache NiFi. Learn how NiFi simplifies data engineering workflows, allowing you to focus on creativity over technical complexities. I'll guide you through practical examples, showcasing NiFi's automation impact from ingestion to delivery. Whether you're a seasoned data engineer or new to GenAI, this talk offers valuable insights into optimizing workflows. Join us to unlock the potential of real-time streaming and witness how NiFi makes data engineering a breeze for GenAI applications! Real-Time Analytics in the Corporate World: How Apache Pinot® Powers Industry Leaders | Viktor Gamov, StarTree Explore how industry leaders like LinkedIn, Uber Eats, and Stripe are mastering real-time data with Viktor as your guide. Discover how Apache Pinot transforms data into actionable insights instantly. Viktor will showcase Pinot's features, including the Star-Tree Index, and explain why it's a game-changer in data strategy. This session is for everyone, from data geeks to business gurus, eager to uncover the future of tech. Join us and be wowed by the power of real-time analytics with Apache Pinot! ------- Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more.

TCFPro24 Building Real-Time Generative AI Pipelines

https://princetonacm.acm.org/tcfpro/ 18th Annual IEEE IT Professional Conference (ITPC) Armstrong Hall at The College of New Jersey Friday, March 15th, 2024 | 10:00 AM to 5:00 PM IT Professional Conference at Trenton Computer Festival IEEE Information Technology Professional Conference on Friday, March 15th, 2024 TCFPro24 Building Real-Time Generative AI Pipelines Building Real-Time Generative AI Pipelines In this talk, Tim will delve into the exciting realm of building real-time generative AI pipelines with streaming capabilities. The discussion will revolve around the integration of cutting-edge technologies to create dynamic and responsive systems that harness the power of generative algorithms. From leveraging streaming data sources to implementing advanced machine learning models, the presentation will explore the key components necessary for constructing a robust real-time generative AI pipeline. Practical insights, use cases, and best practices will be shared, offering a comprehensive guide for developers and data scientists aspiring to design and implement dynamic AI systems in a streaming environment. Tim will show a live demo showing we can use Apache NiFi to provide a live chat between a person in Slack and several LLM models all orchestrated with Apache NiFi, Apache Kafka and Python. We will use RAG against Chroma and Pinecone vector data stores, Hugging Face and WatsonX.AI LLM, and add additional context with NiFi lookups of stocks, weather and other data streams in real-time. Timothy Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

2024 Build Generative AI for Non-Profits

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...

2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipelines https://www.meetup.com/futureofdata-newyork/events/298660453/ Unlocking Financial Data with Real-Time Pipelines (Flink Analytics on Stocks with SQL ) By Timothy Spann Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. I will be utilizing NiFi 2.0 with Python and Vector Databases. Timothy Spann Principal Developer Advocate, Cloudera Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Kafka, Apache Pulsar, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. https://twitter.com/PaaSDev https://www.linkedin.com/in/timothyspann/ https://medium.com/@tspann https://github.com/tspannhw/FLiPStackWeekly/

Conf42-Python-Building Apache NiFi 2.0 Python Processors

Conf42-Python-Building Apache NiFi 2.0 Python Processors https://www.conf42.com/Python_2024_Tim_Spann_apache_nifi_2_processors Building Apache NiFi 2.0 Python Processors Abstract Let’s enhance real-time streaming pipelines with smart Python code. Adding code for vector databases and LLM. Summary Tim Spann: I'm going to be talking today, be building Apache 9520 Python processors. One of the main purposes of supporting Python in the streaming tool Apache Nifi is to interface with new machine learning and AI and Gen AI. He says Python is a real game changer for Cloudera. You're just going to add some metadata around it. It's a great way to pass a file along without changing it too substantially. We really need you to have Python 310 and again JDK 21 on your machine. You got to be smart about how you use these models. There are a ton of python processors available. You can use them in multiple ways. We're still in the early world of Python processors, so now's the time to start putting yours out there. Love to see a lot of people write their own. When we are parsing documents here, again, this is the Python one I'm picking PDF. Lots of different things you could do. If you're interested on writing your own python code for Apache Nifi, definitely reach out and thank.

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg with Stock Data and LLM Abstract In this talk, we’ll discuss how to use Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg to process and analyze stock data. We demonstrated the ingestion, processing, and analysis of stock data. Additionally, we illustrated how to use an LLM to generate predictions from the analyzed data. Karin Wolok Developer Relations, Dev Marketing, and Community Programming @ Project Elevate Karin Wolok's LinkedIn account Karin Wolok's twitter account Tim Spann Principal Developer Advocate @ Cloudera Tim Spann's LinkedIn account Tim Spann's twitter account https://www.conf42.com/Python_2024_Karin_Wolok_Tim_Spann_nifi__kafka_risingwave_iceberg_llm

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines

2024 Feb AI Meetup NYC GenAI_LLMs_ML_Data Codeless Generative AI Pipelines https://www.aicamp.ai/event/eventdetails/W2024022214 apache nifi llm generative ai gen ai ml dl machine learning apache kafka apache flink postgresql python AI Meetup (NYC): GenAI, LLMs, ML and Data Feb 22, 05:30 PM EST Welcome to the monthly in-person AI meetup in New York City, in collaboration with Microsoft. Join us for deep dive tech talks on AI, GenAI, LLMs and machine learning, food/drink, networking with speakers and fellow developers Agenda: * 5:30pm~6:00pm: Checkin, Food/drink and networking * 6:00pm~6:10pm: Welcome/community update * 6:10pm~8:30pm: Tech talks * 8:30pm: Q&A, Open discussion Tech Talk: Searching and Reasoning Over Multimedia Data with Vector Databases and LMMs Speaker: Zain Hasan (Weaviate LinkedIn) Abstract: In this talk, Zain Hasan will discuss how we can use open-source multimodal embedding models in conjunction with large generative multimodal models that can that can see, hear, read, and feel data(!), to perform cross-modal search(searching audio with images, videos with text etc.) and multimodal retrieval augmented generation (MM-RAG) at the billion-object scale with the help of open source vector databases. I will also demonstrate, with live code demos, how being able to perform this cross-modal retrieval in real-time can enables users to use LLMs that can reason over their enterprise multimodal data. This talk will revolve around how we can scale the usage of multimodal embedding and generative models in production. Tech Talk: Codeless Generative AI Pipelines Speaker: Timothy Spann (Cloudera LinkedIn) Abstract: Join us for an insightful talk on leveraging the power of real-time streaming tools, specifically Apache NiFi, to revolutionize GenAI data engineering. In this session, we’ll explore how the integration of Apache NiFi can automate the entire process of prompt building, making it a seamless and efficient task. Speakers/Topics: Stay tuned as we are updating speakers and schedules. If you have a keen interest in speaking to our community, we invite you to submit topics for consideration: Submit Topics Sponsors: We are actively seeking sponsors to support our community. Whether it is by offering venue spaces, providing food/drink, or cash sponsorship. Sponsors will have the chance to speak at the meetups, receive prominent recognition, and gain exposure to our extensive membership base of 20,000+ local or 300K+ developers worldwide. Venue: Microsoft NYC - Times Square, 11 Times Square, New York, NY 10036 Room Name: Central Park West 6501 Community on Slack/Discord - Event chat: chat and connect with speakers and attendees - Sharing blogs, events, job openings, projects collaborations Join Slack (search and join the #newyork channel) | Join Discord

DBA Fundamentals Group: Continuous SQL with Kafka and Flink

DBA Fundamentals Group: Continuous SQL with Kafka and Flink 20-Feb-2024 In this talk, I will walk through how someone can set up and run continuous SQL queries against Kafka topics utilizing Apache Flink. We will walk through creating Kafka topics, schemas, and publishing data. We will then cover consuming Kafka data, joining Kafka topics, and inserting new events into Kafka topics as they arrive. This basic overview will show hands-on techniques, tips, and examples of how to do this. Tim Spann Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK...

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines Unlocking Financial Data with Real-Time Pipelines Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. Key Points to be Covered: Introduction to Real-Time Data Pipelines: a. The limitations of traditional batch processing in the financial domain. b. Understanding the need for real-time data processing. Apache Flink: Powering Real-Time Stream Processing: a. Overview of Apache Flink and its role in real-time stream processing. b. Use cases for Apache Flink in the financial industry. c. How Flink enables fast, scalable, and fault-tolerant processing of streaming financial data. Apache Kafka: Building Resilient Event Streaming Platforms: a. Introduction to Apache Kafka and its role as a distributed streaming platform. b. Kafka's capabilities in handling high-throughput, fault-tolerant, and real-time data streaming. c. Integration of Kafka with financial data sources and consumers. Apache NiFi: Data Ingestion and Flow Management: a. Overview of Apache NiFi and its role in data ingestion and flow management. b. Data integration and transformation capabilities of NiFi for financial data. c. Utilizing NiFi to collect and process financial data from diverse sources. Iceberg: Efficient Data Lake Management: a. Understanding Iceberg and its role in managing large-scale data lakes. b. Iceberg's schema evolution and table-level metadata capabilities. c. How Iceberg simplifies data lake management in financial institutions. Real-World Use Cases: a. Real-time fraud detection using Flink, Kafka, and NiFi. b. Portfolio risk analysis with Iceberg and Flink. c. Streamlined regulatory reporting leveraging all four technologies. Best Practices and Considerations: a. Architectural considerations when building real-time financial data pipelines. b. Ensuring data integrity, security, and compliance in real-time pipelines. c. Scalability an

Building Real-Time Travel Alerts

Building Real-time Travel Alerts In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts. We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development. Let's get streaming. Apache Flink Apache Kafka Apache NiFi FLaNK Stack Tim Spann Big Data Conference Europe 2023

JConWorld_ Continuous SQL with Kafka and Flink