The State of Streaming.pdf

•

0 likes•6 views

Slides from the talk The State of Streaming & Modern Data by Pavan Keshavamurthy of Platformatory at Bengaluru Streams June 2023 meetup. A navigator to the initiated and uninitiated into the world of streaming: Cover the breadth of drivers, reference architectures, blueprints and tooling within the Apache Fn ecosystem

Technology

“The State of Streaming”
Presented at: Bengaluru Streams Meetup - 17
June, 2023
A practitioner’s guide to modern data architecture

whoami
● ಬೆಂಗಳೂರು boy.
● Cofounder, handyman @
platformatory.io
● OSS → ArchLinux, Envoy, Apache
Kafka, Kong (amongst others)
● Functional Programming,
Distributed systems, Himalayas,
Karnataka Music
- https://in.linkedin.com/in/
pavankmurthy
- https://grahana.net/
- https://twitter.com/p6

TOC
- Fast data beats slow data
- Some fundamental shifts in data engineering
- The modern data stack
- Hint, it has streaming in between
- A tale of two architectures
- Lambda
- Kappa
- A view of the streaming ecosystem
- Kafka is the CNS
- Data Movement
- Stream proc will intersect converge the
operational and analytical planes
- Streaming databases is where a lot of analytical
and BI loads will move to
- Data Mesh is the new architecture paradigm for a
modern data estatehe
- The greatest beneﬁciary will be AI/ML

328.77 M TB/d
120 ZB/y
*Protip: Big data getting bigger and
faster.

Fast Data > Slow Data
- MTTI = Mean Time To Insight
- MTTA = Mean Time To (Insight Driven, hopefully useful) Action

Traditional Data
Architecture * just
can’t keep up with
the explosion of
data
** includes
- Warehouses
- Marts
- Lakes
- Swamps

A few foundational
shifts for the
modern
data-driven
enterprise
1. Absolutely everything leads to the cloud
2. Real-time processing will be relevant in almost all
mission critical use-cases
3. Best-in-breed platforms beat packaged platforms
4. Data fan-out at scale over point to point
connectivity
5. Domain based architecture is the only way to break
the data monolith
6. A product approach to data is not only useful but
also necessary
McKinsey: How to build a data architecture to drive innovation

Streaming is hard,
but it is worth it 1. Stream as a core primitive across operational
and analytical planes
2. Data Sourcing & Movement
3. Storage
4. Processing
5. Querying
6. Cross-cutting concerns (Security, Observability,
Governance, etc)

Some uniﬁed data infrastructure archetypes emerge: Courtesy A16z

The Stories
20Trillion
events/day
400Billion
events/day
1Trillion+
evets/day

- Streaming is hotter than ever
- Apache Kafka: The de-facto protocol for
eventing
- Stream Processing Engines have ﬁnally come
off age: Apache Flink, Spark Streaming, KSQL,
Materialize, RisingWave and a whole host of
streaming SQL
- Lake-house architectures are open: Apache
Hudi, Iceberg
- Real Time Analytics now comes with a modern
ﬂavour: Apache Pinot, Druid, Clickhouse…
- AI/ML centric ops will increasingly converge
into streaming
A practitioner’s
view and closing
notes

Similar to The State of Streaming.pdf

“The mainframe is going away” is as true now as it was 10, 20 and 30 years ago. Mainframes are still crucial in handling critical business transactions, they were however built for an era where batch data movement was the norm and can be difficult to integrate into today’s data-driven, real-time, analytics-focused business processes as well as the environments that support them. Until now. Join experts from Confluent, Attunity, and Capgemini for a one-hour online talk session where you’ll learn how to: Unlock your mainframe data with unique change data capture (CDC) functionality without incurring the complexity and expense that come with sending ongoing queries into the mainframe database How using CDC benefits advanced analytics approaches such as deep machine learning and predictive analytics Deliver ongoing streams of data in real-time to the most demanding analytics environments Ensure that your analytics environment includes the broadest possible range of data sources and destinations while ensuring true enterprise-grade functionality Identify use cases that can help you get started delivering value to the business moving from POC to Pilot to Production

Leveraging Mainframe Data for Modern Analytics

confluent

Etl is Dead; Long Live Streams

confluent

Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture. Things we’ll discuss decoupling your system at scale via event-driven architectures include: – Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows. – Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern – How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea) – Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way. – Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems – Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem. – The simplicity and power of streaming SQL with microservices

Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...

confluent

London Spark Meetup 2014-11-11 @Skimlinks http://www.meetup.com/Spark-London/events/217362972/ To paraphrase the immortal crooner Don Ho: "Tiny Batches, in the wine, make me happy, make me feel fine." http://youtu.be/mlCiDEXuxxA Apache Spark provides support for streaming use cases, such as real-time analytics on log files, by leveraging a model called discretized streams (D-Streams). These "micro batch" computations operated on small time intervals, generally from 500 milliseconds up. One major innovation of Spark Streaming is that it leverages a unified engine. In other words, the same business logic can be used across multiple uses cases: streaming, but also interactive, iterative, machine learning, etc. This talk will compare case studies for production deployments of Spark Streaming, emerging design patterns for integration with popular complementary OSS frameworks, plus some of the more advanced features such as approximation algorithms, and take a look at what's ahead — including the new Python support for Spark Streaming that will be in the upcoming 1.2 release. Also, let's chat a bit about the new Databricks + O'Reilly developer certification for Apache Spark…

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Paco Nathan

Spark Streaming the Industrial IoT

Jim Haughwout

Data streaming

Alberto Paro

"Real-Time Analytics with Spark Streaming" presented at QCon São Paulo, 2015-03-26 http://qconsp.com/presentation/real-time-analytics-spark-streaming This talk presents an overview of Spark and its history and applications, then focuses on the Spark Streaming component used for real-time analytics. We compare it with earlier frameworks such as MillWheel and Storm, and explore industry motivations for open-source micro-batch streaming at scale. The talk will include demos for streaming apps that include machine-learning examples. We also consider public case studies of production deployments at scale. We’ll review the use of open-source sketch algorithms and probabilistic data structures that get leveraged in streaming – for example, the trade-off of 4% error bounds on real-time metrics for two orders of magnitude reduction in required memory footprint of a Spark app.

QCon São Paulo: Real-Time Analytics with Spark Streaming

Paco Nathan

Big Data Processing Beyond MapReduce by Dr. Flavio Villanustre

HPCC Systems

Connect K of SMACK:pykafka, kafka-python or?

Micron Technology

Databricks Meetup @ Los Angeles Apache Spark User Group

Paco Nathan

Big data apache spark + scala

Juantomás García Molina

Streaming data systems have been growing rapidly in importance to the modern data stack. Kafka’s kSQL provides an interface for analytic tools that speak SQL. Apache Superset, the most popular modern open-source visualization and analytics solution, plugs into nearly any data source that speaks SQL, including Kafka. Here, we review and compare methods for connecting Kafka to Superset to enable streaming analytics use cases including anomaly detection, operational monitoring, and online data integration.

Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset

HostedbyConfluent

Apache Phoenix with Actor Model (Akka.io) for real-time Big Data Programming...

Trieu Nguyen

Typesafe did a survey of Spark usage last year and found that a large percentage of Spark users combine it with Cassandra and Kafka. This talk focuses on streaming data scenarios that demonstrate how these three tools complement each other for building robust, scalable, and flexible data applications. Cassandra provides resilient and scalable storage, with flexible data format and query options. Kafka provides durable, scalable collection of streaming data with message-queue semantics. Spark provides very flexible analytics, everything from classic SQL queries to machine learning and graph algorithms, running in a streaming model based on "mini-batches", offline batch jobs, or interactive queries. We'll consider best practices and areas where improvements are needed.

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...

DataStax Academy

Stories About Spark, HPC and Barcelona by Jordi Torres

Spark Summit

Delivered this talk as part of Spark & Kafka Summit 2017 organized by Unicom Learning Conference. Big data processing is undoubtedly one of the most exciting areas in computing today, and remains an area of fast evolution and introduction of new ideas. Apache Spark is at the cusp of overtaking MapReduce to emerge as the de-facto standard for big data processing. Thanks to its multi-functional capabilities (SQL, Structured Streaming, ML Pipelines and GraphX) under one unified platform , Spark is now a dominant compute technology across various industry use cases and real-time analytics applications. Apache Spark in past few years has seen successful production and commercial deployments across E-Commerce, Healthcare and Travel industry. Session gave audience an understanding about the latest and upcoming trends in Big-Data Analytics and the role of Spark in enabling those future use-cases of advanced analytics. Session explored the latest concepts from Apache Spark 2.x and introduction to various ML/DL frameworks that can run Spark along with some real-life use-cases and applications from Retail and IoT verticals.

Apache Spark and future of advanced analytics

Muralidhar Somisetty

“Next Generation Grid – HPC Cloud” proposes a toolkit capturing current capabilities of Apache Hadoop, Spark, Flink and Heron as well as MPI and Asynchronous Many Task systems from HPC. This supports a Cloud-HPC-Edge (Fog, Device) Function as a Service Architecture. Note this "new grid" is focussed on data and IoT; not computing. Use interoperable common abstractions but multiple polymorphic implementations.

Next Generation Grid: Integrating Parallel and Distributed Computing Runtimes...

Geoffrey Fox

Facing Open Banking regulation, rapidly increasing transaction volumes and increasing customer expectations, Nationwide took the decision to take load off their back-end systems through real-time streaming of data changes into Kafka. Hear about how Nationwide started their journey with Kafka, from their initial use case of creating a real-time data cache using Change Data Capture, Kafka and Microservices to how Kafka allowed them to build a stream processing backbone used to reengineer the entire banking experience including online banking, payment processing and mortgage applications. See a working demo of the system and what happens to the system when the underlying infrastructure breaks. Technologies covered include: Change Data Capture, Kafka (Avro, partitioning and replication) and using KSQL and Kafka Streams Framework to join topics and process data.

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

confluent

Evolving Beyond the Data Lake: A Story of Wind and Rain

MapR Technologies

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Luan Moreno Medeiros Maciel

Similar to The State of Streaming.pdf (20)

Leveraging Mainframe Data for Modern Analytics

Etl is Dead; Long Live Streams

Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Spark Streaming the Industrial IoT

Data streaming

QCon São Paulo: Real-Time Analytics with Spark Streaming

Big Data Processing Beyond MapReduce by Dr. Flavio Villanustre

Connect K of SMACK:pykafka, kafka-python or?

Databricks Meetup @ Los Angeles Apache Spark User Group

Big data apache spark + scala

Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset

Apache Phoenix with Actor Model (Akka.io) for real-time Big Data Programming...

Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...

Stories About Spark, HPC and Barcelona by Jordi Torres

Apache Spark and future of advanced analytics

Next Generation Grid: Integrating Parallel and Distributed Computing Runtimes...

Introducing Events and Stream Processing into Nationwide Building Society (Ro...

Evolving Beyond the Data Lake: A Story of Wind and Rain

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Recently uploaded

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Scaling API-first – The story of a global engineering organization

Radu Cotescu

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

Evaluating the top large language models.pdf

ChristopherTHyatt

CNv6 Instructor Chapter 6 Quality of Service

giselly40

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Recently uploaded (20)

GenCyber Cyber Security Day Presentation

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Boost Fertility New Invention Ups Success Rates.pdf

Exploring the Future Potential of AI-Enabled Smartphone Processors

Axa Assurance Maroc - Insurer Innovation Award 2024

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Scaling API-first – The story of a global engineering organization

Boost PC performance: How more available memory can improve productivity

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Presentation on how to chat with PDF using ChatGPT code interpreter

Finology Group – Insurtech Innovation Award 2024

Artificial Intelligence: Facts and Myths

Handwritten Text Recognition for manuscripts and early printed texts

GenAI Risks & Security Meetup 01052024.pdf

08448380779 Call Girls In Civil Lines Women Seeking Men

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Evaluating the top large language models.pdf

CNv6 Instructor Chapter 6 Quality of Service

Partners Life - Insurer Innovation Award 2024

presentation ICT roal in 21st century education

The State of Streaming.pdf

1. “The State of Streaming” Presented at: Bengaluru Streams Meetup - 17 June, 2023 A practitioner’s guide to modern data architecture

2. whoami ● ಬೆಂಗಳೂರು boy. ● Cofounder, handyman @ platformatory.io ● OSS → ArchLinux, Envoy, Apache Kafka, Kong (amongst others) ● Functional Programming, Distributed systems, Himalayas, Karnataka Music - https://in.linkedin.com/in/ pavankmurthy - https://grahana.net/ - https://twitter.com/p6

3. TOC - Fast data beats slow data - Some fundamental shifts in data engineering - The modern data stack - Hint, it has streaming in between - A tale of two architectures - Lambda - Kappa - A view of the streaming ecosystem - Kafka is the CNS - Data Movement - Stream proc will intersect converge the operational and analytical planes - Streaming databases is where a lot of analytical and BI loads will move to - Data Mesh is the new architecture paradigm for a modern data estatehe - The greatest beneﬁciary will be AI/ML

4. 328.77 M TB/d 120 ZB/y *Protip: Big data getting bigger and faster.

5. Fast Data > Slow Data - MTTI = Mean Time To Insight - MTTA = Mean Time To (Insight Driven, hopefully useful) Action

6. Traditional Data Architecture * just can’t keep up with the explosion of data ** includes - Warehouses - Marts - Lakes - Swamps

7. A few foundational shifts for the modern data-driven enterprise 1. Absolutely everything leads to the cloud 2. Real-time processing will be relevant in almost all mission critical use-cases 3. Best-in-breed platforms beat packaged platforms 4. Data fan-out at scale over point to point connectivity 5. Domain based architecture is the only way to break the data monolith 6. A product approach to data is not only useful but also necessary McKinsey: How to build a data architecture to drive innovation

8. Streaming is hard, but it is worth it 1. Stream as a core primitive across operational and analytical planes 2. Data Sourcing & Movement 3. Storage 4. Processing 5. Querying 6. Cross-cutting concerns (Security, Observability, Governance, etc)

9. Some uniﬁed data infrastructure archetypes emerge: Courtesy A16z

10. Modern BI

11. Enterprise Multi-Modal processing

12. AI/ML

13. The Stories 20Trillion events/day 400Billion events/day 1Trillion+ evets/day

14. - Streaming is hotter than ever - Apache Kafka: The de-facto protocol for eventing - Stream Processing Engines have ﬁnally come off age: Apache Flink, Spark Streaming, KSQL, Materialize, RisingWave and a whole host of streaming SQL - Lake-house architectures are open: Apache Hudi, Iceberg - Real Time Analytics now comes with a modern ﬂavour: Apache Pinot, Druid, Clickhouse… - AI/ML centric ops will increasingly converge into streaming A practitioner’s view and closing notes

The State of Streaming.pdf

Recommended

Recommended

More Related Content

Similar to The State of Streaming.pdf

Similar to The State of Streaming.pdf (20)

More from AvinashUpadhyaya3

More from AvinashUpadhyaya3 (6)

Recently uploaded

Recently uploaded (20)

The State of Streaming.pdf