Events Everywhere: Enabling Digital Transformation in the Public Sector
•
0 likes•245 views
Events are driving a paradigm shift in application development from commands to reactions to events. Event-driven architectures using streaming platforms allow building real-time, scalable applications. Apache Kafka is an event streaming platform that can be used to build event-driven microservices and data pipelines to integrate various data sources. This enables building applications that deliver value to customers through real-time processing of event streams. The public sector can leverage these approaches to enable digital transformation through software-defined, event-driven applications across various domains.
Report
Share
Report
Share
Download to read offline
Recommended
Tale of two streaming frameworks (Karthik D - Walmart)
Speaker: Karthik Deivasigamani (https://www.linkedin.com/in/karthikdeivasigamani/)
Talk presented during Bangalore Kafka group's stream processing meetup at Walmart
https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/
Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka ...
Apache Kafka is an open source event streaming platform. It is often used to complement or even replace existing middleware to integrate applications and build microservice architectures. Apache Kafka is already used in various projects in almost every bigger company today. Understood, battled-tested, highly scalable, reliable, real-time.
Blockchain is a different story. This technology is a lot in the news, especially related to cryptocurrencies like Bitcoin. But what is the added value for software architectures? Is blockchain just hype and adds complexity? Or will it be used by everybody in the future, like a web browser or mobile app today? And how is it related to an integration architecture and event streaming platform?
This session explores use cases for blockchains and discusses different alternatives such as Hyperledger, Ethereum and a Kafka-native tamper-proof blockchain implementation. Different architectures are discussed to understand when blockchain really adds value and how it can be combined with the Apache Kafka ecosystem to integrate blockchain with the rest of the enterprise architecture to build a highly scalable and reliable event streaming infrastructure.
Speakers:
Kai Waehner, Technology Evangelist, Confluent
Stephen Reed, CTO, Co-Founder, AiB
Concepts and Patterns for Streaming Services with Kafka
Cloud Native Night March 2020, Mainz: Talk by Perry Krol (@perkrol, Confluent)
=== Please download slides if blurred! ===
Abstract: Proven approaches such as service-oriented and event-driven architectures are joined by newer techniques such as microservices, reactive architectures, DevOps, and stream processing. Many of these patterns are successful by themselves, but they provide a more holistic and compelling approach when applied together. In this session Confluent will provide insights how service-based architectures and stream processing tools such as Apache Kafka® can help you build business-critical systems. You will learn why streaming beats request-response based architectures in complex, contemporary use cases, and explain why replayable logs such as Kafka provide a backbone for both service communication and shared datasets.
Based on these principles, we will explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches, apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as “inside out databases” and “event streams as a source of truth”.
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing...
Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of our architecture is the development of a universal eventing framework to accommodate transformation into an event-driven architecture (EDA). Our application provides a representational state transfer (REST) and remote procedure call (gRPC) interface that allows development teams to publish and consume events with a simple Noun-Verb-Object (NVO) syntax. Embedded within the framework are structured schema evolutions with Confluent Schema Registry and AVRO, configurable (self-service) event-routing with K-Tables, dynamic event-aggregation with Kafka Streams, distributed event-tracing with Jaeger, and event querying against a MongoDB event-store hydrated by Kafka Connect. Lastly, we developed techniques to handle long-term event storage within Kafka; specifically surrounding the automated deletion of expired events and re-hydration of missing events. In Centene's first business use case, events related to claim processing of provider reconsiderations was used to provide real-time updates to providers on the status of their claim appeals. To satisfy the business requirement, multiple monolith systems independently leveraged the event framework, to stream status updates for display on the Centene Provider Portal instantly. This provided a capability that was brand new to Centene: the ability to interact and engage with our providers in real-time through the use of event streams. In this presentation, we will walk you through the architecture of the eventing framework and showcase how our business requirements within our claims adjudication domain were able to be solved leveraging the Kafka Stream DSL and the Confluent Platform. And more importantly, how Centene plans on leveraging this framework, written on-top of Kafka Streams, to change our culture from batch processing to real-time stream processing.
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Your mainframe and IBM i platforms do hard work for your business, supporting essential computing transactions every day. However, mainframe data does not easily integrate with the cloud platforms driving data-driven, real-time, analytics-focused business processes. Integrating data from this critical technology often results in high costs, missed deadlines, and unhappy customers. So, what can you do? Join us to hear how Precisely Connect can help use the power of Apache Kafka to eliminate data silos and make cloud-based, event-driven data architectures a reality. Start your cloud transformation journey today, knowing you don’t need to leave essential transaction data behind! Learn more about: • Where to begin your cloud transformation journey using mainframe and IBM i data and Apache Kafka • What you need to move mainframe and IBM i data to the cloud while reducing costs, modernizing architectures, and using the staff you have today • How Precisely Connect customers are using change data capture and Apache Kafka to deliver real-time insights to the cloud
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
Life doesn't happen in batch mode which is why application engineers and data architects need to closely cooperate to get the best out of streaming platforms like Apache Kafka and NoSQL data stores such as MongoDB. This session explores ways and means to integrate both worlds in a streaming fashion.
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
Responding to a global pandemic presents a unique set of technical and public health challenges. The real challenge is the ability to gather data coming in via many data streams in variety of formats influences the real-world outcome and impacts everyone. The Centers for Disease Control and Prevention CELR (COVID Electronic Lab Reporting) program was established to rapidly aggregate, validate, transform, and distribute laboratory testing data submitted by public health departments and other partners. Confluent Kafka with KStreams and Connect play a critical role in program objectives to:
o Track the threat of COVID-19 virus
o Provide comprehensive data for local, state, and federal response
o Better understand locations with an increase in incidence
Amsterdam meetup at ING June 18, 2019
The document describes an Amsterdam Apache Kafka Meetup. The meetup agenda includes three presentations: 1) Using Kafka to integrate data warehouses and cloud-based big data systems by Mic Hussey of Confluent, 2) Topic management at scale by several speakers from ING, and 3) Real-time investment alerts using Apache Kafka and Spring Kafka at ING Bank by two additional speakers from ING. The meetup concludes with a drinks and networking session. Attendees are encouraged to join the Confluent community on Slack and local Kafka meetup groups.
Recommended
Tale of two streaming frameworks (Karthik D - Walmart)
Speaker: Karthik Deivasigamani (https://www.linkedin.com/in/karthikdeivasigamani/)
Talk presented during Bangalore Kafka group's stream processing meetup at Walmart
https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/
Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka ...
Apache Kafka is an open source event streaming platform. It is often used to complement or even replace existing middleware to integrate applications and build microservice architectures. Apache Kafka is already used in various projects in almost every bigger company today. Understood, battled-tested, highly scalable, reliable, real-time.
Blockchain is a different story. This technology is a lot in the news, especially related to cryptocurrencies like Bitcoin. But what is the added value for software architectures? Is blockchain just hype and adds complexity? Or will it be used by everybody in the future, like a web browser or mobile app today? And how is it related to an integration architecture and event streaming platform?
This session explores use cases for blockchains and discusses different alternatives such as Hyperledger, Ethereum and a Kafka-native tamper-proof blockchain implementation. Different architectures are discussed to understand when blockchain really adds value and how it can be combined with the Apache Kafka ecosystem to integrate blockchain with the rest of the enterprise architecture to build a highly scalable and reliable event streaming infrastructure.
Speakers:
Kai Waehner, Technology Evangelist, Confluent
Stephen Reed, CTO, Co-Founder, AiB
Concepts and Patterns for Streaming Services with Kafka
Cloud Native Night March 2020, Mainz: Talk by Perry Krol (@perkrol, Confluent)
=== Please download slides if blurred! ===
Abstract: Proven approaches such as service-oriented and event-driven architectures are joined by newer techniques such as microservices, reactive architectures, DevOps, and stream processing. Many of these patterns are successful by themselves, but they provide a more holistic and compelling approach when applied together. In this session Confluent will provide insights how service-based architectures and stream processing tools such as Apache Kafka® can help you build business-critical systems. You will learn why streaming beats request-response based architectures in complex, contemporary use cases, and explain why replayable logs such as Kafka provide a backbone for both service communication and shared datasets.
Based on these principles, we will explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches, apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as “inside out databases” and “event streams as a source of truth”.
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing...
Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of our architecture is the development of a universal eventing framework to accommodate transformation into an event-driven architecture (EDA). Our application provides a representational state transfer (REST) and remote procedure call (gRPC) interface that allows development teams to publish and consume events with a simple Noun-Verb-Object (NVO) syntax. Embedded within the framework are structured schema evolutions with Confluent Schema Registry and AVRO, configurable (self-service) event-routing with K-Tables, dynamic event-aggregation with Kafka Streams, distributed event-tracing with Jaeger, and event querying against a MongoDB event-store hydrated by Kafka Connect. Lastly, we developed techniques to handle long-term event storage within Kafka; specifically surrounding the automated deletion of expired events and re-hydration of missing events. In Centene's first business use case, events related to claim processing of provider reconsiderations was used to provide real-time updates to providers on the status of their claim appeals. To satisfy the business requirement, multiple monolith systems independently leveraged the event framework, to stream status updates for display on the Centene Provider Portal instantly. This provided a capability that was brand new to Centene: the ability to interact and engage with our providers in real-time through the use of event streams. In this presentation, we will walk you through the architecture of the eventing framework and showcase how our business requirements within our claims adjudication domain were able to be solved leveraging the Kafka Stream DSL and the Confluent Platform. And more importantly, how Centene plans on leveraging this framework, written on-top of Kafka Streams, to change our culture from batch processing to real-time stream processing.
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Your mainframe and IBM i platforms do hard work for your business, supporting essential computing transactions every day. However, mainframe data does not easily integrate with the cloud platforms driving data-driven, real-time, analytics-focused business processes. Integrating data from this critical technology often results in high costs, missed deadlines, and unhappy customers. So, what can you do? Join us to hear how Precisely Connect can help use the power of Apache Kafka to eliminate data silos and make cloud-based, event-driven data architectures a reality. Start your cloud transformation journey today, knowing you don’t need to leave essential transaction data behind! Learn more about: • Where to begin your cloud transformation journey using mainframe and IBM i data and Apache Kafka • What you need to move mainframe and IBM i data to the cloud while reducing costs, modernizing architectures, and using the staff you have today • How Precisely Connect customers are using change data capture and Apache Kafka to deliver real-time insights to the cloud
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
Life doesn't happen in batch mode which is why application engineers and data architects need to closely cooperate to get the best out of streaming platforms like Apache Kafka and NoSQL data stores such as MongoDB. This session explores ways and means to integrate both worlds in a streaming fashion.
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
Responding to a global pandemic presents a unique set of technical and public health challenges. The real challenge is the ability to gather data coming in via many data streams in variety of formats influences the real-world outcome and impacts everyone. The Centers for Disease Control and Prevention CELR (COVID Electronic Lab Reporting) program was established to rapidly aggregate, validate, transform, and distribute laboratory testing data submitted by public health departments and other partners. Confluent Kafka with KStreams and Connect play a critical role in program objectives to:
o Track the threat of COVID-19 virus
o Provide comprehensive data for local, state, and federal response
o Better understand locations with an increase in incidence
Amsterdam meetup at ING June 18, 2019
The document describes an Amsterdam Apache Kafka Meetup. The meetup agenda includes three presentations: 1) Using Kafka to integrate data warehouses and cloud-based big data systems by Mic Hussey of Confluent, 2) Topic management at scale by several speakers from ING, and 3) Real-time investment alerts using Apache Kafka and Spring Kafka at ING Bank by two additional speakers from ING. The meetup concludes with a drinks and networking session. Attendees are encouraged to join the Confluent community on Slack and local Kafka meetup groups.
Real time analytics in Azure IoT
1) Sam Vanhoutte discusses using Azure services like IoT Edge, IoT Hub, Stream Analytics, and Azure Databricks for real-time data analytics in IoT from edge to cloud.
2) A traffic camera scenario is presented where IoT Edge is used at the edge for tasks like license plate recognition while the cloud is used for analytics like detecting speeding tickets and suspicious vehicles.
3) Stream Analytics is used both at the edge and in the cloud to process streaming data in real-time while Azure Databricks is used for structured streaming and continuous aggregations using Apache Spark.
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Guru Sattanathan, Confluent, Senior Solutions Engineer
Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats!
Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts?
If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today.
In this session, we will go in this order:
-Recap on Enterprise Integration technologies
-What are the key flaws & What needs improvement?
-What is Apache Kafka?
-Rethinking Integration using Apache Kafka
https://www.meetup.com/KafkaMelbourne/events/280590162/
Stream me to the Cloud (and back) with Confluent & MongoDB
In this online talk, we’ll explore how and why companies are leveraging Confluent and MongoDB to modernize their architecture and leverage the scalability of the cloud and the velocity of streaming. Based upon a sample retail business scenario, we will explain how changes in an on-premise database are streamed via the Confluent Cloud to MongoDB Atlas and back.
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
New Features in Confluent Platform 6.0 / Apache Kafka 2.6, including REST Proxy and API, Tiered Storage for AWS S3 and GCP GCS, Cluster Linking (On-Premise, Edge, Hybrid, Multi-Cloud), Self-Balancing Clusters), ksqlDB.
Real time data processing and model inferncing platform with Kafka streams (N...
Speaker: Navinder Pal Singh Brar (https://www.linkedin.com/in/navinderpalsinghbrar/)
Talk presented during Bangalore Kafka group's stream processing meetup at Walmart
https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior.
While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement.
In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Presenters: Rachel Pedreschi, Senior Director, Solutions Engineering, Imply.io + Josh Treichel, Partner Solutions Architect, Confluent
Analytic pipelines running purely on batch processing systems can suffer from hours of data lag, resulting in accuracy issues with analysis and overall decision-making. Join us for a demo to learn how easy it is to integrate your Apache Kafka® streams in Apache Druid (incubating) to provide real-time insights into the data.
In this online talk, you’ll hear about ingesting your Kafka streams into Imply’s scalable analytic engine and gaining real-time insights via a modern user interface.
Register now to learn about:
-The benefits of combining a real-time streaming platform with a comprehensive analytics stack
-Building an analytics pipeline by integrating Confluent Platform and Imply
-How KSQL, streaming SQL for Kafka, can easily transform and filter streams of data in real time
-Querying and visualizing streaming data in Imply
-Practical ways to implement Confluent Platform and Imply to address common use cases such as analyzing network flows, collecting and monitoring IoT data and visualizing clickstream data
Confluent Platform, developed by the creators of Kafka, enables the ingest and processing of massive amounts of real-time event data. Imply, the complete analytics stack built on Druid, can ingest, store, query and visualize streaming data from Confluent Platform, enabling end-to-end real-time analytics. Together, Confluent and Imply can provide low latency data delivery, data transform, and data querying capabilities to power a range of use cases.
Kafka summit SF 2019 - the art of the event-streaming app
Have you ever imagined what it would be like to build a massively scalable streaming application on Kafka, the challenges, the patterns and the thought process involved? How much of the application can be reused? What patterns will you discover? How does it all fit together? Depending upon your use case and business, this can mean many things. Starting out with a data pipeline is one thing, but evolving into a company-wide real-time application that is business critical and entirely dependent upon a streaming platform is a giant leap. Large-scale streaming applications are also called event streaming applications. They are classically different from other data systems; event streaming applications are viewed as a series of interconnected streams that are topologically defined using stream processors; they hold state that models your use case as events. Almost like a deconstructed realtime database.
In this talk, I step through the origins of event streaming systems, understanding how they are developed from raw events to evolve into something that can be adopted at an organizational scale. I start with event-first thinking, Domain Driven Design to build data models that work with the fundamentals of Streams, Kafka Streams, KSQL and Serverless (FaaS). Building upon this, I explain how to build common business functionality by stepping through patterns for Scalable payment processing Run it on rails: Instrumentation and monitoring Control flow patterns (start, stop, pause) Finally, all of these concepts are combined in a solution architecture that can be used at enterprise scale. I will introduce enterprise patterns such as events-as-a-backbone, events as APIs and methods for governance and self-service. You will leave talk with an understanding of how to model events with event-first thinking, how to work towards reusable streaming patterns and most importantly, how it all fits together at scale.
dotScale 2017 Keynote: The Rise of Real Time by Neha Narkhede
Slides from Neha Narkhede's keynote at the dotScale conference in Paris on April 24th, 2017.
There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it.
This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.
Redis and Kafka - Advanced Microservices Design Patterns Simplified
The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.
Bridge Your Kafka Streams to Azure Webinar
With a fully managed Apache Kafka(R) as-a-service on Microsoft Azure, businesses can focus on building applications and not managing clusters. Build a persistent bridge from on-premises data systems to the cloud with a hybrid Kafka service or stream across public clouds for multi-cloud data pipelines.
In this session for business and technical data leaders, you can learn about powering business applications with the managed Kafka service that streams data into Azure SQL Data Warehouse, Cosmos DB, Azure Data Lake Storage and Azure Blob Storage.
Why Build an Apache Kafka® Connector
The document discusses why companies should build Apache Kafka connectors. It begins with an agenda that covers what event streaming is, Kafka and connecting to Kafka, the value of building a connector, and a Q&A session. It then discusses how event streaming is enabling new business models and outcomes across many industries. The Confluent Verified Integrations Program helps partners build and verify connectors, with benefits like marketing support and a multi-vendor case management process. The presentation provides information on the verification levels (Gold for Kafka Connect API connectors, Standard for other integrations), requirements, submission process, and how Confluent can help support partners. It encourages attendees to sign up for the program or contact them with any questions.
Kakfa summit london 2019 - the art of the event-streaming app
Have you ever imagined what it would be like to build a massively scalable streaming application on Kafka, the challenges, the patterns and the thought process involved? How much of the application can be reused? What patterns will you discover? How does it all fit together? Depending upon your use case and business, this can mean many things. Starting out with a data pipeline is one thing, but evolving into a company-wide real-time application that is business critical and entirely dependent upon a streaming platform is a giant leap. Large-scale streaming applications are also called event streaming applications. They are classically different from other data systems; event streaming applications are viewed as a series of interconnected streams that are topologically defined using stream processors; they hold state that models your use case as events. Almost like a deconstructed real-time database.
In this talk, I step through the origins of event streaming systems, understanding how they are developed from raw events to evolve into something that can be adopted at an organizational scale. I start with event-first thinking, Domain Driven Design to build data models that work with the fundamentals of Streams, Kafka Streams, KSQL and Serverless (FaaS).
Building upon this, I explain how to build common business functionality by stepping through the patterns for: – Scalable payment processing – Run it on rails: Instrumentation and monitoring – Control flow patterns Finally, all of these concepts are combined in a solution architecture that can be used at an enterprise scale. I will introduce enterprise patterns such as events-as-a-backbone, events as APIs and methods for governance and self-service. You will leave talk with an understanding of how to model events with event-first thinking, how to work towards reusable streaming patterns and most importantly, how it all fits together at scale.
New Approaches for Fraud Detection on Apache Kafka and KSQL
This document discusses new approaches for fraud detection using Apache Kafka and KSQL. It introduces KSQL, an open-source streaming SQL engine for Apache Kafka. KSQL can be used to perform streaming ETL, anomaly detection, and event monitoring using SQL-like queries on streaming data. The document demonstrates how to run KSQL locally or in a client-server configuration, and how Arcadia Data provides a visualization layer on top of KSQL to enable visual analytics on streaming data.
Patterns of Distributed Application Design
This webinar by Orkhan Gasimov (Senior Solution Architect, Consultant, GlobalLogic) was delivered at Java Community Webinar #3 on October 16, 2020.
During webinar we had simplified overview of classical and modern architecture patterns and concepts that are used for development of distributed applications during the last decade.
More details and presentation: https://www.globallogic.com/ua/about/events/java-community-webinar-3/
APAC Kafka Summit - Best Of
Watch the replay here: https://videos.confluent.io/watch/iLBZiCiAbHhUPwHcbTQ9cn?
Speaker: David Peterson
GCP for Apache Kafka® Users: Stream Ingestion and Processing
Watch this talk here: https://www.confluent.io/online-talks/gcp-for-apache-kafka-users-stream-ingestion-processing
In private and public clouds, stream analytics commonly means stateless processing systems organized around Apache Kafka® or a similar distributed log service. GCP took a somewhat different tack, with Cloud Pub/Sub, Dataflow, and BigQuery, distributing the responsibility for processing among ingestion, processing and database technologies.
We compare the two approaches to data integration and show how Dataflow allows you to join and transform and deliver data streams among on-prem and cloud Apache Kafka clusters, Cloud Pub/Sub topics and a variety of databases. The session will have a mix of architectural discussions and practical code reviews of Dataflow-based pipelines.
Evolving from Messaging to Event Streaming
- Event streaming is evolving as a new paradigm for data integration that addresses challenges with traditional messaging queue and ETL-based approaches. Kafka in particular provides an event streaming platform that can be used to build a data mesh architecture.
- Traditional integration approaches rely on message queues, service buses, and batch ETL processes which limit flexibility, scalability, and the ability to access data in real-time. Event streaming addresses these challenges.
- A data mesh architecture built on event streaming decentralizes control of data, makes it more accessible and reusable across organizational boundaries via self-service APIs. This enables greater agility, data sharing, and real-time analytics compared to traditional integration.
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Despite great advances in Kafka's SaaS offerings it can still be challenging to create a sustainable event-driven ecosystem. Often platform engineers become de facto ‘gatekeepers’ of events & topics, yet their day job is not about data modelling or domain expertise. We've all seen the bottlenecks these unsustainable processes create.
Realising the potential of event streams requires much more than infrastructure. Beyond an event-driven mindset, it requires domain experts to lead creation of well-defined discoverable events through fit-for-purpose governance. AsyncAPI is the OpenAPI for events that can form the basis of the required self-governing, self-service eventing framework.
This session will introduce a self-governing framework using AsyncAPI and share how the Bank of New Zealand applied this framework to leverage a passionate Kafka community and embed event-driven thinking. You’ll leave with a tangible set of ideas to give your own events a bit more swagger using AsyncAPI.
Now You See Me, Now You Compute: Building Event-Driven Architectures with Apa...
Talk URL: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/77360
Abstract: Would you cross the street with traffic information that’s a minute old? Certainly not. Modern businesses have the same needs nowadays, whether it’s due to competitive pressure or because their customers have much higher expectations of how they want to interact with a product or service. At the heart of this movement are events: in today’s digital age, events are everywhere. Every digital action—across online purchases to ride-sharing requests to bank deposits—creates a set of events around transaction amount, transaction time, user location, account balance, and much more. The technology that allows businesses to read, write, store, and compute and process these events in real-time are event-streaming platforms, and tens of thousands of companies like Netflix, Audi, PayPal, Airbnb, Uber, and Pinterest have picked Apache Kafka as the de facto choice to implement event-driven architectures and reshape their industries.
Michael Noll explores why and how you can use Apache Kafka and its growing ecosystem to build event-driven architectures that are elastic, scalable, robust, and fault tolerant, whether it’s on-premises, in the cloud, on bare metal machines, or in Kubernetes with Docker containers. Specifically, you’ll look at Kafka as the storage and publish and subscribe layer; Kafka’s Connect framework for integrating external data systems such as MySQL, Elastic, or S3 with Kafka; and Kafka’s Streams API and KSQL as the compute layer to implement event-driven applications and microservices in Java and Scala and streaming SQL, respectively, that process the events flowing through Kafka in real time. Michael provides an overview of the most relevant functionality, both current and upcoming, and shares best practices and typical use cases so you can tie it all together for your own needs.
Streaming real time data with Vibe Data Stream
The process of streaming real-time data from a wide variety of machine data sources and entities can be very complex and unwieldy. Using an agent-based approach, Informatica has invented a new technique and open access product that makes this process much more user friendly and efficient, even when dealing with multiple environments such as Hadoop, Cassandra, Storm, Amazon Kinesis and Complex Event Processing.
More Related Content
What's hot
Real time analytics in Azure IoT
1) Sam Vanhoutte discusses using Azure services like IoT Edge, IoT Hub, Stream Analytics, and Azure Databricks for real-time data analytics in IoT from edge to cloud.
2) A traffic camera scenario is presented where IoT Edge is used at the edge for tasks like license plate recognition while the cloud is used for analytics like detecting speeding tickets and suspicious vehicles.
3) Stream Analytics is used both at the edge and in the cloud to process streaming data in real-time while Azure Databricks is used for structured streaming and continuous aggregations using Apache Spark.
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Guru Sattanathan, Confluent, Senior Solutions Engineer
Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats!
Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts?
If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today.
In this session, we will go in this order:
-Recap on Enterprise Integration technologies
-What are the key flaws & What needs improvement?
-What is Apache Kafka?
-Rethinking Integration using Apache Kafka
https://www.meetup.com/KafkaMelbourne/events/280590162/
Stream me to the Cloud (and back) with Confluent & MongoDB
In this online talk, we’ll explore how and why companies are leveraging Confluent and MongoDB to modernize their architecture and leverage the scalability of the cloud and the velocity of streaming. Based upon a sample retail business scenario, we will explain how changes in an on-premise database are streamed via the Confluent Cloud to MongoDB Atlas and back.
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
New Features in Confluent Platform 6.0 / Apache Kafka 2.6, including REST Proxy and API, Tiered Storage for AWS S3 and GCP GCS, Cluster Linking (On-Premise, Edge, Hybrid, Multi-Cloud), Self-Balancing Clusters), ksqlDB.
Real time data processing and model inferncing platform with Kafka streams (N...
Speaker: Navinder Pal Singh Brar (https://www.linkedin.com/in/navinderpalsinghbrar/)
Talk presented during Bangalore Kafka group's stream processing meetup at Walmart
https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/266777028/
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior.
While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement.
In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Presenters: Rachel Pedreschi, Senior Director, Solutions Engineering, Imply.io + Josh Treichel, Partner Solutions Architect, Confluent
Analytic pipelines running purely on batch processing systems can suffer from hours of data lag, resulting in accuracy issues with analysis and overall decision-making. Join us for a demo to learn how easy it is to integrate your Apache Kafka® streams in Apache Druid (incubating) to provide real-time insights into the data.
In this online talk, you’ll hear about ingesting your Kafka streams into Imply’s scalable analytic engine and gaining real-time insights via a modern user interface.
Register now to learn about:
-The benefits of combining a real-time streaming platform with a comprehensive analytics stack
-Building an analytics pipeline by integrating Confluent Platform and Imply
-How KSQL, streaming SQL for Kafka, can easily transform and filter streams of data in real time
-Querying and visualizing streaming data in Imply
-Practical ways to implement Confluent Platform and Imply to address common use cases such as analyzing network flows, collecting and monitoring IoT data and visualizing clickstream data
Confluent Platform, developed by the creators of Kafka, enables the ingest and processing of massive amounts of real-time event data. Imply, the complete analytics stack built on Druid, can ingest, store, query and visualize streaming data from Confluent Platform, enabling end-to-end real-time analytics. Together, Confluent and Imply can provide low latency data delivery, data transform, and data querying capabilities to power a range of use cases.
Kafka summit SF 2019 - the art of the event-streaming app
Have you ever imagined what it would be like to build a massively scalable streaming application on Kafka, the challenges, the patterns and the thought process involved? How much of the application can be reused? What patterns will you discover? How does it all fit together? Depending upon your use case and business, this can mean many things. Starting out with a data pipeline is one thing, but evolving into a company-wide real-time application that is business critical and entirely dependent upon a streaming platform is a giant leap. Large-scale streaming applications are also called event streaming applications. They are classically different from other data systems; event streaming applications are viewed as a series of interconnected streams that are topologically defined using stream processors; they hold state that models your use case as events. Almost like a deconstructed realtime database.
In this talk, I step through the origins of event streaming systems, understanding how they are developed from raw events to evolve into something that can be adopted at an organizational scale. I start with event-first thinking, Domain Driven Design to build data models that work with the fundamentals of Streams, Kafka Streams, KSQL and Serverless (FaaS). Building upon this, I explain how to build common business functionality by stepping through patterns for Scalable payment processing Run it on rails: Instrumentation and monitoring Control flow patterns (start, stop, pause) Finally, all of these concepts are combined in a solution architecture that can be used at enterprise scale. I will introduce enterprise patterns such as events-as-a-backbone, events as APIs and methods for governance and self-service. You will leave talk with an understanding of how to model events with event-first thinking, how to work towards reusable streaming patterns and most importantly, how it all fits together at scale.
dotScale 2017 Keynote: The Rise of Real Time by Neha Narkhede
Slides from Neha Narkhede's keynote at the dotScale conference in Paris on April 24th, 2017.
There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it.
This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.
Redis and Kafka - Advanced Microservices Design Patterns Simplified
The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.
Bridge Your Kafka Streams to Azure Webinar
With a fully managed Apache Kafka(R) as-a-service on Microsoft Azure, businesses can focus on building applications and not managing clusters. Build a persistent bridge from on-premises data systems to the cloud with a hybrid Kafka service or stream across public clouds for multi-cloud data pipelines.
In this session for business and technical data leaders, you can learn about powering business applications with the managed Kafka service that streams data into Azure SQL Data Warehouse, Cosmos DB, Azure Data Lake Storage and Azure Blob Storage.
Why Build an Apache Kafka® Connector
The document discusses why companies should build Apache Kafka connectors. It begins with an agenda that covers what event streaming is, Kafka and connecting to Kafka, the value of building a connector, and a Q&A session. It then discusses how event streaming is enabling new business models and outcomes across many industries. The Confluent Verified Integrations Program helps partners build and verify connectors, with benefits like marketing support and a multi-vendor case management process. The presentation provides information on the verification levels (Gold for Kafka Connect API connectors, Standard for other integrations), requirements, submission process, and how Confluent can help support partners. It encourages attendees to sign up for the program or contact them with any questions.
Kakfa summit london 2019 - the art of the event-streaming app
Have you ever imagined what it would be like to build a massively scalable streaming application on Kafka, the challenges, the patterns and the thought process involved? How much of the application can be reused? What patterns will you discover? How does it all fit together? Depending upon your use case and business, this can mean many things. Starting out with a data pipeline is one thing, but evolving into a company-wide real-time application that is business critical and entirely dependent upon a streaming platform is a giant leap. Large-scale streaming applications are also called event streaming applications. They are classically different from other data systems; event streaming applications are viewed as a series of interconnected streams that are topologically defined using stream processors; they hold state that models your use case as events. Almost like a deconstructed real-time database.
In this talk, I step through the origins of event streaming systems, understanding how they are developed from raw events to evolve into something that can be adopted at an organizational scale. I start with event-first thinking, Domain Driven Design to build data models that work with the fundamentals of Streams, Kafka Streams, KSQL and Serverless (FaaS).
Building upon this, I explain how to build common business functionality by stepping through the patterns for: – Scalable payment processing – Run it on rails: Instrumentation and monitoring – Control flow patterns Finally, all of these concepts are combined in a solution architecture that can be used at an enterprise scale. I will introduce enterprise patterns such as events-as-a-backbone, events as APIs and methods for governance and self-service. You will leave talk with an understanding of how to model events with event-first thinking, how to work towards reusable streaming patterns and most importantly, how it all fits together at scale.
New Approaches for Fraud Detection on Apache Kafka and KSQL
This document discusses new approaches for fraud detection using Apache Kafka and KSQL. It introduces KSQL, an open-source streaming SQL engine for Apache Kafka. KSQL can be used to perform streaming ETL, anomaly detection, and event monitoring using SQL-like queries on streaming data. The document demonstrates how to run KSQL locally or in a client-server configuration, and how Arcadia Data provides a visualization layer on top of KSQL to enable visual analytics on streaming data.
Patterns of Distributed Application Design
This webinar by Orkhan Gasimov (Senior Solution Architect, Consultant, GlobalLogic) was delivered at Java Community Webinar #3 on October 16, 2020.
During webinar we had simplified overview of classical and modern architecture patterns and concepts that are used for development of distributed applications during the last decade.
More details and presentation: https://www.globallogic.com/ua/about/events/java-community-webinar-3/
APAC Kafka Summit - Best Of
Watch the replay here: https://videos.confluent.io/watch/iLBZiCiAbHhUPwHcbTQ9cn?
Speaker: David Peterson
GCP for Apache Kafka® Users: Stream Ingestion and Processing
Watch this talk here: https://www.confluent.io/online-talks/gcp-for-apache-kafka-users-stream-ingestion-processing
In private and public clouds, stream analytics commonly means stateless processing systems organized around Apache Kafka® or a similar distributed log service. GCP took a somewhat different tack, with Cloud Pub/Sub, Dataflow, and BigQuery, distributing the responsibility for processing among ingestion, processing and database technologies.
We compare the two approaches to data integration and show how Dataflow allows you to join and transform and deliver data streams among on-prem and cloud Apache Kafka clusters, Cloud Pub/Sub topics and a variety of databases. The session will have a mix of architectural discussions and practical code reviews of Dataflow-based pipelines.
Evolving from Messaging to Event Streaming
- Event streaming is evolving as a new paradigm for data integration that addresses challenges with traditional messaging queue and ETL-based approaches. Kafka in particular provides an event streaming platform that can be used to build a data mesh architecture.
- Traditional integration approaches rely on message queues, service buses, and batch ETL processes which limit flexibility, scalability, and the ability to access data in real-time. Event streaming addresses these challenges.
- A data mesh architecture built on event streaming decentralizes control of data, makes it more accessible and reusable across organizational boundaries via self-service APIs. This enables greater agility, data sharing, and real-time analytics compared to traditional integration.
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Despite great advances in Kafka's SaaS offerings it can still be challenging to create a sustainable event-driven ecosystem. Often platform engineers become de facto ‘gatekeepers’ of events & topics, yet their day job is not about data modelling or domain expertise. We've all seen the bottlenecks these unsustainable processes create.
Realising the potential of event streams requires much more than infrastructure. Beyond an event-driven mindset, it requires domain experts to lead creation of well-defined discoverable events through fit-for-purpose governance. AsyncAPI is the OpenAPI for events that can form the basis of the required self-governing, self-service eventing framework.
This session will introduce a self-governing framework using AsyncAPI and share how the Bank of New Zealand applied this framework to leverage a passionate Kafka community and embed event-driven thinking. You’ll leave with a tangible set of ideas to give your own events a bit more swagger using AsyncAPI.
What's hot (20)
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Stream me to the Cloud (and back) with Confluent & MongoDB
Stream me to the Cloud (and back) with Confluent & MongoDB
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
Real time data processing and model inferncing platform with Kafka streams (N...
Real time data processing and model inferncing platform with Kafka streams (N...
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply
Kafka summit SF 2019 - the art of the event-streaming app
Kafka summit SF 2019 - the art of the event-streaming app
dotScale 2017 Keynote: The Rise of Real Time by Neha Narkhede
dotScale 2017 Keynote: The Rise of Real Time by Neha Narkhede
Redis and Kafka - Advanced Microservices Design Patterns Simplified
Redis and Kafka - Advanced Microservices Design Patterns Simplified
Kakfa summit london 2019 - the art of the event-streaming app
Kakfa summit london 2019 - the art of the event-streaming app
New Approaches for Fraud Detection on Apache Kafka and KSQL
New Approaches for Fraud Detection on Apache Kafka and KSQL
GCP for Apache Kafka® Users: Stream Ingestion and Processing
GCP for Apache Kafka® Users: Stream Ingestion and Processing
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Similar to Events Everywhere: Enabling Digital Transformation in the Public Sector
Now You See Me, Now You Compute: Building Event-Driven Architectures with Apa...
Talk URL: https://conferences.oreilly.com/strata/strata-ny/public/schedule/detail/77360
Abstract: Would you cross the street with traffic information that’s a minute old? Certainly not. Modern businesses have the same needs nowadays, whether it’s due to competitive pressure or because their customers have much higher expectations of how they want to interact with a product or service. At the heart of this movement are events: in today’s digital age, events are everywhere. Every digital action—across online purchases to ride-sharing requests to bank deposits—creates a set of events around transaction amount, transaction time, user location, account balance, and much more. The technology that allows businesses to read, write, store, and compute and process these events in real-time are event-streaming platforms, and tens of thousands of companies like Netflix, Audi, PayPal, Airbnb, Uber, and Pinterest have picked Apache Kafka as the de facto choice to implement event-driven architectures and reshape their industries.
Michael Noll explores why and how you can use Apache Kafka and its growing ecosystem to build event-driven architectures that are elastic, scalable, robust, and fault tolerant, whether it’s on-premises, in the cloud, on bare metal machines, or in Kubernetes with Docker containers. Specifically, you’ll look at Kafka as the storage and publish and subscribe layer; Kafka’s Connect framework for integrating external data systems such as MySQL, Elastic, or S3 with Kafka; and Kafka’s Streams API and KSQL as the compute layer to implement event-driven applications and microservices in Java and Scala and streaming SQL, respectively, that process the events flowing through Kafka in real time. Michael provides an overview of the most relevant functionality, both current and upcoming, and shares best practices and typical use cases so you can tie it all together for your own needs.
Streaming real time data with Vibe Data Stream
The process of streaming real-time data from a wide variety of machine data sources and entities can be very complex and unwieldy. Using an agent-based approach, Informatica has invented a new technique and open access product that makes this process much more user friendly and efficient, even when dealing with multiple environments such as Hadoop, Cassandra, Storm, Amazon Kinesis and Complex Event Processing.
The Zero-ETL Approach: Enhancing Data Agility and Insight
In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method.
We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy.
Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations.
Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today.
Join us to discover how Zero-ETL can elevate your organization's data strategy.
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).
CONNECTING THE REAL WORLD TO ENTERPRISE IT – HOW IoT DRIVES OUR ENERGY TRANSI...
Enterprise IT systems are deaf, blind and highly insensitive. They do not know what is going on in the outside world. Through Internet of Things technology, we provide eyes, ears and hands that allow enterprises to learn about and react in real time to events in the physical world. The energy transition at a major Dutch energy company (Eneco) is powered by IoT technology – to steer and sometimes curtail windmills and solar farms and to coordinate local energy production and trade. This session shows you how the physical world was connected to the customer portals and apps, asset management systems and Kafka platform through the Azure cloud based IoT Hub en Edge, digital twin, serverless functions, timeseries datastores and streaming data analysis. It is a story about technological innovation on top of existing foundations and of a vision for business and our society at large.
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Für die Automobilindustrie ist die digitale Transformation wie für jede andere Branche zugleich eine digitale Revolution: Neue Marktspieler, neue Technologien und die in immer größeren Mengen anfallenden Daten schaffen neue Chancen, aber auch neue Herausforderungen – und erfordern neben neuen IT-Architekturen auch völlig neue Denkansätze.
60% der Fortune500-Unternehmen setzen zur Umsetzung ihrer Daten-Streaming-Projekte auf die umfassende verteilte Streaming-Plattform Apache Kafka®, darunter auch die AUDI AG.
Erfahren Sie in diesem Webinar:
Wie Kafka als Grundlage sowohl für Daten-Pipelines als auch für Anwendungen dient, die Echtzeit-Datenströme konsumieren und verarbeiten.
Wie Kafka Connect und Kafka Streams geschäftskritische Anwendungen unterstützt
Wie Audi mithilfe von Kafka und Confluent eine Fast Data IoT-Plattform umgesetzt hat, die den Bereich „Connected Car“ revolutioniert
Sprecher:
David Schmitz, Principal Architect, Audi Electronics Venture GmbH
Kai Waehner, Technology Evangelist, Confluent
Building an Enterprise Eventing Framework
Watch this talk here: https://www.confluent.io/online-talks/building-an-enterprise-eventing-framework-on-demand
Learn how Centene improved their ability to interact and engage with healthcare providers in real time with MongoDB and Confluent Platform.
Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of their architecture is the development of a universal eventing framework designed to accommodate transformation into an event-driven architecture (EDA).
The business requirements within Centene's claims adjudication domain were solved leveraging the Kafka Stream DSL, Confluent Platform and MongoDB. Most importantly, Centene discusses how they plan on leveraging this framework to change their culture from batch processing to real-time stream processing.
Confluent Partner Tech Talk with BearingPoint
This document discusses best practices for debugging client applications in Kafka streams. It begins by asking a question about debugging practices for producers, consumers, and Kafka streams applications. It then describes a Partner Technical Sales Enablement offering that includes live sessions and on-demand learning paths on topics like Confluent fundamentals and use cases. It outlines additional support for partners through technical workshops, coaching, and solution discovery sessions. The document concludes by stating the goal of Partner Tech Talks is to provide insights and inspiration through use case discussions.
Hadoop Summit 2016 - Evolution of Big Data Pipelines At Intuit
This document summarizes the evolution of Intuit's big data pipelines over time. It discusses how the pipelines started as disparate and moved to a single cohesive pipeline. The stages of evolution included moving to real-time processing, high availability with mirroring, and integrating streaming and batch processing. Key components of the modern pipeline include Apache Kafka for event collection, real-time processing engines, and data storage in Hive. Testing ensures the pipeline meets throughput, latency, and data integrity requirements as the volumes and complexity increase.
Spark meetup stream processing use cases
Stream processing is key for enabling real-time insights from big data. It allows continuous processing of data as it flows through a system, enabling instantaneous actions via alerts. This is unlike traditional batch processing where data is analyzed only after it has been stored. Stream processing is becoming more important as organizations need faster decision making and new data sources like sensors produce high volumes of data quickly. It has broad applications across industries like financial services, telecom, online advertising, and IoT. Example use cases include fraud detection, dynamic bidding, infrastructure management, and predictive maintenance. Open source technologies now make stream processing more feasible at large scale for real-time analytics.
The Evolution of Big Data Pipelines at Intuit
The document summarizes the evolution of Intuit's big data pipelines over time from disparate and chaotic early stages to their current integrated cloud-based architecture. It describes how Intuit transitioned from siloed data storage to a single cohesive data pipeline using Apache Kafka and real-time processing. It outlines the key components of their current big data pipeline including real-time data collection, processing, profile storage, and monitoring systems and how this pipeline supports use cases like personalization, fraud detection and more.
Real time data integration best practices and architecture
This document discusses best practices and architectures for real-time data integration. It outlines how traditional integration approaches are no longer sufficient due to business demands for more timely, accurate information. Real-time integration can reduce decision latency and improve responsiveness. The document describes different real-time integration patterns like transactional data processing, data replication, and event-driven architectures. It provides examples of how to implement real-time integration through a data integration hub and event processing. The key benefits of real-time integration are also summarized.
2016 DSG Webinar Azure HDInsight 2 V4
This document discusses how to implement real-time analytics on big data using Microsoft Azure services. It begins with an introduction and overview of real-time analytics. It then discusses how real-time fraud detection in the telecom industry can help reduce losses. The presentation demonstrates using Azure Event Hubs to capture streaming call record data, Spark Streaming to process it, Hive for storage, and Power BI for visualization and real-time alerts. It provides an architecture diagram and steps for setting up such a real-time fraud detection system.
2016 DSG Webinar Azure HDInsight 2 V4
This document discusses how to implement real-time analytics on big data using Microsoft Azure services. It begins with an introduction and overview of real-time analytics. It then discusses how real-time fraud detection in the telecom industry can help reduce losses. The presentation demonstrates using Azure Event Hubs to capture streaming call record data, Spark Streaming to process it, storing the results in Hive, and using Power BI for visualization and real-time alerts. It provides an architecture diagram and steps for setting up such a real-time fraud detection system.
The Zero-ETL Approach: Enhancing Data Agility and Insight
In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method.
We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy.
Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations.
Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today.
Join us to discover how Zero-ETL can elevate your organization's data strategy.
Splunk Discovery Day Hamburg - Data Driven Insights
This document discusses how Splunk provides data-driven insights into IT operations to support digital transformation. It notes that digital services are causing an explosion in machine-generated data and that insights are needed to keep up. The customer experience is more digital and tied to the digital journey across websites, apps, and devices. Splunk collects machine data from all sources and provides rapid search, correlation, visualization, alerts, and machine learning to gain operational insights that were previously difficult to obtain. It is used by many industries and roles to monitor apps, networks, security, and more. Over 13,000 customers use Splunk in over 110 countries.
Building Reactive Real-time Data Pipeline
Topic: Building reactive real-time data pipeline at FPT ?
1) What is “Data Pipeline” ?
2) Big Data Problems at FPT
+ VnExpress: pageview and heat-map
+ eClick: real-time reactive advertising
3) Solutions and Patterns
4) Fast Data Architecture at FPT
5) Wrap up
Digital Business Transformation in the Streaming Era
Enterprises are rapidly adopting stream computing backbones, in-memory data stores, change data capture, and other low-latency approaches for end-to-end applications. As businesses modernize their data architectures over the next several years, they will begin to evolve toward all-streaming architectures. In this webcast, Wikibon, Attunity, and MemSQL will discuss how enterprise data professionals should migrate their legacy architectures in this direction. They will provide guidance for migrating data lakes, data warehouses, data governance, and transactional databases to support all-streaming architectures for complex cloud and edge applications. They will discuss how this new architecture will drive enterprise strategies for operationalizing artificial intelligence, mobile computing, the Internet of Things, and cloud-native microservices.
Link to the Wikibon report - wikibon.com/wikibons-2018-big-data-analytics-trends-forecast
Link to Attunity Streaming CDC Book Download - http://www.bit.ly/cdcbook
Link to MemSQL's Free Data Pipeline Book - http://go.memsql.com/oreilly-data-pipelines
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
This document summarizes a presentation about a new real-time stream processing solution implemented by GetInData for telecommunications company Kcell. The new solution processes over 160,000 events per second from 10 million subscribers using Apache Flink on Apache Hadoop for reliability and scalability. It provides real-time insights for use cases like detecting fraudulent roaming or offering targeted promotions. The implementation leverages open source technologies like Apache Kafka, HDFS, and Druid to cost-effectively meet Kcell's growing analytics needs. Future work includes adding more data sources and machine learning to further enhance the platform.
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
http://flink-forward.org/kb_sessions/flink-in-zalandos-world-of-microservices/
In this talk we present Zalando’s microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing with Apache Flink for near-real time business intelligence.
Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization.
We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach – Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams.
We no longer live in a world of static data sets, but are instead confronted with endless streams of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch.
With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL.
Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements.
On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability.
Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.
Similar to Events Everywhere: Enabling Digital Transformation in the Public Sector (20)
Now You See Me, Now You Compute: Building Event-Driven Architectures with Apa...
Now You See Me, Now You Compute: Building Event-Driven Architectures with Apa...
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
CONNECTING THE REAL WORLD TO ENTERPRISE IT – HOW IoT DRIVES OUR ENERGY TRANSI...
CONNECTING THE REAL WORLD TO ENTERPRISE IT – HOW IoT DRIVES OUR ENERGY TRANSI...
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Hadoop Summit 2016 - Evolution of Big Data Pipelines At Intuit
Hadoop Summit 2016 - Evolution of Big Data Pipelines At Intuit
Real time data integration best practices and architecture
Real time data integration best practices and architecture
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
Splunk Discovery Day Hamburg - Data Driven Insights
Splunk Discovery Day Hamburg - Data Driven Insights
Digital Business Transformation in the Streaming Era
Digital Business Transformation in the Streaming Era
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
Assisting millions of active users in real-time - Alexey Brodovshuk, Kcell; K...
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
More from confluent
Building API data products on top of your real-time data infrastructure
This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products.
You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering.
You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks.
Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the
Real-time Streaming and AI Era
Andrew Foo
Customer Solutions @ Confluent
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Santander Stream Processing with Apache Flink
Flink is becoming the de facto standard for stream processing due to its scalability, performance, fault tolerance, and language flexibility. It supports stream processing, batch processing, and analytics through one unified system. Developers choose Flink for its robust feature set and ability to handle stream processing workloads at large scales efficiently.
Unlocking the Power of IoT: A comprehensive approach to real-time insights
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flink
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
AWS Immersion Day Mapfre - Confluent
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Eventos y Microservicios - Santander TechTalk
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.
Citi TechTalk Session 2: Kafka Deep Dive
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Build real-time streaming data pipelines to AWS with Confluent
Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.
Q&A with Confluent Professional Services: Confluent Service Mesh
No matter whether you are migrating your Kafka cluster to Confluent Cloud, running a cloud-hybrid environment or are in a different situation where data protection and encryption of sensitive information is required, Confluent Service Mesh allows you to transparently encrypt your data without the need to make code changes to you existing applications.
Citi Tech Talk: Event Driven Kafka Microservices
Microservices have become a dominant architectural paradigm for building systems in the enterprise, but they are not without their tradeoffs. Learn how to build event-driven microservices with Apache Kafka
Confluent & GSI Webinars series - Session 3
An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities.
It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks.
This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.
Citi Tech Talk: Messaging Modernization
This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.
Citi Tech Talk: Data Governance for streaming and real time data
Getting to grips with governance on realtime data: Data Discovery, Data Catalogues, Schema Management, Lineage
Confluent & GSI Webinars series: Session 2
This session will show why the old paradigm does not work and that a new approach to the data strategy needs to be taken. It aims to show how a Data Streaming Platform is integral to the evolution of a company’s data strategy and how Confluent is not just an integration layer but the central nervous system for an organisation
Data In Motion Paris 2023
Vous apprendrez également à :
• Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données
• Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience
• Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés
More from confluent (20)
Building API data products on top of your real-time data infrastructure
Building API data products on top of your real-time data infrastructure
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
Recently uploaded
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
What began over 115 years ago as a supplier of precision gauges to the automotive industry has evolved into being an industry leader in the manufacture of product branding, automotive cockpit trim and decorative appliance trim. Value-added services include in-house Design, Engineering, Program Management, Test Lab and Tool Shops.
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Java is for many years one of the most popular programming languages, but it used to have hard times in the Serverless community. Java is known for its high cold start times and high memory footprint, comparing to other programming languages like Node.js and Python. In this talk I'll look at the general best practices and techniques we can use to decrease memory consumption, cold start times for Java Serverless development on AWS including GraalVM (Native Image) and AWS own offering SnapStart based on Firecracker microVM snapshot and restore and CRaC (Coordinated Restore at Checkpoint) runtime hooks. I'll also provide a lot of benchmarking on Lambda functions trying out various deployment package sizes, Lambda memory settings, Java compilation options and HTTP (a)synchronous clients and measure their impact on cold and warm start times.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable.
In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed:
Key Takeaways:
Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement.
Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers.
Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
From Natural Language to Structured Solr Queries using LLMs
This talk draws on experimentation to enable AI applications with Solr. One important use case is to use AI for better accessibility and discoverability of the data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints.
That is where AI – and most importantly, Natural Language Processing and Large Language Model techniques – could make a difference. This natural language, conversational engine could facilitate access and usage of the data leveraging the semantics of any data source.
The objective of the presentation is to propose a technical approach and a way forward to achieve this goal.
The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries based on the Solr index’s metadata.
This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents within Apache Solr.
The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook. The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application.
In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy.
We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.
"What does it really mean for your system to be available, or how to define w...
We will talk about system monitoring from a few different angles. We will start by covering the basics, then discuss SLOs, how to define them, and why understanding the business well is crucial for success in this exercise.
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Principle of conventional tomography-Bibash Shahi ppt..pptx
before the computed tomography, it had been widely used.
Recently uploaded (20)
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Events Everywhere: Enabling Digital Transformation in the Public Sector
- 1. Events Everywhere Enabling Digital Transformation in the Public Sector
- 2. 2
- 3. What’s the big idea?
- 5. I walk into a room, flip the light switch, and the light turns on This is a Command
- 6. I walk into a room, generate an “entered room” event and the light turns on This is a reaction to an event
- 7. It is a fundamental paradigm shift of how applications are developed
- 9. Aren’t new… …but… An Architecture for Well-Conditioned, Scalable Internet Services - white paper Focusing on Events blog post - Martin Fowler 2001 2006 Your coffee shop doesn’t use two-phase commit - Gregor Hohpe 2005
- 10. The world is changing.
- 11. What is an Event? SOMETHING HAPPENED!
- 13. Business Digitization Trends are Revolutionizing your Data Flow
- 14. How did we get here?
- 16. Pre-Streaming
- 19. The Rise of Event Streaming 99% 19 1% - Non-Transactional “Event Data” - Transactional “Database State”
- 21. Event driven apps look like this
- 22. A series of related events ..is a stream..
- 23. A stream captures behavior
- 24. The events are the data… persisted, in order, in Kafka
- 25. In a log like a commit-log in a database
- 26. Streaming Correctness delivery guarantees, timestamp, temporal consistency, scale model
- 27. Streams?
- 28. Driven by Events • • • • {Survey:100} {IVR:150} {Twitter:190} Stream processor Data driven logic goes here! {Sat_Score:90} {Sat_Score:60}{Sat_Score:40}
- 29. Using external processing systems leads to complicated architectures DB CONNECTOR APP APP DB STREAM PROCESSING APPDB CONNECTOR CONNECTOR
- 30. We can put it back together in a simpler way DB APP APP DB APP PULL PUSH CONNECTORS STREAM PROCESSING STATE STORES ksqlDB
- 31. CREATE STREAM unhappy_users AS SELECT * FROM survey WHERE sat_score < 40; With ksqlDB - Not a single line of Java or Scala code needed. Stream processing with Kafka
- 32. Evolvability user experience? how many users affected? has it happened before? new old supports data change, logic change, logic extension, schema evolution, loose coupling, add processors, A/B path
- 33. Case Study: Life as a Stream of Events https://www.confluent.io/online-talks/how-apache-kafka-helps-to-create-data-culture/ https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/ Parent information Healthcare provider Child benefits Child birth registration Parental leave
- 34. OUTCOME: Build Apps that Deliver Value To our customers and our business (Often, this involves data pipelines)
- 35. Apache Kafka Connect API: Import and Export Data In & Out of Kafka Kafka Connect API Kafka Pipeline Sources Sinks
- 36. Instantly Connect Popular Data Sources & Sinks Data Diode 100+ pre-built connectors 80+ Confluent Supported 20+ Partner Supported, Confluent Verified
- 37. Patterns that start to emerge ● Data pipelines (ETL, integration) ● Monitoring and alerting, including log collection and aggregation ● Event-driven microservices ● Enterprise wide event-driven microservices ● IoT ● Customer 360 ● Central nervous system/digital nervous system
- 38. Change Data Capture (CDC) • Table/Row updates detected from commit log { eventKey: user-100 username: felix address: blah membership: 1234 } table: users connector topic: users
- 39. Internet of Things • MQTT • Device centric { eventKey: device-100 lat: 144.123 long: 120.133 value: 0.0007 } device: geo-sensor MQTT topic: geo-sensors
- 40. Microservices • Events • Transactional • Plus some magic with local-state { eventKey: user-100 username: felix address: london membership: 123 }user-enrichment topic: enriched-users Stream processor
- 41. New Streaming-first Paradigm • Scalable • Real-time • Stream processing Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Contextual Event-Driven Applications Universal Event Pipeline Data Stores Logs 3rd Party Apps Custom Apps/Microservices EVENT STREAMING PLATFORM ksqlDB
- 42. Every Industry is Moving from Batch/Manual to Software-Defined Auto / Transport Monolithic Applications Software-defined Spreadsheet-driven driver schedule Real-time ETA Banking Nightly credit-card fraud checks Real-time credit card fraud prevention Retail Batch inventory updates Real-time inventory management Healthcare Batch claims processing Real-time claims processing Oil and Gas Batch analytics Real-time analytics Manufacturing Scheduled equipment maintenance Automated, predictive maintenance Cybersecurity Reactive cyber-security forensics Automated SIEM and Anomaly Detection U.S. Defense Agencies 42
- 43. How ‘Events Everywhere’ can Enable Digital Transformation in the Public Sector!
- 44. Free eBooks Kafka: The Definitive Guide Neha Narkhede, Gwen Shapira, Todd Palino Making Sense of Stream Processing Martin Kleppmann I ❤ Logs Jay Kreps Designing Event-Driven Systems Ben Stopford http://cnfl.io/book-bundle
- 45. Interacting with Confluent 45 Join the Confluent Slack Channel https://launchpass.com/confluentcommunity Local meetups https://www.confluent.io/community/ Try a hands-on demo https://github.com/confluentinc/examples Developer resources https://developer.confluent.io/
- 46. Project Metamorphosis Unveiling the next-gen event streaming platform Listen to replay and Sign up for updates cnfl.io/pm Jay Kreps Co-founder and CEO Confluent
- 47. Thank you! More talks at www.confluent.io/events/ Kafka Summit www.kafka-summit.org