Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020

•

0 likes•3,269 views

Are you a Kafka Streams application developer who needs a faster, more efficient way to reproduce a bug or issue from past events? Do you need to test a new algorithm patch near a discrete point-in-time? Do you have a standard methodology for investigating these issues, or does each team member devise an ad hoc solution? How much time does your simulation take? It is technically impossible to solve this challenge using only changelog topics, since a compact, topic-based changelog doesn't capture the full change history. The Derivatives Data team at Bloomberg augments this through periodic snapshots. To make the snapshotted state accessible to our replay system, we built a query service that leverages Interactive Queries using the Kafka Streams API, along with gRPC-based coordination, to fetch the distributed snapshot states from different snapshotting instances. In this talk, we will cover: ● Overview of this system architecture ● Deep dive into the mechanics of snapshots with Kafka Streams, state-store changelogs, and a query service to serve replay requests ● How two modes, normal and replay, are used on the Kafka Streams application runtime ● Some use cases that benefit from this replay system

Technology

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Replaying KStreams Apps
Using State Snapshots
Kafka Summit 2020
August 25, 2020
Nish Sinha, Software Engineer
Yan Wang, Software Engineer

© 2018 Bloomberg Finance L.P. All rights reserved.
transformfilter
State
transform
Market data
(bid/ask/trade)
Kafka Streams Processors
Algorithm-
generated price
State
filter
LOGGING + METRICS + ALARMS
Kafka Streams Stateful Pipelines

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Motivation: Real Scenarios
• Debug a bad output that appeared on last Thursday @ 2:30 PM
• Backtest a new model
• Backfill by correcting bad data

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Logging Your Problem Away

© 2018 Bloomberg Finance L.P. All rights reserved.
Another Way: Play The Point In Time From Start

© 2018 Bloomberg Finance L.P. All rights reserved.
Yet Another Way: Play The Point In Time From Nearby Snapshots

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Wish list
• Make Real Replay (align with the states) achievable
• Minimal load on Kafka clusters
• Minimal impact on production system
• Whole Nine Yards: Replay the application itself

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Design Considerations
• Snapshotting vs. continuous data capture
• How much historical time?
• Scale out?

© 2018 Bloomberg Finance L.P. All rights reserved.
External Non-Intrusive Snapshots

© 2018 Bloomberg Finance L.P. All rights reserved.
System Overview

© 2018 Bloomberg Finance L.P. All rights reserved.
Snapshotter

© 2018 Bloomberg Finance L.P. All rights reserved.
Storage Layer

© 2018 Bloomberg Finance L.P. All rights reserved.
Query Snapshots

© 2018 Bloomberg Finance L.P. All rights reserved.
State Orchestrator

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
State Exposer

© 2018 Bloomberg Finance L.P. All rights reserved.
Align Data with States

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Summary

What's hot

Using FLaNK with InfluxDB for EdgeAI IoT at Scale Timothy from StreamNative take you on a hands-on deep-dive on using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team run deep learning models on the edge devices and send images, and capture real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream real-time to their data space. These edge applications classify images and sensor readings real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

InfluxData

Kafka and Kafka Streams in the Global Schibsted Data Platform

Fredrik Vraalsen

One challenge of widespread adoption of any technology within an organisation is balancing organic growth and maintaining standards and best practice. At AO.com - one of the UK's largest online electrical retailers - we’ve invested in tooling to simplify application onboarding into the event processing platform. This includes creating topics, defining access to the platform and supporting our governance functions. A key part of any data function is data governance and discoverability. Through standardised definitions of Kafka topics and use of Avro schemas, we can map which topics exist, what data they contain and who has access to them. This allows us to support multiple cross-functional teams using automatic data gathering. In this session, AO.com and Confluent Professional Services will share how we tackled the challenge of platform adoption and provide hands-on examples of the open-source tools, ""Kafka Clusterstate Tools"" and ""Kafka Streams Inspector"", we developed.

Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...

HostedbyConfluent

Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021

InfluxData

(Bob Lehmann, Bayer) Kafka Summit SF 2018 You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why? In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform. In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

Introduction to networking options available in Confluent Cloud Self Serve provisioning of confluent Kafka clusters. VPC Peering, VNet Peering, Transit Gateway and Private Link Options for AWS, GCP, Azure networking offering. Caveats of confluent's cloud networking solutions customers should be aware of. Details of two major pieces of the architecture of Confluent Cloud - Data Plane Network and Control Plane.

Confluent Cloud Networking | Rajan Sundaram, Confluent

HostedbyConfluent

Stream Processing as helped to turn many monolithic database-centric applications into fast, scalable, and flexible real time applications. However, there are still entire classes of applications that are built against databases, because today's streaming processing model is not yet rich enough to support these applications. We present what we believe is still missing in today's stream processing models, and how we envision to evolve stream processing for the next wave of application to be able to move to a stream processing architecture.

Flink Forward Berlin 2018: Stephan Ewen - Keynote: "Unlocking the next wave o...

Flink Forward

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.

Comparing three data ingestion approaches where Apache Kafka integrates with ...

HostedbyConfluent

Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

InfluxData

Nilden Tutular, Volkan Balikci, Uygar Zubari [Eldor Corporation] | MQTT - Mac...

InfluxData

Ana-Maria Calin [InfluxData] | Migrating from OSS to InfluxDB Cloud | InfluxD...

InfluxData

Many enterprises faced with silo’ed, batch-oriented, legacy systems struggle to compete in this new digital-first world. Adhering to the ‘If it’s not broken don’t fix it’ mentality leaves the door wide open for native digital challengers to grow and succeed. To stay competitive, your organization must respond in real time to every customer experience transaction, sale, and market movement. But how do you get there? First, you must change your mindset. As streaming platforms become central to data strategies, companies both small and large are re-thinking their enterprise architecture with real-time context at the forefront. Monoliths are evolving into microservices. Datacenters are moving to the cloud. What was once a ‘batch’ mindset is quickly being replaced with stream processing as the demands of the business impose real-time requirements on technology leaders. Join Argyle, in partnership with Confluent, in our 2018 CIO Virtual Event: The Digital Transformation Mindset – More Than Just Technology. During the webinar we’ll learn how leading companies across industries rely on a streaming platform to make event-driven architectures central to: • How data strategies and IT initiatives are improving the digital customer experiences • How executives are reducing risk with real time monitoring and anomaly detection • Increasing operational agility with microservices and IoT architectures within organizations

Digital Transformation Mindset - More Than Just Technology

confluent

Building a Streaming Data Pipeline for Trains Delays Processing

Databricks

Sam Dillard [InfluxData] | Performance Optimization in InfluxDB | InfluxDays...

InfluxData

Kapacitor Stream Processing

InfluxData

TeleHealth Platform: DevOps-Based Progressive Delivery The talk covers a real-life experience related to building a DevOps Delivery-powered AI platform for doctors’ community and telehealth support for patients during COVID-19 lockdown. The doctors’ community interacts related to cases and triage for different patient cases. They can extend telehealth support using medical practice management solutions. Patients can order medicines online through integrated pharmacies on the platform. AI Platform has digital, voice, and knowledge assistants to provide information to the doctor. DevOps is enabled using Jenkins on AWS which helps in continuous integration and progressive delivery of features to Mobile and web apps (Apple & Google app stores). Historical data is used for predictive analytics by the machine learning platform. The platform helps healthcare enterprises: 1. Deploy voice tech to facilitate clinical documentation 2. Reduce physicians’ administrative burden 3. Increase patient volume and billable revenue 4. Eliminate transcription costs 5. Use voice to increase touchpoints and increase patient engagement.

Bhagvan Kommadi [Value Momentum] | TeleHealth Platform: DevOps-Based Progress...

InfluxData

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

InfluxData

Speaker: Nauman Hafiz Level: 100 (Beginner) Track: Atlas The R/GA Connected Work platform enables a new level of operational management and interactivity across our global offices. The platform is a global, data-driven, api-first service that powers fluid web content, dynamic signage, native mobile applications, IoT devices, and chatbots. The platform collects data in real-time, integrates with a variety of services, and provides a rich ecosystem of tools - such as the ability to quickly find the nearest available meeting room, discover the office art, or explore our rich portfolio of work and content. In this session we will discuss our original move to MongoDB, the operations and tooling required to manage the AWS infrastructure, and how we plan on benefiting from Atlas. What You Will Learn: - Considerations for self-hosting MongoDB infrastructure - Key opportunities for performance and cost optimization - The benefits of moving to MongoDB Atlas

MongoDB and the Future of Workspaces

MongoDB

In this talk we walk you through our operational journey for building and operating large-scale production streaming applications at King. Our Flink streaming jobs consume over 40 billion events every day maintaining over 10 TBs of user state with Flink's strong consistency guarantees. We will focus on the evolution of our in-house stream processing platform, and the different challenges we have faced at different times in it's 1.5 years of production lifeline. Topics we'll cover: - Overview of our platform - Cluster environment, and deployment procedures - Flink job management (configuration, versioning, etc.) - Monitoring and debugging practices - Some Flink features that we find super amazing.

Flink Forward Berlin 2017: Gyula Fora - Building and operating large-scale st...

Flink Forward

Running Consul on Kubernetes and Beyond

Mitchell Pronschinske

What's hot (20)

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

Kafka and Kafka Streams in the Global Schibsted Data Platform

Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...

Alex Nauda [Nobl9] | How Not to Build an SLO Platform | InfluxDays NA 2021

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

Confluent Cloud Networking | Rajan Sundaram, Confluent

Flink Forward Berlin 2018: Stephan Ewen - Keynote: "Unlocking the next wave o...

Comparing three data ingestion approaches where Apache Kafka integrates with ...

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

Nilden Tutular, Volkan Balikci, Uygar Zubari [Eldor Corporation] | MQTT - Mac...

Ana-Maria Calin [InfluxData] | Migrating from OSS to InfluxDB Cloud | InfluxD...

Digital Transformation Mindset - More Than Just Technology

Building a Streaming Data Pipeline for Trains Delays Processing

Sam Dillard [InfluxData] | Performance Optimization in InfluxDB | InfluxDays...

Kapacitor Stream Processing

Bhagvan Kommadi [Value Momentum] | TeleHealth Platform: DevOps-Based Progress...

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

MongoDB and the Future of Workspaces

Flink Forward Berlin 2017: Gyula Fora - Building and operating large-scale st...

Running Consul on Kubernetes and Beyond

Similar to Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020

(Lei Chen, Bloomberg, L.P.) Kafka Summit SF 2018 At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing. We’ll cover: -Our system architecture -Best practices of using the Processor API and State Store API -Dynamic gap session implementation -Historical data re-processing practice in KStreams app -Chaining multiple KStreams apps with Spark Streaming job

Real-Time Market Data Analytics Using Kafka Streams

confluent

Multi-Tenant HBase Cluster - HBaseCon2018-final

Biju Nair

HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies

Michael Stack

Serving queries at low latency using HBase

Biju Nair

VOGIN-IP-lezing-edgar_meij

voginip

Users of financial data require their queries to return results with very low latency. As a financial data service provider, Bloomberg needs to consistently meet these requirements for our clients. HBase promises millisecond latency, auto-sharding, and an open schema. However, as HBase is a NoSQL database, support for transaction processing is not trivial. This talk will discuss a data modelling technique and use case where effective transaction processing can be achieved. We will also discuss how this data model helps us achieve real-time streaming, scalability, and millisecond read-write latency for billions of queries each day.

How data modelling helps serve billions of queries in millisecond latency wit...

DataWorks Summit

Market Intelligence Briefing: The Department of Defense FY17 Budget

Tania Anderson

Speaker: Clive Charlton, AWS Level: 300 Day zero was the term coined by the City of Cape Town for the day they would have to turn the water off for citizens, potentially making Cape Town the first major city to run out of water. Thankfully day zero was avoided in 2018 but the danger still lingers. Learn how AWS empowers citizens to use water responsibly using AWS IoT, AWS Lambda, Amazon DynamoDB, Amazon S3, Amazon Cognito.

AWS IoT: enabling responsible water use - AWS Summit Cape Town 2018

Amazon Web Services

Law enforcement agencies have to do more with less, yet the challenges of public safety are becoming more complex. Despite funding hurdles, several agencies are finding success tapping into technologies like predictive policing and data analytics to do their jobs. Tune in to Market Intelligence SLED consultant, Rachel Eckert, as she explores how law enforcement is using technology to stretch finite resources and offers insight into how companies should target law enforcement, what programs agencies are undertaking and what solutions are in demand.

Law Enforcement Needs a Boost--Here's How Technology Can Help

immixGroup

“There are only two hard things in Computer Science: cache invalidation and naming things.” — Phil Karlton Today’s microservice-based architectures rely on caches to hide latency and provide more responsive experiences for users. Unfortunately, these caches can be tricky to get right. Who places the data in the cache? When should the cached data expire? What happens when the cache or its datastore fail? We found that by using an event-based push model, we could avoid most of the pitfalls associated with traditional caches. This talk will cover the basic concept of push-based caches and their implications. It will delve into how you might build such a cache and what to do when your dataset is large, as well as look at our experience using these kinds of caches in production at Bloomberg.

The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit...

confluent

Amazon Elasticsearch Service (Amazon ES) is a fully managed service that enables you to deploy, secure, and operate Amazon ES at scale with zero down time. Discover best practices to get the most out of search applications. Hear highlights from how Municipal Securities Rulemaking Board (MSRB) used Amazon ES to modernize their analytics platform. The MSRB engaged Hitachi Vantara to develop a prototype data analytics platform that enabled natural language and free-text search. Join us to learn more about how to leverage the cloud to generate tremendous and transformative value. Speaker: Santosh Karla, Principal Cloud Architect, Hitachi Vantara

Taking a Dip in the Muni Bond Data Lake

Amazon Web Services

TipoTapp For The Customer

Paul Armstrong

Site | https://www.infoq.com/qconai2018/ Youtube | https://www.youtube.com/watch?v=2h0biIli2F4&t=19s At PayPal, data engineers, analysts and data scientists work with a variety of datasources (Messaging, NoSQL, RDBMS, Documents, TSDB), compute engines (Spark, Flink, Beam, Hive), languages (Scala, Python, SQL) and execution models (stream, batch, interactive). Due to this complex matrix of technologies and thousands of datasets, engineers spend considerable time learning about different data sources, formats, programming models, APIs, optimizations, etc. which impacts time-to-market (TTM). To solve this problem and to make product development more effective, PayPal Data Platform developed "Gimel", a unified analytics data platform which provides access to any storage through a single unified data API and SQL, that are powered by a centralized data catalog. In this session, we will introduce you to the various components of Gimel - Compute Platform, Data API, PCatalog, GSQL and Notebooks. We will provide a demo depicting how Gimel reduces TTM by helping our engineers write a single line of code to access any storage without knowing the complexity behind the scenes.

QCon 2018 | Gimel | PayPal's Analytic Platform

Deepak Chandramouli

Meetup 18/10/2018 - Artificiële intelligentie en mobiliteit

Digipolis Antwerpen

Next Generation of Treasury Technology Cash Adventure- Brad Teaver.pptx

Nitin790564

On this week’s episode of The Internet Report, Archana and I cover some newsworthy updates that we’ve seen over the past week. We discuss a notable Facebook SDK outage that had ripple effects on other popular services that leverage its log-in functionality, including Spotify and Tik Tok. We also discuss a blog from AWS sharing their thoughts on the JEDI contract. We’re also joined by Arash Molavi, the lead Internet researcher here at ThousandEyes. Arash shares his insight into outages we’re seeing, discusses what constitutes an outage, and why loss, latency and jitter can impact end-user experience in various ways depending on the context. Last, we cover our usual availability check of ISP, public cloud, and collaboration app provider networks.

The Internet Report: Episode 7 Week of May 4 - May 10

ThousandEyes

Compliance departments within banks and other financial institutions are turning to machine learning for improving their Anti Money Laundering compliance activities. Today, the systems that aim to detect potentially suspicious activity are commonly rule-based, and suffer from ultra-high false positive rates. DataRobot will discuss how their Automated Machine Learning platform was successfully used for a real use case to reduce their false positives and to enhance their Anti-Money Laundering activities.

Data Natives meets DataRobot | "Build and deploy an anti-money laundering mo...

Dataconomy Media

Building csm while going from on premise to saa s

Jessica Osborn

Rok overview 2018 may

RockwellAutomationIR

Rok overview 2018 may

RockwellAutomationIR

Similar to Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020 (20)

Real-Time Market Data Analytics Using Kafka Streams

Multi-Tenant HBase Cluster - HBaseCon2018-final

HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies

Serving queries at low latency using HBase

VOGIN-IP-lezing-edgar_meij

How data modelling helps serve billions of queries in millisecond latency wit...

Market Intelligence Briefing: The Department of Defense FY17 Budget

AWS IoT: enabling responsible water use - AWS Summit Cape Town 2018

Law Enforcement Needs a Boost--Here's How Technology Can Help

The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit...

Taking a Dip in the Muni Bond Data Lake

TipoTapp For The Customer

QCon 2018 | Gimel | PayPal's Analytic Platform

Meetup 18/10/2018 - Artificiële intelligentie en mobiliteit

Next Generation of Treasury Technology Cash Adventure- Brad Teaver.pptx

The Internet Report: Episode 7 Week of May 4 - May 10

Data Natives meets DataRobot | "Build and deploy an anti-money laundering mo...

Building csm while going from on premise to saa s

Rok overview 2018 may

More from confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

confluent

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

confluent

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

confluent

Confluent Partner Tech Talk with Synthesis

confluent

The Future of Application Development - API Days - Melbourne 2023

confluent

The Playful Bond Between REST And Data Streams

confluent

More from confluent (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams

Recently uploaded

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

ICT role in 21st century education and its challenges

rafiqahmad00786416

Exploring Multimodal Embeddings with Milvus

Zilliz

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Manulife - Insurer Transformation Award 2024

The Digital Insurer

DBX First Quarter 2024 Investor Presentation

Dropbox

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Recently uploaded (20)

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

ICT role in 21st century education and its challenges

Exploring Multimodal Embeddings with Milvus

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Strategies for Landing an Oracle DBA Job as a Fresher

Cyberprint. Dark Pink Apt Group [EN].pdf

Axa Assurance Maroc - Insurer Innovation Award 2024

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

presentation ICT roal in 21st century education

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Why Teams call analytics are critical to your entire business

[BuildWithAI] Introduction to Gemini.pdf

FWD Group - Insurer Innovation Award 2024

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

AWS Community Day CPH - Three problems of Terraform

Manulife - Insurer Transformation Award 2024

DBX First Quarter 2024 Investor Presentation

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020

1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Replaying KStreams Apps Using State Snapshots Kafka Summit 2020 August 25, 2020 Nish Sinha, Software Engineer Yan Wang, Software Engineer

2. © 2018 Bloomberg Finance L.P. All rights reserved. transformfilter State transform Market data (bid/ask/trade) Kafka Streams Processors Algorithm- generated price State filter LOGGING + METRICS + ALARMS Kafka Streams Stateful Pipelines

3. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Motivation: Real Scenarios • Debug a bad output that appeared on last Thursday @ 2:30 PM • Backtest a new model • Backfill by correcting bad data

7. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Wish list • Make Real Replay (align with the states) achievable • Minimal load on Kafka clusters • Minimal impact on production system • Whole Nine Yards: Replay the application itself

8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Design Considerations • Snapshotting vs. continuous data capture • How much historical time? • Scale out?

18. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Questions? Nish Sinha, nsinha35@bloomberg.net Yan Wang, ywang1445@bloomberg.net

19. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. We are hiring! Thank you! https://www.bloomberg.com/careers

Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020

Similar to Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020 (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Bloomberg) Kafka Summit 2020