Real-Time Market Data Analytics Using Kafka Streams

•

3 likes•11,011 views

(Lei Chen, Bloomberg, L.P.) Kafka Summit SF 2018 At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing. We’ll cover: -Our system architecture -Best practices of using the Processor API and State Store API -Dynamic gap session implementation -Historical data re-processing practice in KStreams app -Chaining multiple KStreams apps with Spark Streaming job

Technology

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Lei Chen
Software Engineer, Team Lead
lchen576@bloomberg.net
Real-time* Market Data Processing
Using Kafka Streams
* Actually, just low-latency

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Agenda
• Use cases and challenges
• Why Kafka Streams
• Deep dive into our implementation
• Some other tips & tricks

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Who we are & What we do
• Derivative market data group in Bloomberg
• Builds market data pipelines
• Apply big data & AI technologies in financial domain

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Streaming use cases
Market movement
(bid/ask/trade)
Composite price
(Bloomberg Generated Market Indicator)
Option price
(Calculate option price using
calculated volatilities)

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Challenges and why Kafka Streams
• Zero data loss
• Ultra-Low latency
• Huge data volume
• Large state
• Corporate DR compliance
• Maintenance
‘Exactly once’ delivery
Super fast
Highly scalable
State store
Fault tolerant
Minimal management overhead
*Data already in Kafka!

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Deep dive into our implementation

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Key takeaways
• transform()/process() combines the best of both DSL and PAPI
• Kryo for state serialization/deserialization
• Think stream/table duality
• Avoid unnecessary DSL call by accessing state directly
• Use Kubernetes as runtime
• Monitoring is important (leverage built-in metrics)

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
DSL vs Processor API
• Declarative vs Imperative
• Usability vs Flexibility
• High-level API vs low-level programming model

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Monitoring
• Built-in webserver
• Queryable state
• Metrics
• Internal topics
• Lags (topic/partition)

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Hot load configuration
KStream
KTable
Stream & Table Join
Joined Stream
KStream
GlobalKTable
Joined Stream
Stream & Table Join
Conf. topic
Data topic
Comdb2 Kafka Connect
Conf. topic
Data topic
DB Kafka Connect
KStream
KTable state

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Community Involvement
• KIP-362 - Dynamic Gap Session Window
— Versus fixed-gap session window
session1 session2 session3
gap3gap2gap1

© 2018 Bloomberg Finance L.P. All rights reserved.
Some other tips & tricks

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Thread model and depth first topology
APP
KAFKA
STREAMS
topology

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Trick 1 - Batch Processing In Kafka Streams
10/15/18
Micro batch?
Possible!
Batch?
Harder!
watermark
State
App2
App1

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Trick 2 - Chain Multiple Kafka Streams App
• Kafka as message bus
• Compose pipeline using multiple Kafka Streams apps
• Could leverage third-party pipeline framework – Spring Cloud Data Flow, etc.
KSTREAMS
APP1
topology
KSTREAMS
APPN
topology
KSTREAMS
APP2
topology

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Our Streaming Platform

© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
We are hiring!
Questions?
https://tinyurl.com/y7bepre9

What's hot

Presentation at Strata Data Conference 2018, New York The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure. Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker. Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.

A Deep Dive into Kafka Controller

confluent

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Databricks

Companies are increasingly moving to the cloud to store and process data. In this talk, we will talk through how companies can use tag-based policies in Apache Ranger to protect access to data both in on-premises environments as well in AWS-based cloud environments. We will go into details of how tag-based policies work and the integration with Apache Atlas and various services. We will also talk through how companies can leverage Ranger’s policies to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Kafka, Apache Hive, Apache Spark, or plain old ETL using MapReduce. We will also deep dive into Ranger’s proposed integration with S3 and other cloud-native systems. We will wrap it up with an end-to-end demo showing how tags and tag-based masking policies can be used to anonymize sensitive data and track how tags are propagated within the system and how sensitive data can be protected using tag-based policies Speakers Don Bosco Durai, Chief Security Architect, Privacera Madhan Neethiraj, Sr. Director of Engineering, Hortonworks

Securing data in hybrid environments using Apache Ranger

DataWorks Summit

Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.

Apache Kafka at LinkedIn

Discover Pinterest

NATS Streaming - an alternative to Apache Kafka?

Anton Zadorozhniy

Mario Molina, Software Engineer CDC systems are usually used to identify changes in data sources, capture and replicate those changes to other systems. Companies are using CDC to sync data across systems, cloud migration or even applying stream processing, among others. In this presentation we’ll see CDC patterns, how to use it in Apache Kafka, and do a live demo! https://www.meetup.com/Mexico-Kafka/events/277309497/

CDC patterns in Apache Kafka®

confluent

PostgreSQL major version upgrade using built in Logical Replication

Atsushi Torikoshi

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Evening out the uneven: dealing with skew in Flink

Flink Forward

CoC23_ Looking at the New Features of Apache NiFi Apache NiFi has a lot of new features, processors and best practices that have arrived in the last year or so. I will walk through building flows using the latest tips, techniques and processor. I will and change a number of data flows utilizing the latest NiFi version and point out gotchas and some never dos. The deck will act as a take-away with notes, tips and guides to what we covered. ===> Any NiFi 1.23+ and 2.0 in progress features people want to see?

CoC23_ Looking at the New Features of Apache NiFi

Timothy Spann

The world of real-time data processing is constantly evolving, with new technologies and platforms emerging to meet the ever-increasing demands of modern data-driven businesses. Apache Flink and RisingWave are two powerful stream processing solutions that have gained significant traction in recent years. But which platform is right for your organization? Karin Wolok and Yingjun Wu go head-to-head to compare and contrast the strengths and limitations of Flink and RisingWave. They’ll also share real-world use cases, best practices for optimizing performance and efficiency, and key considerations for selecting the right solution for your specific business needs.

Battle of the Stream Processing Titans – Flink versus RisingWave

Yingjun Wu

Grafana.pptx

Bhushan Rane

Watch this talk here: https://www.confluent.io/online-talks/integrating-apache-kafka-into-your-environment-on-demand Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams. This session is part 4 of 4 in our Fundamentals for Apache Kafka series.

Integrating Apache Kafka Into Your Environment

confluent

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Apache Sedona: how to process petabytes of agronomic data with Spark

Luiz Henrique Zambom Santana

Kafka 101

Aparna Pillai

Meetup: Streaming Data Pipeline Development In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/ You can join the meeting virtually here: https://cloudera.zoom.us/j/91603330726 Speaker - Tim Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Meetup: Streaming Data Pipeline Development

Timothy Spann

Delta from a Data Engineer's Perspective

Databricks

Change Data Feed is a new feature of Delta Lake on Databricks that is available as a public preview since DBR 8.2. This feature enables a new class of ETL workloads such as incremental table/view maintenance and change auditing that were not possible before. In short, users will now be able to query row level changes across different versions of a Delta table. In this talk we will dive into how Change Data Feed works under the hood and how to use it with existing ETL jobs to make them more efficient and also go over some new workloads it can enable.

Change Data Feed in Delta

Databricks

Delight (https://www.datamechanics.co/delight) is a free & cross-platform monitoring dashboard for Apache Spark, which display system metrics (CPU Usage, Memory Usage) along with Spark information (jobs, stages, tasks) on the same timeline. Delight is a great complement to the Spark UI when it comes to troubleshooting your Spark application and understanding its performance bottleneck. It works freely on top of any Spark platform (whether it’s open-source or commercial, in the cloud or on-premise). You can install it using an open-sourced Spark agent (https://github.com/datamechanics/delight). In this session, the co-founders of Data Mechanics will take you through performance troubleshooting sessions with Delight on real-world data engineering pipelines. You will see how Delight and the Spark UI can jointly help you spot the performance bottleneck of your applications, and how you can use these insights to make your applications more cost-effective and stable.

Delight: An Improved Apache Spark UI, Free, and Cross-Platform

Databricks

Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput. This talk provides a comprehensive overview of Kafka architecture and internal functions, including: -Topics, partitions and segments -The commit log and streams -Brokers and broker replication -Producer basics -Consumers, consumer groups and offsets This session is part 2 of 4 in our Fundamentals for Apache Kafka series.

Apache Kafka Architecture & Fundamentals Explained

confluent

What's hot (20)

A Deep Dive into Kafka Controller

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Securing data in hybrid environments using Apache Ranger

Apache Kafka at LinkedIn

NATS Streaming - an alternative to Apache Kafka?

CDC patterns in Apache Kafka®

PostgreSQL major version upgrade using built in Logical Replication

Evening out the uneven: dealing with skew in Flink

CoC23_ Looking at the New Features of Apache NiFi

Battle of the Stream Processing Titans – Flink versus RisingWave

Grafana.pptx

Integrating Apache Kafka Into Your Environment

Batch Processing at Scale with Flink & Iceberg

Apache Sedona: how to process petabytes of agronomic data with Spark

Kafka 101

Meetup: Streaming Data Pipeline Development

Delta from a Data Engineer's Perspective

Change Data Feed in Delta

Delight: An Improved Apache Spark UI, Free, and Cross-Platform

Apache Kafka Architecture & Fundamentals Explained

Similar to Real-Time Market Data Analytics Using Kafka Streams

After migrating much of our tech stack to Kafka Streams and Kafka Connect, some of our newer streaming applications encountered the same problem: while their primary input data was available in Kafka and their static side input data could be ingested using traditional connectors, they still needed access to semi-static data that has historically been acquired through in-line synchronous requests and responses. External service calls made from within a Kafka Streams application are highly discouraged, as they make error-handling difficult and typically slow down the application. To solve this problem, we used Kafka Connect components to build a large-scale data subscription service which subscribes to, polls, and disseminates auxiliary data to Kafka topics for immediate consumption by our streaming applications. This talk will cover: - An overview of our subscription service architecture - A deep-dive into our Kafka Connector and each of its custom components - Design considerations and various pitfalls that we encountered along the way

Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay V...

confluent

In a micro-services environment, a clean slate is the ideal starting point for writing Kafka Streams applications. However, the reality is that, in most established organizations, existing legacy systems need to be replaced, while the current client base is maintained. In this talk, we will review our journey in migrating from an existing, legacy system to a more scalable Kafka Streams application, all the while maintaining the stability and integrity of our data for existing customers. We will concentrate on how two technologies – Kafka Connect and KSQL – were instrumental during this migration. We will cover: * Our motivation for migrating to KStreams * How Kafka Connect and KStreams concepts (KTable and GlobalKTable) allowed us to incorporate an external data store into the KStreams app * How KSQL, using stream-to-stream JOINs, provided a fast and easy way to achieve A/B testing * Kafka Connect and KSQL internals, pitfalls, and tips

Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...

confluent

Kafka Streams performance monitoring and tuning is important for many reasons, including identifying bottlenecks, achieving greater throughput, and capacity planning. In this talk we’ll share the techniques we used to achieve greater performance and save on compute, storage, and cost. We’ll cover: Identifying design bottlenecks in by reviewing logs, metrics, and serdes. State store access patterns, design, and optimization Using profiling tools such as JMX, YourKit etc. Performance tuning of Kafka and Kafka Streams configuration and properties. JVM optimization for correct heap size and garbage collection strategies. Functional programming and imperative programming trade offs.

Why My Streaming Job is Slow - Profiling and Optimizing Kafka Streams Apps (L...

confluent

Are you a Kafka Streams application developer who needs a faster, more efficient way to reproduce a bug or issue from past events? Do you need to test a new algorithm patch near a discrete point-in-time? Do you have a standard methodology for investigating these issues, or does each team member devise an ad hoc solution? How much time does your simulation take? It is technically impossible to solve this challenge using only changelog topics, since a compact, topic-based changelog doesn't capture the full change history. The Derivatives Data team at Bloomberg augments this through periodic snapshots. To make the snapshotted state accessible to our replay system, we built a query service that leverages Interactive Queries using the Kafka Streams API, along with gRPC-based coordination, to fetch the distributed snapshot states from different snapshotting instances. In this talk, we will cover: ● Overview of this system architecture ● Deep dive into the mechanics of snapshots with Kafka Streams, state-store changelogs, and a query service to serve replay requests ● How two modes, normal and replay, are used on the Kafka Streams application runtime ● Some use cases that benefit from this replay system

Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Blo...

confluent

HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies

Michael Stack

Serving queries at low latency using HBase

Biju Nair

Multi-Tenant HBase Cluster - HBaseCon2018-final

Biju Nair

Apache Geode (incubating) is the core of Pivotal Gemfire now available as an open source project governed by Apache Software Foundation Incubator. The legacy of Pivotal Gemfire and the ASF community uniquely position Geode as a secret ingredient for modern-day data management architectures. These types of architectures require a robust in-memory data grid solution to handle a variety of use cases, ranging from enterprise-wide caching to real-time transactional applications at scale. In addition, as memory size and network bandwidth growth continues to outpace those of disk, the importance of managing large pools of RAM at scale increases. It is essential to innovate at the same pace. Apache Geode (incubating) has all the right ingredients to do for RAM what HDFS has done for direct attach disks. The excitement (and funding!) in this area of big data ecosystem is palpable, and the ASF is the place where the innovation is happening. Come to this session to understand: a brief history of Geode, architecture and use cases, design philosophy and principles, but most importantly: how you too can participate in the in-memory data center revolution.

IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire

In-Memory Computing Summit

Site | https://www.infoq.com/qconai2018/ Youtube | https://www.youtube.com/watch?v=2h0biIli2F4&t=19s At PayPal, data engineers, analysts and data scientists work with a variety of datasources (Messaging, NoSQL, RDBMS, Documents, TSDB), compute engines (Spark, Flink, Beam, Hive), languages (Scala, Python, SQL) and execution models (stream, batch, interactive). Due to this complex matrix of technologies and thousands of datasets, engineers spend considerable time learning about different data sources, formats, programming models, APIs, optimizations, etc. which impacts time-to-market (TTM). To solve this problem and to make product development more effective, PayPal Data Platform developed "Gimel", a unified analytics data platform which provides access to any storage through a single unified data API and SQL, that are powered by a centralized data catalog. In this session, we will introduce you to the various components of Gimel - Compute Platform, Data API, PCatalog, GSQL and Notebooks. We will provide a demo depicting how Gimel reduces TTM by helping our engineers write a single line of code to access any storage without knowing the complexity behind the scenes.

QCon 2018 | Gimel | PayPal's Analytic Platform

Deepak Chandramouli

We presented Gimel at Dataworks Summit in San Jose. Gimel is the open source unified data API which enables connectivity to any data store with a single API. Along with the API which works with Scala and Python, we are also surfacing a SQL interface to access any data store with just SQL. Now data scientists and analysts can directly consume data from big data platforms like Kafka for real-time streaming data access or Elastic for search-related data all with SQL just like they can access Oracle or Teradata. On the other hand, data engineers can relax now with this abstracted API since it isolates the ever-changing world of big data infrastructure from their code. No longer do they need to worry about API versions, connector versions, data store-specific semantics, or compute engine and version. Gimel is also tightly integrated with Jupyter notebooks so all of the power is now available to anyone with a browser. gimel.io unifieddatacatalog.io ppextensions.io Ping me on LinkedIn for more info!

Gimel at Dataworks Summit San Jose 2018

Romit Mehta

https://dataworkssummit.com/san-jose-2018/expo-theatre/gimel-paypals-analytics-data-platform/ At PayPal, data engineers, analysts, and data scientists work with a variety of data sources (Messaging, NoSQL, RDBMS, Documents, TSDB), compute engines (Spark, Flink, Beam, Hive), languages (Scala, Python, SQL) and execution models (stream, batch, interactive). Due to this complex matrix of technologies and thousands of datasets, engineers spend considerable time learning about different data sources, formats, programming models, APIs, optimizations, etc. which impacts time-to-market (TTM). To solve this problem and to make product development more effective, PayPal Data Platform developed “Gimel”, a unified analytics data platform which provides access to any storage through a single unified data API and SQL, that is powered by a centralized data catalog. In this session, we will introduce you to the various components of Gimel – Compute Platform, Data API, PCatalog, GSQL, and Notebooks. We will provide a demo depicting how Gimel reduces TTM by helping our engineers write a single line of code to access any storage without knowing the complexity behind the scenes.

Dataworks | 2018-06-20 | Gimel data platform

Deepak Chandramouli

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...

Lucidworks

One key feature that differentiates HBase from other distributed databases is its support of coprocessors. Bloomberg develops and manages some very low-latency systems that service real-time requests. In order to achieve real-time speeds, it was necessary to utilize coprocessors, which are similar to traditional stored procedures. As a result, we were able to match the average latency of an HBase cluster with that of a traditional database. This was done by using coprocessors to parallelize a lot of data computation and reduce the number of round-trips to the cluster by a factor of 5, thereby lowering the amount of data sent over the wire by 5. However, there are also significant challenges to managing coprocessors in a production environment. In this talk, I will to review the use case for HBase coprocessors and some practical tips on how to properly develop and deploy them. Some of the key topics covered in this talk are: Type of coprocessors Development challenges Deployment challenges Speakers Amit Anand, Senior Software Developer, Bloomberg LP Esther Kundin, Senior Software Engineer, Bloomberg LP

HBase coprocessors, Uses, Abuses, Solutions

DataWorks Summit

Tracing is always a challenge, no matter what your architecture is. Creating an application with serverless functions, such as with AWS Lambda, provides agility and scalability to your application, but it also creates an added challenge for code tracing. In this session, we review Datadog's distributed tracing capabilities and how Trek10 uses those capabilities to improve its customers’ applications. Learn how to use AWS X-Ray in a serverless environment. Also, learn strategies for working with traces and logs that explain application errors. Finally, learn how Trek10 uses AWS X-Ray with Datadog to measure and improve its applications' performance. This session is brought to you by AWS partner, Datadog.

How Trek10 Uses Datadog's Distributed Tracing to Improve AWS Lambda Projects ...

Amazon Web Services

From Mainframe to Microservices with Pivotal Platform and Kafka: Bridging the...

VMware Tanzu

Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise

VMware Tanzu

Are you curious about KNIME Software? Do you know the difference between KNIME Analytics Platform and KNIME Server? Which data sources can KNIME connect to? Can you run an R script from within a KNIME workflow? A Python script? Which other integrations are available? How can KNIME help with ETL, data preparation, and general data manipulation? Which machine learning algorithms can KNIME offer? This webinar answers all of these questions! There’s also information about connecting to big data clusters and how you can run the whole or part of your analysis on a big data platform. It also covers everything you need to know about Microsoft Azure and Amazon AWS

KNIME Software Overview

KNIMESlides

Users of financial data require their queries to return results with very low latency. As a financial data service provider, Bloomberg needs to consistently meet these requirements for our clients. HBase promises millisecond latency, auto-sharding, and an open schema. However, as HBase is a NoSQL database, support for transaction processing is not trivial. This talk will discuss a data modelling technique and use case where effective transaction processing can be achieved. We will also discuss how this data model helps us achieve real-time streaming, scalability, and millisecond read-write latency for billions of queries each day.

How data modelling helps serve billions of queries in millisecond latency wit...

DataWorks Summit

In this session, we discuss best practices for building serverless applications that handle high throughput and bursty data using Amazon SQS, Amazon SNS, and AWS Lambda, including new features such as message filtering and SNS/SQS as event sources for Lambda will be covered in depth. Hear from our customers Enel and Letgo as they share their experiences and deployment strategies. Enel is a multinational energy company that is present in 34 countries across 5 continents, and serves nearly 71 million end-users. The company uses AWS as its platform for IoT and energy management. Letgo is one of the largest and fastest-growing apps to buy and sell locally, with over 100 million downloads and hundreds of millions of listings. Letgo uses AWS to process their online marketplace transactions.

Build High-Throughput, Bursty Data Apps with Amazon SQS, SNS, & Lambda (API30...

Amazon Web Services

Similar to Real-Time Market Data Analytics Using Kafka Streams (20)

Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay V...

Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...

Why My Streaming Job is Slow - Profiling and Optimizing Kafka Streams Apps (L...

Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Blo...

HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies

Serving queries at low latency using HBase

Multi-Tenant HBase Cluster - HBaseCon2018-final

IMCSummit 2015 - 1 IT Business - The Evolution of Pivotal Gemfire

QCon 2018 | Gimel | PayPal's Analytic Platform

Gimel at Dataworks Summit San Jose 2018

Dataworks | 2018-06-20 | Gimel data platform

Dynamic Rule-based Real-time Market Data Alerts

Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...

HBase coprocessors, Uses, Abuses, Solutions

How Trek10 Uses Datadog's Distributed Tracing to Improve AWS Lambda Projects ...

From Mainframe to Microservices with Pivotal Platform and Kafka: Bridging the...

Pivotal Digital Transformation Forum: Journey to Become a Data-Driven Enterprise

KNIME Software Overview

How data modelling helps serve billions of queries in millisecond latency wit...

Build High-Throughput, Bursty Data Apps with Amazon SQS, SNS, & Lambda (API30...

More from confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

confluent

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

confluent

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

confluent

Confluent Partner Tech Talk with Synthesis

confluent

The Future of Application Development - API Days - Melbourne 2023

confluent

The Playful Bond Between REST And Data Streams

confluent

More from confluent (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams

Recently uploaded

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Real Time Object Detection Using Open CV

Khem

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Recently uploaded (20)

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Why Teams call analytics are critical to your entire business

Partners Life - Insurer Innovation Award 2024

GenAI Risks & Security Meetup 01052024.pdf

HTML Injection Attacks: Impact and Mitigation Strategies

Real Time Object Detection Using Open CV

Strategies for Landing an Oracle DBA Job as a Fresher

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Artificial Intelligence: Facts and Myths

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

presentation ICT roal in 21st century education

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Top 10 Most Downloaded Games on Play Store in 2024

Apidays New York 2024 - The value of a flexible API Management solution for O...

Scaling API-first – The story of a global engineering organization

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Data Cloud, More than a CDP by Matt Robison

AWS Community Day CPH - Three problems of Terraform

Boost Fertility New Invention Ups Success Rates.pdf

Real-Time Market Data Analytics Using Kafka Streams

1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Lei Chen Software Engineer, Team Lead lchen576@bloomberg.net Real-time* Market Data Processing Using Kafka Streams * Actually, just low-latency

2. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Agenda • Use cases and challenges • Why Kafka Streams • Deep dive into our implementation • Some other tips & tricks

3. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Who we are & What we do • Derivative market data group in Bloomberg • Builds market data pipelines • Apply big data & AI technologies in financial domain

4. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Streaming use cases Market movement (bid/ask/trade) Composite price (Bloomberg Generated Market Indicator) Option price (Calculate option price using calculated volatilities)

5. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Challenges and why Kafka Streams • Zero data loss • Ultra-Low latency • Huge data volume • Large state • Corporate DR compliance • Maintenance ‘Exactly once’ delivery Super fast Highly scalable State store Fault tolerant Minimal management overhead *Data already in Kafka!

7. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Key takeaways • transform()/process() combines the best of both DSL and PAPI • Kryo for state serialization/deserialization • Think stream/table duality • Avoid unnecessary DSL call by accessing state directly • Use Kubernetes as runtime • Monitoring is important (leverage built-in metrics)

8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. DSL vs Processor API • Declarative vs Imperative • Usability vs Flexibility • High-level API vs low-level programming model

9. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Monitoring • Built-in webserver • Queryable state • Metrics • Internal topics • Lags (topic/partition)

10. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Hot load configuration KStream KTable Stream & Table Join Joined Stream KStream GlobalKTable Joined Stream Stream & Table Join Conf. topic Data topic Comdb2 Kafka Connect Conf. topic Data topic DB Kafka Connect KStream KTable state

11. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Community Involvement • KIP-362 - Dynamic Gap Session Window — Versus fixed-gap session window session1 session2 session3 gap3gap2gap1

14. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Trick 1 - Batch Processing In Kafka Streams 10/15/18 Micro batch? Possible! Batch? Harder! watermark State App2 App1

15. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Trick 2 - Chain Multiple Kafka Streams App • Kafka as message bus • Compose pipeline using multiple Kafka Streams apps • Could leverage third-party pipeline framework – Spring Cloud Data Flow, etc. KSTREAMS APP1 topology KSTREAMS APPN topology KSTREAMS APP2 topology

Real-Time Market Data Analytics Using Kafka Streams

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Real-Time Market Data Analytics Using Kafka Streams

Similar to Real-Time Market Data Analytics Using Kafka Streams (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Real-Time Market Data Analytics Using Kafka Streams