Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processing with Apache Kafka

•

0 likes•227 views

confluent

Watch the webcast here: https://videos.confluent.io/watch/yaWXJTKELvUGfPx9cnmhZ8 Speaker: Kenneth Cheung

Technology

Tech Talk #2
Introduction to Streaming Data and
Stream Processing with Apache Kafka
Kenneth Cheung
Sr. Solutions Engineer, Greater China
Kenneth@conﬂuent.io

Schedule
Tech Talks Date/Time
TT#1 Dive into Apache Kafka® 12th June 2020, Thursday
TT#2 Introduction to Streaming Data and Stream Processing with Apache
Kafka
2nd July 2020, Thursday
TT#3 Conﬂuent Schema Registry 6th
August 2020, Thursday
TT#4 Kafka Connect 3rd
September 2020, Thursday
TT#5 Avoiding Pitfalls with Large-Scale Kafka Deployments 8th
October 2020, Thursday

Conﬂuent Platform
Dynamic Performance & Elasticity
Auto Data Balancer | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center
Efﬁcient
Operations at Scale
Freedom of Choice
Committer-driven Expertise
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Global Resilience
Multi-region Clusters | Replicator
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
ARCHITECTOPERATORDEVELOPER
Open Source | Community licensed
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Fully Managed Cloud ServiceSelf-managed Software
Training Partners
Enterprise
Support
Professional
Services
Apache Kafka

72
• Apache Kafka Stream Processing Book Bundle
• Online Talks
• Conﬂuent Platform Datasheet
• Streaming Use Cases Financial Services
https://events.conﬂuent.io/conﬂuentandbankofchinahktecht
Conﬂuent Resources

Interacting with Conﬂuent
73
Join the Conﬂuent Slack Channel
https://launchpass.com/conﬂuentcommunity
Local meetups
https://www.conﬂuent.io/community/
KafkaSummit 2020
Aug 24 - 25
https://kafka-summit.org/

Project Metamorphosis
Unveiling the next-gen event
streaming platform
Listen to replay and
Sign up for updates
cnﬂ.io/pm
Jay Kreps
Co-founder and CEO
Conﬂuent

Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processing with Apache Kafka

What's hot

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

Kafka has fast become the center of streaming analytics applications in the modern digital enterprise. Kafka operates in the context of a broad ecosystem of data lifecycle components which need a consistent platform of security, monitoring, management and governance. This problem becomes paramount when your streaming architectures go hybrid by spanning from on-premises to the cloud. Throw in the reality of a multi-cloud setup that a lot of enterprises are facing and now, you have a complex streaming architecture that is difficult to operationally manage, monitor, secure or govern. Cloudera remains committed to an open community driven approach and increasing the ease of use and visibility for Kafka based solutions. Attend this session to understand more about how streaming architectures can be extended easily to the hybrid cloud and multi-cloud. Also, learn about our plans for further community contributions.

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

HostedbyConfluent

Developing cloud native microservices introduced us to many new challenges. One of the most difficult is to build reliable microservices integrations and their data exchange patterns. In this session I will share my 10 years of experience with building microservices and application runtime platforms with some of the largest European organisations. I will introduce basic principles of developing Java Spring Boot with Apache Kafka. These patterns can be used for: microservices communication decoupling, implementing microservices state stores, avoiding dependencies on traditional database systems. This session is targeted for developers who are interested in learning new cloud native development practices and understanding how event streaming microservices improve their current work. Demo application code will be available to participants.

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...

HostedbyConfluent

All Streams Ahead! ksqlDB Workshop ANZ

confluent

Eine weitere neue sicherheitsrelevante Funktion in Confluent Platform 5.4 sind Structured Audit Logs. Jetzt ist natürlich alles in Kafka ein Log, aber Kafka protokolliert nicht, was Kafka mit Kafka macht - nur das, was in einen Topics geschrieben wird. Im dritten Teil der Deep Dive Sessions besprechen wir neben den Structured Audit Logs außerdem die "Weiterentwicklung" der bereits bekannten Schema Registry: Die Schema Validation agiert auf dem Topic-Level und stellt sicher, dass jede einzelne Message, die zu einem bestimmten Topic erstellt wird in der Schema Registry überprüft wird. Mehr dazu erklären wir in unserem Deep Dive #3.

Deep Dive Series #3: Schema Validation + Structured Audit Logs

confluent

Building a Web Application with Kafka as your Database

confluent

Watch this talk here: https://www.confluent.io/online-talks/using-apache-kafka-to-optimize-real-time-analytics-financial-services-iot-applications When it comes to the fast-paced nature of capital markets and IoT, the ability to analyze data in real time is critical to gaining an edge. It’s not just about the quantity of data you can analyze at once, it’s about the speed, scale, and quality of the data you have at your fingertips. Modern streaming data technologies like Apache Kafka and the broader Confluent platform can help detect opportunities and threats in real time. They can improve profitability, yield, and performance. Combining Kafka with Panopticon visual analytics provides a powerful foundation for optimizing your operations. Use cases in capital markets include transaction cost analysis (TCA), risk monitoring, surveillance of trading and trader activity, compliance, and optimizing profitability of electronic trading operations. Use cases in IoT include monitoring manufacturing processes, logistics, and connected vehicle telemetry and geospatial data. This online talk will include in depth practical demonstrations of how Confluent and Panopticon together support several key applications. You will learn: -Why Apache Kafka is widely used to improve performance of complex operational systems -How Confluent and Panopticon open new opportunities to analyze operational data in real time -How to quickly identify and react immediately to fast-emerging trends, clusters, and anomalies -How to scale data ingestion and data processing -Build new analytics dashboards in minutes

Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...

confluent

Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve! In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions: - Whether to store data contiguously or using an index - How many storage tiers are best? - Where should metadata live? - And more. It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.

Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...

HostedbyConfluent

Quality Matters … and as event-driven architectures (EDA) become increasingly popular in the microservices space, ensuring the delivery and performance of your EDA increases in importance. But while it’s powerful architecture, it does come with its challenges, especially from a testing perspective. For example, most organizations are not reliant on Kafka alone, but a multitude of interconnected APIs like REST, GraphQL and gRPC. One of the questions that arise from this challenge: How do you build end-to-end tests when the APIs are completely different technologies—without relying on fragile scripts? In our talk, we’ll tackle this question and many more when it comes to the testing of Apache Kafka endpoints and your services architecture. We’ll cover what makes testing in EDA difficult; technologies that can help you; and how we at SmartBear are thinking about these testing problems and, most importantly, how we are trying to solve for them.

Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...

HostedbyConfluent

Server-Sent Events (SSE) is a server push technology where clients receive automatic server updates through the secure http connection. SSE can be used in apps like live stock updates, that use one way data communications and also helps to replace long polling by maintaining a single connection and keeping a continuous event stream going through it. We used a simple Kafka producer to publish messages onto Kafka topics and developed a reactive Kafka consumer by leveraging Spring Webflux to read data from Kafka topic in non-blocking manner and send data to clients that are registered with Kafka consumer without closing any http connections. This implementation allows us to send data in a fully asynchronous & non-blocking manner and allows us to handle a massive number of concurrent connections. We’ll cover: •Push data to external or internal apps in near real time •Push data onto the files and securely copy them to any cloud services •Handle multiple third-party apps integrations

Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...

HostedbyConfluent

James Watters, Pivotal | Kafka Summit NYC 2019 Keynote (Spring Boot+Kafka: Th...

confluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

HostedbyConfluent

Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage. Agenda: - Data in Motion with Event Streaming and Apache Kafka - Streaming ETL Pipelines - IT Modernisation and Hybrid Multi-Cloud - Customer Experience and Customer 360 - IoT and Big Data Processing - Machine Learning and Analytics

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Kai Wähner

Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services

confluent

If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...

HostedbyConfluent

Introduction to Apache Kafka and Confluent... and why they matter!

Paolo Castagna

Event-Driven Architectures (EDA ) are perceived as mythical objects that instantly transform your systems into ""real-time"" ones! BUT, come to think of it, aren't they already ""real-time""? I mean, adding an item to the cart is pretty much instant in ( most ) webshops. In fact, EDA solves an entirely different set of problems and with the help of Apache Kafka, we will walk through the (re)evolution path. Microservices are easy to get started with, but once we do, we keep stumbling across the same issues: data access, consistency, and failures ( sounds familiar? ). The solution? Patterns, patterns, patterns … You’ve probably heard about terms such as “Event Notification”, “Event-carried State Transfer”, or even “Event Sourcing”, but how can they be used to solve our problems? And more importantly, how can we use Apache Kafka to take advantage of these patterns? I guess we will find out soon!

Demystifying Event-Driven Architectures with Apache Kafka | Bogdan Sucaciu, P...

HostedbyConfluent

How Priceline uses Kafka Streams technology to effectively save TBs on daily licenses of our monitoring systems. Kafka Streams powers a big part of our analytics and monitoring pipelines and delivers operational metrics transformations in real time. All logs and operational metrics from all of the APIs of Priceline’s products flow into Kafka and is ingested into our Monitoring System Splunk for Alerting and Monitoring. We have now implemented data transformations, aggregations and summarizations using Kafka Streams technologies to effectively eliminate PCI/PII violations on the log data; do aggregations on metrics to avoid ingesting sub-second metrics and ingest metrics only at the granularity that we need to. We will cover the need for custom Serdes, custom partitioners, and why we don’t use the confluent registry. You will also learn how Priceline uses a self service model to configure its streams, topics and consumers using Data Collection Console, which is our UI for managing the Kafka streaming pipelines.

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...

confluent

Bank of China (HK) Tech Talk 1: Dive Into Apache Kafka

confluent

What's hot (20)