Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra

•Download as PPTX, PDF•

3 likes•1,544 views

Kafka is used as a "dial tone for data" to ingest large amounts of data at HomeAway. An experiment using Kafka and Camus to ingest over 1TB of data per day from various systems and applications was successful. It allowed for various use cases like SLA reporting, fraud detection, search and clickstream analysis, and traveler segmentation. Key lessons included the importance of the schema registry for decoupling producers and consumers, and making stream processing easy through tools like Samza. The presentation concludes that Kafka allows building systems of engagement and intelligence on top of ingested data.

Engineering

© Copyright 2016 HomeAway, Inc.
Kafka: The “Dial Tone” for Data

HomeAway
The world leader for vacation rentals
> 1 million listings
(and growing!)

Agenda
© Copyright 2016 HomeAway, Inc.
• Overview
• The Problem
• The Experiment
• Results: Use Cases
• Lessons Learned
• Next Steps

© Copyright 2016 HomeAway, Inc.
Overview

Difference between Dinosaurs and Unicorns
© Copyright 2016 HomeAway, Inc.

In the old days: “Dial Tone” looked like this
© Copyright 2016 HomeAway, Inc.
ATDT

Today: Kafka is the modern “Dial Tone” for Data
© Copyright 2016 HomeAway, Inc.
Producer
Consumer

The Problem
© Copyright 2016 HomeAway, Inc.

Our original problem/motivation
© Copyright 2016 HomeAway, Inc.
search head
indexer
indexer
app server forwarder
app server forwarder
1 TB/day ingress and growing!
40,000 calls/sec

Also… Historical Analytic Pipeline was slow/expensive
© Copyright 2016 HomeAway, Inc.
app server
OLTP OLAP
analytics
ETL

Fill the Lake! Alternatives
?
Problem: Fill Hadoop!
Problem Data Lake
© Copyright 2016 HomeAway, Inc.

What we wanted… the Big Idea
© Copyright 2016 HomeAway, Inc.
If you can log it… … you can analyze it!

How to build self-service?
© Copyright 2016 HomeAway, Inc.

Hypothesis: Use Kafka!
© Copyright 2016 HomeAway, Inc.
2 ms median
latency
http://bit.ly/jay_on_logs
the log
2 Million Events / Sec!
(3 cheap machines)
http://goo.gl/pv5GoL
“Benchmarking Apache
Kafka”

© Copyright 2016 HomeAway, Inc.
The Experiment

HACommonsLogging
• KafkaAppender
Schema-on-read
• KafkaAvroLogger
Schema-on-write
Experiment: Schema-on-Read, Schema-on-Write
Data Lake
© Copyright 2016 HomeAway, Inc.
Schema
Registry

Architecture: Kafka + Camus = BigData Ingress
© Copyright 2016 HomeAway, Inc.
Camus

© Copyright 2016 HomeAway, Inc.
The Results

Use Cases: ITOA / SLA Reporting
© Copyright 2016 HomeAway, Inc.

Use Cases: Fraud
© Copyright 2016 HomeAway, Inc.

Use Cases: Search + ClickStream
© Copyright 2016 HomeAway, Inc.
User Behavior
Search Requests
A/B Test Readouts
Proctor
EDAP

Use Cases: Search + ClickStream
© Copyright 2016 HomeAway, Inc.

Use Cases: Traveler Segmentation
© Copyright 2016 HomeAway, Inc.
EDAP
Data
Model

Lessons Learned
© Copyright 2016 HomeAway, Inc.

Lesson #1: The Schema [registry] is Everything!
Data Lake
© Copyright 2016 HomeAway, Inc.
Schema
Registry
• Decouples producers from
consumers
• Enforces backwards
compatibility
• Enables self-service /
democratization
• SOT for schemas in the pipe

Lesson #2: A Kafka/SR governance module is helpful
Data Lake
© Copyright 2016 HomeAway, Inc.
• TURN OFF Auto Topic Creation!
• Need a place for developers
to request topics
• Retention Policy
• Expected Load
• Compaction
• Partition Size / Partition Key
• Owner
• LTS Date

Lesson #3: Make it easy to do stream processing
© Copyright 2016 HomeAway, Inc.
Schema
Registry
• samza-archetype
• samza-job-deployer
• Will evaluate k-streams!!!!
http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple

Next Steps
© Copyright 2016 HomeAway, Inc.

Consistency : 3 types of Data
© Copyright 2016 HomeAway, Inc.
Event
Document
Transactional

Kafka Producer Spooling
© Copyright 2016 HomeAway, Inc.

Conclusion
© Copyright 2016 HomeAway, Inc.

Yesterday
© Copyright 2016 HomeAway, Inc.
Systems of Record

Today
© Copyright 2016 HomeAway, Inc.
Systems of Engagement

Tomorrow
© Copyright 2016 HomeAway, Inc.
Systems of Intelligence

Don’t be a dinosaur…
© Copyright 2016 HomeAway, Inc.
ATDT

Thank you
© Copyright 2016 HomeAway, Inc.

End of Presentation
© Copyright 2016 HomeAway, Inc.

Viewers also liked

Microservices in the Apache Kafka Ecosystem

confluent

This talk is about the beauty of sequential access and append-only data structures. We'll do this in the context of a little-known paper entitled “Log Structured Merge Trees”. LSM describes a surprisingly counterintuitive approach to storing and accessing data in a sequential fashion. It came to prominence in Google's Big Table paper and today, the use of Logs, LSM and append-only data structures drive many of the world's most influential storage systems: Cassandra, HBase, RocksDB, Kafka and more. Finally, we'll look at how the beauty of sequential access goes beyond database internals, right through to how applications communicate, share data and scale.

Power of the Log: LSM & Append Only Data Structures

confluent

Apache Flink: API, runtime, and project roadmap

Kostas Tzoumas

Connected Homes is at the forefront of IoT in the UK. Spun out of British Gas in 2012, its expanding Hive IoT product range & its access to the largest pool of UK smart meter data uniquely positions it as a key player in the UK market. We will share with you how Apache Kafka has become a strategic technology used throughout the business and explore some of our use cases. We will give a brief overview of Connected Homes and why Apache Kafka is being adopted in teams for operational, feature and realtime data science purposes. Deeper technical insights will be shown around smart meter customers and how we use Apache Kafka to provide realtime alerting.

Never at Rest - IoT and Data Streaming at British Gas Connected Homes, Paul M...

confluent

Ingesting Healthcare Data, Micah Whitacre

confluent

The initial deployment of Espresso relies on MySQL’s built-in mechanism for Master-Slave replication. Storage hosts running MySQL masters service HTTP requests to store and retrieve documents, while hosts running slave replicas remain mostly idle. Since replication is at the MySQL instance level, masters and slaves must contain the exact same partitions – precluding flexible and dynamic partition placement and migration within the cluster. Espresso is migrating to a new deployment topology where each Storage Node may host a combination of master and slave partitions; thus distributing the application requests equally across all available hardware resources. This topology requires per-partition replication between master and slave nodes. Kafka will be used as the transport for replication between partitions. For use as the replication stream for the source-of-truth data store for LinkedIn’s most valuable data, Kafka must be as reliable as MySQL replication. The session will cover Kafka configuration options to ensure highly reliable, in-order message delivery. Additionally, the application logic maintains state both within the Kafka event stream and externally to detect message re-delivery, out of order delivery, and messages inserted out-of-band. These application protocols to guarantee high fidelity will be discussed.

Espresso Database Replication with Kafka, Tom Quiggle

confluent

101 mistakes FINN.no has made with Kafka (Baksida meetup)

Henning Spjelkavik

In the financial industry, losing data is unacceptable. Financial firms are adopting Kafka for their critical applications. Kafka provides the low latency, high throughput, high availability, and scale that these applications require. But can it also provide complete reliability? As a system architect, when asked “Can you guarantee that we will always get every transaction,” you want to be able to say “Yes” with total confidence. In this session, we will go over everything that happens to a message – from producer to consumer, and pinpoint all the places where data can be lost – if you are not careful. You will learn how developers and operation teams can work together to build a bulletproof data pipeline with Kafka. And if you need proof that you built a reliable system – we’ll show you how you can build the system to prove this too.

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

confluent

BY Jun Rao From the Bay Area Apache Kafka September 2016 Meetup. Abstract: To manage the ever-increasing volume and velocity of data within your company you have successfully made the transition from single machines and one-off solutions to large, distributed stream infrastructures in your data center powered by Apache Kafka. But what needs to be done if one data center is not enough? In this session we describe building resilient data pipelines with Apache Kafka that span multiple data centers and points of presence. We provide an overview of best practices and common patterns while covering key areas such as architecture guidelines, data replication and mirroring as well as disaster scenarios and failure handling.

Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...

confluent

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. Many organizations understand the use cases around their data – fraud detection, quality of service and technical operations, user behavior analysis, for example – but are not necessarily data infrastructure experts. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes an hour of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. Attendees will leave this session knowing not just which open source projects go into a system such as this, but how they work together, what tradeoffs and decisions need to be addressed, and how to present a single general purpose data platform to multiple applications. This session should be attended by data infrastructure engineers and architects planning, building, or maintaining similar systems.

Building an Event-oriented Data Platform with Kafka, Eric Sammer

confluent

Presented by Michael Noll, Product Manager, Confluent. Why are there so many stream processing frameworks that each define their own terminology? Are the components of each comparable? Why do you need to know about spouts or DStreams just to process a simple sequence of records? Depending on your application’s requirements, you may not need a full framework at all. Processing and understanding your data to create business value is the ultimate goal of a stream data platform. In this talk we will survey the stream processing landscape, the dimensions along which to evaluate stream processing technologies, and how they integrate with Apache Kafka. Particularly, we will learn how Kafka Streams, the built-in stream processing engine of Apache Kafka, compares to other stream processing systems that require a separate processing infrastructure.

A Practical Guide to Selecting a Stream Processing Technology

confluent

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

At Dropbox we are currently handling approximately 10,000,000 messages per second at peak across our handful of Kafka clusters. The largest of which has hit throughputs of 7,000,000 per second (~30 Gbps) on only 20 nodes. We’ll walk you through the steps we took to get where we are, the design that works for us — and those that didn’t. We’ll talk about the tooling we had to build and what we want to see exist. We’ll dive deeper into configuration and provide a blueprint you can follow. We’ll talk about the trials and tribulations of using Kafka — including ways we’ve set our clusters on fire, ways we’ve lost data, ways we’ve turned our hairs gray, and ways we’ve heroically saved the day for our users. Finally, we’ll spend time on some of the work we’re doing to handle consumer coordination across our many different systems and to integrate Kafka into a well established corporate infrastructure. (I.e., making Kafka “”play nice”” with everybody.)

Deploying Kafka at Dropbox, Mark Smith, Sean Fellows

confluent

The Rise of Real Time

confluent

Do you think that analytics are run on stored data sets? Think again, the combination of Apache Kafka and Stream Processing enables analytics on real-time data streams. First, we give a brief overview of stream processing and how it differs from the request response model of analytics on stored data. Next, we cover the characteristics of Kafka which make it such a good fit for Stream processing and why they matter. Finally, we show a number of use cases which highlight how stream processing is being used to do real-time analytics at scale with very low latency.

Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer

confluent

Flink and Kafka are popular components to build an open source stream processing infrastructure. We present how Flink integrates with Kafka to provide a platform with a unique feature set that matches the challenging requirements of advanced stream processing applications. In particular, we will dive into the following points: Flink’s support for event-time processing, how it handles out-of-order streams, and how it can perform analytics on historical and real-time streams served from Kafka’s persistent log using the same code. We present Flink’s windowing mechanism that supports time-, count- and session- based windows, and intermixing event and processing time semantics in one program. How Flink’s checkpointing mechanism integrates with Kafka for fault-tolerance, for consistent stateful applications with exactly-once semantics. We will discuss “”Savepoints””, which allows users to save the state of the streaming program at any point in time. Together with a durable event log like Kafka, savepoints allow users to pause/resume streaming programs, go back to prior states, or switch to different versions of the program, while preserving exactly-once semantics. We explain the techniques behind the combination of low-latency and high throughput streaming, and how latency/throughput trade-off can configured. We will give an outlook on current developments for streaming analytics, such as streaming SQL and complex event processing.

Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen

confluent

Building a real-time pipeline from scratch that is able to handle billion+ transactions per day, store, analyze and visualize it all in real-time has never been easier. In this build-as-we-go talk, we’ll create a front-to-back architecture that does exactly that. * we’ll start with a simple producer emitting a few messages and publishing them onto a Kafka queue * on consuming end of the queue a Spark-based Streamliner process will pick them up and store in MemSQL * ZoomData will connect to MemSQL for real-time visualization where we’ll be able to ask various questions and see answers change as data is flowing through the system * we’ll quickly make the entire pipeline more complex by increasing the amount of data as well as complexity of the data, until reaching 100K transactions per second As we walk through this demo, we will touch on cross data-center Kafka and MemSQL set-ups, speed limitations if any as well as echo back to real-life use cases of a similar set-up used in Goldman’s Asset Management division for the purposes of Portfolio Management & Trading.

Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, An...

confluent

The session will discuss how Uber evolved its stream processing system to handle a number of use cases in Uber Marketplace, with a focus on how Apache Kafka and Apache Samza played an important role in building a robust and efficient data pipeline. The use cases include but not limited to realtime aggregation of geospatial time series, computing key metrics as well as forecasting of marketplace dynamics, and extracting patterns from various event streams. The session will present how Kafka and Samza are used to meet the requirements of the use cases, what additional tools are needed, and lessons learned from operating the pipeline.

Stream Processing with Kafka in Uber, Danny Yuan

confluent

www.confluent.io/online-talks/ This talk will introduce Kafka Streams and help you understand how to map practical data problems to stream processing and how to write applications that process streams of data at scale using Kafka Streams. We will also cover what is stream processing, why one should care about stream processing, where Apache Kafka and Kafka Streams fit in, the hard parts of stream processing, and how Kafka Streams solves those problems; along with a concrete example of how these ideas tie together in Kafka Streams and in the big picture of your data center.

Demystifying Stream Processing with Apache Kafka

confluent

Protecting your data at rest with Apache Kafka by Confluent and Vormetric

confluent

Viewers also liked (20)

Microservices in the Apache Kafka Ecosystem

Power of the Log: LSM & Append Only Data Structures

Apache Flink: API, runtime, and project roadmap

Never at Rest - IoT and Data Streaming at British Gas Connected Homes, Paul M...

Ingesting Healthcare Data, Micah Whitacre

Espresso Database Replication with Kafka, Tom Quiggle

101 mistakes FINN.no has made with Kafka (Baksida meetup)

When it Absolutely, Positively, Has to be There: Reliability Guarantees in Ka...

Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...

Building an Event-oriented Data Platform with Kafka, Eric Sammer

A Practical Guide to Selecting a Stream Processing Technology

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

Deploying Kafka at Dropbox, Mark Smith, Sean Fellows

The Rise of Real Time

Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer

Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen

Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, An...

Stream Processing with Kafka in Uber, Danny Yuan

Demystifying Stream Processing with Apache Kafka

Protecting your data at rest with Apache Kafka by Confluent and Vormetric

Similar to Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra

Kafka is quickly gaining momentum as a very popular and very fast messaging platform that is very good at integrating different types of data quickly. Kafka makes this data available as a real-time data stream for consumption by enterprise users.There is so much hidden data available in our Oracle databases. How can we turn the database inside out to make this data available real-time to Kafka along with the other data sources in our enterprise. This paper will present the use cases of Oracle real-time data streaming as well as an introduction into Kafka and how to use Oracle logical replication to get Oracle real time into Kafka. This paper will include a real life real time demo from Oracle into Kafka.

[db tech showcase Tokyo 2016] E22: Getting real time Oracle data into Kafka a...

Insight Technology, Inc.

Design Patterns for working with Fast Data

MapR Technologies

Apache Kafka is an open-source message broker project that provides a platform for storing and processing real-time data feeds. In this presentation Ian Downard describes the concepts that are important to understand in order to effectively use the Kafka API. He describes how to prepare a development environment from scratch, how to write a basic publish/subscribe application, and how to run it on a variety of cluster types, including simple single-node clusters, multi-node clusters using Heroku’s “Kafka as a Service”, and enterprise-grade multi-node clusters using MapR’s Converged Data Platform. Video: https://vimeo.com/188045894 Ian also discusses strategies for working with "fast data" and how to maximize the throughput of your Kafka pipeline. He describes which Kafka configurations and data types have the largest impact on performance and provide some useful JUnit tests, combined with statistical analysis in R, that can help quantify how various configurations effect throughput.

Design Patterns for working with Fast Data in Kafka

Ian Downard

Content lake architecture can evolve the media workflow by providing efficiency from content security all the way to value-added services, such as machine learning and content monetization. In this session, technical leaders from 21st Century Fox, Warner Bros., and Astro Malaysia discuss the migration of their petabyte-scale video libraries (production and distribution archives) to the cloud in order to increase the customer reach and value of their media archives. Discover some of the lessons learned, the TCO analysis around various different storage tiers, the challenges and best practices from 10s of petabytes ingest, storage, and value-added compute at scale.

Hollywood's Cloud-Based Content Lakes: Modernized Media Archives (MAE203) - A...

Amazon Web Services

GOES-16 is a source of critical data for monitoring smoke, flooding impacts, burn scars, volcanic ash, and weather. However, finding and using this data can require significant investment. Element 84 married video compression and streaming technology with NASA’s Cumulus data processing pipeline, plus AWS Managed Services, to make the entire GOES-16 archive interactive on an array of formats. Users can now easily identify dates of interest for events like natural disasters, and stage a subset of the archive for analysis. And all of this scales down to $0 when not in use.

How Element 84 Raises the Bar on Streaming Satellite Data

Amazon Web Services

Amazon Aurora Serverless is an on-demand, autoscaling configuration for Aurora (MySQL-compatible edition) where the database automatically starts up, shuts down, and scales up or down capacity based on your application's needs. It enables you to run your database in the cloud without managing any database instances. Aurora Serverless is a simple, cost-effective option for infrequent, intermittent, or unpredictable workloads. In this session, we explore these use cases, take a look under the hood, and delve into the future of serverless databases. We also hear a case study from a customer building new functionality on top of Aurora Serverless.

Aurora Serverless: Scalable, Cost-Effective Application Deployment (DAT336) -...

Amazon Web Services

AWS Storage State of the Union

Amazon Web Services

by Drew Meyer, Sr. Product Marketing Manager, AWS This session will provide an overview of the AWS storage portfolio, including block, file, object, and cloud data migration services. We will touch on new offerings, outline some of the most common use cases, and prepare you for the individual deep dive sessions, customer sessions, and new announcements. The session will also address our partner network and what it means for a storage provider to have the APN Storage Competency.

AWS Storage State of the Union & APN Storage Ecosystem

Amazon Web Services

Apache Pulsar at Yahoo! Japan

StreamNative

Going Deep on Amazon Aurora Serverless (DAT427-R1) - AWS re:Invent 2018

Amazon Web Services

Advanced Threat Detection on Streaming Data

Carol McDonald

Within the last years, governmental bodies have been futilely trying to fight against dark web hosted marketplaces. Shortly after the closing of “The Silk Road” by the FBI and Europol in 2013, new successors have been established. Through the combination of cryptocurrencies and nonstandard communication protocols and tools, agents can anonymously trade in a marketplace for illegal items without leaving any record. This talk will presents a research carried out to gain insights on the products and services sold within one of the larger marketplaces for drugs, fake ids and weapons on the Internet, Agora, and on new developments after the demise of Agora.

Dark web markets: from the silk road to alphabay, trends and developments

Andres Baravalle

Come scalare da zero ai tuoi primi 10 milioni di utenti.pdf

Amazon Web Services

In this webinar, Mike Stonebraker explains the tradeoffs of performance, scale, and required programming “heroics” in capturing the value of fast data with different stream processing alternatives. Then hear from John Hugg, Founding Engineer of VoltDB, as he discusses the VoltDB approach to fast data. Learn what’s possible when systems integrate event processing with state management in a consistent, transactional way. You can also view the webinar recording here: http://learn.voltdb.com/WRStonebrakerRTeventprocessing.html

Mike Stonebraker on Designing An Architecture For Real-time Event Processing

VoltDB

At Mist, we believe the best way of achieving reliability is by embracing uncontrolled unreliability. We run our microservices and stream processing entirely on Amazon EC2 Spot Instances, which introduces uncontrolled chaos. In this session, we start with an overview of our system and discuss how we achieve high reliability on our largest service, Live Aggregators (LA). LA is our in-house, real-time, timeseries aggregation system that processes over a billion messages a day while aggregating over 100 million timeseries. We also share how we are autoscaling LA by predicting and adapting to changing loads that results in server utilizations of over 70%.

Cost-Effectively Running Distributed Systems at Scale in the Cloud (CMP349) -...

Amazon Web Services

Transitioning Geoscience Research to the Cloud: Opportunities and Challenges

Amazon Web Services

Spoofing and Denial of Service: A risk to the decentralized Internet

APNIC

DDoS And Spoofing, a risk to the decentralized internet

Tom Paseka

Hybrid Cloud Enablement Technologies

hybrid cloud

Overview of AWS Ground Station

AWS Daily News

Similar to Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra (20)

[db tech showcase Tokyo 2016] E22: Getting real time Oracle data into Kafka a...

Design Patterns for working with Fast Data

Design Patterns for working with Fast Data in Kafka

Hollywood's Cloud-Based Content Lakes: Modernized Media Archives (MAE203) - A...

How Element 84 Raises the Bar on Streaming Satellite Data

Aurora Serverless: Scalable, Cost-Effective Application Deployment (DAT336) -...

AWS Storage State of the Union

AWS Storage State of the Union & APN Storage Ecosystem

Apache Pulsar at Yahoo! Japan

Going Deep on Amazon Aurora Serverless (DAT427-R1) - AWS re:Invent 2018

Advanced Threat Detection on Streaming Data

Dark web markets: from the silk road to alphabay, trends and developments

Come scalare da zero ai tuoi primi 10 milioni di utenti.pdf

Mike Stonebraker on Designing An Architecture For Real-time Event Processing

Cost-Effectively Running Distributed Systems at Scale in the Cloud (CMP349) -...

Transitioning Geoscience Research to the Cloud: Opportunities and Challenges

Spoofing and Denial of Service: A risk to the decentralized Internet

DDoS And Spoofing, a risk to the decentralized internet

Hybrid Cloud Enablement Technologies

Overview of AWS Ground Station

More from confluent

Speed Wins: From Kafka to APIs in Minutes

confluent

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Workshop híbrido: Stream Processing con Flink

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

AWS Immersion Day Mapfre - Confluent

confluent

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Confluent & GSI Webinars series - Session 3

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Messaging Modernization

confluent

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Data In Motion Paris 2023

confluent

Confluent Partner Tech Talk with Synthesis

confluent

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

Recently uploaded

Introduction to Machine Learning Unit-4 Notes for II-II Mechanical Engineering

C Sai Kiran

Multivibrator and its types defination and usges.pptx

alijaker017

Software Engineering - Part 2 which describes the following topics: Introduction, Modelling Concepts and Class Modelling: What is Object orientation? What is OO development? OO Themes; Evidence for usefulness of OO development; OO Modelling history. Modelling as Design technique: Modelling, abstraction, The Three models. Class Modelling: Object and Class Concept, Link and associations concepts, Generalization and Inheritance, A sample class model, Navigation of class models, and UML diagrams. Building the Analysis Models: Requirement Analysis, Analysis Model Approaches, Data Modelling Concepts.

Software Engineering - Modelling Concepts + Class Modelling + Building the An...

Prakhyath Rai

Our vision is to make ONLINE CAR SERVICING SYSTEM is needed for the country as technology grows fast in the world. The purpose is to develop this project to provide a better solution to the problems that the customers faces. This website will provide customers to reserve their car services from their home or from office. Customer feels hard to send their car for the service, either they need to reserve using mail system or walk in to send their car for service. At the end of these project, a prototype of justified system will develop which will provide a solution for the identify problem to improve the organizations revenue and performance. This system will be web based system where it is able to conduct an overall process of online registration and bookings too. The data will be stored in keep as privacy for each customer who has register.

ONLINE CAR SERVICING SYSTEM PROJECT REPORT.pdf

Kamal Acharya

Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf

JNTUA

Research Methodolgy & Intellectual Property Rights Series 2

T.D. Shashikala

The Project deals with the development of the computerized system for maintaining the regular records and services that are undertaken in the furniture business. This project titled "Web Based Integrated Furniture Showroom Management System" has been aimed to design and computerized system that can handle various activities are been carried out at the Furniture Showroom. This application has been developed using PHP Programming Language as its front end and the back end is MYSQL Server In the existing system all the activities and record maintenance of the furniture showroom are done manually by the manager. The Project deals with the development of the computerized system for maintaining the regular records and services that are undertaken in this most important and large business oriented furniture business. This Project also enables the users to perform all the day to day business operations in the furniture showroom business most efficiently.

Furniture showroom management system project.pdf

Kamal Acharya

The ONLINE QUIZ is a web application for to take online test in an efficient manner and no time wasting for checking the paper. The main objective of ONLINE QUIZ is to efficiently evaluate the candidate thoroughly through a fully automated system that not only saves lot of time but also gives fast results. For students they give papers according to their convenience and time and there is no need of using extra thing like paper, pen etc. This application is basically create in WORDPRESS. It is a web software you can use to create a beautiful website or blog. we like to say that WORDPRESS is both free and priceless at the same time.

Quiz application system project report..pdf

Kamal Acharya

The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...

Roi Lipman

Lab Manual Arduino UNO Microcontrollar.docx

RashidFaridChishti

Complex Plane, Modulus, Argument, and Graphical Representation of a Complex Number Hello, my name is Mohammad Ali Nayeem, a student at DIU studying Software Engineering. In Math 102, we delve into various concepts related to complex numbers, including the complex plane, modulus, argument, and their graphical representation. Complex Plane The complex plane is a two-dimensional plane used to visually represent complex numbers. It consists of a horizontal axis (the real axis) and a vertical axis (the imaginary axis). Any complex number 𝑧 = 𝑎 + 𝑏 𝑖 z=a+bi can be plotted on this plane, where 𝑎 a is the real part and 𝑏 b is the imaginary part. Modulus The modulus (or absolute value) of a complex number 𝑧 = 𝑎 + 𝑏 𝑖 z=a+bi is the distance of the point representing 𝑧 z from the origin (0, 0) in the complex plane. It is denoted by ∣ 𝑧 ∣ ∣z∣ and calculated using the formula: ∣ 𝑧 ∣ = 𝑎 2 + 𝑏 2 ∣z∣= a 2 +b 2 The modulus provides a measure of the magnitude of the complex number. Argument The argument of a complex number is the angle 𝜃 θ between the positive real axis and the line segment connecting the origin to the point representing the complex number. It is denoted by arg ⁡ ( 𝑧 ) arg(z) and is usually measured in radians. The argument can be found using the formula: 𝜃 = tan ⁡ − 1 ( 𝑏 𝑎 ) θ=tan −1 ( a b ) The argument indicates the direction of the complex number in the complex plane. Graphical Representation To graphically represent a complex number 𝑧 = 𝑎 + 𝑏 𝑖 z=a+bi, plot the point ( 𝑎 , 𝑏 ) (a,b) on the complex plane. The modulus is the length of the line from the origin to this point, and the argument is the angle this line makes with the positive real axis. For example, the complex number 3 + 4 𝑖 3+4i can be plotted as the point (3, 4). Its modulus is: ∣ 3 + 4 𝑖 ∣ = 3 2 + 4 2 = 5 ∣3+4i∣= 3 2 +4 2 =5 Its argument is: 𝜃 = tan ⁡ − 1 ( 4 3 ) ≈ 0.93 radians θ=tan −1 ( 3 4 )≈0.93 radians Understanding these concepts helps in visualizing and manipulating complex numbers, which are fundamental in many areas of engineering and applied mathematics. Best regards, Mohammad Ali Nayeem DIU Software Engineering Student

Complex plane, Modulus, Argument, Graphical representation of a complex numbe...

MohammadAliNayeem

The project is on Supermarket Billing. Supermarket is the place where customers come to purchase their daily using products & pay for that. So there is a need to calculate how many products are sold & to generate the bill for the customers. In this project, we have 3 users. First is the data entry operator who will enter the products in the database. Second one is the administrator who will decide the taxes & commissions on the products & can see the report of any product. Third one is the bill calculating operator who will calculate the bill & print.

Supermarket billing system project report..pdf

Kamal Acharya

Now a day’s education plays a great role in development of any country. Many of education organizations try to increase education quality. One of the aspects of this improvement is managing of school resources. Education Management System carried on by any individual or institution engaged in providing a services to students, teachers, guardians and other persons are intermediary that performs one or more of the following functionalities – Student Admission, Employee Registration, Student List, Employee List, Student Attendance, Employee Attendance, Student Routine, Result Management, Payroll & Accounts. Education Management System (EMS) is such a service which provides all services for an educational institute to make your life easier and faster by assuring its performance. Easy User Management System, Easy Admission Process, Easy Attendance System. EMS is a system that will provide you a bird’s eye view of the functioning of the entire educational institution. It is a management information system helps to manage the different processes in an educational institution like General Administration, Staff Management, Academics, Student Management, and Accounts etc. The information is made using the latest technologies and help’s to make decision making a lot faster, effective and easier than ever before. Also helps to improve the overall quality of education of the institution. We use database and database technology are having a major impact on the growing use of computers. The implementation of the system was done using c# and SQL Server 2012 technologies, allowing system to be run in Windows OS. In a nutshell, Education Management Software managed your education institution by simplifying and automating processes and addressing the needs of all stakeholders helping them to be more efficient in their respective roles.

School management system project report.pdf

Kamal Acharya

Electrostatic field in a coaxial transmission line

JulioCesarSalazarHer1

ROAD CONSTRUCTION PRESENTATION.PPTX.pptx

GagandeepKaur617299

Introduction to Artificial Intelligence and History of AI

Sheetal Jain

Electronic Shop Management software helps Electronic showrooms owners and management staff by producing different kind of financial and stock tracking reports, etc. This software is able to manage all electronic stocks. In this software shop owner can manage the data of customer and buyers. Also tax information and other government charges including recycle charges. Electrical shop management system is workable application for retail store inventory and account management. It keeps a list STOCKS and products at a store and can do operation on them. The most important operation on them. The most important operation is a PURCHASE, all the transactions and the billing details and stock purchasing details involved on it

Electrical shop management system project report.pdf

Kamal Acharya

2024 DevOps Pro Europe - Growing at the edge

Paco Orozco

Its an "United Nations Park" Construction site. We have visited this place to gain some practical experience which is related to our academic topic. In this site we have seen a various engineering materials like hollow bricks, Concrete block, plain tiles. And how to make a water fountain, and pathway which is aesthetically pleasing to see. This construction was directed by our Honorable Sir Md. Abul Hasan. The Name of the developer is "STRUCXEL" which have the glorious history in the development in Chittagong.

"United Nations Park" Site Visit Report.

MdManikurRahman

Arduino based vehicle speed tracker project

Rased Khan

Recently uploaded (20)

Introduction to Machine Learning Unit-4 Notes for II-II Mechanical Engineering

Multivibrator and its types defination and usges.pptx

Software Engineering - Modelling Concepts + Class Modelling + Building the An...

ONLINE CAR SERVICING SYSTEM PROJECT REPORT.pdf

Involute of a circle,Square, pentagon,HexagonInvolute_Engineering Drawing.pdf

Research Methodolgy & Intellectual Property Rights Series 2

Furniture showroom management system project.pdf

Quiz application system project report..pdf

The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...

Lab Manual Arduino UNO Microcontrollar.docx

Complex plane, Modulus, Argument, Graphical representation of a complex numbe...

Supermarket billing system project report..pdf

School management system project report.pdf

Electrostatic field in a coaxial transmission line

ROAD CONSTRUCTION PRESENTATION.PPTX.pptx

Introduction to Artificial Intelligence and History of AI

Electrical shop management system project report.pdf

2024 DevOps Pro Europe - Growing at the edge

"United Nations Park" Site Visit Report.

Arduino based vehicle speed tracker project

Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra

2. HomeAway The world leader for vacation rentals > 1 million listings (and growing!)

15. Hypothesis: Use Kafka! © Copyright 2016 HomeAway, Inc. 2 ms median latency http://bit.ly/jay_on_logs the log 2 Million Events / Sec! (3 cheap machines) http://goo.gl/pv5GoL “Benchmarking Apache Kafka”

27. Lesson #1: The Schema [registry] is Everything! Data Lake © Copyright 2016 HomeAway, Inc. Schema Registry • Decouples producers from consumers • Enforces backwards compatibility • Enables self-service / democratization • SOT for schemas in the pipe

28. Lesson #2: A Kafka/SR governance module is helpful Data Lake © Copyright 2016 HomeAway, Inc. • TURN OFF Auto Topic Creation! • Need a place for developers to request topics • Retention Policy • Expected Load • Compaction • Partition Size / Partition Key • Owner • LTS Date

29. Lesson #3: Make it easy to do stream processing © Copyright 2016 HomeAway, Inc. Schema Registry • samza-archetype • samza-job-deployer • Will evaluate k-streams!!!! http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple

Editor's Notes

Is often the difference between dinosaurs and unicorns.
“We are living at a dawn of a new age, where how we listen to data…
Today, we @ HomeAway believe that Kafka is the “Dial Tone” for data, enabling present businesses to turn themselves into rainbows and unicorns.

Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra

Similar to Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra

Editor's Notes