Rootconf
•
1 like•567 views
LinkedIn uses Kafka for log aggregation and monitoring. It ingests over 500,000 metrics per minute from applications into Kafka, which then writes to storage clusters. It also uses Kafka to transport log data to ELK (Elasticsearch, Logstash, Kibana) for real-time search, analysis and visualization of metrics and logs. Kafka provides benefits like horizontal scalability, multiple consumer groups, and no overhead on client applications.
Report
Share
Report
Share
Download to read offline
Recommended
GraphQL API on a Serverless Environment
The document discusses using GraphQL to build a serverless API for a mobile app that detects construction errors. Key points include:
1. GraphQL provides an efficient way to fetch data from microservices in a single roundtrip and allows flexible field selection.
2. The previous monolithic REST API was split into microservices for data fetching, state management, and serving the API to improve performance and scalability.
3. Serverless is a good fit for the stateless GraphQL architecture as resolvers can trigger other serverless functions and the API can scale up and down easily.
Lambda architecture
The document describes AppsFlyer's lambda architecture for real-time analytics. It ingests raw event data from mobile apps into Kafka in real-time, processes the data using Apache Spark for reports and dashboards, and stores the data in S3 for batch processing. It monitors the system using Grafana, Kibana, and end-to-end tests to ensure reliable reporting on metrics like lifetime value, retention, and cohorts.
2018 07-11 - kafka integration patterns
During this meet up we will see different approach to integrate Apache Kafka in an enterprise architrecture
Productionize spark structured streaming
Ivan Kosianenko discusses how AppsFlyer has evolved its data pipeline from using MemSQL and Druid to handle 12 billion events daily to now using ClickHouse and Druid to handle 38 billion events daily. He then explains how building a custom data streaming solution with Clojure can become complex due to issues like distributed orchestration, monitoring, and delivery guarantees. As an alternative, Ivan recommends using Spark Structured Streaming for its built-in functionality addressing these issues like write-ahead logging and exactly-once processing. He demonstrates how AppsFlyer leverages Spark Structured Streaming to ingest streaming data from Kafka into ClickHouse while monitoring streaming queries and end-to-end latency.
Logging in The World of DevOps
This document summarizes Cecep Mahbub's experience working on logging infrastructure for various companies and discusses best practices for centralized logging. It outlines Cecep's background maintaining logging systems at KMK and BBM, and their current role setting up Bukalapak's logging infrastructure on Google Cloud Platform. Key topics covered include common uses of logs, benefits of a centralized logging server, examples of logging servers and SaaS options, and integrating logs into DevOps processes.
Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...
This document summarizes Jan Svoboda's presentation on building event streaming microservices with Spring Boot and Apache Kafka. Jan discusses his motivation for learning Kafka and microservices development. He outlines a typical journey from strangling monoliths to building event-driven architectures using Kafka. Jan explains how Spring and Kafka integrate well together. The remainder of the presentation demonstrates building a sample application using these techniques and dives into common architecture patterns like event sourcing, CQRS, scaling stateful services, and reverse proxy routing.
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
Kafka and Kafka Streams in the Global Schibsted Data Platform
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
Recommended
GraphQL API on a Serverless Environment
The document discusses using GraphQL to build a serverless API for a mobile app that detects construction errors. Key points include:
1. GraphQL provides an efficient way to fetch data from microservices in a single roundtrip and allows flexible field selection.
2. The previous monolithic REST API was split into microservices for data fetching, state management, and serving the API to improve performance and scalability.
3. Serverless is a good fit for the stateless GraphQL architecture as resolvers can trigger other serverless functions and the API can scale up and down easily.
Lambda architecture
The document describes AppsFlyer's lambda architecture for real-time analytics. It ingests raw event data from mobile apps into Kafka in real-time, processes the data using Apache Spark for reports and dashboards, and stores the data in S3 for batch processing. It monitors the system using Grafana, Kibana, and end-to-end tests to ensure reliable reporting on metrics like lifetime value, retention, and cohorts.
2018 07-11 - kafka integration patterns
During this meet up we will see different approach to integrate Apache Kafka in an enterprise architrecture
Productionize spark structured streaming
Ivan Kosianenko discusses how AppsFlyer has evolved its data pipeline from using MemSQL and Druid to handle 12 billion events daily to now using ClickHouse and Druid to handle 38 billion events daily. He then explains how building a custom data streaming solution with Clojure can become complex due to issues like distributed orchestration, monitoring, and delivery guarantees. As an alternative, Ivan recommends using Spark Structured Streaming for its built-in functionality addressing these issues like write-ahead logging and exactly-once processing. He demonstrates how AppsFlyer leverages Spark Structured Streaming to ingest streaming data from Kafka into ClickHouse while monitoring streaming queries and end-to-end latency.
Logging in The World of DevOps
This document summarizes Cecep Mahbub's experience working on logging infrastructure for various companies and discusses best practices for centralized logging. It outlines Cecep's background maintaining logging systems at KMK and BBM, and their current role setting up Bukalapak's logging infrastructure on Google Cloud Platform. Key topics covered include common uses of logs, benefits of a centralized logging server, examples of logging servers and SaaS options, and integrating logs into DevOps processes.
Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...
This document summarizes Jan Svoboda's presentation on building event streaming microservices with Spring Boot and Apache Kafka. Jan discusses his motivation for learning Kafka and microservices development. He outlines a typical journey from strangling monoliths to building event-driven architectures using Kafka. Jan explains how Spring and Kafka integrate well together. The remainder of the presentation demonstrates building a sample application using these techniques and dives into common architecture patterns like event sourcing, CQRS, scaling stateful services, and reverse proxy routing.
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
Kafka and Kafka Streams in the Global Schibsted Data Platform
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
Putting the Spark into Functional Fashion Tech Analystics
Metail uses Apache Spark and a functional programming approach to process and analyze data from its fashion recommendation application. It collects data through various pipelines to understand user journeys and optimize business processes like photography. Metail's data pipeline is influenced by functional paradigms like immutability and uses Spark on AWS to operate on datasets in a distributed, scalable manner. The presentation demonstrated Metail's use of Clojure, Spark, and AWS services to build a functional data pipeline for analytics purposes.
Apache Flink
This presentation gives an overview of the Apache Flink project. It explains Flink in terms of its architecture, use cases and the manner in which it works.
Links for further information and connecting
http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
https://nz.linkedin.com/pub/mike-frampton/20/630/385
https://open-source-systems.blogspot.com/
Elastic Stack Basic - All The Capabilities in 6.3!
In Elastic Stack 6.3 we have taken a bold step and opened the X-Pack code for viewing, comment, and bug tracking. In addition, we’ve now included all of the FREE X-Pack Features in the default 6.3 distribution. Learn how you can benefit from these changes to accelerate your projects and get started with the Elastic Stack today!
Session 03 data_migration_at_scale_by_sameer
In this session Sameer discussed his strategy of migrating database of aroung 100TB size from on-premise servers to AWS RDS (Mysql database).
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...
If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.
Building scalable data with kafka and spark
Slides for the talk given on 20-07-2019 at Nairobi JVM. It was a talk about building data pipelines with Apache Kafka as a message broker or enterprise bus and Apache spark as a distributed computing engine that enables processing of large volume of data efficiently.
Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft
The document discusses Kafka connectors for Cosmos DB that allow for seamless integration between the two services without requiring complex application code. It provides an overview of Kafka Connect and connectors, use cases for integrating Cosmos DB and Kafka, and the architecture of source and sink connectors that can read from and write to Cosmos DB and Kafka. It also previews a demo of the connectors and suggests ways to take integration further.
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.
Improving Mobile Payments With Real time Spark
This document discusses improving mobile payments by implementing real-time analytics using Apache Spark streaming. The initial solution involved batch processing of mobile payment event data. The new solution uses Spark streaming to analyze data in real-time from sources like Amazon Kinesis. This allows for automatic alerts and a closed feedback loop. Challenges in moving from batch to streaming processing and optimizing the Python code are also covered.
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
This document summarizes Riot Games' journey to establishing a global Kafka aggregation platform. It describes how Riot previously had complex, siloed architectures with operational data challenges. It then outlines how Riot transitioned to using Kafka for scalable, easy aggregation across regions. The document details Riot's current regional collection and global aggregation approach using Kafka Connect. It also discusses challenges encountered and solutions implemented around areas like message replication, partition reassignment, and low latency needs. Finally, it previews Riot's plans for real-time analytics, bi-directional messaging, streaming metrics, and handling of personal information with their Kafka platform.
Running R on AWS Lambda by Ana-Maria Niculescu
This document discusses running R code on Amazon Lambda. It presents a solution using custom R runtimes provided by Bakdata to deploy R packages and functions to Lambda. Shell scripts automate the deployment process, creating the necessary AWS infrastructure including VPC, S3, API Gateway and Lambda function. A simple example R function is shown running on Lambda. While Lambda has limitations like memory and timeout restrictions, it is suitable for running modularized R code on an as-needed basis without managing servers.
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag.
Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information.
For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well.
You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being.
We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand.
In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.
Master thesis
My master thesis presentation: "Avoiding CRUD operations lock-in in NoSQL databases: extension of the CPIM library"
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
This document discusses using fast data and stream processing with Kafka to improve supply chain planning. It describes problems with traditional sequential and batch-oriented systems and proposes using Kafka streams to process continuous data in real-time. Examples are given of using Kafka streams for message translation, splitting messages, aggregation, and integrating data from multiple topics to generate reports. Challenges with testing integration points and data quality are also mentioned.
Librecon 2016 bilbao: kappa architecture IoT of the cars
This document discusses Kappa Architecture, which is a data processing architecture using Apache Kafka as the centralized data backbone. It summarizes Jay Kreps' original definition of Kappa Architecture and describes how it provides a more flexible way to do real-time and batch processing compared to the traditional Lambda Architecture. The document also provides an example of how a company implemented Kappa Architecture using Kafka and Spark Streaming to monitor car telemetry data from many vehicles and support flexible real-time queries over that data.
Building a Self-Service Hadoop Platform at Linkedin with Azkaban
LinkedIn developed the Azkaban workflow manager to schedule and run Hadoop jobs. They created versions 1.0, 2.0, and 2.5 of Azkaban, adding new features like plug-ins, authentication, and a redesigned UI. Azkaban is now used by over 1,000 LinkedIn users to run 2,500 workflows and 30,000 jobs daily across multiple Hadoop clusters.
A Tool For Big Data Analysis using Apache Spark
Building a REST based generic big data tool for solving business problems across verticles using Apache Spark
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
This document summarizes Haitao Wang's experience working on streaming platforms at Alibaba and Microsoft. It describes Alibaba's data infrastructure challenges in handling large volumes of streaming data. It introduces Alibaba Blink, a distribution of Apache Flink that was developed to meet Alibaba's scale needs. Blink has achieved unprecedented throughput of 472 million events per second with latency of 10s of milliseconds. The document outlines improvements made in Blink's runtime, declarative SQL support, and use cases at Alibaba including real-time A/B testing, search index building, and online machine learning.
From Config Management Sucks to #cfgmgmtlove
This document summarizes Kris Buytaert's talk on the evolution of config management from the 1990s to present day. It discusses early approaches like manual installations and system imaging tools. It then covers the rise of infrastructure as code using tools like Puppet, Chef, and Docker. The talk addresses challenges like getting operations teams to adopt new methods and complexities that can arise from dependencies and modules. It promotes treating infrastructure like code with development practices for versioning, testing, and continuous integration/deployment.
Mesoscon 2015
This document discusses Mesos implementation at Bloomberg. It notes that Bloomberg runs one of the largest private networks and was an early adopter of cloud computing and software as a service. It describes how Mesos is used to provide elastic data processing and analytics across Bloomberg's 3000+ developers. Key parts of the Mesos implementation include using Marathon for application deployment, Kafka for processing topologies, and ELK/InfluxDB/Grafana for centralized monitoring. The document also discusses lessons learned around access control, Zookeeper protection, and cleaning up sandbox data.
More Related Content
What's hot
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
Putting the Spark into Functional Fashion Tech Analystics
Metail uses Apache Spark and a functional programming approach to process and analyze data from its fashion recommendation application. It collects data through various pipelines to understand user journeys and optimize business processes like photography. Metail's data pipeline is influenced by functional paradigms like immutability and uses Spark on AWS to operate on datasets in a distributed, scalable manner. The presentation demonstrated Metail's use of Clojure, Spark, and AWS services to build a functional data pipeline for analytics purposes.
Apache Flink
This presentation gives an overview of the Apache Flink project. It explains Flink in terms of its architecture, use cases and the manner in which it works.
Links for further information and connecting
http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
https://nz.linkedin.com/pub/mike-frampton/20/630/385
https://open-source-systems.blogspot.com/
Elastic Stack Basic - All The Capabilities in 6.3!
In Elastic Stack 6.3 we have taken a bold step and opened the X-Pack code for viewing, comment, and bug tracking. In addition, we’ve now included all of the FREE X-Pack Features in the default 6.3 distribution. Learn how you can benefit from these changes to accelerate your projects and get started with the Elastic Stack today!
Session 03 data_migration_at_scale_by_sameer
In this session Sameer discussed his strategy of migrating database of aroung 100TB size from on-premise servers to AWS RDS (Mysql database).
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...
If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.
Building scalable data with kafka and spark
Slides for the talk given on 20-07-2019 at Nairobi JVM. It was a talk about building data pipelines with Apache Kafka as a message broker or enterprise bus and Apache spark as a distributed computing engine that enables processing of large volume of data efficiently.
Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft
The document discusses Kafka connectors for Cosmos DB that allow for seamless integration between the two services without requiring complex application code. It provides an overview of Kafka Connect and connectors, use cases for integrating Cosmos DB and Kafka, and the architecture of source and sink connectors that can read from and write to Cosmos DB and Kafka. It also previews a demo of the connectors and suggests ways to take integration further.
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.
Improving Mobile Payments With Real time Spark
This document discusses improving mobile payments by implementing real-time analytics using Apache Spark streaming. The initial solution involved batch processing of mobile payment event data. The new solution uses Spark streaming to analyze data in real-time from sources like Amazon Kinesis. This allows for automatic alerts and a closed feedback loop. Challenges in moving from batch to streaming processing and optimizing the Python code are also covered.
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
This document summarizes Riot Games' journey to establishing a global Kafka aggregation platform. It describes how Riot previously had complex, siloed architectures with operational data challenges. It then outlines how Riot transitioned to using Kafka for scalable, easy aggregation across regions. The document details Riot's current regional collection and global aggregation approach using Kafka Connect. It also discusses challenges encountered and solutions implemented around areas like message replication, partition reassignment, and low latency needs. Finally, it previews Riot's plans for real-time analytics, bi-directional messaging, streaming metrics, and handling of personal information with their Kafka platform.
Running R on AWS Lambda by Ana-Maria Niculescu
This document discusses running R code on Amazon Lambda. It presents a solution using custom R runtimes provided by Bakdata to deploy R packages and functions to Lambda. Shell scripts automate the deployment process, creating the necessary AWS infrastructure including VPC, S3, API Gateway and Lambda function. A simple example R function is shown running on Lambda. While Lambda has limitations like memory and timeout restrictions, it is suitable for running modularized R code on an as-needed basis without managing servers.
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag.
Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information.
For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well.
You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being.
We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand.
In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.
Master thesis
My master thesis presentation: "Avoiding CRUD operations lock-in in NoSQL databases: extension of the CPIM library"
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
This document discusses using fast data and stream processing with Kafka to improve supply chain planning. It describes problems with traditional sequential and batch-oriented systems and proposes using Kafka streams to process continuous data in real-time. Examples are given of using Kafka streams for message translation, splitting messages, aggregation, and integrating data from multiple topics to generate reports. Challenges with testing integration points and data quality are also mentioned.
Librecon 2016 bilbao: kappa architecture IoT of the cars
This document discusses Kappa Architecture, which is a data processing architecture using Apache Kafka as the centralized data backbone. It summarizes Jay Kreps' original definition of Kappa Architecture and describes how it provides a more flexible way to do real-time and batch processing compared to the traditional Lambda Architecture. The document also provides an example of how a company implemented Kappa Architecture using Kafka and Spark Streaming to monitor car telemetry data from many vehicles and support flexible real-time queries over that data.
Building a Self-Service Hadoop Platform at Linkedin with Azkaban
LinkedIn developed the Azkaban workflow manager to schedule and run Hadoop jobs. They created versions 1.0, 2.0, and 2.5 of Azkaban, adding new features like plug-ins, authentication, and a redesigned UI. Azkaban is now used by over 1,000 LinkedIn users to run 2,500 workflows and 30,000 jobs daily across multiple Hadoop clusters.
A Tool For Big Data Analysis using Apache Spark
Building a REST based generic big data tool for solving business problems across verticles using Apache Spark
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
This document summarizes Haitao Wang's experience working on streaming platforms at Alibaba and Microsoft. It describes Alibaba's data infrastructure challenges in handling large volumes of streaming data. It introduces Alibaba Blink, a distribution of Apache Flink that was developed to meet Alibaba's scale needs. Blink has achieved unprecedented throughput of 472 million events per second with latency of 10s of milliseconds. The document outlines improvements made in Blink's runtime, declarative SQL support, and use cases at Alibaba including real-time A/B testing, search index building, and online machine learning.
What's hot (20)
Real-Time Dynamic Data Export Using the Kafka Ecosystem
Real-Time Dynamic Data Export Using the Kafka Ecosystem
Putting the Spark into Functional Fashion Tech Analystics
Putting the Spark into Functional Fashion Tech Analystics
Elastic Stack Basic - All The Capabilities in 6.3!
Elastic Stack Basic - All The Capabilities in 6.3!
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...
Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...
Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft
Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
Kafka Summit SF 2017 - Fast Data in Supply Chain Planning
Librecon 2016 bilbao: kappa architecture IoT of the cars
Librecon 2016 bilbao: kappa architecture IoT of the cars
Building a Self-Service Hadoop Platform at Linkedin with Azkaban
Building a Self-Service Hadoop Platform at Linkedin with Azkaban
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Viewers also liked
From Config Management Sucks to #cfgmgmtlove
This document summarizes Kris Buytaert's talk on the evolution of config management from the 1990s to present day. It discusses early approaches like manual installations and system imaging tools. It then covers the rise of infrastructure as code using tools like Puppet, Chef, and Docker. The talk addresses challenges like getting operations teams to adopt new methods and complexities that can arise from dependencies and modules. It promotes treating infrastructure like code with development practices for versioning, testing, and continuous integration/deployment.
Mesoscon 2015
This document discusses Mesos implementation at Bloomberg. It notes that Bloomberg runs one of the largest private networks and was an early adopter of cloud computing and software as a service. It describes how Mesos is used to provide elastic data processing and analytics across Bloomberg's 3000+ developers. Key parts of the Mesos implementation include using Marathon for application deployment, Kafka for processing topologies, and ELK/InfluxDB/Grafana for centralized monitoring. The document also discusses lessons learned around access control, Zookeeper protection, and cleaning up sandbox data.
Building Product from ground up using Open Source Technologies
This is a presentation given by Amit Goel, Founder & CEO, Patterbuzz at Open Source India 2014 conference in Bangalore. . He spoke about the story of Patterbuzz and its journey of 2 years since its beginning to explain the product development with the pace of a start up. It covered all the necessary evils and building the technology form scratch using open source softwares and available technologies and how to leverage its advantage.
Patterbuzz builds on the core philosophy of unbundling content from the publishing industry and making it available to consumers in a device reading friendly format with pay as you go model. For example: users can buy single articles if they like rather than purchasing the complete magazine.
Currently, Patterbuzz enables magazines from India / Indian publishing industry. Most popular magazines available are Popular Science, Mother & Baby, Electronics for you, open source for you, computer active, Gfiles, Power Politics, Hard News, Car India, Bike India etc...
Data science team, a practice to setup
This document outlines roles and processes for building an effective data science team. It recommends that teams include people with skills in machine learning, data engineering, analytics, mathematics, and visualization. The document describes the CRISP-DM process for data science projects and a lambda architecture for continuous analysis of big data using a pipeline of jobs, messaging queues, and data processing applications. It emphasizes building cross-disciplinary teams that can work together to ask important questions and create useful data products.
Send that (damn) elevator down !
This document contains notes from a leadership summit talk given by Ekta Grover. The main topics discussed include Maslow's hierarchy of needs, strategically creating opportunities, focusing on execution over ideas, growing strong teams, managing priorities, having a global perspective, building social capital, failing forward with resilience, and cultivating confidence through competence.
Introducing ELK
This document introduces the (B)ELK stack, which consists of Beats, Elasticsearch, Logstash, and Kibana. It describes each component and how they work together. Beats are lightweight data shippers that collect data from logs and systems. Logstash processes and transforms data from inputs like Beats. Elasticsearch stores and indexes the data. Kibana provides visualization and analytics capabilities. The document provides examples of using each tool and tips for working with the ELK stack.
Experiences in ELK with D3.js for Large Log Analysis and Visualization
This document discusses experiences using the ELK stack (Elasticsearch, Logstash, Kibana) and D3.js for large log analysis and visualization. It begins with an overview of network traffic logging at Kasetsart University, which generates over 30 terabytes of log data per day. It then demonstrates setting up an ELK testbed to index these logs in real-time for fast search and exploration in Kibana. Finally, it shows how D3.js can be used to create dynamic, real-time visualizations of the logged data.
Monitoring with ElasticSearch
This document discusses using the ELK stack (Elasticsearch, Logstash, Kibana) for log collection, aggregation, and visualization. It describes how to use Logstash to collect logs from various sources, apply filters like Grok to parse logs into structured data, and output to Elasticsearch for analysis and long-term storage. Metrics can also be sent to Graphite for long-term storage and visualization. The ELK stack provides a way for both developers and operations teams to analyze logs.
Elastic Stackにハマった話
This document summarizes Kazuhiro Kosaka's talk about being stuck using the Elastic Stack. Some key points included being stuck with data loss between Filebeat and Logstash due to Logstash's lack of a persistent queue in versions below 5.1. Kosaka was also stuck with Elasticsearch writing issues due to JVM memory pressure on a single m3.medium instance that was resolved by upgrading to 3 m3.large instances. Restrictions of the Amazon Elasticsearch Service were also discussed, as well as next steps of migrating to Elastic Stack 5.x and considering Elastic Cloud or a private cluster.
Monitoring using Open source technologies
In this presentation, we have talked about building an end to end solution for monitoring using open source technologies such as Graphite, Grafana, Apache Spark, Apache Kafka, OpenTSDB, Prometheus, etc.
The Rise of Real Time
Multi-tenancy allows a single instance of software to serve multiple customers. It is an architecture where a single instance of the software runs on a server and serves multiple tenants. This allows for efficient use of computing resources and reduces maintenance costs as updates only need to be applied to a single code base.
Keystone - Leverage Big Data 2016
The document discusses Netflix's real-time data infrastructure called Keystone, which processes over 1 trillion events per day. It describes how Netflix built Keystone over 1 year to handle the massive scale of data from its growing user base and content catalog. Key points include:
- Keystone uses Kafka as its main data backbone to ingest, process and move data across clusters. It has over 3000 instances across 24 Kafka clusters.
- Keystone processes an average of 700 billion unique events per day and 1 trillion total events daily across 1.4 petabytes of data.
- The system was designed based on Netflix's culture of freedom and responsibility and best practices of full automation, self-healing and minimizing single points
How Did BuzzFeed Harvest One Million Email Subscribers?
How exactly did Buzzfeed managed to accumulate 1 million subscribers in 2014 alone? We have unpacked 20 steps the media news company took to reach this staggering number which looks set to be far higher in 2015.
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Slides for our solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
Kafka is Uber's real-time data infrastructure that powers many of its core systems and products. It processes both real-time and batch data from many different sources and consumers across Uber's distributed systems. Over time, Uber has improved Kafka to handle larger volumes of data across more data centers and languages. Looking forward, Uber envisions Kafka enabling even more dynamic and real-time systems through continued innovation.
Scaling an ELK stack at bol.com
A presentation about the deployment of an ELK stack at bol.com
At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure.
The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling.
These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.
Building an IoT Kafka Pipeline in Under 5 Minutes
This document discusses building an IoT Kafka pipeline using MemSQL in under 5 minutes. It begins with an overview of IoT, Kafka, and operational data warehouses. It then discusses MemSQL and how it functions as an operational data warehouse by continuously loading and querying data in real-time. The document demonstrates launching a MemSQL cluster, creating schemas and pipelines to ingest, transform, persist and analyze IoT data from Kafka. It emphasizes MemSQL's ability to handle different data types and scales from IoT at high throughput with low latency.
Building a Modern Data Architecture on AWS - Webinar
Amazon Web Services gives you fast access to flexible and low cost IT resources, so you can rapidly scale and build virtually any big data application including data warehousing, clickstream analytics, fraud detection, recommendation engines, event-driven ETL, serverless computing, and internet-of-things processing regardless of volume, velocity, and variety of data.
https://aws.amazon.com/webinars/anz-webinar-series/
ELK at LinkedIn - Kafka, scaling, lessons learned
How LinkedIn uses and scale ELK clusters using Kafka. Lessons learned.
There are useful notes in the PowerPoint version.
BDX 2016- Monal daxini @ Netflix
Netflix built a scalable event streaming pipeline called Keystone to replace their legacy system, ingesting over 1 trillion events per day. Keystone utilizes Apache Kafka, Samza, and Spark Streaming to reliably process and route streaming data at massive scale across Netflix's infrastructure. The success of Keystone was due in large part to Netflix's culture of freedom and responsibility that empowered a small team to build and operate the new system without separate management oversight.
Viewers also liked (20)
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source Technologies
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
How Did BuzzFeed Harvest One Million Email Subscribers?
How Did BuzzFeed Harvest One Million Email Subscribers?
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Real-Time Log Analysis with Apache Mesos, Kafka and Cassandra
Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
Building a Modern Data Architecture on AWS - Webinar
Building a Modern Data Architecture on AWS - Webinar
Similar to Rootconf
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
At NMC (Nielsen Marketing Cloud) we provide our customers (marketers and publishers) real-time analytics tools to profile their target audiences. To achieve that, we need to ingest billions of events per day into our big data stores, and we need to do it in a scalable yet cost-efficient manner.
In this session, we will discuss how we continuously transform our data infrastructure to support these goals. Specifically, we will review how we went from CSV files and standalone Java applications all the way to multiple Kafka and Spark clusters, performing a mixture of Streaming and Batch ETLs, and supporting 10x data growth We will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty). We will present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services’ costs. Topics include:
Kafka and Spark Streaming for stateless and stateful use-cases
Spark Structured Streaming as a possible alternative
Combining Spark Streaming with batch ETLs
”Streaming” over Data Lake using Kafka
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.
Introduction to apache kafka
Intro to Apache Kafka I gave at the Big Data Meetup in Geneva in June 2016. Covers the basics and gets into some more advanced topics. Includes demo and source code to write clients and unit tests in Java (GitHub repo on the last slides).
Netflix Data Pipeline With Kafka
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.
Netflix Data Pipeline With Kafka
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.
Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency
theScore is one of the leading sports news and media platforms in North America. With the recent growth of sports betting, theScore is diving in head-first. Province and state regulations often enforce strict data-locality requirements often prohibiting the use of centralized hosted data storage solutions.
In this talk, we'll learn how theScore built Datadex, a geographically distributed system for low-latency queries and real-time updates. We'll touch on the architecture of the system (Aggregator Leaf Tailer), the underlying technologies (RocksDB, Kafka, and the JVM), as well as some nitty-gritty tips for optimizing the JVM and RocksDB for high throughput.
Activity feeds (and more) at mate1
The document provides an overview of Mate1, an online dating company, and their transition to using Kafka and Cassandra for their activity feed and data storage. Some key points:
- Mate1 started in 2003 and has grown to around 40 employees, with a 13 person engineering team. They own their own hardware.
- They previously used a "fan-out on read" approach where activity data was stored across different databases and querying them to generate feeds was slow.
- They transitioned to a "fan-out on write" approach using Kafka to queue activity events and Cassandra to store aggregated user feeds and counters. This improved performance and scalability.
- Events are published to Kafka topics and
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
At NMC (Nielsen Marketing Cloud) we provide our customers (marketers and publishers) real-time analytics tools to profile their target audiences.
To achieve that, we need to ingest billions of events per day into our big data stores, and we need to do it in a scalable yet cost-efficient manner.
In this session, we will discuss how we continuously transform our data infrastructure to support these goals.
Specifically, we will review how we went from CSV files and standalone Java applications all the way to multiple Kafka and Spark clusters, performing a mixture of Streaming and Batch ETLs, and supporting 10x data growth.
We will share our experience as early-adopters of Spark Streaming and Spark Structured Streaming, and how we overcame technical barriers (and there were plenty...).
We will present a rather unique solution of using Kafka to imitate streaming over our Data Lake, while significantly reducing our cloud services' costs.
Topics include :
* Kafka and Spark Streaming for stateless and stateful use-cases
* Spark Structured Streaming as a possible alternative
* Combining Spark Streaming with batch ETLs
* "Streaming" over Data Lake using Kafka
Laskar: High-Velocity GraphQL & Lambda-based Software Development Model
Sale Stock Engineering, represented by Garindra Prahandono, presents "High-Velocity GraphQL & Lambda-based Software Development Model" in BandungJS event on May 14th, 2018.
Apache Spark vs Apache Flink
This document compares Apache Spark and Apache Flink. Both are open-source platforms for distributed data processing. Spark was created in 2009 at UC Berkeley and donated to the Apache Foundation in 2013. It uses resilient distributed datasets (RDDs) and lazy evaluation. Flink was started in 2010 as a collaboration between universities in Germany and became an Apache project in 2014. It uses cyclic data flows and supports both batch and stream processing. While Spark is currently more mature with more components and community support, Flink claims to be faster for stream and batch processing. Overall, both platforms continue to evolve and improve.
Apache Kafka - Martin Podval
Apache Kafka is a distributed messaging system that allows for publishing and subscribing to streams of records, known as topics, in a fault-tolerant and scalable way. It is used for building real-time data pipelines and streaming apps. Producers write data to topics which are committed to disks across partitions and replicated for fault tolerance. Consumers read data from topics in a decoupled manner based on offsets. Kafka can process streaming data in real-time and at large volumes with low latency and high throughput.
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.
Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.
How Uber scaled its Real Time Infrastructure to Trillion events per day
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.
Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Talk on Netflix Keystone by Peter Bakas at SF Data Engineering Meetup on 2/23/2016.
Topics covered:
- Architectural design and principles for Keystone
- Technologies that Keystone is leveraging
- Best practices
http://www.meetup.com/SF-Data-Engineering/events/228293610/
Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter
Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 200M messages per second. In this talk, we will share the learning and challenges in our journey moving to Apache Kafka.
Timothy Spann: Apache Pulsar for ML
Timothy Spann: Apache Pulsar for ML
Data Science Online Camp 2023 Winter
Website: https://dscamp.org
Youtube: https://www.youtube.com/channel/UCeHtPZ_ZLZ-nHFMUCXY81RQ
FB: https://www.facebook.com/people/Data-Science-Camp/100064240830422/
Event driven architectures with Kinesis
Mark Harrison presented on using Amazon Kinesis for event-driven microservices architectures. He discussed the limitations of traditional monolithic and microservice architectures, and how Kinesis can help address issues like tight coupling, high latency, and lack of event broadcasting. Key concepts around Kinesis included streams, shards, producers, consumers, and constraints like throughput limits. Harrison demonstrated a custom Scala/Akka client for Kinesis that provides asynchronous, non-blocking producers and consumers with features like throttling and checkpointing. Performance tests showed throughput scales linearly with additional shards. In closing, Harrison invited the audience to learn more about open source and job opportunities with Weight Watchers.
The Fn Project: A Quick Introduction (December 2017)
A quick introduction to the Fn Project (https://fnproject.io), an open source serverless/functions platform by Oracle.
Centralized Logging System Using ELK Stack
Centralized Logging System using ELK Stack
The document discusses setting up a centralized logging system (CLS) using the ELK stack. The ELK stack consists of Logstash to capture and filter logs, Elasticsearch to index and store logs, and Kibana to visualize logs. Logstash agents on each server ship logs to Logstash, which filters and sends logs to Elasticsearch for indexing. Kibana queries Elasticsearch and presents logs through interactive dashboards. A CLS provides benefits like log analysis, auditing, compliance, and a single point of control. The ELK stack is an open-source solution that is scalable, customizable, and integrates with other tools.
2015 03-16-elk at-bsides
Jeremy Cohoe presented on using the ELK (Elasticsearch, Logstash, Kibana) stack for log analysis. He began with an overview of what ELK is and its components - Logstash parses logs, Elasticsearch is the database, and Kibana provides the GUI. Cohoe then demonstrated using ELK to monitor 802.11 client probes with a software defined radio and parse Flex pager signals. Finally, he discussed implementing ELK in production for a Linux central syslog system, including scaling out with Redis, common plugins, and cluster monitoring tools.
Similar to Rootconf (20)
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency
Aggregator Leaf Tailer: Bringing Data to Your Users with Ultra Low Latency
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
Laskar: High-Velocity GraphQL & Lambda-based Software Development Model
Laskar: High-Velocity GraphQL & Lambda-based Software Development Model
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
How Uber scaled its Real Time Infrastructure to Trillion events per day
How Uber scaled its Real Time Infrastructure to Trillion events per day
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter
Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter
The Fn Project: A Quick Introduction (December 2017)
The Fn Project: A Quick Introduction (December 2017)
Recently uploaded
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Recently uploaded (20)
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Rootconf
- 1. How LinkedIn is leveraging kafka for log aggregation and monitoring Akbar KM, Muzammil Rehman SRE Linkedin
- 2. Agenda ● What is Pub Sub? ● Kafka basics ● Kafka for monitoring ● inGraph’s ● ELK at Linkedin
- 3. What is pub sub ? Producer Consumer Producer Consumer Topic 1 Topic 2 Topic 3 subscribe publish(topic, msg) Publish subscribe system msg msg
- 4. Kafka
- 6. inGraph ● inGraphs is a visualization frontend for a linkedin’s metrics ● Agent is a library used by all java application ● Data is transported over kafka ● Collector consume and write to rrd ● Features ○ Rest based ○ Self service ○ Little over 500,000 metrics collected per minute ○ Each RRD is roughly 815k. Each RRD is written to two collectors to maintain data integrity ○ We currently have 1.4TB of SSD in production colo
- 7. ELK at linkedin ● ELK (Elasticsearch, Logstash, Kibana) ○ E -> Near Real-Time Search & Analysis ○ L -> Plumbing and Glue for all your data ○ K -> Line graphs, pie charts, dashboard ● Logstash kafka plugin consumes logs and end up written into elasticsearch
- 8. Why Kafka 1. Near real time delivery 2. Supports multiple consumers groups aka queuing 3. No overhead on clients that comes up logstash agent 4. Kafka scales horizontally 5. Supports REST out of the box 6. In house support