Ceilo componentization diagrams
•Download as PPTX, PDF•
1 like•478 views
Ceilometer is an OpenStack telemetry service that collects measurements of OpenStack cloud usage. It consists of central and compute agents that collect metrics and event data from OpenStack services and push it to a central agent. The central agent then sends the data to collectors that store it in a database and make it available via the Ceilometer APIs for metrics, alarms, and events.
Report
Share
Report
Share
Recommended
Ceilosca
See what happens when you overlay Ceilometer functionalities on top of a scalable solution like Monasca ... This is Ceilosca
Blueprint: Kafka Publisher of Ceilometer
The blueprint of Kafka publisher of Ceilometer project.
Launchpad:
https://blueprints.launchpad.net/ceilometer/+spec/kafka-publisher
Enforcing Application SLA with Congress and Monasca
This document discusses using Congress and Monasca to enforce service level agreements (SLAs) for applications and infrastructure. It proposes using Congress policies to define SLA thresholds that trigger Monasca alarms, which then notify relevant parties. Examples are given for notifying operators if servers are underutilized, and evacuating critical business applications from unhealthy hosts. The current state integrates Monasca metrics with Congress, with future work including developing a Monasca alarm datasource and converting policies to alarms.
Ceilometer presentation ods havana final - published
Presentation on OpenStack Ceilometer made by Nick Barcet & Julien Danjou during OpenStack Summit Portland - April 2013
OpenStack in Action 4! Nick Barcet & Julien Danjou - From ceilometer to telem...
Paris, 5th December 2013 : OpenStack in Action 4! organized by eNovance, brings together members of the OpenStack community.
Ceilometer presentation ODS Grizzly.pdf
Ceilometer is OpenStack's metering project that collects usage data from OpenStack components like Nova, Glance, and Cinder. It allows for single collection of per-user/tenant usage data across all resources. Ceilometer is designed to be scalable, provide a simple REST API, and be extensible via plugins. DreamHost plans to use Ceilometer to measure usage for billing by customizing it to track instance hours, storage, and bandwidth usage.
Ceilometer + Heat = Alarming
Ceilometer is a tool that collects usage and performance data, while Heat orchestrates complex deployments on top of OpenStack. Heat aims to autoscale its deployments, scaling up when they're running hot and scaling back when idle.
Ceilometer can access decisive data and trigger the appropriate actions in Heat. The result of these two OpenStack projects meeting is value creation in the form of an alarming API in Ceilometer and its consumption in Heat.
Slides presented at the Fall OpenStack Design Summit in Hong Kong
OpenStack Ceilometer
Ceilometer is the metering component of OpenStack that collects usage data from OpenStack systems. It provides a REST API to retrieve this data for purposes like billing, chargeback, and monitoring. Ceilometer uses agents running on compute nodes and central servers to collect data through polling and notifications. The collected data is sent to a collector and stored in a database. An API server then exposes the stored data via a REST interface.
Recommended
Ceilosca
See what happens when you overlay Ceilometer functionalities on top of a scalable solution like Monasca ... This is Ceilosca
Blueprint: Kafka Publisher of Ceilometer
The blueprint of Kafka publisher of Ceilometer project.
Launchpad:
https://blueprints.launchpad.net/ceilometer/+spec/kafka-publisher
Enforcing Application SLA with Congress and Monasca
This document discusses using Congress and Monasca to enforce service level agreements (SLAs) for applications and infrastructure. It proposes using Congress policies to define SLA thresholds that trigger Monasca alarms, which then notify relevant parties. Examples are given for notifying operators if servers are underutilized, and evacuating critical business applications from unhealthy hosts. The current state integrates Monasca metrics with Congress, with future work including developing a Monasca alarm datasource and converting policies to alarms.
Ceilometer presentation ods havana final - published
Presentation on OpenStack Ceilometer made by Nick Barcet & Julien Danjou during OpenStack Summit Portland - April 2013
OpenStack in Action 4! Nick Barcet & Julien Danjou - From ceilometer to telem...
Paris, 5th December 2013 : OpenStack in Action 4! organized by eNovance, brings together members of the OpenStack community.
Ceilometer presentation ODS Grizzly.pdf
Ceilometer is OpenStack's metering project that collects usage data from OpenStack components like Nova, Glance, and Cinder. It allows for single collection of per-user/tenant usage data across all resources. Ceilometer is designed to be scalable, provide a simple REST API, and be extensible via plugins. DreamHost plans to use Ceilometer to measure usage for billing by customizing it to track instance hours, storage, and bandwidth usage.
Ceilometer + Heat = Alarming
Ceilometer is a tool that collects usage and performance data, while Heat orchestrates complex deployments on top of OpenStack. Heat aims to autoscale its deployments, scaling up when they're running hot and scaling back when idle.
Ceilometer can access decisive data and trigger the appropriate actions in Heat. The result of these two OpenStack projects meeting is value creation in the form of an alarming API in Ceilometer and its consumption in Heat.
Slides presented at the Fall OpenStack Design Summit in Hong Kong
OpenStack Ceilometer
Ceilometer is the metering component of OpenStack that collects usage data from OpenStack systems. It provides a REST API to retrieve this data for purposes like billing, chargeback, and monitoring. Ceilometer uses agents running on compute nodes and central servers to collect data through polling and notifications. The collected data is sent to a collector and stored in a database. An API server then exposes the stored data via a REST interface.
From Ceilometer to Telemetry: not so alarming!
Presentation of Ceilometer (OpenStack Telemetry) new features in OpenStack Havana and a look at the features coming in IceHouse. Joint presentation done with Julien Danjou at the OpenStack In Action 4 (Dec 5th 2013)
Telemetry Updates - Juno Edition
Eoghan Glynn, Ceilometer Project PTL, outlines the changes made in the Icehouse release as well as upcoming updates for Juno.
Learn more about Ceilometer here: https://wiki.openstack.org/wiki/Ceilometer
Container Monitoring Best Practices Using AWS and InfluxData by Gunnar Aasen
In this InfluxDays NYC 2019 talk by Gunnar Aasen (Manager of Partner Engineering at InfluxData), you will get an overview of the AWS Container Monitoring Stack as well as how you can use InfluxDB on AWS for container monitoring. This session will include a demo of the solution.
Stabilizing the Jenga tower: Scaling out Ceilometer
This document discusses the evolution of Ceilometer, an OpenStack component for collecting and managing telemetry data. Ceilometer collects metrics from OpenStack services and stores them for later retrieval and analysis. The architecture has changed over time to improve scalability, with a shift to active-active workload partitioning in Kilo and integration with Gnocchi in Liberty for more efficient storage of time-series metric data. Best practices are provided around data collection, storage, and deployment scenarios to help operators reliably collect and manage telemetry at scale.
goto; London: Keeping your Cloud Footprint in Check
Presented on the "Lean" track at goto; London September 17th, 2015. Covers how Netflix manages cloud cost efficiency in light of innovation and reliability drivers.
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Netflix carefully plans holiday capacity to ensure availability while maintaining efficiency. A cross-functional team works with engineering teams to understand capacity needs through bottoms-up and tops-down modeling. They prepare for spikes by coordinating large service migrations to optimize "trough borrowing" and manage failover capacity. The goal is to fulfill their charter of ensuring efficient, available cloud capacity so engineering can prioritize innovation and availability.
PowerStream Demo
At Spark Summit East in New York, we unveil PowerStream, an Internet of Things (IoT) simulation with visualizations and alerts based on real-time data from 2 million sensors across global wind farms.
Running a Massively Parallel Self-serve Distributed Data System At Scale
Nearly any Internet-connected screen is capable of streaming Netflix content. Sitting on top of a cloud-native microservice architecture, the entire ecosystem generates over 1 trillion events every day to feed critical Netflix systems to monitor service health, to detect fraudulent behaviors, and to improve customer experience.
Keystone is the critical piece of Netflix backend infrastructure to ensure massive amount of events are processed in near real time, reliably, at scale, and in face of failures in a cloud-native microservices environment.
Turns out, such an embarrassingly parallel stream processing system is not embarrassingly easy to develop and operate, especially given the challenges of unpredictable failures in a cloud-native environment, self-serve multi-tenancy support, and assumptions of maintaining extremely high development/operation agility.
This talk will shed light on how we built an elastic, resilient, reactive, and self-healing distributed system in the cloud. Zhenzhong will present * High-level cloud-native microservice based Keystone architecture. * A deep dive on how we built the system based on ideas such as declarative reconciliation, container based immutable deployment, logical workload isolation, and chaos exercise. * Insights into our operation best practices, such as capacity provisioning, delivery semantics, deployment tradeoffs, backpressure management, etc.
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
The document summarizes Netflix's cloud capacity planning operations. It notes that Netflix tracks over 2,400 applications running on 80 instance types deployed across 30 zones and 60 accounts in AWS. To provide transparency and actionable insights at scale, Netflix built interactive dashboards and focuses on deviations and trends. Its tool called Picsou comprehensively monitors Netflix's current cloud footprint, forecasts future capacity needs, and optimizes on-demand and reserved capacity for cost and reliability. Going forward, Picsou will provide capacity alerts, forecasts, and recommendations to optimize reserved capacity usage.
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Reative UI
This document discusses Reactive UI, a cross-platform MVVM framework built on Reactive Extensions (Rx). It provides an overview of Rx basics like Observables, Observers, and queries. It also describes how Reactive UI uses Rx concepts like ReactiveObject for view models, Bindings for view-model binding, and ReactiveCommand for executing asynchronous commands. The document concludes with a demo of Reactive UI in action and resources for learning more about Rx and Reactive UI.
Monitoring Microservices
By Tom Wilkie, delivered at London Microservices User Group on 2/12/15
The rise of microservice-based applications has had many knock-on effects, not least on the complexity of monitoring your application. Order-of-magnitude increase in the number of moving parts and rate of change of the application require us to reassess traditional monitoring techniques.
In this talk we will discuss some different approaches to monitoring, visualising and tracing containerised, microservices-based applications. We’ll present different techniques to some of the emergent problems, and try not to rant too much.
Virtual Flink Forward 2020: Data driven matchmaking streaming at Hyperconnect...
HyperConnect's 1to1 video matchmaking system is consist of various machine learning techniques to maximize user satisfaction. Our matchmaking system manages large user context containing actions a few seconds ago, and reacts in milliseconds to produce meaningful new results in each user session. It's difficult in traditional way. So, distributed streaming is essential to handle in this cases. Topics include: - Why our team choose Apache Flink in comparison with alternatives - Matchmaking streaming architecture with detail abstraction levels based on Flink operator - Pairwise scoring microservice management with Flink - Stateful matchmaking computation with low latency, fault-tolerance, and scalability - How to manage large-scale events: classifying feature types, collecting with a multi-window stream - Applications: personalization, multi-armed-bandit on stream.
Data & Analytics Forum: Moving Telcos to Real Time
MemSQL is a real-time database that allows users to simultaneously ingest, serve, and analyze streaming data and transactions. It is an in-memory distributed relational database that supports SQL, key-value, documents, and geospatial queries. MemSQL provides real-time analytics capabilities through Streamliner, which allows one-click deployment of Apache Spark for real-time data pipelines and analytics without batch processing. It is available in free community and paid enterprise editions with support and additional features.
Time and ordering in streaming distributed systems
The document discusses time and ordering in distributed streaming systems through three lenses: uniformity of time, arrow of time, and perception of time. It explains that in distributed systems, clocks are not perfectly synchronized and time is not uniform. It also discusses how the direction of time can be blurred by out of order event processing. Finally, it explores how perceptions of time and ordering depend on an observer's frame of reference, and there is no single global ordering of events in a distributed system.
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
In this session, we will look at how Apache Flink can be used to stream anonymized API request and response data from a production environment to make sure staging environments are up-to-date and reflect the most recent features (and bugs) that comprise a service. The talk will also examine how to deal with issues of data retention, throttling, and persistence, finishing with recommendations for how to use these sandbox environments to rapidly prototype and test new features and fixes.
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Speaker: Yupeng Fu, Staff Engineer, Uber
High availability and reliability are important requirements to Uber services, and the services shall tolerate datacenter failures in a region and fail over to another region. In this talk, we will present the active-active Apache Kafka® at Uber and how it facilitates disaster discovery across regions for Uber services. In particular, we will highlight the key components including topic replication, topic aggregation, offsets sync and then walk through several use cases of their disaster recovery strategy using active-active Kafka. Lastly, we will present several interesting challenges and the future work planned.
Yupeng Fu is a staff engineer in Uber Data Org leading the streaming data platform. Previously, he worked at Alluxio and Palantir, building distributed data analysis and storage platforms. Yupeng holds a B.S. and an M.S. from Tsinghua University and did his Ph.D. research on databases at UCSD.
Monitoring as Software Validation
This document discusses monitoring systems and infrastructure. It recommends monitoring everything, including networks, machines, and applications, to learn from infrastructure, anticipate failures, and speed up changes. It presents Graphite as an open-source tool for storing and visualizing real-time time-series data efficiently. Graphite includes components for receiving metrics data, storing data long-term in Whisper, and visualizing data in Graphite Web. It also discusses using StatsD and CollectD to monitor application and system metrics and send them to Graphite. Case studies show how two companies use monitoring to track simulations and the interactions of image processing applications. The document emphasizes that monitoring and testing are both important but serve different purposes.
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
As many industries, banking is undergoing a fundamental change because of the software revolution. No longer are banks competing only on interest rates and having the best traders, these days customer experience and having the best engineers are the focus. In this changing world, banks compete with new start-ups, the so-called Fintechs, and with large platform organisations such as Google, Facebook and Apple. At ING, we believe that staying ahead of the game means changing how we interact with our customers, no longer a traditional model of waiting for the customers to come to the bank through our website or apps, but to actively reach out to the customer with information that is relevant to him or her in order to make their financial life frictionless. Many of these changes are driven by reacting to all events that are relevant to the customer, and using streaming analytics to be able to reach out to the customer in milliseconds after the event occurs. Apache Flink is key for ING to achieve this. This presentation addresses how ING approaches the challenge, the role that Apache Flink plays, and the consequences regulations have on how we work with Open Source in general, and with Apache Flink (and data Artisans) in particular. This keynote takes place at Kino 3.
Storing VMs with Cinder and Ceph RBD.pdf
The document discusses different ways of storing virtual machine images and volumes using Ceph storage. It describes growing from using proprietary storage hardware to using Ceph with its open source, scalable, and fault-tolerant design. It outlines how virtual machine images can be stored in Ceph using the Cinder volume driver and RADOS Block Device (RBD) for efficient cloning and live migration. Client access to volumes is provided through a Linux kernel module while container/VM access is through device drivers.
More Related Content
What's hot
From Ceilometer to Telemetry: not so alarming!
Presentation of Ceilometer (OpenStack Telemetry) new features in OpenStack Havana and a look at the features coming in IceHouse. Joint presentation done with Julien Danjou at the OpenStack In Action 4 (Dec 5th 2013)
Telemetry Updates - Juno Edition
Eoghan Glynn, Ceilometer Project PTL, outlines the changes made in the Icehouse release as well as upcoming updates for Juno.
Learn more about Ceilometer here: https://wiki.openstack.org/wiki/Ceilometer
Container Monitoring Best Practices Using AWS and InfluxData by Gunnar Aasen
In this InfluxDays NYC 2019 talk by Gunnar Aasen (Manager of Partner Engineering at InfluxData), you will get an overview of the AWS Container Monitoring Stack as well as how you can use InfluxDB on AWS for container monitoring. This session will include a demo of the solution.
Stabilizing the Jenga tower: Scaling out Ceilometer
This document discusses the evolution of Ceilometer, an OpenStack component for collecting and managing telemetry data. Ceilometer collects metrics from OpenStack services and stores them for later retrieval and analysis. The architecture has changed over time to improve scalability, with a shift to active-active workload partitioning in Kilo and integration with Gnocchi in Liberty for more efficient storage of time-series metric data. Best practices are provided around data collection, storage, and deployment scenarios to help operators reliably collect and manage telemetry at scale.
goto; London: Keeping your Cloud Footprint in Check
Presented on the "Lean" track at goto; London September 17th, 2015. Covers how Netflix manages cloud cost efficiency in light of innovation and reliability drivers.
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Netflix carefully plans holiday capacity to ensure availability while maintaining efficiency. A cross-functional team works with engineering teams to understand capacity needs through bottoms-up and tops-down modeling. They prepare for spikes by coordinating large service migrations to optimize "trough borrowing" and manage failover capacity. The goal is to fulfill their charter of ensuring efficient, available cloud capacity so engineering can prioritize innovation and availability.
PowerStream Demo
At Spark Summit East in New York, we unveil PowerStream, an Internet of Things (IoT) simulation with visualizations and alerts based on real-time data from 2 million sensors across global wind farms.
Running a Massively Parallel Self-serve Distributed Data System At Scale
Nearly any Internet-connected screen is capable of streaming Netflix content. Sitting on top of a cloud-native microservice architecture, the entire ecosystem generates over 1 trillion events every day to feed critical Netflix systems to monitor service health, to detect fraudulent behaviors, and to improve customer experience.
Keystone is the critical piece of Netflix backend infrastructure to ensure massive amount of events are processed in near real time, reliably, at scale, and in face of failures in a cloud-native microservices environment.
Turns out, such an embarrassingly parallel stream processing system is not embarrassingly easy to develop and operate, especially given the challenges of unpredictable failures in a cloud-native environment, self-serve multi-tenancy support, and assumptions of maintaining extremely high development/operation agility.
This talk will shed light on how we built an elastic, resilient, reactive, and self-healing distributed system in the cloud. Zhenzhong will present * High-level cloud-native microservice based Keystone architecture. * A deep dive on how we built the system based on ideas such as declarative reconciliation, container based immutable deployment, logical workload isolation, and chaos exercise. * Insights into our operation best practices, such as capacity provisioning, delivery semantics, deployment tradeoffs, backpressure management, etc.
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
The document summarizes Netflix's cloud capacity planning operations. It notes that Netflix tracks over 2,400 applications running on 80 instance types deployed across 30 zones and 60 accounts in AWS. To provide transparency and actionable insights at scale, Netflix built interactive dashboards and focuses on deviations and trends. Its tool called Picsou comprehensively monitors Netflix's current cloud footprint, forecasts future capacity needs, and optimizes on-demand and reserved capacity for cost and reliability. Going forward, Picsou will provide capacity alerts, forecasts, and recommendations to optimize reserved capacity usage.
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Reative UI
This document discusses Reactive UI, a cross-platform MVVM framework built on Reactive Extensions (Rx). It provides an overview of Rx basics like Observables, Observers, and queries. It also describes how Reactive UI uses Rx concepts like ReactiveObject for view models, Bindings for view-model binding, and ReactiveCommand for executing asynchronous commands. The document concludes with a demo of Reactive UI in action and resources for learning more about Rx and Reactive UI.
Monitoring Microservices
By Tom Wilkie, delivered at London Microservices User Group on 2/12/15
The rise of microservice-based applications has had many knock-on effects, not least on the complexity of monitoring your application. Order-of-magnitude increase in the number of moving parts and rate of change of the application require us to reassess traditional monitoring techniques.
In this talk we will discuss some different approaches to monitoring, visualising and tracing containerised, microservices-based applications. We’ll present different techniques to some of the emergent problems, and try not to rant too much.
Virtual Flink Forward 2020: Data driven matchmaking streaming at Hyperconnect...
HyperConnect's 1to1 video matchmaking system is consist of various machine learning techniques to maximize user satisfaction. Our matchmaking system manages large user context containing actions a few seconds ago, and reacts in milliseconds to produce meaningful new results in each user session. It's difficult in traditional way. So, distributed streaming is essential to handle in this cases. Topics include: - Why our team choose Apache Flink in comparison with alternatives - Matchmaking streaming architecture with detail abstraction levels based on Flink operator - Pairwise scoring microservice management with Flink - Stateful matchmaking computation with low latency, fault-tolerance, and scalability - How to manage large-scale events: classifying feature types, collecting with a multi-window stream - Applications: personalization, multi-armed-bandit on stream.
Data & Analytics Forum: Moving Telcos to Real Time
MemSQL is a real-time database that allows users to simultaneously ingest, serve, and analyze streaming data and transactions. It is an in-memory distributed relational database that supports SQL, key-value, documents, and geospatial queries. MemSQL provides real-time analytics capabilities through Streamliner, which allows one-click deployment of Apache Spark for real-time data pipelines and analytics without batch processing. It is available in free community and paid enterprise editions with support and additional features.
Time and ordering in streaming distributed systems
The document discusses time and ordering in distributed streaming systems through three lenses: uniformity of time, arrow of time, and perception of time. It explains that in distributed systems, clocks are not perfectly synchronized and time is not uniform. It also discusses how the direction of time can be blurred by out of order event processing. Finally, it explores how perceptions of time and ordering depend on an observer's frame of reference, and there is no single global ordering of events in a distributed system.
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
In this session, we will look at how Apache Flink can be used to stream anonymized API request and response data from a production environment to make sure staging environments are up-to-date and reflect the most recent features (and bugs) that comprise a service. The talk will also examine how to deal with issues of data retention, throttling, and persistence, finishing with recommendations for how to use these sandbox environments to rapidly prototype and test new features and fixes.
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Speaker: Yupeng Fu, Staff Engineer, Uber
High availability and reliability are important requirements to Uber services, and the services shall tolerate datacenter failures in a region and fail over to another region. In this talk, we will present the active-active Apache Kafka® at Uber and how it facilitates disaster discovery across regions for Uber services. In particular, we will highlight the key components including topic replication, topic aggregation, offsets sync and then walk through several use cases of their disaster recovery strategy using active-active Kafka. Lastly, we will present several interesting challenges and the future work planned.
Yupeng Fu is a staff engineer in Uber Data Org leading the streaming data platform. Previously, he worked at Alluxio and Palantir, building distributed data analysis and storage platforms. Yupeng holds a B.S. and an M.S. from Tsinghua University and did his Ph.D. research on databases at UCSD.
Monitoring as Software Validation
This document discusses monitoring systems and infrastructure. It recommends monitoring everything, including networks, machines, and applications, to learn from infrastructure, anticipate failures, and speed up changes. It presents Graphite as an open-source tool for storing and visualizing real-time time-series data efficiently. Graphite includes components for receiving metrics data, storing data long-term in Whisper, and visualizing data in Graphite Web. It also discusses using StatsD and CollectD to monitor application and system metrics and send them to Graphite. Case studies show how two companies use monitoring to track simulations and the interactions of image processing applications. The document emphasizes that monitoring and testing are both important but serve different purposes.
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
As many industries, banking is undergoing a fundamental change because of the software revolution. No longer are banks competing only on interest rates and having the best traders, these days customer experience and having the best engineers are the focus. In this changing world, banks compete with new start-ups, the so-called Fintechs, and with large platform organisations such as Google, Facebook and Apple. At ING, we believe that staying ahead of the game means changing how we interact with our customers, no longer a traditional model of waiting for the customers to come to the bank through our website or apps, but to actively reach out to the customer with information that is relevant to him or her in order to make their financial life frictionless. Many of these changes are driven by reacting to all events that are relevant to the customer, and using streaming analytics to be able to reach out to the customer in milliseconds after the event occurs. Apache Flink is key for ING to achieve this. This presentation addresses how ING approaches the challenge, the role that Apache Flink plays, and the consequences regulations have on how we work with Open Source in general, and with Apache Flink (and data Artisans) in particular. This keynote takes place at Kino 3.
What's hot (20)
Container Monitoring Best Practices Using AWS and InfluxData by Gunnar Aasen
Container Monitoring Best Practices Using AWS and InfluxData by Gunnar Aasen
Stabilizing the Jenga tower: Scaling out Ceilometer
Stabilizing the Jenga tower: Scaling out Ceilometer
goto; London: Keeping your Cloud Footprint in Check
goto; London: Keeping your Cloud Footprint in Check
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Running a Massively Parallel Self-serve Distributed Data System At Scale
Running a Massively Parallel Self-serve Distributed Data System At Scale
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Virtual Flink Forward 2020: Data driven matchmaking streaming at Hyperconnect...
Virtual Flink Forward 2020: Data driven matchmaking streaming at Hyperconnect...
Data & Analytics Forum: Moving Telcos to Real Time
Data & Analytics Forum: Moving Telcos to Real Time
Time and ordering in streaming distributed systems
Time and ordering in streaming distributed systems
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Viewers also liked
Storing VMs with Cinder and Ceph RBD.pdf
The document discusses different ways of storing virtual machine images and volumes using Ceph storage. It describes growing from using proprietary storage hardware to using Ceph with its open source, scalable, and fault-tolerant design. It outlines how virtual machine images can be stored in Ceph using the Cinder volume driver and RADOS Block Device (RBD) for efficient cloning and live migration. Client access to volumes is provided through a Linux kernel module while container/VM access is through device drivers.
Webinar Monitoring in era of cloud computing
Create-Net is a research center that offers cloud computing research, consulting, training, and webinars. This webinar discusses monitoring in the cloud computing era, beginning with introductions to Ceilometer and Monasca. Ceilometer is OpenStack's metering framework that collects data from OpenStack services through agents and notifications. It stores data in a database and provides an API. Monasca is a monitoring as a service platform that processes metrics and events at scale through microservices and stores data for querying and visualization. The webinar concludes with a discussion of trends in cloud monitoring.
openstackの仮想マシンHA機能の現状と今後の方向性
OpenStack HAチームで仮想マシンHA機能の方式に関する議論が行われている，Austinサミットでの議論では，いくつかのアプローチが提案されたが、それぞれに長所短所があり、 まだコミュニティとしての方向性が固まっていない．本資料では，仮想マシンHA機能についてAustinサミットでの議論内容と今後のコミュニティの方向性について説明します．
Ceph and OpenStack - Feb 2014
Ceph is an open-source distributed storage system that provides object, block, and file storage in a single unified platform. It has become popular for OpenStack deployments due to its ability to provide scalable storage using commodity hardware. Ceph uses a pseudo-random placement algorithm called CRUSH to distribute data and replicas across its clusters to provide fault tolerance and load balancing.
Ceph and Mirantis OpenStack
On January 14, 2014, Dmitry Borodaenko presented information on Ceph and OpenStack. This is the slide deck for that presentation.
Your 1st Ceph cluster
This document provides an overview and planning guidelines for a first Ceph cluster. It discusses Ceph's object, block, and file storage capabilities and how it integrates with OpenStack. Hardware sizing examples are given for a 1 petabyte storage cluster with 500 VMs requiring 100 IOPS each. Specific lessons learned are also outlined, such as realistic IOPS expectations from HDD and SSD backends, recommended CPU and RAM per OSD, and best practices around networking and deployment.
Viewers also liked (7)
Recently uploaded
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications.
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Communications Mining Series - Zero to Hero - Session 1
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentationsGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Recently uploaded (20)
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Ceilo componentization diagrams
- 1. IaaS Services OpenStack Msg Bus Ceilometer Notification Agent Ceilometer Central Agent Ceilometer API (metrics, events, alarms, statistics) Collector Storage Ceilometer Compute Agent Ceilo Msg Bus Ceilometer (as today)
- 2. IaaS Services OpenStack Msg Bus Ceilometer Notification Agent Ceilometer Central Agent Ceilometer Events API Event Collector Event Storage Ceilometer Compute Agent Ceilo Msg BusMetrics Collector Ceilometer Metrics API Metrics Storage Ceilometer Alarms API Alarms Storage Alarm Evaluator/N otifier Ceilometer Components (repos) Events Alarms Metrics Collection