Statistics for Engineers
•
8 likes•4,227 views
1) Heinrich Hartmann presented on statistics and monitoring for engineers. He discussed various methods for API monitoring including external monitoring, log analysis, and measuring latency averages and percentiles. 2) Histograms were presented as another method that involves dividing the latency and time scales into bands and reporting periods to count samples, allowing flexible analysis while enabling aggregation. 3) Takeaways included being wary of line graphs, not aggregating percentiles but instead using histograms, keeping all raw data, and striving for meaningful metrics.
Report
Share
Report
Share
Download to read offline
Recommended
Scalable Online Analytics for Monitoring
Monitoring systems will get smarter in order to keep up with the demands of tomorrow's IT architectures. Features like anomaly detection, root cause analysis, and forecasting tools will be critical components of this next level of monitoring. At the same time, the data that monitoring systems ingest is ever increasing in amount and velocity.
This session covers architectural models for advanced online analytics. We argue that stateful online computations provide a means to realize machine learning on high-velocity data. We show how alerting systems, event engines, stream aggregators, and time-series databases interact to support smart, scalable, and resilient monitoring solutions.
Heinrich Hartmann is the Chief Data Scientist at Circonus. He is driving the development of analytics methods that transform monitoring data into actionable information as part of the Circonus monitoring platform. In his prior life, Heinrich pursued an academic career as a mathematician (PhD in Bonn, Oxford). Later he transitioned into computer science and worked as consultant for a number of different companies and research institutions.
Circonus: Design failures - A Case Study
SRECon18 Keynote https://www.usenix.org/conference/srecon18europe/presentation/schlossnagle
Joint talk Theo Schlossnagle (p1-26), Heinrich Hartmann (p26-44ff).
Latency SLOs Done Right @ SREcon EMEA 2019
Latency is a key indicator of service quality, and important to measure and track. However, measuring latency correctly is not easy. In contrast to familiar metrics like CPU utilization or request counts, the "latency" of a service is not easily expressed in numbers. Percentile metrics have become a popular means to measure the request latency, but have several shortcomings, especially when it comes to aggregation. The situation is particularly dire if we want to use them to specify Service Level Objectives (SLOs) that quantify the performance over a longer time horizons. In the talk we will explain these pitfalls, and suggest three practical methods how to implement effective Latency SLOs.
Real Time Experiment Analytics at Pinterest with Apache Flink - Ben Liu & Par...
This document discusses Pinterest's real-time experiment analytics system built on Apache Flink. It provides an overview of the system, describes the high volume of streaming data being processed, and goes into detail on the NumeratorComputer operator and how it calculates metrics for A/B tests. It also covers challenges faced like checkpoint failures and how root causes were identified and addressed. The goal is to provide faster insights from experiments compared to existing batch systems by computing metrics continuously from streamed event data.
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using F...
Distributed tracing is used to analyze performance and error cases in service oriented architectures. The Observability team at Airbnb recently created Upshot, a data pipeline that uses Flink to analyze over 40 million trace events per minute. Summaries of the resulting data are sent to Druid, Datadog, and other downstream datastores. This talk will focus on how we use Flink and how we analyzed and addressed scaling issues we encountered while building Upshot.
Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join dataset...
Streaming engines like Apache Flink are redefining ETL and data processing. Data can be extracted, transformed, filtered and written out in real-time with an ease matching that of batch processing. However the real challenge of matching the prowess of batch ETL remains in doing joins, in maintaining state and to have the data be paused or rested dynamically. Netflix has a microservices architecture. Different microservices serve and record different kind of user interactions with the product. Some of these live services generate millions of events per second, all carrying meaningful but often partial information. Things start to get exciting when we want to combine the events coming from one high-traffic microservice to another. Joining these raw events generates rich datasets that are used to train the machine learning models that serve Netflix recommendations. Historically we have done this joining of large volume data-sets in batch. However we asked ourselves if the data is being generated in real-time, why must it not be processed downstream in real time? Why wait a full day to get information from an event that was generated a few mins ago? In this talk, we will share how we solved a complex join of two high-volume event streams using Flink. We will talk about maintaining large state, fault tolerance of a stateful application and strategies for failure recovery.
How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitig...
Robinhood is democratizing the financial systems by offering commission-free investing and trading with the use of your phone or desktop. As exciting as that sounds to the outside world, internally, the team at Robinhood must understand the different risk vectors and build engineering solutions to mitigate these risks. In this talk, Allison will talk about how they build a real-time risk monitoring system with InfluxDB and Faust, an open-source Python stream processing library. She will review the architecture behind the system which will involve both the time series anomaly detection part (InfluxDB) and the real-time stream processing part (Faust/Kafka).
Flink Forward Berlin 2018: Krzysztof Zarzycki & Alexey Brodovshuk - "Assistin...
Nowadays many companies become data rich and intensive. They have millions of users generating billions of interactions and events per day. These massive streams of complex events can be processed and reacted upon to e.g. offer new products, next best actions, communicate to users or detect frauds, and quicker we can do it, the higher value we can generate. Our presentation will be based on our recent experience in building a real-time data analytics platform for telco events. This platform has been jointly built by GetInData and Kcell - the leading telco in Kazakhstan - in just a few months and it currently runs in production at the scale of 10M subscribers and 160K events per second on a still small cluster. It's used as a backbone for personalized marketing campaigns, detecting frauds, cross-sell & up-sell by following the behavior of millions of users in real-time and reacting to it instantly. We will share how we build such platform using current best of breed open-source projects like Flink, Kafka, and Nifi. We won't skimp on the details how we designed and optimized our Flink applications for high load and performance. We will also describe challenges that we faced during development and try to provide some tips what one should pay attention to when developing similar solutions, not only for telco, but also for banks, e-commerce, IoT and other industries.
Recommended
Scalable Online Analytics for Monitoring
Monitoring systems will get smarter in order to keep up with the demands of tomorrow's IT architectures. Features like anomaly detection, root cause analysis, and forecasting tools will be critical components of this next level of monitoring. At the same time, the data that monitoring systems ingest is ever increasing in amount and velocity.
This session covers architectural models for advanced online analytics. We argue that stateful online computations provide a means to realize machine learning on high-velocity data. We show how alerting systems, event engines, stream aggregators, and time-series databases interact to support smart, scalable, and resilient monitoring solutions.
Heinrich Hartmann is the Chief Data Scientist at Circonus. He is driving the development of analytics methods that transform monitoring data into actionable information as part of the Circonus monitoring platform. In his prior life, Heinrich pursued an academic career as a mathematician (PhD in Bonn, Oxford). Later he transitioned into computer science and worked as consultant for a number of different companies and research institutions.
Circonus: Design failures - A Case Study
SRECon18 Keynote https://www.usenix.org/conference/srecon18europe/presentation/schlossnagle
Joint talk Theo Schlossnagle (p1-26), Heinrich Hartmann (p26-44ff).
Latency SLOs Done Right @ SREcon EMEA 2019
Latency is a key indicator of service quality, and important to measure and track. However, measuring latency correctly is not easy. In contrast to familiar metrics like CPU utilization or request counts, the "latency" of a service is not easily expressed in numbers. Percentile metrics have become a popular means to measure the request latency, but have several shortcomings, especially when it comes to aggregation. The situation is particularly dire if we want to use them to specify Service Level Objectives (SLOs) that quantify the performance over a longer time horizons. In the talk we will explain these pitfalls, and suggest three practical methods how to implement effective Latency SLOs.
Real Time Experiment Analytics at Pinterest with Apache Flink - Ben Liu & Par...
This document discusses Pinterest's real-time experiment analytics system built on Apache Flink. It provides an overview of the system, describes the high volume of streaming data being processed, and goes into detail on the NumeratorComputer operator and how it calculates metrics for A/B tests. It also covers challenges faced like checkpoint failures and how root causes were identified and addressed. The goal is to provide faster insights from experiments compared to existing batch systems by computing metrics continuously from streamed event data.
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using F...
Distributed tracing is used to analyze performance and error cases in service oriented architectures. The Observability team at Airbnb recently created Upshot, a data pipeline that uses Flink to analyze over 40 million trace events per minute. Summaries of the resulting data are sent to Druid, Datadog, and other downstream datastores. This talk will focus on how we use Flink and how we analyzed and addressed scaling issues we encountered while building Upshot.
Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join dataset...
Streaming engines like Apache Flink are redefining ETL and data processing. Data can be extracted, transformed, filtered and written out in real-time with an ease matching that of batch processing. However the real challenge of matching the prowess of batch ETL remains in doing joins, in maintaining state and to have the data be paused or rested dynamically. Netflix has a microservices architecture. Different microservices serve and record different kind of user interactions with the product. Some of these live services generate millions of events per second, all carrying meaningful but often partial information. Things start to get exciting when we want to combine the events coming from one high-traffic microservice to another. Joining these raw events generates rich datasets that are used to train the machine learning models that serve Netflix recommendations. Historically we have done this joining of large volume data-sets in batch. However we asked ourselves if the data is being generated in real-time, why must it not be processed downstream in real time? Why wait a full day to get information from an event that was generated a few mins ago? In this talk, we will share how we solved a complex join of two high-volume event streams using Flink. We will talk about maintaining large state, fault tolerance of a stateful application and strategies for failure recovery.
How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitig...
Robinhood is democratizing the financial systems by offering commission-free investing and trading with the use of your phone or desktop. As exciting as that sounds to the outside world, internally, the team at Robinhood must understand the different risk vectors and build engineering solutions to mitigate these risks. In this talk, Allison will talk about how they build a real-time risk monitoring system with InfluxDB and Faust, an open-source Python stream processing library. She will review the architecture behind the system which will involve both the time series anomaly detection part (InfluxDB) and the real-time stream processing part (Faust/Kafka).
Flink Forward Berlin 2018: Krzysztof Zarzycki & Alexey Brodovshuk - "Assistin...
Nowadays many companies become data rich and intensive. They have millions of users generating billions of interactions and events per day. These massive streams of complex events can be processed and reacted upon to e.g. offer new products, next best actions, communicate to users or detect frauds, and quicker we can do it, the higher value we can generate. Our presentation will be based on our recent experience in building a real-time data analytics platform for telco events. This platform has been jointly built by GetInData and Kcell - the leading telco in Kazakhstan - in just a few months and it currently runs in production at the scale of 10M subscribers and 160K events per second on a still small cluster. It's used as a backbone for personalized marketing campaigns, detecting frauds, cross-sell & up-sell by following the behavior of millions of users in real-time and reacting to it instantly. We will share how we build such platform using current best of breed open-source projects like Flink, Kafka, and Nifi. We won't skimp on the details how we designed and optimized our Flink applications for high load and performance. We will also describe challenges that we faced during development and try to provide some tips what one should pay attention to when developing similar solutions, not only for telco, but also for banks, e-commerce, IoT and other industries.
Debunking Common Myths in Stream Processing
This document discusses stream processing with Apache Flink. It begins by defining streaming as the continuous processing of never-ending data streams. It then debunks four common myths about stream processing: 1) that there is always a throughput/latency tradeoff, showing that Flink can achieve high throughput and low latency; 2) that exactly-once processing is not possible, but Flink provides exactly-once state guarantees with checkpoints; 3) that streaming is only for real-time applications, whereas it can also be used for historical data; and 4) that streaming is too hard, whereas most data problems are actually streaming problems. The document concludes by discussing Flink's community and examples of companies using Flink in production.
Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrati...
In modern applications of streaming frameworks, stateful streaming is arguably one of the most important usage cases. Flink, as a well-supported streaming framework for stateful streaming, readily helps developers spend less efforts on system deployment and focus more on the business logic. Nevertheless, upgrading from an existing production system to a new one with stateful streaming can still be a challenging task for any development team. In this talk, we will share our experience in migrating an existing system at Appier (an AI-based startup specialized with B2B solutions) to stateful streaming with Flink. We will first discuss how stateful streaming matches our business logic and its potential benefits. Then, we review the obstacles that we have encountered during migration, and present our solutions to conquer them. We hope that our experience and tips shared in this talk hints future users to prepare themselves towards applying Flink in their production systems more painlessly.
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
In this session, we will look at how Apache Flink can be used to stream anonymized API request and response data from a production environment to make sure staging environments are up-to-date and reflect the most recent features (and bugs) that comprise a service. The talk will also examine how to deal with issues of data retention, throttling, and persistence, finishing with recommendations for how to use these sandbox environments to rapidly prototype and test new features and fixes.
Robust Stream Processing With Apache Flink
My talk from Reactive Summit 2016 - Not terribly interesting without the actual talk since most of it was live coding and demo.
Flink Forward Berlin 2017: Steffen Hausmann - Build a Real-time Stream Proces...
The increasing number of available data sources in today's application stacks created a demand to continuously capture and process data from various sources to quickly turn high volume streams of raw data into actionable insights. Apache Flink addresses many of the challenges faced in this domain as it's specifically tailored to distributed computations over streams. While Flink provides all the necessary capabilities to process streaming data, provisioning and maintaining a Flink cluster still requires considerable effort and expertise. We will discuss how cloud services can remove most of the burden of running the clusters underlying your Flink jobs and explain how to build a real-time processing pipeline on top of AWS by integrating Flink with Amazon Kinesis and Amazon EMR. We will furthermore illustrate how to leverage the reliable, scalable, and elastic nature of the AWS cloud to effectively create and operate your real-time processing pipeline with little operational overhead.
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
http://flink-forward.org/kb_sessions/declarative-stream-processing-with-streamsql-and-cep/
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at sub-second latency. Systems like Apache Flink enable applications that include typical CEP features as well as heavy aggregations. In this talk we will show how Apache Flink unifies CEP and stream analytics workloads. Guided by examples, we introduce Flink’s CEP-enriched StreamSQL interface and discuss how queries are compiled, optimized, and executed on Flink.
Keynote: Stephan Ewen - Stream Processing as a Foundational Paradigm and Apac...
Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays.
The fact that stream processing is gaining rapid adoption is also due to more powerful and maturing technology (much of it open source at the ASF) that has solved many of the hard technical challenges.
We discuss Apache Flink's approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.
Making Sense of Streaming Sensor Data: How Uber Detects on Trip Car Crashes -...
Safety is one of the most crucial concerns of Uber’s ride-sharing platform. To advance the timeliness in response to the safety issues in daily business, Uber runs a Flink pipeline that joins together multiple high volume (>10 TB/day) streaming sources of sensor data along with trip information in order to extract a number of contextual features. It deploys a deep learning model on top of TensorFlow in Flink to detect potential car crashes and identify general trip and driving anomalies. This knowledge is then passed down to the business operational teams to proactively reach out to riders and drivers to confirm their user experiences and provide prompt safety assistance if needed.
Stream Loops on Flink - Reinventing the wheel for the streaming era
This document discusses adding iterative processing capabilities to stream processing systems like Apache Flink. It proposes programming model extensions that treat iterative computations as structured loops over windows. Progress would be tracked using progress timestamps rather than watermarks to allow for arbitrary loop structures. Challenges include managing state and cyclic flow control to avoid deadlocks while encouraging iteration completion.
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Over 137 million members worldwide are enjoying TV series, feature films across a wide variety of genres and languages on Netflix. It leads to petabyte scale of user behavior data. At Netflix, our client logging platform collects and processes this data to empower recommendations, personalization and many other services to enhance user experience. Built with Apache Flink, this platform processes 100s of billion events and a petabyte data per day, 2.5 million events/sec in sub milliseconds latency. The processing involves a series of data transformations such as decryption and data enrichment of customer, geo, device information using microservices based lookups.
The transformed and enriched data is further used by multiple data consumers for a variety of applications such as improving user-experience with A/B tests, tracking application performance metrics, tuning algorithms. This causes redundant reads of the dataset by multiple batch jobs and incurs heavy processing costs. To avoid this, we have developed a config driven, centralized, managed platform, on top of Apache Flink, that reads this data once and routes it to multiple streams based on dynamic configuration. This has resulted in improved computation efficiency, reduced costs and reduced operational overhead.
Stream processing at scale while ensuring that the production systems are scalable and cost-efficient brings interesting challenges. In this talk, we will share about how we leverage Apache Flink to achieve this, the challenges we faced and our learnings while running one of the largest Flink application at Netflix.
Insight Demo
This document discusses developing a crowd detector system using location data from mobile phones. It proposes collecting location data from phones, performing a random walk on the data, and using k-means clustering with K=500 clusters to identify crowded areas of San Francisco. The document outlines challenges in choosing parameters like batch interval and processing time. It suggests using Kafka to handle data streaming and limiting input/processing rates to have a stable system with constant delay between batches.
Insight Recent Demo
This document discusses developing a crowd detector system using location data from mobile phones. It proposes collecting location data from phones, performing a random walk on the data, and using k-means clustering with K=500 clusters to identify crowded areas of San Francisco. The goals are to tune parameters like batch processing time and update intervals to have a stable, low-latency system. Challenges addressed include choosing K, limiting input/processing rates, and understanding Spark Streaming complexity. The proposed solution engineers location data from Yelp to simulate real phone data.
Continuously Updating Query Results over Real-Time Linked Data
This document discusses continuously updating query results over real-time linked data. It proposes moving continuous query evaluation from the server to the client to lower server load. Dynamic data is represented in RDF and annotated with time validity using methods like reification, graphs or implicit graphs. A query streamer engine exposes dynamic data through a Triple Pattern Fragments interface and sends query results to clients, offloading work from the server. An evaluation compares annotation methods, measures query execution times and server CPU usage, finding the query streamer approach has better scalability by distributing load to clients.
Using Kafka to integrate DWH and Cloud Based big data systems
Mic Hussey, Senior Systems Engineer, Confluent
Using Kafka to integrate DWH and Cloud Based big data systems
https://www.meetup.com/Stockholm-Apache-Kafka-Meetup-by-Confluent/events/268636234/
Monitoring Flink with Prometheus
Video: https://data-artisans.com/flink-forward-berlin/resources/monitoring-flink-with-prometheus
Live Demo Code: https://github.com/mbode/flink-prometheus-example
Prometheus is a cloud-native monitoring system prioritizing reliability and simplicity – and Flink works really well with it! This session will show you how to leverage the Flink metrics system together with Pronetheus to improve the observability of your jobs. There will be a live demo establishing how everything ties in together. The talk is aimed at people already building and running Flink jobs who would like to gain more insight into them. It is fine if you are not familiar with Prometheus yet as the basic concepts will be introduced. If you have ever wondered how you could use modern monitoring tools to be alerted in the middle of the night in case your Flink job‘s 99th percentile end-to-end latency degraded for some reason, this might just be the talk you are looking for.
Continuous Self-Updating Query Results over Dynamic Linked Data
This document discusses continuously updating query results over dynamic linked data. It proposes moving continuous query evaluation from the server to the client to lower server load. Key points:
- Dynamic linked data streams, like sensor data, add new triples over time and typically query the current value.
- Existing approaches have servers continuously evaluate queries, causing low availability due to high load.
- The document proposes storing dynamic data efficiently, transmitting it to clients, and having clients evaluate queries to reduce server load.
- A preliminary test showed the proposed approach moves more load to clients but scales server load better than existing solutions that rely on continuous server-side query evaluation.
Ceilometer lsf-intergration-openstack-summit
CERN uses ceilometer as a single source of truth for accounting data from both virtual machines and batch computing. Ceilometer implements a plugin to poll CERN's batch accounting database for unpublished records, which are then pushed to RabbitMQ. A ceilometer collector consumes the messages and inserts the data into a ceilometer database. This decreases load on the OpenStack messaging server by using separate instances and RabbitMQ servers for VM and batch metering data. Most batch data is published within two runs processing around 200,000 job records each, which takes approximately 5 hours to complete. The average publishing rate to the batch RabbitMQ server is 11 records per second.
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
As a distributed stream processing engine, Flink provides users with convenient operators to manipulate data on the fly. Among all these operators, join could be the most complicated one as it requires the capability to cross-analyze various sources simultaneously. In this talk, we aim to give a comprehensive introduction to the stream join in Flink. Specifically, we'll first provide an overview of the different join types and which of them are currently supported by Flink DataStream and Table & SQL APIs. Then we'll discuss some key points when performing distributed stream join. After that, we'd like to focus the rationale and implementation details of the time-windowed join launched in version 1.4. Since there are still a lot of improvements can be made, we'll end our talk by sharing some proposals for the future work.
Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...
Back to the program
Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Analytics
Thursday 17th
from 18:00 to 18:40
Theatre 19
-
Keynote
In this talk I’ll give a very short introduction to stream processing in general and then dive into event-time based stream processing. I will outline how this is important for IoT applications and also why it is such a challenging topic. Afterwards we’ll look at some real-world IoT use cases that are enabled by the support for robust event-time based stream processing provided by Apache Flink™. We will especially focus on easy of use and on correctness of results in the face of errors.
In the first half of the talk we’ll cover the basics of stream processing. We will look at the differences between event-time based and processing-time and at stateful stream processing. While on this, we’ll also highlight how the combination of these features is essential for doing robust stream processing in an IoT setting.
In the second part, we will look at how Flink solves some of the challenges that arise in event-time based processing and how that enables novel applications in the IoT space. We will do the latter by looking at a collection of real-world IoT use cases.
Some of the topics covered will be:
- Apache Flink
- Stateful Stream Processing
- Event Time vs. Processing Time Windowing
- Processing of out-of-order events
- IoT use cases
OSMC 2015: Monitoring at Spotify-When things go ping in the night by Martin Parm
When Spotify started in 2006, with just 20 people, they were more worried about selling the idea of music streaming than of setting up monitoring systems. Fast forward to 2015 and
more than 400 engineers are collecting more than 30 million time series from more than 10000 hosts; so how did we get here? The journey has been a long one, with plenty of false starts and growing pains, from scaling systems to scaling teams to scaling the business itself; challenging what we thought we knew about operational monitoring at every step.
This talk is about some of the more interesting challenges we've faced along the way, and about what we've learned so far; covering some of the technical details but primarily focusing on the human aspects, and how our monitoring solutions have both shaped and been shaped by organizational structures and changing engineering practices.
Monitoring at Facebook - Ran Leibman, Facebook - DevOpsDays Tel Aviv 2015
This document summarizes Ran Leibman's presentation on monitoring tools, components, and mentality at Facebook. It describes Facebook's monitoring architecture including the operational data store (ODS) for storing metrics, Scuba for real-time log monitoring, the alarm system for creating alerts, Facebook Auto-Remediation (FBAR) for automating issue resolution, notifications and subscriptions for alerting engineers, and dashboards for visualizing data. The presentation emphasizes treating metrics as important data, empowering developers to monitor, automating problem resolution, and using monitoring to surface previously unknown issues.
More Related Content
What's hot
Debunking Common Myths in Stream Processing
This document discusses stream processing with Apache Flink. It begins by defining streaming as the continuous processing of never-ending data streams. It then debunks four common myths about stream processing: 1) that there is always a throughput/latency tradeoff, showing that Flink can achieve high throughput and low latency; 2) that exactly-once processing is not possible, but Flink provides exactly-once state guarantees with checkpoints; 3) that streaming is only for real-time applications, whereas it can also be used for historical data; and 4) that streaming is too hard, whereas most data problems are actually streaming problems. The document concludes by discussing Flink's community and examples of companies using Flink in production.
Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrati...
In modern applications of streaming frameworks, stateful streaming is arguably one of the most important usage cases. Flink, as a well-supported streaming framework for stateful streaming, readily helps developers spend less efforts on system deployment and focus more on the business logic. Nevertheless, upgrading from an existing production system to a new one with stateful streaming can still be a challenging task for any development team. In this talk, we will share our experience in migrating an existing system at Appier (an AI-based startup specialized with B2B solutions) to stateful streaming with Flink. We will first discuss how stateful streaming matches our business logic and its potential benefits. Then, we review the obstacles that we have encountered during migration, and present our solutions to conquer them. We hope that our experience and tips shared in this talk hints future users to prepare themselves towards applying Flink in their production systems more painlessly.
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
We present I², an interactive development environment for real-time analysis pipelines, which is based on Apache Flink and Apache Zeppelin. The sheer amount of available streaming data frequently makes it impossible to visualize all data points at the same time. I² coordinates running Flink jobs and corresponding visualizations such that only the currently depicted data points are processed in Flink and transferred towards the front end. We show how Flink jobs can adapt to changed visualization properties at runtime to allow interactive data exploration on high bandwidth data streams. Moreover, we present a data reduction technique which minimizes data transfer while providing loss free time-series plots. We show I² in a live demonstration in which we replay recorded sensor data from a football match (ca. 12k event/s). I² was first presented at EDBT'17 where it was awarded as best demonstration. The demonstration is available as open source at github.com/TU-Berlin-DIMA/i2.
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
In this session, we will look at how Apache Flink can be used to stream anonymized API request and response data from a production environment to make sure staging environments are up-to-date and reflect the most recent features (and bugs) that comprise a service. The talk will also examine how to deal with issues of data retention, throttling, and persistence, finishing with recommendations for how to use these sandbox environments to rapidly prototype and test new features and fixes.
Robust Stream Processing With Apache Flink
My talk from Reactive Summit 2016 - Not terribly interesting without the actual talk since most of it was live coding and demo.
Flink Forward Berlin 2017: Steffen Hausmann - Build a Real-time Stream Proces...
The increasing number of available data sources in today's application stacks created a demand to continuously capture and process data from various sources to quickly turn high volume streams of raw data into actionable insights. Apache Flink addresses many of the challenges faced in this domain as it's specifically tailored to distributed computations over streams. While Flink provides all the necessary capabilities to process streaming data, provisioning and maintaining a Flink cluster still requires considerable effort and expertise. We will discuss how cloud services can remove most of the burden of running the clusters underlying your Flink jobs and explain how to build a real-time processing pipeline on top of AWS by integrating Flink with Amazon Kinesis and Amazon EMR. We will furthermore illustrate how to leverage the reliable, scalable, and elastic nature of the AWS cloud to effectively create and operate your real-time processing pipeline with little operational overhead.
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
http://flink-forward.org/kb_sessions/declarative-stream-processing-with-streamsql-and-cep/
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at sub-second latency. Systems like Apache Flink enable applications that include typical CEP features as well as heavy aggregations. In this talk we will show how Apache Flink unifies CEP and stream analytics workloads. Guided by examples, we introduce Flink’s CEP-enriched StreamSQL interface and discuss how queries are compiled, optimized, and executed on Flink.
Keynote: Stephan Ewen - Stream Processing as a Foundational Paradigm and Apac...
Stream Processing is emerging as a popular paradigm for data processing architectures, because it handles the continuous nature of most data and computation and gets rid of artificial boundaries and delays.
The fact that stream processing is gaining rapid adoption is also due to more powerful and maturing technology (much of it open source at the ASF) that has solved many of the hard technical challenges.
We discuss Apache Flink's approach to high performance stream processing with state, strong consistency, low latency, and sophisticated handling of time. With such building blocks, Apache Flink can handle classes of problems previously considered out of reach for stream processing. We also take a sneak preview at the next steps for Flink.
Making Sense of Streaming Sensor Data: How Uber Detects on Trip Car Crashes -...
Safety is one of the most crucial concerns of Uber’s ride-sharing platform. To advance the timeliness in response to the safety issues in daily business, Uber runs a Flink pipeline that joins together multiple high volume (>10 TB/day) streaming sources of sensor data along with trip information in order to extract a number of contextual features. It deploys a deep learning model on top of TensorFlow in Flink to detect potential car crashes and identify general trip and driving anomalies. This knowledge is then passed down to the business operational teams to proactively reach out to riders and drivers to confirm their user experiences and provide prompt safety assistance if needed.
Stream Loops on Flink - Reinventing the wheel for the streaming era
This document discusses adding iterative processing capabilities to stream processing systems like Apache Flink. It proposes programming model extensions that treat iterative computations as structured loops over windows. Progress would be tracked using progress timestamps rather than watermarks to allow for arbitrary loop structures. Challenges include managing state and cyclic flow control to avoid deadlocks while encouraging iteration completion.
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Over 137 million members worldwide are enjoying TV series, feature films across a wide variety of genres and languages on Netflix. It leads to petabyte scale of user behavior data. At Netflix, our client logging platform collects and processes this data to empower recommendations, personalization and many other services to enhance user experience. Built with Apache Flink, this platform processes 100s of billion events and a petabyte data per day, 2.5 million events/sec in sub milliseconds latency. The processing involves a series of data transformations such as decryption and data enrichment of customer, geo, device information using microservices based lookups.
The transformed and enriched data is further used by multiple data consumers for a variety of applications such as improving user-experience with A/B tests, tracking application performance metrics, tuning algorithms. This causes redundant reads of the dataset by multiple batch jobs and incurs heavy processing costs. To avoid this, we have developed a config driven, centralized, managed platform, on top of Apache Flink, that reads this data once and routes it to multiple streams based on dynamic configuration. This has resulted in improved computation efficiency, reduced costs and reduced operational overhead.
Stream processing at scale while ensuring that the production systems are scalable and cost-efficient brings interesting challenges. In this talk, we will share about how we leverage Apache Flink to achieve this, the challenges we faced and our learnings while running one of the largest Flink application at Netflix.
Insight Demo
This document discusses developing a crowd detector system using location data from mobile phones. It proposes collecting location data from phones, performing a random walk on the data, and using k-means clustering with K=500 clusters to identify crowded areas of San Francisco. The document outlines challenges in choosing parameters like batch interval and processing time. It suggests using Kafka to handle data streaming and limiting input/processing rates to have a stable system with constant delay between batches.
Insight Recent Demo
This document discusses developing a crowd detector system using location data from mobile phones. It proposes collecting location data from phones, performing a random walk on the data, and using k-means clustering with K=500 clusters to identify crowded areas of San Francisco. The goals are to tune parameters like batch processing time and update intervals to have a stable, low-latency system. Challenges addressed include choosing K, limiting input/processing rates, and understanding Spark Streaming complexity. The proposed solution engineers location data from Yelp to simulate real phone data.
Continuously Updating Query Results over Real-Time Linked Data
This document discusses continuously updating query results over real-time linked data. It proposes moving continuous query evaluation from the server to the client to lower server load. Dynamic data is represented in RDF and annotated with time validity using methods like reification, graphs or implicit graphs. A query streamer engine exposes dynamic data through a Triple Pattern Fragments interface and sends query results to clients, offloading work from the server. An evaluation compares annotation methods, measures query execution times and server CPU usage, finding the query streamer approach has better scalability by distributing load to clients.
Using Kafka to integrate DWH and Cloud Based big data systems
Mic Hussey, Senior Systems Engineer, Confluent
Using Kafka to integrate DWH and Cloud Based big data systems
https://www.meetup.com/Stockholm-Apache-Kafka-Meetup-by-Confluent/events/268636234/
Monitoring Flink with Prometheus
Video: https://data-artisans.com/flink-forward-berlin/resources/monitoring-flink-with-prometheus
Live Demo Code: https://github.com/mbode/flink-prometheus-example
Prometheus is a cloud-native monitoring system prioritizing reliability and simplicity – and Flink works really well with it! This session will show you how to leverage the Flink metrics system together with Pronetheus to improve the observability of your jobs. There will be a live demo establishing how everything ties in together. The talk is aimed at people already building and running Flink jobs who would like to gain more insight into them. It is fine if you are not familiar with Prometheus yet as the basic concepts will be introduced. If you have ever wondered how you could use modern monitoring tools to be alerted in the middle of the night in case your Flink job‘s 99th percentile end-to-end latency degraded for some reason, this might just be the talk you are looking for.
Continuous Self-Updating Query Results over Dynamic Linked Data
This document discusses continuously updating query results over dynamic linked data. It proposes moving continuous query evaluation from the server to the client to lower server load. Key points:
- Dynamic linked data streams, like sensor data, add new triples over time and typically query the current value.
- Existing approaches have servers continuously evaluate queries, causing low availability due to high load.
- The document proposes storing dynamic data efficiently, transmitting it to clients, and having clients evaluate queries to reduce server load.
- A preliminary test showed the proposed approach moves more load to clients but scales server load better than existing solutions that rely on continuous server-side query evaluation.
Ceilometer lsf-intergration-openstack-summit
CERN uses ceilometer as a single source of truth for accounting data from both virtual machines and batch computing. Ceilometer implements a plugin to poll CERN's batch accounting database for unpublished records, which are then pushed to RabbitMQ. A ceilometer collector consumes the messages and inserts the data into a ceilometer database. This decreases load on the OpenStack messaging server by using separate instances and RabbitMQ servers for VM and batch metering data. Most batch data is published within two runs processing around 200,000 job records each, which takes approximately 5 hours to complete. The average publishing rate to the batch RabbitMQ server is 11 records per second.
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
As a distributed stream processing engine, Flink provides users with convenient operators to manipulate data on the fly. Among all these operators, join could be the most complicated one as it requires the capability to cross-analyze various sources simultaneously. In this talk, we aim to give a comprehensive introduction to the stream join in Flink. Specifically, we'll first provide an overview of the different join types and which of them are currently supported by Flink DataStream and Table & SQL APIs. Then we'll discuss some key points when performing distributed stream join. After that, we'd like to focus the rationale and implementation details of the time-windowed join launched in version 1.4. Since there are still a lot of improvements can be made, we'll end our talk by sharing some proposals for the future work.
Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...
Back to the program
Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Analytics
Thursday 17th
from 18:00 to 18:40
Theatre 19
-
Keynote
In this talk I’ll give a very short introduction to stream processing in general and then dive into event-time based stream processing. I will outline how this is important for IoT applications and also why it is such a challenging topic. Afterwards we’ll look at some real-world IoT use cases that are enabled by the support for robust event-time based stream processing provided by Apache Flink™. We will especially focus on easy of use and on correctness of results in the face of errors.
In the first half of the talk we’ll cover the basics of stream processing. We will look at the differences between event-time based and processing-time and at stateful stream processing. While on this, we’ll also highlight how the combination of these features is essential for doing robust stream processing in an IoT setting.
In the second part, we will look at how Flink solves some of the challenges that arise in event-time based processing and how that enables novel applications in the IoT space. We will do the latter by looking at a collection of real-world IoT use cases.
Some of the topics covered will be:
- Apache Flink
- Stateful Stream Processing
- Event Time vs. Processing Time Windowing
- Processing of out-of-order events
- IoT use cases
What's hot (20)
Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrati...
Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrati...
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
Virtual Flink Forward 2020: How Streaming Helps Your Staging Environment and ...
Flink Forward Berlin 2017: Steffen Hausmann - Build a Real-time Stream Proces...
Flink Forward Berlin 2017: Steffen Hausmann - Build a Real-time Stream Proces...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Keynote: Stephan Ewen - Stream Processing as a Foundational Paradigm and Apac...
Keynote: Stephan Ewen - Stream Processing as a Foundational Paradigm and Apac...
Making Sense of Streaming Sensor Data: How Uber Detects on Trip Car Crashes -...
Making Sense of Streaming Sensor Data: How Uber Detects on Trip Car Crashes -...
Stream Loops on Flink - Reinventing the wheel for the streaming era
Stream Loops on Flink - Reinventing the wheel for the streaming era
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Continuously Updating Query Results over Real-Time Linked Data
Continuously Updating Query Results over Real-Time Linked Data
Using Kafka to integrate DWH and Cloud Based big data systems
Using Kafka to integrate DWH and Cloud Based big data systems
Continuous Self-Updating Query Results over Dynamic Linked Data
Continuous Self-Updating Query Results over Dynamic Linked Data
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...
Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...
Viewers also liked
OSMC 2015: Monitoring at Spotify-When things go ping in the night by Martin Parm
When Spotify started in 2006, with just 20 people, they were more worried about selling the idea of music streaming than of setting up monitoring systems. Fast forward to 2015 and
more than 400 engineers are collecting more than 30 million time series from more than 10000 hosts; so how did we get here? The journey has been a long one, with plenty of false starts and growing pains, from scaling systems to scaling teams to scaling the business itself; challenging what we thought we knew about operational monitoring at every step.
This talk is about some of the more interesting challenges we've faced along the way, and about what we've learned so far; covering some of the technical details but primarily focusing on the human aspects, and how our monitoring solutions have both shaped and been shaped by organizational structures and changing engineering practices.
Monitoring at Facebook - Ran Leibman, Facebook - DevOpsDays Tel Aviv 2015
This document summarizes Ran Leibman's presentation on monitoring tools, components, and mentality at Facebook. It describes Facebook's monitoring architecture including the operational data store (ODS) for storing metrics, Scuba for real-time log monitoring, the alarm system for creating alerts, Facebook Auto-Remediation (FBAR) for automating issue resolution, notifications and subscriptions for alerting engineers, and dashboards for visualizing data. The presentation emphasizes treating metrics as important data, empowering developers to monitor, automating problem resolution, and using monitoring to surface previously unknown issues.
Prometheus (Monitorama 2016)
Prometheus is a open-source time series database with a powerful query language designed for operational monitoring.
Contact us at prometheus@robustperception.io
Production testing through monitoring
Identifying and fixing issues in new code before deploying it to production is important for every software development cycle. However, relying on traditional testing methods in the age of Internet-scale data driven problems may prove to be incomplete. Identifying and fixing the issues in production quickly is crucial, but it requires insight into usage patterns and trends across the whole architecture and application logic. In this talk I touch on inefficiencies of some of the most common testing methods, provide real world examples of discovering odd edge cases with monitoring and offer recommendations on top-down metric instrumentation to help DevOps organizations with identifying and acting on business-effecting problems.
2016 metrics-as-culture
We don't always think of it this way, but your metrics *are* your culture... Your metrics shape behavior and incentives, which really is the heart of culture.
SREcon 2016 Performance Checklists for SREs
Talk from SREcon2016 by Brendan Gregg. Video: https://www.usenix.org/conference/srecon16/program/presentation/gregg . "There's limited time for performance analysis in the emergency room. When there is a performance-related site outage, the SRE team must analyze and solve complex performance issues as quickly as possible, and under pressure. Many performance tools and techniques are designed for a different environment: an engineer analyzing their system over the course of hours or days, and given time to try dozens of tools: profilers, tracers, monitoring tools, benchmarks, as well as different tunings and configurations. But when Netflix is down, minutes matter, and there's little time for such traditional systems analysis. As with aviation emergencies, short checklists and quick procedures can be applied by the on-call SRE staff to help solve performance issues as quickly as possible.
In this talk, I'll cover a checklist for Linux performance analysis in 60 seconds, as well as other methodology-derived checklists and procedures for cloud computing, with examples of performance issues for context. Whether you are solving crises in the SRE war room, or just have limited time for performance engineering, these checklists and approaches should help you find some quick performance wins. Safe flying."
Monitoring As a Service
Building a modern monitoring environment is more than just using the latest awesome tools, collecting all of the data, displaying numerous graphs and knowing when things go wrong. A modern monitoring environment is more than tools and infrastructure. It's a service. A service you provide to your whole team: developers, operations, security, and the business. This talk is about how you can build monitoring environments (or extend your existing environment) that are customer-focussed rather than infrastructure focussed. We'll see how you can treat your needs and the needs of your organization as customer requirements and build monitoring that is consumable and configurable on demand.
Building a modern monitoring environment is more than just using the latest awesome tools, collecting all of the data, displaying numerous graphs and knowing when things go wrong. A modern monitoring environment is more than tools and infrastructure. It's a service. A service you provide to your whole team: developers, operations, security, and the business. This talk is about how you can build monitoring environments (or extend your existing environment) that are customer-focussed rather than infrastructure focussed. We'll see how you can treat your needs and the needs of your organization as customer requirements and build monitoring that is consumable and configurable on demand.
Monitoring Challenges - Monitorama 2016 - Monitoringless
Opening talk at Monitorama, talks about the problems of monitoring, challenges of creating monitoring tools and why monitoring vendors keep getting disrupted. Ended with a discussion of simulation testing and serverless architectures - Monitorless.
Scaling Pinterest's Monitoring
This document summarizes Brian Overstreet's talk on scaling Pinterest's monitoring system over time as the company and traffic grew. It describes how Pinterest started with just Ganglia for system metrics and no application metrics. They introduced Graphite but faced challenges with packet loss and metrics being dropped. They then introduced OpenTSDB which users were happier with due to its querying speed. Pinterest developed an agent-based pipeline using Kafka and Storm to address packet loss issues and allow over 1.5 million points per second to be ingested by OpenTSDB. Key lessons included the need to educate users, control incoming metrics, and ensure the monitoring system scales with engineers rather than just site users.
DevOpsDays Amsterdam - Monitoring at Service Provider Scale
At Rackspace, we're implementing DevOps principles to enable scale and faster flow of business value, for us that means customer satisfaction. There's lots of work for us to do to complete our implementation, but this talk shows a little about our belief system for DevOps and how we're trying to change monitoring into a source of customer delight.
Bruce Lawson: Progressive Web Apps: the future of Apps
Native Apps, like Flash, are a bridging technology. Progressive Web Apps are a new suite of technologies that combine the user experience of native, with the immediacy and reach of the web. Learn why we have them, and how to make them.
Microservices Workshop All Topics Deck 2016
Just about all of my current technical content in one 364 slide mega-deck. Source files at https://github.com/adrianco/slides
Sections on:
Scene Setting
State of the Cloud
What Changes?
Product Processes
Microservices
State of the Art
Segmentation
What’s Missing?
Monitoring
Challenges
Migration
Response Times
Serverless
Lock-In
Teraservices
Wrap-Up
Julieta Sieyra - CV Especialidades
Este documento presenta el perfil profesional de Julieta Sieyra, una ingeniera química argentina con experiencia en la industria petrolera. Obtuvo un MBA de la Universidad de Texas A&M y tiene más de 15 años de experiencia liderando equipos de perforación, terminación e intervención de pozos en empresas como BP, Chevron y Repsol YPF. Actualmente ofrece sus servicios como coach de eficiencia organizacional enfocada en mejora continua, gestión de proyectos, liderazgo y desarrollo de indicadores clave de desemp
Engineering Math
This document outlines the lecture topics for Engineering Mathematics covering Fourier series, integrals, and transforms taught by Dr. A. Selk Ghafari. The lecture covers Fourier series, Fourier series for periodic functions, even and odd functions, half range expansions, Fourier integrals, Fourier transforms, Fourier sine and cosine transforms, transforms of derivatives, and complex Fourier transforms. Examples from the textbook are referenced for several topics.
10CSL67 CG LAB PROGRAM 9
Program to fill any given polygon using scan-line area filling algorithms. (Use appropriate data structures.)
Building Scalable Systems: an asynchronous approach
This document discusses building scalable systems using an asynchronous approach. It begins by introducing the author and their background in scalable internet architectures. It then discusses some concepts around big data, NoSQL databases, and using the cloud. Key points made include that big data is a marketing term, NoSQL databases are chosen based on data model and CAP constraints, and that scaling in the cloud requires an inherently scalable architecture. It also notes some challenges with quality of service, patching, and assuming scaling is easy in the cloud. The document advocates measuring system metrics before making architecture changes.
How to Speak "Manager"
Keynote address given at Campus LISA, UCSD, July 2014. Abstract: Technical professionals act as brokers in organizations, communicating across functional and organizational boundaries. As brokers, tech professionals straddle two worlds, and can leverage this space if they know how. This talk will focus on understanding the gap between the IT function and management and how to bridge that gap in order to increase respect for IT among management, and improve your relationship with management. The talk will also cover identifying your (or your team’s) role within the organization, effective communication with upper management, positioning yourself and your team to increase visibility, and becoming a strategic partner.
Monitorama: How monitoring can improve the rest of the company
Monitoring can improve the entire company by sharing data and techniques across teams. By implementing structured logging, automatic metrics collection, and common data visualization tools, monitoring can become the central data platform. This allows all teams like developers, analysts, and executives to access insights that help improve products, prioritize issues, and make data-driven decisions.
Grokking Grok: Monitorama PDX 2015
The document discusses using Logstash and regular expressions (Grok) to parse log files from Cisco firewalls and Check Point firewalls. It provides Grok patterns to extract fields from sample log messages for each vendor. It also shows how to further transform the extracted fields using Logstash filters like mutate. The goal is to get the log data into a structured format that can be indexed and analyzed.
2014 devops conferences
This document lists several upcoming DevOps, infrastructure, and cloud computing conferences in 2014 and 2015, including DevOpsDays, O'Reilly Velocity, Surge, YOW!, Monitorama, LISA, Flowcon, PuppetConf, ChefConf, and Dockercon. It also mentions some language-specific and regional conferences to watch out for next year such as PyconAU, LCA Auckland, RubyConf, and FOSDEM.
Viewers also liked (20)
OSMC 2015: Monitoring at Spotify-When things go ping in the night by Martin Parm
OSMC 2015: Monitoring at Spotify-When things go ping in the night by Martin Parm
Monitoring at Facebook - Ran Leibman, Facebook - DevOpsDays Tel Aviv 2015
Monitoring at Facebook - Ran Leibman, Facebook - DevOpsDays Tel Aviv 2015
Monitoring Challenges - Monitorama 2016 - Monitoringless
Monitoring Challenges - Monitorama 2016 - Monitoringless
DevOpsDays Amsterdam - Monitoring at Service Provider Scale
DevOpsDays Amsterdam - Monitoring at Service Provider Scale
Bruce Lawson: Progressive Web Apps: the future of Apps
Bruce Lawson: Progressive Web Apps: the future of Apps
Building Scalable Systems: an asynchronous approach
Building Scalable Systems: an asynchronous approach
Monitorama: How monitoring can improve the rest of the company
Monitorama: How monitoring can improve the rest of the company
Similar to Statistics for Engineers
OSMC 2016 - Friends and foes by Heinrich Hartmann
Heinrich Hartmann ist Chief Data Scientist bei der Circonus Monitoring and Analytics Plattform. Als solcher treibt er die Entwicklung von Analyse Methoden an, die Monitoring Daten zu verwertbare Informationen umwandeln. Früher verfolgte er eine Karriere als Mathematiker. Dann wechselte er zur Computer Wissenschaft, und arbeitete als Berater für eine Vielzahl Firmen und Forschungseinrichtungen.
OSMC 2016 | Friends and foes in API Monitoring by Heinrich Hartmann
APIs sind überall. In modernen (mircro-) service orientierten Architekturen nehmen APIs eine zentrale Rolle ein, deren Performance man im Auge behalten muss.
Zentrale Fragen sind:
Hat sich die User Experience für einige/alle Nutzer gerade verschlechtert?
Wo liegen die Ursachen?
Was sind die Business Implications?
Diese Fragen sind keineswegs leicht zu beantworten. Im Talk stellen wir verschiedene Ansätze zum API monitoring vor, diskutieren Stärken und Schwächen, und zeigen häufig angetroffene Missverständnisse auf. Erstaunlicher Weise gibt es einen klaren Gewinner, der noch wenig Verbreitung gefunden hat.
How to Measure Latency
Measuring Latency for Monitoring and Benchmarking purposes is notoriously difficult. There are a lot of pitfalls with collecting, aggregating and analyzing latency data.
In the talk, we will make an effort to visit this topic from a top-down perspective and compile known complications and best-practice approaches on how to avoid them. This will include:
Measurement Overhead
Queuing effects – Coordinated omission
Histograms for Aggregation and Visualization
Percentile aggregation
Latency bands and burn-down charts
Latency comparison methods (QQ Plots, KS-Distance)
Lego-like building blocks of Storm and Spark Streaming Pipelines
This document discusses an approach to building streaming analytics applications using reusable building blocks like Lego blocks. It presents a centralized approach where common streaming patterns like ingestion, filtering, classification, enrichment etc. are defined as reusable pipelines. These pre-built pipelines can then be connected together to rapidly build streaming analytics apps for various use cases across industries. The document demonstrates how different streaming engines can be used within pipelines and how routing between pipelines can be configured dynamically for A/B testing and model optimization. This approach aims to provide a unified visual platform for collaborative development of efficient streaming analytics solutions at scale.
Webinar Monitoring in era of cloud computing
Create-Net is a research center that offers cloud computing research, consulting, training, and webinars. This webinar discusses monitoring in the cloud computing era, beginning with introductions to Ceilometer and Monasca. Ceilometer is OpenStack's metering framework that collects data from OpenStack services through agents and notifications. It stores data in a database and provides an API. Monasca is a monitoring as a service platform that processes metrics and events at scale through microservices and stores data for querying and visualization. The webinar concludes with a discussion of trends in cloud monitoring.
Shikha fdp 62_14july2017
Introduction of streaming data, difference between batch processing and stream processing, Research issues in streaming data processing, Performance evaluation metrics , tools for stream processing.
Lean Six Sigma methodology
Lean Six Sigma is a process improvement methodology that relies on a collaborative team effort to improve performance by systematically removing waste, combining Lean and Six Sigma to eliminate the eight kinds of waste
Lean Six Sigma projects comprise aspects of Lean's waste elimination and the Six Sigma focus on reducing defects
5.1 mining data streams
This document discusses techniques for mining data streams. It begins by defining different types of streaming data like time-series data and sequence data. It then discusses the characteristics of data streams like their huge volume, fast changing nature, and requirement for real-time processing. The key challenges in stream query processing are the unbounded memory requirements and need for approximate query answering. The document outlines several synopsis data structures and techniques used for mining data streams, including random sampling, histograms, sketches, and randomized algorithms. It also discusses architectures for stream query processing and classification of dynamic data streams.
Streaming analytics state of the art
This document provides an overview of streaming analytics, including definitions, common use cases, and key concepts like streaming engines, processing models, and guarantees. It also provides examples of analyzing data streams using Apache Spark Structured Streaming, Apache Flink, and Kafka Streams APIs. Code snippets demonstrate windowing, triggers, and working with event-time.
Product Engineer Certified Lean Six Sigma Black Belt by IASSC
This document is a resume for Mohamed Hakkache. It summarizes his professional experience and education. He currently works as a Product Engineer at Lear Corporation and has various technical certifications in areas like Lean Six Sigma, deep learning, machine learning, and Apache Hadoop. Previously he has worked on process improvement projects and as a Production Supervisor. He is looking to apply his data science knowledge gained from online courses at places like Stanford, IBM, and Intel.
Apache Flink: Real-World Use Cases for Streaming Analytics
This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications.
In this talk, you learn more about:
1. What is Apache Flink Stack?
2. Batch vs. Streaming Analytics
3. Key Differentiators of Apache Flink for Streaming Analytics
4. Real-World Use Cases with Flink for Streaming Analytics
5. Who is using Flink?
6. Where do you go from here?
Digital and data journey demystified: how it all works
Learn how innovations will deliver business value as part of a larger digital transformation underway in your business. The innogy data stack with example of a pro-active retention initiative based on predictive analytics. How in-house data science competences can help you to innovate
Predicting Stock Market Price Using Support Vector Regression
The document presents a methodology for predicting stock market prices using support vector machine regression (SVR) with different windowing techniques. It involves collecting historical stock market data, preprocessing the data using various windowing approaches to convert the time series to a supervised learning format, training SVR models on the windowed data with different parameters, and evaluating the models' ability to predict stock prices on testing data. The results show that de-flattening and 5-day windows achieved the lowest prediction errors compared to the actual stock prices in the testing period.
Christian Kreuzfeld – Static vs Dynamic Stream Processing
The document discusses static vs dynamic stream processing. It covers using stream processing for the first time, increasing use cases, implementation issues, and requirements for stream processing frameworks. It then summarizes the SPQR and Apache Flink frameworks, highlighting how SPQR allows no-code topology definition while Flink provides many extension points. Finally, it discusses future directions, including using Apache Zeppelin for its support of dynamic queries on streaming data.
Optimizing connected system performance md&m-anaheim-sandhi bhide 02-07-2017
Sandhiprakash Bhide presenting at the Smart Manufacturing Innovation Summit/Industry 4.0 event on "Optimizing Connected System Performance and Establishing Tangible Goals for Sensor Use"
Big Data Architectures @ JAX / BigDataCon 2016
Mit der Architektur steht und fällt jedes IT-Projekt. Das gilt in noch stärkerem Maße für Big-Data-Projekte, denn hier konnten noch keine Standards über Jahrzehnte ihre Tauglichkeit beweisen. Dennoch verbreiten und etablieren sich auch hier gute und effektive Lösungen. Der Vortrag erklärt, welche Bausteine wichtig für die verschiedenen Einsatzmöglichkeiten im Big-Data-Umfeld sind, und wie sie in konkrete Lösungen gegossen werden können. Dabei beleuchtet er sowohl traditionelle Big-Data-Architekturen als auch aktuelle Ansätze, wie z. B. die Lambda- und die Kappa-Architektur. Ebenfalls ein Thema sind Stream-Processing-Infrastrukturen und ihre Kombination mit Big-Data-Technologien. Ausgehend von einer produkt- und technologieunabhängigen Referenzarchitektur stellt dieser Vortrag verschiedene Lösungsmöglichkeiten auf Basis von Open-Source-Komponenten vor.
Solving Cybersecurity at Scale
With the rise of IoT and the increasing complexity of applications, clouds, networks and infrastructure, the battle to keep your data and your infrastructure safe from attackers is getting harder. As groups of bad actors collaborate, sharing information and offering illegal access, and botnets as a service, terabits of attack can be launched cheaply. Meanwhile, it’s hard to find enough security analysts to catch and prevent these attacks.
This is where community collaboration and open source efforts like Apache Metron come in. Metron presents a comprehensive framework for application and network, security built on Apache Hadoop and open source Streaming Analytics(ie Apache Nifi, Apache Kafka) tool’s highly scalable data management and processing stacks. Advanced features like profiling, machine learning, and visualization work with real-time streaming detection to make your SOC analysts more efficient, while the intrinsic extensibility of open source helps your data scientists get security insights out of the lab and into production fast.
We will discuss and demonstrate how some real-world businesses and managed service providers are using Apache Metron to identify and solve security threats at scale, and some approaches and ideas for how the platform can fit into your security architecture.
Speaker: Laurence Da Luz, Senior Solutions Architect, Hortonworks
Data Stream Algorithms in Storm and R
Streaming data presents new challenges for statistics and machine learning on extremely large data sets. Tools such as Apache Storm, a stream processing framework, can power range of data analytics but lack advanced statistical capabilities. These slides are from the Apache.con talk, which discussed developing streaming algorithms with the flexibility of both Storm and R, a statistical programming language.
At the talk I dicsussed issues of why and how to use Storm and R to develop streaming algorithms; in particular I focused on:
• Streaming algorithms
• Online machine learning algorithms
• Use cases showing how to process hundreds of millions of events a day in (near) real time
See: https://apacheconna2015.sched.org/event/09f5a1cc372860b008bce09e15a034c4#.VUf7wxOUd5o
Revealing Differences in Designer‘s and Users‘Perspectives
A Tool-supported Process for Visual Attention Prediction for
Designing HMIs for Maritime Monitoring Tasks
Monitoring complex systems includes scanning, aggregating and processing data from various sources. The design of graphical interfaces for monitoring tasks involves a fine-grained exploration of the importance and expected frequency of events that an operator needs to be informed about.
The Human Efficiency Evaluator is a tool for the prediction of human behavior. We extended it to predict the distribution of operator’s attention while monitoring interfaces. The prediction is based on the SEEV model. We show that our tool can be used by experts with different backgrounds to generate predictions following a structured, semi-automated process.
In a qualitative study with subject matter experts, we analyzed different HMI designs for a navigation task in the maritime domain. We evaluated their mod-eling time, tested different prediction result visualizations, and investigated in the model differences between the subjects. Different to what we originally expected, the study revealed that the models created by the subjects substantially differ depending on their perspectives. Heat maps visualizing the predicted attention allocation were appreciated by the subjects and enabled them to argue about their perspective.
Self-Tuning Data Centers
Self-tuning data centers aim to minimize human intervention through machine learning techniques. Current challenges include meeting service level agreements for performance and uptime while maximizing efficiency of resources and minimizing costs. A self-tuning architecture uses monitoring data to detect issues and make recommendations for scaling, migration, or tuning of resources without human input. This approach aims to optimize data centers so they can scale efficiently to support growing workloads and applications.
Similar to Statistics for Engineers (20)
OSMC 2016 | Friends and foes in API Monitoring by Heinrich Hartmann
OSMC 2016 | Friends and foes in API Monitoring by Heinrich Hartmann
Lego-like building blocks of Storm and Spark Streaming Pipelines
Lego-like building blocks of Storm and Spark Streaming Pipelines
Product Engineer Certified Lean Six Sigma Black Belt by IASSC
Product Engineer Certified Lean Six Sigma Black Belt by IASSC
Apache Flink: Real-World Use Cases for Streaming Analytics
Apache Flink: Real-World Use Cases for Streaming Analytics
Digital and data journey demystified: how it all works
Digital and data journey demystified: how it all works
Predicting Stock Market Price Using Support Vector Regression
Predicting Stock Market Price Using Support Vector Regression
Christian Kreuzfeld – Static vs Dynamic Stream Processing
Christian Kreuzfeld – Static vs Dynamic Stream Processing
Optimizing connected system performance md&m-anaheim-sandhi bhide 02-07-2017
Optimizing connected system performance md&m-anaheim-sandhi bhide 02-07-2017
Revealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘Perspectives
More from Heinrich Hartmann
Linux System Monitoring with eBPF
Ignite talk given at dodkiel18 and dodzh18.
The long version is here: https://www.circonus.com/2018/05/linux-system-monitoring-with-ebpf/
Geometric Aspects of LSA
The document discusses latent semantic analysis (LSA) for determining the similarity of topics between documents. It outlines that LSA uses singular value decomposition to project documents into a lower dimensional space where similar topics are closer. The presentation will cover the geometry of SVD used in LSA and how the extremal vector in the reduced space represents the dominant topic that is common across documents.
Seminar on Complex Geometry
This seminar will cover the basics of complex manifolds and geometry through readings and presentations on topics from reference book [2]. The seminar will be organized as a reading group, with participants reading assigned material each week and one person giving a talk on a particular topic. Some of the topics to be covered include holomorphic functions, complex manifolds, vector bundles, divisors, projective space, differential forms, Kähler manifolds, and applications to curves.
Seminar on Motivic Hall Algebras
This document outlines a seminar on motivic Hall algebras. The seminar will cover topics including Grothendieck rings of varieties, motivic invariants like Euler characteristics and Hodge polynomials, algebraic spaces, and stacks. The main focus will be T. Bridgeland's paper on motivic Hall algebras, which uses motives and stacks in its construction of motivic Hall algebras.
GROUPOIDS, LOCAL SYSTEMS AND DIFFERENTIAL EQUATIONS
This document discusses groupoids, local systems, and their relationships to differential equations on manifolds. Some key points:
1) Groupoids generalize groups by allowing multiple objects and isomorphisms between them. Representations of groupoids correspond to local systems on manifolds.
2) Local systems on a manifold X are sheaves of vector spaces that are locally isomorphic to a constant sheaf. They correspond to representations of the fundamental groupoid of X.
3) Vector bundles with connections on a Riemann surface B are equivalent to local systems on B. Global sections of bundles generate differential equations, whose solutions can be studied via the bundle's local system or groupoid representation.
Topics in Category Theory
This document discusses topics in category theory, including set-functors, adjunctions, and limits. It begins by defining set-functors and natural transformations between them. It notes that a natural transformation is uniquely determined by its value on an initial element of a functor. It then introduces adjunctions and decomposes them into left and right adjoints. It shows that a left adjoint exists if and only if certain set-functors are representable. Finally, it defines limits of diagrams (I-systems) over an index category I. It shows that a limit exists if and only if the cone functor is representable.
Related-Work.net at WeST Oberseminar
This document discusses Related-Work.net, a scientific discussion platform and open database of papers and citations. It describes the vision for the platform to be a social community for scientists with open data and free software. The document outlines the history of Related-Work.net and its merger with OpenCitations.net. It also describes some of the challenges in building the platform, such as matching citations to papers and identifying authors, as well as examples of potential data mining applications.
Komplexe Zahlen
Schülerprojekt 2002
Inhalt:
* Einführung Komplexe Zahlen
* Exponentialfunktion und DeMoiver Formeln
* Penningfallen
* Rollkurven
* Fraktale
Pushforward of Differential Forms
1) The document discusses fiber integrals of differential forms along oriented submersions. It defines the pushforward or fiber integral f* of a differential form along a submersion f, which maps forms on the total space to forms on the base space.
2) Properties of the fiber integral are proven, including normalization, the Fubini theorem, base change formulas, and Stokes' theorem.
3) The document also discusses orientations of vector bundles and submersions, which are needed to define the fiber integral.
Polynomproblem
Beweis der folgenden Aussage:
Sei f ∈ C ∞ (R) mit der Eigenschaft, dass ∀ x ∈ R ∃ n ∈ N mit f (n) (x) = 0. Dann ist f ein Polynom.
Hecke Curves and Moduli spcaes of Vector Bundles
This are the notes of a seminar talk delivered in summer 2008 at Bonn. Let SU (2, 1) be the
moduli space of rank 2 bundles with a fixed determinant of rank 1 over a curve C of genus g ≥ 2.
This is a Fano manifold of Picard rank 1. We discuss the example g = 2 where SU (2, 1) is the
intersection of two quadrics in P5 . In this case the minimal rational curves are lines. There is a
very interesting class of rational curves on SU (2, 1), called Hecke curves, which are constructed
by extending a given bundle by torsion sheaves. In the case g ≥ 3 we will see that Hecke curves
have minimal anti-canonical degree (4) and that any rational curve passing through a generic
point is a Hecke curve.
Local morphisms are given by composition
(1) Local morphisms between function-like ringed spaces are given by composition of functions. A morphism is local if the preimage of the maximal ideal at each point is the maximal ideal of the image point.
(2) A morphism is C-linear if it maps each function to the function composed with the continuous map. For a function-like ringed space, a morphism is local and C-linear if and only if it is given by composition of functions.
(3) The maximal ideal at each point of a function-like ringed space is the intersection of the maximal ideal of vanishing functions and the structure sheaf at that point.
Notes on Intersection theory
Notes on intersection theory written for a seminar in Bonn in 2010.
Following Fulton's book the following topics are covered:
- Motivation of intersection theory
- Cones and Segre Classes
- Chern Classes
- Gauss-Bonet Formula
- Segre classes under birational morphisms
- Flat pull back
Cusps of the Kähler moduli space and stability conditions on K3 surfaces
Presentation about the paper with the same title http://arxiv.org/abs/1012.3121
Abstract:
In [Ma1] S. Ma established a bijection between Fourier--Mukai partners of a K3 surface and cusps of the K\"ahler moduli space. The K\"ahler moduli space can be described as a quotient of Bridgeland's stability manifold. We study the relation between stability conditions σ near to a cusp and the associated Fourier--Mukai partner Y in the following ways. (1) We compare the heart of σ to the heart of coherent sheaves on Y. (2) We construct Y as moduli space of σ-stable objects.
An appendix is devoted to the group of auto-equivalences of the derived category which respect the component Stab†(X) of the stability manifold
Log Files
Log files provide insights into systems like web servers and databases by recording details of requests, responses, and operations over time. They can be used for monitoring systems, troubleshooting issues, and analyzing usage patterns. However, the large volumes of log data produced require efficient processing and aggregation approaches to gain these insights in real-time or through batch analysis. Common techniques include shipping logs to a central aggregator, using group communication protocols for real-time distribution, and batch processing with Hadoop/MapReduce.
More from Heinrich Hartmann (19)
GROUPOIDS, LOCAL SYSTEMS AND DIFFERENTIAL EQUATIONS
GROUPOIDS, LOCAL SYSTEMS AND DIFFERENTIAL EQUATIONS
Cusps of the Kähler moduli space and stability conditions on K3 surfaces
Cusps of the Kähler moduli space and stability conditions on K3 surfaces
Recently uploaded
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Trusted Execution Environment for Decentralized Process Mining
Presentation of the paper "Trusted Execution Environment for Decentralized Process Mining" given during the CAiSE 2024 Conference in Cyprus on June 7, 2024.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
dbms calicut university B. sc Cs 4th sem.pdf
Its a seminar ppt on database management system using sql
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Operating System Used by Users in day-to-day life.pptx
Dive into the realm of operating systems (OS) with Pravash Chandra Das, a seasoned Digital Forensic Analyst, as your guide. 🚀 This comprehensive presentation illuminates the core concepts, types, and evolution of OS, essential for understanding modern computing landscapes.
Beginning with the foundational definition, Das clarifies the pivotal role of OS as system software orchestrating hardware resources, software applications, and user interactions. Through succinct descriptions, he delineates the diverse types of OS, from single-user, single-task environments like early MS-DOS iterations, to multi-user, multi-tasking systems exemplified by modern Linux distributions.
Crucial components like the kernel and shell are dissected, highlighting their indispensable functions in resource management and user interface interaction. Das elucidates how the kernel acts as the central nervous system, orchestrating process scheduling, memory allocation, and device management. Meanwhile, the shell serves as the gateway for user commands, bridging the gap between human input and machine execution. 💻
The narrative then shifts to a captivating exploration of prominent desktop OSs, Windows, macOS, and Linux. Windows, with its globally ubiquitous presence and user-friendly interface, emerges as a cornerstone in personal computing history. macOS, lauded for its sleek design and seamless integration with Apple's ecosystem, stands as a beacon of stability and creativity. Linux, an open-source marvel, offers unparalleled flexibility and security, revolutionizing the computing landscape. 🖥️
Moving to the realm of mobile devices, Das unravels the dominance of Android and iOS. Android's open-source ethos fosters a vibrant ecosystem of customization and innovation, while iOS boasts a seamless user experience and robust security infrastructure. Meanwhile, discontinued platforms like Symbian and Palm OS evoke nostalgia for their pioneering roles in the smartphone revolution.
The journey concludes with a reflection on the ever-evolving landscape of OS, underscored by the emergence of real-time operating systems (RTOS) and the persistent quest for innovation and efficiency. As technology continues to shape our world, understanding the foundations and evolution of operating systems remains paramount. Join Pravash Chandra Das on this illuminating journey through the heart of computing. 🌟
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Recently uploaded (20)
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Statistics for Engineers
- 1. Monitorama PDX, June 29th 2016 Statistics for Engineers Heinrich Hartmann, Circonus
- 2. @HeinrichHartman Hi, I am Heinrich · Lives in Munich, EU · Refugee from Academia (Ph.D.) · Analytics Lead at Circonus, Monitoring and Analytics Platform heinrich.hartmann@circonus.com @HeinrichHartman(n)
- 3. @HeinrichHartman #StatsForEngineers has been around for a while [1] Statistics for Engineers @ ACM Queue [2] Statistics for Engineers Workshop Material @ GitHub [3] Spike Erosion @ circonus.com [4] T. Schlossnagle - The Problem with Math @ circonus.com [5] T. Schlossnagle - Percentages are not People @ circonus.com [6] W. Vogels - Service Level Agreements in Amazon’s Dynamo/Sec. 2.2 [7] G. Schlossnagle - API Performance Monitoring @ Velocity Bejing 2015 Upcoming [8] 3h workshop “Statistics for Engineers” @ SRECon 2016 in Dublin
- 4. @HeinrichHartman A tale of API Monitoring
- 5. @HeinrichHartman “Attic” - a furniture webstore · Attic is a (fictional) furniture webstore · Web API serving their catalog · Loses money if requests take too long Monitoring Goals 1. Measure user experience / quality of service 2. Determine (financial) implications of service degradation 3. Define sensible SLA-targets for the Dev- and Ops-teams
- 7. @HeinrichHartman {1} External API Monitoring Method 1. Make a synthetic request every minute 2. Measure and store request latency Good for · Measure Availability · Alert on outages Bad for · Measuring user experience Latencies of synthetic requests over time
- 8. @HeinrichHartman <!> Spike Erosion </!> · On long time ranges, aggregated / rolled-up data is commonly displayed · This practice “erodes” latency spikes heavily! · Store all data and use alternative aggregation methods (min/max) to get full picture, cf. [3]. 1d max all samples as Heatmap / ‘dirt’
- 10. @HeinrichHartman Method Write to log file: - time of completion, - request latency, and further metadata. Discussion · Rich information source for all kinds of analysis · Easy instrumentation (printf) · Slow. Long delay (minutes) before data is indexed and becomes accessible for analysis · Expensive. Not feasibile for high volume APIs {2} Log Analysis Internal view of an API - “UML” version.
- 11. @HeinrichHartman Numerical Digest: The Request-Latency Chart a concise visualization of the API usage Latency on the y- axis time the request was completed
- 12. @HeinrichHartman Construction of the Request-Latency Chart (RLC) Request Latency UML Diagram Request Latency Chart
- 13. @HeinrichHartman Math view on APIs (A) Latency distribution (B) Arrival/Completion times (C) Queuing theory
- 14. @HeinrichHartman “Requests are People” If you care about your users, you care about their requests. Every single one.
- 15. @HeinrichHartman {3} Monitoring Latency Averages
- 16. @HeinrichHartman {3} What are latency mean values? reporting period
- 17. @HeinrichHartman {3} Mean Request Latency Monitoring Method 1. Select a reporting period (e.g. 1 min) 2. For each period report the mean latency Pro/Con + Measure requests by actual people + Cheap to collect store and analyze - Easily skewed by outliers at the high end (complex, long running requests) - ... and the low end (cached responses) “Measuring the average latency is like measuring the average temperature in a hospital.” -- Dogan @ Optimizely
- 18. @HeinrichHartman {3} Mean Request Latency in practice
- 19. @HeinrichHartman {3} Mean Request Latency - Robust Variants 1. Median Latency - Sort latency values in reporting period - The median is the ‘central’ value. 2. Truncated Means - Take out min and max latencies in reporting period (k-times). - Then compute the mean value 3. Collect Deviation Measures - Avoid standdard deviations, use - Use Mean absolute deviation Construction of the median latency
- 21. @HeinrichHartman {4} What are Percentiles?
- 22. @HeinrichHartman {4} Percentile Monitoring Method 1. Select a reporting period (e.g. 1 min) 2. For each reporting period measure the 50%, 90%, 99%, 99.9% latency percentile 3. Alert when percentiles are over a threshold value Pro/Con + Measure requests by actual people + Cheap to collect store and analyze + Robust to Outliers - Up-front choice of percentiles needed - Can not be aggregated
- 23. {5} How it looks in practice Latency percentiles 50,90,99 computed over 1m reporting periods
- 24. <!> Percentiles can’t be aggregated </!> The median of two medians is NOT the total median. If you store percentiles you need to: A. Keep all your data. Never take average rollups! B. Store percentiles for all aggregation levels separately, e.g. ○ per Node / Rack / DC ○ per Endpoint / Service C. Store percentiles for all reporting periods you are interested in, e.g. per min / h / day D. Store all percentiles you will ever be interested in, e.g. 50, 75, 90, 99, 99.9 Further Reading: [4] T. Schlossnagle - The Problem with Math @ circonus.com
- 25. @HeinrichHartman {5} API Monitoring with Histograms
- 26. {5} API Monitoring with Histograms Method 1. Divide latency scale into bands 2. Divide the time scale into reporting periods 3. Count the number of samples in each latency band x reporting period Discussion · Summary of full RLC, with reduced precision · Extreme compression compared to logs · Percentiles, averages, medians, etc. can be derived · Aggregation across time and nodes trivial · Allows more meaningful metrics latency sample count time
- 27. {5} Histogram Monitoring in Practice Histograms can be visualized as heatmaps. Aggregate data from all nodes serving “web-api” .. across windows of 10min.
- 28. {5} Histogram Monitoring in Practice All kinds of metrics can be derived from histograms
- 29. @HeinrichHartman {6} The search for meaningful metrics
- 30. {6} Users offended per minute
- 31. {6} Total users offended so far
- 32. @HeinrichHartman Takeaways · Don’t trust line graphs (at least on large scale) · Don’t aggregate percentiles. Aggregate histograms. · Keep your data · Strive for meaningful metrics