Atmosphere 2016 - Albert Lacki, Jaroslaw Bloch - Real user monitoring at scale - dream lab
•
1 like•282 views
A pretty detailed story of how we built a real-time user monitoring platform, gathering data of millions of users. Using the joint forces of CDN, Cloud and BigData, we created a tool for developers and product owners to guide them towards right (and data-driven) product decisions.
Report
Share
Report
Share
Download to read offline
Recommended
Real User Monitoring at Scale @ Atmosphere Conference 2016
This document summarizes a real user monitoring system at scale. The key points are:
1. The system monitors over 20 million real users, 7 million requests per minute, and 150 million daily page views across 3 data centers and 3,000 servers.
2. Data is collected from over 40 teams and 250 daily deployments using script injection and sent to 3 data centers. It is ingested into Kafka and replicated for redundancy.
3. The ingested data is processed by Storm and aggregated metrics are stored in Graphite for presentation and analysis to provide insights for over 300 specialists. Custom alerts are also used.
Server architecture
The document discusses server architecture and different types of servers. It describes common server roles like web servers, application servers, and proxy servers. It provides examples of simple web servers written in Node.js and Ruby. Popular web servers mentioned include Nginx and Apache. Different hosting options for servers are also covered, such as virtual dedicated servers (VDS), virtual private servers (VPS), and cloud servers hosted on platforms like Amazon AWS, Google Cloud, and Microsoft Azure.
ATS Internals
The document provides an overview of the internal components of ATS (Apache Traffic Server) and outlines plans for analyzing and documenting the source code. It includes a directory of chapters planned to cover various subsystems like the event system, I/O core, session handling, state machines, caching, and more. It also summarizes some of the key classes and interfaces in the event system, I/O core networking subsystem, and how network connections are handled. The author reflects on improvements that could be made and plans to support documentation in multiple languages.
Data Collection & Prometheus Scraping with Sensu 2.0
Applications are complex systems. Their many moving parts, component and dependency services, may span any number of infrastructure technologies and platforms, from bare metal to serverless. As the number of services increases, teams responsible for them will naturally develop their own preferences, such as how they instrument their code or how and when they receive alerts.
Cometd / ShortBus Slide Presentation
ShortBus is a system for sending and receiving events over HTTP connections that allows for long-lived persistent connections, event replay, and pluggable data formats like JSON. It includes a Perlbal plugin that runs as a proxy service, JavaScript and Perl libraries, and supports uses like live chat, stock tickers, multiplayer games, and synchronizing content across multiple user devices. Events are authorized and clients connected by the web server, which can then acknowledge receipt of events.
Liquid Stream Processing Across Web Browsers and Web Servers
ICWE2015 presentation on the control infrastructure of the Web Liquid Streams framework, a data stream framework developed at the University of Lugano, Switzerland.
OSMC 2018 | Monitoring with Sensu 2.0 by Sean Porter
Sean Porter, CTO of Sensu Inc., discussed monitoring in modern, complex infrastructures using Sensu Go. He outlined three methods for collecting monitoring data: 1) service checks using scripts and exit codes, 2) an events API for entity management and external checks, and 3) StatsD for metrics aggregation. Porter emphasized that ephemeral infrastructure is now common, requiring monitoring solutions that can scale with containers, microservices, and cloud-based infrastructure. Sensu Go provides monitoring as infrastructure scales in complexity over time.
Rest is bad
5 minutes ignite talk.
Rest based API is the worst choice that you can do with your Microservices ecosystem.
Recommended
Real User Monitoring at Scale @ Atmosphere Conference 2016
This document summarizes a real user monitoring system at scale. The key points are:
1. The system monitors over 20 million real users, 7 million requests per minute, and 150 million daily page views across 3 data centers and 3,000 servers.
2. Data is collected from over 40 teams and 250 daily deployments using script injection and sent to 3 data centers. It is ingested into Kafka and replicated for redundancy.
3. The ingested data is processed by Storm and aggregated metrics are stored in Graphite for presentation and analysis to provide insights for over 300 specialists. Custom alerts are also used.
Server architecture
The document discusses server architecture and different types of servers. It describes common server roles like web servers, application servers, and proxy servers. It provides examples of simple web servers written in Node.js and Ruby. Popular web servers mentioned include Nginx and Apache. Different hosting options for servers are also covered, such as virtual dedicated servers (VDS), virtual private servers (VPS), and cloud servers hosted on platforms like Amazon AWS, Google Cloud, and Microsoft Azure.
ATS Internals
The document provides an overview of the internal components of ATS (Apache Traffic Server) and outlines plans for analyzing and documenting the source code. It includes a directory of chapters planned to cover various subsystems like the event system, I/O core, session handling, state machines, caching, and more. It also summarizes some of the key classes and interfaces in the event system, I/O core networking subsystem, and how network connections are handled. The author reflects on improvements that could be made and plans to support documentation in multiple languages.
Data Collection & Prometheus Scraping with Sensu 2.0
Applications are complex systems. Their many moving parts, component and dependency services, may span any number of infrastructure technologies and platforms, from bare metal to serverless. As the number of services increases, teams responsible for them will naturally develop their own preferences, such as how they instrument their code or how and when they receive alerts.
Cometd / ShortBus Slide Presentation
ShortBus is a system for sending and receiving events over HTTP connections that allows for long-lived persistent connections, event replay, and pluggable data formats like JSON. It includes a Perlbal plugin that runs as a proxy service, JavaScript and Perl libraries, and supports uses like live chat, stock tickers, multiplayer games, and synchronizing content across multiple user devices. Events are authorized and clients connected by the web server, which can then acknowledge receipt of events.
Liquid Stream Processing Across Web Browsers and Web Servers
ICWE2015 presentation on the control infrastructure of the Web Liquid Streams framework, a data stream framework developed at the University of Lugano, Switzerland.
OSMC 2018 | Monitoring with Sensu 2.0 by Sean Porter
Sean Porter, CTO of Sensu Inc., discussed monitoring in modern, complex infrastructures using Sensu Go. He outlined three methods for collecting monitoring data: 1) service checks using scripts and exit codes, 2) an events API for entity management and external checks, and 3) StatsD for metrics aggregation. Porter emphasized that ephemeral infrastructure is now common, requiring monitoring solutions that can scale with containers, microservices, and cloud-based infrastructure. Sensu Go provides monitoring as infrastructure scales in complexity over time.
Rest is bad
5 minutes ignite talk.
Rest based API is the worst choice that you can do with your Microservices ecosystem.
Treasure Data and AWS - Developers.io 2015
This document discusses Treasure Data's data architecture. It describes how Treasure Data collects and imports log data using Fluentd. The data is stored in columnar format in S3 and metadata is stored in PostgreSQL. Treasure Data uses Presto to enable fast analytics on the large datasets. The document provides details on the import process, storage, partitioning, and optimizations to improve query performance.
Supporting Enterprise System Rollouts with Splunk
At Cricket Communications, Splunk started as a way to correlate all of our data into one view to help our operations team keep processes humming. Then we gave secured access to our developers, now they’re addicted. In fact, Splunk is critical in helping us speedup deployment of new systems (like our recent multi-million dollar billing system implementation). Learn how we use Splunk to display key metrics for the business, track overall system health, track transactions, optimize license usage, and support capacity
planning.
From Batch to Streaming ET(L) with Apache Apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
* Pipeline functionality from event source through queryable state for real-time insights.
* API for application development and development process.
* Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
* Stateful processing with event time windowing.
* Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
* Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
* Who is using Apex in production, and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Fitur AppManager - Application Manager ManageEngine
ManageEngine Applications Manager is a solution that monitors IT infrastructure including applications, servers, databases, operating systems, and network services across physical, virtual, and cloud environments. It provides unified monitoring of business infrastructure and diverse applications. Key functions include agentless monitoring of heterogeneous infrastructure, alerts and root cause analysis, reporting, and support for ITIL processes like SLA management. It is available in various product editions for small/medium/enterprise users.
Monitoring patterns for mitigating technical risk
How to define auto healing and alerts for a low latency REST API system. Includes real world riemann code snippets.
5 Years of Progress in Active Data Warehousing
Teradata's Dan Graham, , presentation from the 2010 Teradata User Group meetings on Active Data Warehousing over the last five years.
For more information on Active Data Warehousing, please visit Teradata.com
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
eal-time Processing with Flink for Machine Learning at Netflix
Machine learning plays a critical role in providing a great Netflix member experience. It is used to drive many parts of the site including video recommendations, search results ranking, and selection of artwork images. Providing high-fidelity, near real-time data is increasingly important for these machine learning pipelines, especially as multi-armed bandit and reinforcement learning techniques, in addition to more ""traditional"" supervised learning, become more prevalent. With access to this data, models are able to converge more quickly, features can be updated more frequently, and analysis can be done in a more timely manner.
In this talk, we will focus on the practical details of leveraging Flink to process trillions of events per day, work with the time dimension, and manage large and frequently-changing state. We will discuss different processing schemes and dataflows, scalability and resiliency challenges we tackled, operational considerations, and instrumentation we added for monitoring job health in production.
Digdagによる大規模データ処理の自動化とエラー処理
Digdag can automate large-scale data processing and handle errors. It provides constructs like operators, parameters, and task groups to organize workflows. Operators package tasks to run queries or process data. Parameters allow passing variables between tasks. Task groups modularize and organize workflows. Digdag supports error handling, monitoring, parallelization, versioning, and reproducing workflows across environments.
Log management system for Microservices
How to design Log Management System for microservices architect. Focus on Log classification, with two groups: Business Log and Operation Log.
From Batch to Streaming with Apache Apex Dataworks Summit 2017
This document discusses transitioning from batch to streaming data processing using Apache Apex. It provides an overview of Apex and how it can be used to build real-time streaming applications. Examples are given of how to build an application that processes Twitter data streams and visualizes results. The document also outlines Apex's capabilities for scalable stream processing, queryable state, and its growing library of connectors and transformations.
From Batch to Streaming ET(L) with Apache Apex at Berlin Buzzwords 2017
https://berlinbuzzwords.de/17/session/batch-streaming-etl-apache-apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
Pipeline functionality from event source through queryable state for real-time insights.
API for application development and development process.
Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
Stateful processing with event time windowing.
Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
Recent project development and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Apache Flink @ Tel Aviv / Herzliya Meetup
Slides of my Apache Flink talk "Big Things" meetup at Proofpoint and "Apache Flink" meetup at Clicktale in January 2018
Otimizando servidores web
Easy Taxi está presente em mais de 30 países e tem milhões de usuários, entre passageiros e taxistas. Seu aplicativo roda em dezenas de plataformas móveis e suporta milhares de acessos simultâneos. A aplicação nasceu na nuvem da AWS e faz pleno uso de todos os seus recursos. Nesta apresentação avançada, exploramos a arquitetura da Easy Taxi e analisamos as estratégias de otimização disponíveis para os aplicativos implementados na nuvem AWS.
Plazma - Treasure Data’s distributed analytical database -
This document summarizes Plazma, Treasure Data's distributed analytical database that can import 40 billion records per day. It discusses how Plazma reliably imports and processes large volumes of data through its scalable architecture with real-time and archive storage. Data is imported using Fluentd and processed using its column-oriented, schema-on-read design to enable fast queries. The document also covers Plazma's transaction API and how it is optimized for metadata operations.
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
Presented at LISA18: https://www.usenix.org/conference/lisa18/presentation/huynh
While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:
* Disk latency and io size histograms
* Run queue (scheduler) latency
* Page cache efficiency
* Directory cache efficiency
* LLC (aka L3 cache) efficiency
* Kernel timer counters
* System-wide TCP retransmits
Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it.
If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics.
In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.
Billions of Rows, Millions of Insights, Right Now
Presentation from Tableau Customer Conference 2013 on building a real time reporting/analytics platform. Topics discussed include definitions of big data and real time, technology choices and rationale, use cases for real time big data, architecture, and pitfalls to avoid.
Monitoring with Syslog and EventMachine
This document discusses monitoring systems using syslog and EventMachine. It proposes building a lightweight, polyglot system that aggregates syslog events and displays metrics and visualizations using various protocols like WebSockets, Server-Sent Events, and Graphite. Event sources would send syslog messages which an EventMachine server would parse and pass to an EM:Channel. A JavaScript client could subscribe to the channel for real-time updates.
Scale and Throughput @ Clicktale with Akka
In the world of big data we need to build services that will be able to collect massive data, save it and pass it to processing and analysis. However, building manageable, reliable services that are scalable and cost effective is not an easy task. The choice of eco-system, frameworks and programming language, as well as using solid engineering principles is also crucial for achieving this goal.
I will share our journey and insights from rebuilding a cloud service in Linux eco-system using Scala, Akka Actors and Aerospike DB, at the end of which we gained 10 folds improvement of server usage with a much lighter, stable and reliable system that handles tens of millions of requests per hour.
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache FlinkDataWorks Summit/Hadoop Summit
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at a sub-second latency. One of these systems is Apache Flink and it enables applications that include typical CEP features as well as heavy aggregations.
Guided by examples, I will demonstrate how Apache Flink enables the user to process CEP and stream analytics workloads alike. Starting from aggregations over streams, we will next detect temporal patterns in our data triggering alerts and finally aggregate these alerts to gain more insights from our data. As an outlook, I will present Flink's CEP-enriched StreamSQL interface providing a declarative way to specify temporal patterns in your SQL query.Streaming SQL to unify batch and stream processing: Theory and practice with ...
SQL is the lingua franca for querying and processing data. To this day, it provides non-programmers with a powerful tool for analyzing and manipulating data. But with the emergence of stream processing as a core technology for data infrastructures, can you still use SQL and bring real-time data analysis to a broader audience?
The answer is yes, you can. SQL fits into the streaming world very well and forms an intuitive and powerful abstraction for streaming analytics. More importantly, you can use SQL as an abstraction to unify batch and streaming data processing. Viewing streams as dynamic tables, you can obtain consistent results from SQL evaluated over static tables and streams alike and use SQL to build materialized views as a data integration tool.
Fabian Hueske and Shuyi Chen explore SQL’s role in the world of streaming data and its implementation in Apache Flink and cover fundamental concepts, such as streaming semantics, event time, and incremental results. They also share their experience using Flink SQL in production at Uber, explaining how Uber leverages Flink SQL to solve its unique business challenges and how the unified stream and batch processing platform enables both technical or nontechnical users to process real-time and batch data reliably using the same SQL at Uber scale.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
More Related Content
Similar to Atmosphere 2016 - Albert Lacki, Jaroslaw Bloch - Real user monitoring at scale - dream lab
Treasure Data and AWS - Developers.io 2015
This document discusses Treasure Data's data architecture. It describes how Treasure Data collects and imports log data using Fluentd. The data is stored in columnar format in S3 and metadata is stored in PostgreSQL. Treasure Data uses Presto to enable fast analytics on the large datasets. The document provides details on the import process, storage, partitioning, and optimizations to improve query performance.
Supporting Enterprise System Rollouts with Splunk
At Cricket Communications, Splunk started as a way to correlate all of our data into one view to help our operations team keep processes humming. Then we gave secured access to our developers, now they’re addicted. In fact, Splunk is critical in helping us speedup deployment of new systems (like our recent multi-million dollar billing system implementation). Learn how we use Splunk to display key metrics for the business, track overall system health, track transactions, optimize license usage, and support capacity
planning.
From Batch to Streaming ET(L) with Apache Apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
* Pipeline functionality from event source through queryable state for real-time insights.
* API for application development and development process.
* Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
* Stateful processing with event time windowing.
* Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
* Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
* Who is using Apex in production, and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Fitur AppManager - Application Manager ManageEngine
ManageEngine Applications Manager is a solution that monitors IT infrastructure including applications, servers, databases, operating systems, and network services across physical, virtual, and cloud environments. It provides unified monitoring of business infrastructure and diverse applications. Key functions include agentless monitoring of heterogeneous infrastructure, alerts and root cause analysis, reporting, and support for ITIL processes like SLA management. It is available in various product editions for small/medium/enterprise users.
Monitoring patterns for mitigating technical risk
How to define auto healing and alerts for a low latency REST API system. Includes real world riemann code snippets.
5 Years of Progress in Active Data Warehousing
Teradata's Dan Graham, , presentation from the 2010 Teradata User Group meetings on Active Data Warehousing over the last five years.
For more information on Active Data Warehousing, please visit Teradata.com
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
eal-time Processing with Flink for Machine Learning at Netflix
Machine learning plays a critical role in providing a great Netflix member experience. It is used to drive many parts of the site including video recommendations, search results ranking, and selection of artwork images. Providing high-fidelity, near real-time data is increasingly important for these machine learning pipelines, especially as multi-armed bandit and reinforcement learning techniques, in addition to more ""traditional"" supervised learning, become more prevalent. With access to this data, models are able to converge more quickly, features can be updated more frequently, and analysis can be done in a more timely manner.
In this talk, we will focus on the practical details of leveraging Flink to process trillions of events per day, work with the time dimension, and manage large and frequently-changing state. We will discuss different processing schemes and dataflows, scalability and resiliency challenges we tackled, operational considerations, and instrumentation we added for monitoring job health in production.
Digdagによる大規模データ処理の自動化とエラー処理
Digdag can automate large-scale data processing and handle errors. It provides constructs like operators, parameters, and task groups to organize workflows. Operators package tasks to run queries or process data. Parameters allow passing variables between tasks. Task groups modularize and organize workflows. Digdag supports error handling, monitoring, parallelization, versioning, and reproducing workflows across environments.
Log management system for Microservices
How to design Log Management System for microservices architect. Focus on Log classification, with two groups: Business Log and Operation Log.
From Batch to Streaming with Apache Apex Dataworks Summit 2017
This document discusses transitioning from batch to streaming data processing using Apache Apex. It provides an overview of Apex and how it can be used to build real-time streaming applications. Examples are given of how to build an application that processes Twitter data streams and visualizes results. The document also outlines Apex's capabilities for scalable stream processing, queryable state, and its growing library of connectors and transformations.
From Batch to Streaming ET(L) with Apache Apex at Berlin Buzzwords 2017
https://berlinbuzzwords.de/17/session/batch-streaming-etl-apache-apex
Stream data processing is increasingly required to support business needs for faster actionable insight with growing volume of information from more sources. Apache Apex is a true stream processing framework for low-latency, high-throughput and reliable processing of complex analytics pipelines on clusters. Apex is designed for quick time-to-production, and is used in production by large companies for real-time and batch processing at scale.
This session will use an Apex production use case to walk through the incremental transition from a batch pipeline with hours of latency to an end-to-end streaming architecture with billions of events per day which are processed to deliver real-time analytical reports. The example is representative for many similar extract-transform-load (ETL) use cases with other data sets that can use a common library of building blocks. The transform (or analytics) piece of such pipelines varies in complexity and often involves business logic specific, custom components.
Topics include:
Pipeline functionality from event source through queryable state for real-time insights.
API for application development and development process.
Library of building blocks including connectors for sources and sinks such as Kafka, JMS, Cassandra, HBase, JDBC and how they enable end-to-end exactly-once results.
Stateful processing with event time windowing.
Fault tolerance with exactly-once result semantics, checkpointing, incremental recovery
Scalability and low-latency, high-throughput processing with advanced engine features for auto-scaling, dynamic changes, compute locality.
Recent project development and roadmap.
Following the session attendees will have a high level understanding of Apex and how it can be applied to use cases at their own organizations.
Apache Flink @ Tel Aviv / Herzliya Meetup
Slides of my Apache Flink talk "Big Things" meetup at Proofpoint and "Apache Flink" meetup at Clicktale in January 2018
Otimizando servidores web
Easy Taxi está presente em mais de 30 países e tem milhões de usuários, entre passageiros e taxistas. Seu aplicativo roda em dezenas de plataformas móveis e suporta milhares de acessos simultâneos. A aplicação nasceu na nuvem da AWS e faz pleno uso de todos os seus recursos. Nesta apresentação avançada, exploramos a arquitetura da Easy Taxi e analisamos as estratégias de otimização disponíveis para os aplicativos implementados na nuvem AWS.
Plazma - Treasure Data’s distributed analytical database -
This document summarizes Plazma, Treasure Data's distributed analytical database that can import 40 billion records per day. It discusses how Plazma reliably imports and processes large volumes of data through its scalable architecture with real-time and archive storage. Data is imported using Fluentd and processed using its column-oriented, schema-on-read design to enable fast queries. The document also covers Plazma's transaction API and how it is optimized for metadata operations.
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
Presented at LISA18: https://www.usenix.org/conference/lisa18/presentation/huynh
While there are plenty of readily available metrics for monitoring Linux kernel, many gems remain hidden. With the help of recent developments in eBPF, it is now possible to run safe programs in the kernel to collect arbitrary information with little to no overhead. A few examples include:
* Disk latency and io size histograms
* Run queue (scheduler) latency
* Page cache efficiency
* Directory cache efficiency
* LLC (aka L3 cache) efficiency
* Kernel timer counters
* System-wide TCP retransmits
Practically any event from "perf list" output and any kernel function can be traced, analyzed and turned into a Prometheus metric with almost arbitrary labels attached to it.
If you are already familiar with BCC tools, you may think if ebpf_exporter as bcc tools turned into prometheus metrics.
In this tutorial we’ll go over eBPF basics, how to write programs and get insights into a running system.
Billions of Rows, Millions of Insights, Right Now
Presentation from Tableau Customer Conference 2013 on building a real time reporting/analytics platform. Topics discussed include definitions of big data and real time, technology choices and rationale, use cases for real time big data, architecture, and pitfalls to avoid.
Monitoring with Syslog and EventMachine
This document discusses monitoring systems using syslog and EventMachine. It proposes building a lightweight, polyglot system that aggregates syslog events and displays metrics and visualizations using various protocols like WebSockets, Server-Sent Events, and Graphite. Event sources would send syslog messages which an EventMachine server would parse and pass to an EM:Channel. A JavaScript client could subscribe to the channel for real-time updates.
Scale and Throughput @ Clicktale with Akka
In the world of big data we need to build services that will be able to collect massive data, save it and pass it to processing and analysis. However, building manageable, reliable services that are scalable and cost effective is not an easy task. The choice of eco-system, frameworks and programming language, as well as using solid engineering principles is also crucial for achieving this goal.
I will share our journey and insights from rebuilding a cloud service in Linux eco-system using Scala, Akka Actors and Aerospike DB, at the end of which we gained 10 folds improvement of server usage with a much lighter, stable and reliable system that handles tens of millions of requests per hour.
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache FlinkDataWorks Summit/Hadoop Summit
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at a sub-second latency. One of these systems is Apache Flink and it enables applications that include typical CEP features as well as heavy aggregations.
Guided by examples, I will demonstrate how Apache Flink enables the user to process CEP and stream analytics workloads alike. Starting from aggregations over streams, we will next detect temporal patterns in our data triggering alerts and finally aggregate these alerts to gain more insights from our data. As an outlook, I will present Flink's CEP-enriched StreamSQL interface providing a declarative way to specify temporal patterns in your SQL query.Streaming SQL to unify batch and stream processing: Theory and practice with ...
SQL is the lingua franca for querying and processing data. To this day, it provides non-programmers with a powerful tool for analyzing and manipulating data. But with the emergence of stream processing as a core technology for data infrastructures, can you still use SQL and bring real-time data analysis to a broader audience?
The answer is yes, you can. SQL fits into the streaming world very well and forms an intuitive and powerful abstraction for streaming analytics. More importantly, you can use SQL as an abstraction to unify batch and streaming data processing. Viewing streams as dynamic tables, you can obtain consistent results from SQL evaluated over static tables and streams alike and use SQL to build materialized views as a data integration tool.
Fabian Hueske and Shuyi Chen explore SQL’s role in the world of streaming data and its implementation in Apache Flink and cover fundamental concepts, such as streaming semantics, event time, and incremental results. They also share their experience using Flink SQL in production at Uber, explaining how Uber leverages Flink SQL to solve its unique business challenges and how the unified stream and batch processing platform enables both technical or nontechnical users to process real-time and batch data reliably using the same SQL at Uber scale.
Similar to Atmosphere 2016 - Albert Lacki, Jaroslaw Bloch - Real user monitoring at scale - dream lab (20)
Fitur AppManager - Application Manager ManageEngine
Fitur AppManager - Application Manager ManageEngine
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
Flink Forward San Francisco 2019: Real-time Processing with Flink for Machine...
From Batch to Streaming with Apache Apex Dataworks Summit 2017
From Batch to Streaming with Apache Apex Dataworks Summit 2017
From Batch to Streaming ET(L) with Apache Apex at Berlin Buzzwords 2017
From Batch to Streaming ET(L) with Apache Apex at Berlin Buzzwords 2017
Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
LISA18: Hidden Linux Metrics with Prometheus eBPF Exporter
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Streaming SQL to unify batch and stream processing: Theory and practice with ...
Streaming SQL to unify batch and stream processing: Theory and practice with ...
Recently uploaded
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Recently uploaded (20)
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Atmosphere 2016 - Albert Lacki, Jaroslaw Bloch - Real user monitoring at scale - dream lab
- 1. Real User Monitoring at scale Jarosław Bloch Albert Łącki
- 4. 20 mln real users 7 mln requests every minute 150 mln PV daily
- 5. 3 data centers 3 000 servers and devices 130 Gbps traffic
- 6. 40 teams > 250 deployments daily 300 specialists
- 8. Speed search +0,4s PV/UU -0,5% load +35% Revenue -5% load +250ms PV/UU -1,8% Revenue/UU -4,3%
- 12. User experience Our responsibility Under our control
- 17. ?
- 18. Real User Monitoring ? ? ? collection ingestion processing presentation
- 19. Data collection
- 20. <script> var data = collect(); data.send(); </script> Data collection
- 21. Navigation Timing API onLoad DOM Processing ResponseRequestTCP, DNSRedirects navigationStart fetchStart requestStart responseStart domLoading responseEnd domInteractive domComplete
- 23. JS Errors try { … } catch (ex) { collect(ex); }
- 29. Data ingestion
- 35. Presentation
- 38. Custom alerts
- 40. Druid
- 42. Deployments
- 43. Instant feedback
- 47. Thank you!