How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood
•
2 likes•2,165 views
Robinhood is democratizing the financial systems by offering commission-free investing and trading with the use of your phone or desktop. As exciting as that sounds to the outside world, internally, the team at Robinhood must understand the different risk vectors and build engineering solutions to mitigate these risks. In this talk, Allison will talk about how they build a real-time risk monitoring system with InfluxDB and Faust, an open-source Python stream processing library. She will review the architecture behind the system which will involve both the time series anomaly detection part (InfluxDB) and the real-time stream processing part (Faust/Kafka).
Report
Share
Report
Share
Download to read offline
Recommended
How Olympus Controls Automates Predictive Maintenance with Telit, MQTT and In...
Olympus Controls specializes in simplifying the integration of motion control, machine vision, and robotic technologies through better automation. They partner with their clients from creation to implementation, and nearly half of the Fortune 100 use Olympus Controls to improve manufacturing plants and factories. Discover how Olympus Controls uses a time series database to collect and analyze industrial sensor data, which is ultimately used to increase their customers' competitive advantage.
Join this webinar to learn as Nick Armenta dives into:
· Olympus Controls' strategy to developing hardware and software used to improved IIoT monitoring
· How they use Telit and MQTT to reduce siloed manufacturing operations, including monitoring robotic arms (i.e. vibration and temperature)
· Their approach to developing IoT analysis using InfluxDB, Telegraf and Flux
Timeseries - data visualization in Grafana
Presentation deals with proper handling of the application and resources monitoring. It mentions tools that help with presentation layer - Grafana, storage - InfluxDB, and communication between the measurements and their destination - Telegraf.
Presented by Marek Szymeczko
Beautiful Monitoring With Grafana and InfluxDB
Query your data streams with the time series database InfluxDB and then visualize the results with stunning Grafana dashboards. Quick and easy to set up. Fully scalable to millions of metrics per second.
Near real-time statistical modeling and anomaly detection using Flink!
Flink Forward San Francisco 2022.
At ThousandEyes we receive billions of events every day that allow us to monitor the internet; the most important aspect of our platform is to detect outages and anomalies that have a potential to cause serious impact to customer applications and user experience. Automatic detection of such events at lowest latency and highest accuracy is extremely important for our customers and their business. After launching several resilient and low latency data pipelines in production using Flink we decided to take it up a notch; we leveraged Flink to build statistical models in near real-time and apply them on incoming stream of events to detect anomalies! In this session we will deep dive into the design as well as discuss pitfalls and learnings while developing our real-time platform that leverages Debezium, Kafka, Flink, ElasticCache and DynamoDB to process events at scale!
by
Kunal Umrigar & Balint Kurnasz
HBase Storage Internals
Apache HBase is the Hadoop opensource, distributed, versioned storage manager well suited for random, realtime read/write access. This talk will give an overview on how HBase achieve random I/O, focusing on the storage layer internals. Starting from how the client interact with Region Servers and Master to go into WAL, MemStore, Compactions and on-disk format details. Looking at how the storage is used by features like snapshots, and how it can be improved to gain flexibility, performance and space efficiency.
Start Automating InfluxDB Deployments at the Edge with balena
balena.io helps companies develop, deploy, update, and manage IoT devices. By using Linux containers and other cloud technologies, balena enables teams to quickly and easily build fleets of connected devices. Developers are able to use containers with the language of choice and pull IoT sensor data from 70+ different single board computers into balenaCloud. Discover how to use balena.io to automate your InfluxDB deployments at the edge!
During this one-hour session, experts from balena and InfluxData will demonstrate how to build and deploy your own air quality IoT solution. You will learn:
The fundamentals of IoT sensor deployment and management using balena.
How to use a time series platform to collect and visualize metrics from edge devices.
Tips and tricks to using balenaCloud to automate InfluxDB deployments and Telegraf configurations.
How to use InfluxDB's Edge Data Replication feature to collect sensor data and push it to InfluxDB Cloud for analysis.
No coding experience required, just a curiosity to start your own IoT adventure.
Recommended
How Olympus Controls Automates Predictive Maintenance with Telit, MQTT and In...
Olympus Controls specializes in simplifying the integration of motion control, machine vision, and robotic technologies through better automation. They partner with their clients from creation to implementation, and nearly half of the Fortune 100 use Olympus Controls to improve manufacturing plants and factories. Discover how Olympus Controls uses a time series database to collect and analyze industrial sensor data, which is ultimately used to increase their customers' competitive advantage.
Join this webinar to learn as Nick Armenta dives into:
· Olympus Controls' strategy to developing hardware and software used to improved IIoT monitoring
· How they use Telit and MQTT to reduce siloed manufacturing operations, including monitoring robotic arms (i.e. vibration and temperature)
· Their approach to developing IoT analysis using InfluxDB, Telegraf and Flux
Timeseries - data visualization in Grafana
Presentation deals with proper handling of the application and resources monitoring. It mentions tools that help with presentation layer - Grafana, storage - InfluxDB, and communication between the measurements and their destination - Telegraf.
Presented by Marek Szymeczko
Beautiful Monitoring With Grafana and InfluxDB
Query your data streams with the time series database InfluxDB and then visualize the results with stunning Grafana dashboards. Quick and easy to set up. Fully scalable to millions of metrics per second.
Near real-time statistical modeling and anomaly detection using Flink!
Flink Forward San Francisco 2022.
At ThousandEyes we receive billions of events every day that allow us to monitor the internet; the most important aspect of our platform is to detect outages and anomalies that have a potential to cause serious impact to customer applications and user experience. Automatic detection of such events at lowest latency and highest accuracy is extremely important for our customers and their business. After launching several resilient and low latency data pipelines in production using Flink we decided to take it up a notch; we leveraged Flink to build statistical models in near real-time and apply them on incoming stream of events to detect anomalies! In this session we will deep dive into the design as well as discuss pitfalls and learnings while developing our real-time platform that leverages Debezium, Kafka, Flink, ElasticCache and DynamoDB to process events at scale!
by
Kunal Umrigar & Balint Kurnasz
HBase Storage Internals
Apache HBase is the Hadoop opensource, distributed, versioned storage manager well suited for random, realtime read/write access. This talk will give an overview on how HBase achieve random I/O, focusing on the storage layer internals. Starting from how the client interact with Region Servers and Master to go into WAL, MemStore, Compactions and on-disk format details. Looking at how the storage is used by features like snapshots, and how it can be improved to gain flexibility, performance and space efficiency.
Start Automating InfluxDB Deployments at the Edge with balena
balena.io helps companies develop, deploy, update, and manage IoT devices. By using Linux containers and other cloud technologies, balena enables teams to quickly and easily build fleets of connected devices. Developers are able to use containers with the language of choice and pull IoT sensor data from 70+ different single board computers into balenaCloud. Discover how to use balena.io to automate your InfluxDB deployments at the edge!
During this one-hour session, experts from balena and InfluxData will demonstrate how to build and deploy your own air quality IoT solution. You will learn:
The fundamentals of IoT sensor deployment and management using balena.
How to use a time series platform to collect and visualize metrics from edge devices.
Tips and tricks to using balenaCloud to automate InfluxDB deployments and Telegraf configurations.
How to use InfluxDB's Edge Data Replication feature to collect sensor data and push it to InfluxDB Cloud for analysis.
No coding experience required, just a curiosity to start your own IoT adventure.
Tanny Ng, Nadeem Syed [WP Engine] | How WP Engine Transformed Monitoring Into...
WP Engine powers more of the top 10M sites than anyone else in WordPress. We serve 5.2 billion web requests per day! Our solution gives our customers performance, intelligence and integrations they need to drive their business forward faster. We needed a monitoring solution without single failure points to support our scale. Discover how WP Engine transformed their monitoring solution into an Observability Platform with InfluxDB, gaining better visibility and reducing infrastructure downtime.
Zabbix Monitoring Platform
Introduction Zabbix monitoring platform and compare with other popular tools.
Building a fully managed stream processing platform on Flink at scale for Lin...
Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.
Observability at Scale
Observability has emerged as one of the hottest topics on the DevOps landscape. Organizations seek to improve visibility into their cloud infrastructure and applications and identify production issues that may negatively impact #customerexperience.
➡️ But what are some of the best practices for scaling observability for modernapplications?
➡️ What challenges are #cloudplatforms facing?
Explore how to overcome the challenges and unlock speed, observability, and automation across your DevOps lifecycle.
Monitoring via Datadog
Datadog is a cloud monitoring solution that brings metrics from all of your apps, tools, servers & services into one place. It brings servers, clouds, metrics, apps, and team together by seamlessly aggregating metrics and events across the full devops stack.
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent, and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyse the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Scalability, Availability & Stability Patterns
Overview of scalability, availability and stability patterns, techniques and products.
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.
Introduction to influx db
Presentation for Pervasive Systems class lectured by prof. Ioannis Chatzigiannakis, a.y. 2015-16, about the No-SQL database InfluxDB. The course is intended for students of MS in Engineering in Computer Science at Sapienza - University of Rome.
The complete code for the demo is available on Github:
https://github.com/RobGaud/PervasiveSystemsPersonal
You can also find me on LinkedIn:
https://www.linkedin.com/in/roberto-gaudenzi-4b0422116
Our journey with druid - from initial research to full production scale
Here at the Nielsen Marketing Cloud we use druid.io (http://druid.io/) as one of our main data stores, both for simple counts and for approximate count-distinct (DataSketches).
It’s been more than a year since we started using it, injecting billions of events each day to multiple druid clusters for different use-cases.
In this meet-up, we will share our journey, the challenges we had, the way we overcame them (at least most of them) and the steps we made to optimize the process around Druid to keep the solution cost effective.
Before diving into Druid, we will briefly present our data pipeline architecture, starting from the front-end serving system, deployed in number of geo-locations, to a centralized Kafka cluster in the cloud, and give some examples of the different processes that consume from Kafka and feed our different data sources.
What we've learned from running a PostgreSQL managed service on Kubernetes
In this talk, I will share some of our learnings from running a managed PostgreSQL/TimescaleDB service on Kubernetes on AWS for a little more than a year: I’ll start with the motivation of running managed PostgreSQL on Kubernetes, the benefits and drawbacks. I’ll describe the architecture of the managed PostgreSQL cloud on Kubernetes I’ll zoom in on how we solved some of the Kubernetes-specific issues within our cloud, such as upgrading extensions without downtimes, taming the dreaded OOM killer, and doing regular maintenance and PostgreSQL major upgrades. I’ll share how open-source tools from the PostgreSQL ecosystem helps us to run the service and explain how we use them in a slightly non-trivial way.
This talk was given by Oleksii Kliukin for DoK Day Europe @ KubeCon 2022.
Monitoring With Prometheus
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
How azeti Monitors PLC and SCADA Systems Using MQTT and InfluxDB
How azeti Improves Industry 4.0 Monitoring Using MQTT and InfluxDB
azeti are the creators of an industrial IoT platform that enables customers from process and manufacturing industries to leverage their unused shop floor data in order to lower process complexity, maintenance and operations cost.
By collecting thousands of data points directly from sensors, machine controls (PLC) or control systems (DCS / SCADA), azeti has been able to save customers hundreds of thousands of dollars annually. Discover how azeti uses InfluxDB to enable IIoT use cases like condition monitoring and predictive maintenance for their clients. In this webinar, Florian Hoenigschmid and Sebastian Koch will dive into:
azeti’s approach to enable IIoT use cases
Their methodology to improve machine health and utilization
Why they use a time series database to store vibration, temperature and other sensor data
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
Cisco Systems’ Cisco Live conference is held annually in the winter in Europe and in the summer in the United States. A typical US event hosts 600+ breakout sessions, dozens of keynotes, Certification Testing and walk-in labs. The conference serves over 26,000 attendees and all their mobile devices. The internal network team is responsible for ensuring that 2,200 wireless access points and 800 switches are providing sufficient network connectivity, availability and bandwidth for all attendees, speakers and organizers across 2 million square feet of conference space. Over 9 days (4 days of setup and 5 conference days), event staff and attendees have pushed over 84 terabytes of data from the conference to the internet! Dual 100 Gigabit/second primary links and backup 10 Gigabit/second links handle anything the users can throw at it.
The infrastructure required for this event also includes servers, VMs, and containerized workloads. With the growing need for hybrid events, Cisco’s team also ensures 100% video streaming uptime. Discover how Cisco uses InfluxDB to store key performance metrics across many IT domains alongside their commercial management solutions. The team continues to iterate and improve year-over-year to gain visibility into their network and devices to streamline troubleshooting and quickly respond to events before they become service impacting.
In this webinar, Jason Davis dives into:
Cisco’s approach to using automation, orchestration, Python scripts, SNMP, and streaming telemetry to collect network data
Their methodology to troubleshooting, prioritizing, and scheduling fixes to ensure the best client experience
How a time series platform is crucial to their real-time data analysis
Killzone Shadow Fall: Threading the Entity Update on PS4
On the PS3 the SPUs were too limited for doing typical entity logic (e.g. humanoids walking around and shooting), so all of our entity logic was running on a single thread. For the PS4 we adopted the "entity as a job" approach and developed a set of rules that entities have to adhere to. We also developed a way to validate these rules at runtime so that no race conditions occur. This presentation will also show how we spread out entity updates across frames. The resulting system has proven itself on the PS4 -- and there have been surprisingly few race conditions that were not easily caught.
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache Flink, presented at the Apache Flink and Neo4j meetup, March 2016, Berlin
Challenges of monitoring distributed systems
Back in the days, you had a single machine and you could scroll down the single log file to figure out what is going on. In this Big Data world you need to combine a lot of logs together to figure out what is going on. Data is coming in huge volumes, with high speed so choosing important information and getting rid of noise becomes real challenge. There is a need for a centralized monitoring platform which will aid the engineers operating the systems, and serve the right information at the right time.
This talk will try to help you understand all the challenges and you will get an idea which tools and technology stacks are good fit to successfully monitor Big Data systems. The focus will be on open source and free solutions. The problem can be separated in two domains which both are the subject of this talk: metrics stack to gather simple metrics on central place and log stack to aggregate logs from different machines to central place. We will finish up with a combined stack and ideas how it can be improved even further with alerting and automated failover scenarios.
More Related Content
What's hot
Tanny Ng, Nadeem Syed [WP Engine] | How WP Engine Transformed Monitoring Into...
WP Engine powers more of the top 10M sites than anyone else in WordPress. We serve 5.2 billion web requests per day! Our solution gives our customers performance, intelligence and integrations they need to drive their business forward faster. We needed a monitoring solution without single failure points to support our scale. Discover how WP Engine transformed their monitoring solution into an Observability Platform with InfluxDB, gaining better visibility and reducing infrastructure downtime.
Zabbix Monitoring Platform
Introduction Zabbix monitoring platform and compare with other popular tools.
Building a fully managed stream processing platform on Flink at scale for Lin...
Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.
Observability at Scale
Observability has emerged as one of the hottest topics on the DevOps landscape. Organizations seek to improve visibility into their cloud infrastructure and applications and identify production issues that may negatively impact #customerexperience.
➡️ But what are some of the best practices for scaling observability for modernapplications?
➡️ What challenges are #cloudplatforms facing?
Explore how to overcome the challenges and unlock speed, observability, and automation across your DevOps lifecycle.
Monitoring via Datadog
Datadog is a cloud monitoring solution that brings metrics from all of your apps, tools, servers & services into one place. It brings servers, clouds, metrics, apps, and team together by seamlessly aggregating metrics and events across the full devops stack.
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent, and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyse the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Scalability, Availability & Stability Patterns
Overview of scalability, availability and stability patterns, techniques and products.
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.
Introduction to influx db
Presentation for Pervasive Systems class lectured by prof. Ioannis Chatzigiannakis, a.y. 2015-16, about the No-SQL database InfluxDB. The course is intended for students of MS in Engineering in Computer Science at Sapienza - University of Rome.
The complete code for the demo is available on Github:
https://github.com/RobGaud/PervasiveSystemsPersonal
You can also find me on LinkedIn:
https://www.linkedin.com/in/roberto-gaudenzi-4b0422116
Our journey with druid - from initial research to full production scale
Here at the Nielsen Marketing Cloud we use druid.io (http://druid.io/) as one of our main data stores, both for simple counts and for approximate count-distinct (DataSketches).
It’s been more than a year since we started using it, injecting billions of events each day to multiple druid clusters for different use-cases.
In this meet-up, we will share our journey, the challenges we had, the way we overcame them (at least most of them) and the steps we made to optimize the process around Druid to keep the solution cost effective.
Before diving into Druid, we will briefly present our data pipeline architecture, starting from the front-end serving system, deployed in number of geo-locations, to a centralized Kafka cluster in the cloud, and give some examples of the different processes that consume from Kafka and feed our different data sources.
What we've learned from running a PostgreSQL managed service on Kubernetes
In this talk, I will share some of our learnings from running a managed PostgreSQL/TimescaleDB service on Kubernetes on AWS for a little more than a year: I’ll start with the motivation of running managed PostgreSQL on Kubernetes, the benefits and drawbacks. I’ll describe the architecture of the managed PostgreSQL cloud on Kubernetes I’ll zoom in on how we solved some of the Kubernetes-specific issues within our cloud, such as upgrading extensions without downtimes, taming the dreaded OOM killer, and doing regular maintenance and PostgreSQL major upgrades. I’ll share how open-source tools from the PostgreSQL ecosystem helps us to run the service and explain how we use them in a slightly non-trivial way.
This talk was given by Oleksii Kliukin for DoK Day Europe @ KubeCon 2022.
Monitoring With Prometheus
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
How azeti Monitors PLC and SCADA Systems Using MQTT and InfluxDB
How azeti Improves Industry 4.0 Monitoring Using MQTT and InfluxDB
azeti are the creators of an industrial IoT platform that enables customers from process and manufacturing industries to leverage their unused shop floor data in order to lower process complexity, maintenance and operations cost.
By collecting thousands of data points directly from sensors, machine controls (PLC) or control systems (DCS / SCADA), azeti has been able to save customers hundreds of thousands of dollars annually. Discover how azeti uses InfluxDB to enable IIoT use cases like condition monitoring and predictive maintenance for their clients. In this webinar, Florian Hoenigschmid and Sebastian Koch will dive into:
azeti’s approach to enable IIoT use cases
Their methodology to improve machine health and utilization
Why they use a time series database to store vibration, temperature and other sensor data
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
Cisco Systems’ Cisco Live conference is held annually in the winter in Europe and in the summer in the United States. A typical US event hosts 600+ breakout sessions, dozens of keynotes, Certification Testing and walk-in labs. The conference serves over 26,000 attendees and all their mobile devices. The internal network team is responsible for ensuring that 2,200 wireless access points and 800 switches are providing sufficient network connectivity, availability and bandwidth for all attendees, speakers and organizers across 2 million square feet of conference space. Over 9 days (4 days of setup and 5 conference days), event staff and attendees have pushed over 84 terabytes of data from the conference to the internet! Dual 100 Gigabit/second primary links and backup 10 Gigabit/second links handle anything the users can throw at it.
The infrastructure required for this event also includes servers, VMs, and containerized workloads. With the growing need for hybrid events, Cisco’s team also ensures 100% video streaming uptime. Discover how Cisco uses InfluxDB to store key performance metrics across many IT domains alongside their commercial management solutions. The team continues to iterate and improve year-over-year to gain visibility into their network and devices to streamline troubleshooting and quickly respond to events before they become service impacting.
In this webinar, Jason Davis dives into:
Cisco’s approach to using automation, orchestration, Python scripts, SNMP, and streaming telemetry to collect network data
Their methodology to troubleshooting, prioritizing, and scheduling fixes to ensure the best client experience
How a time series platform is crucial to their real-time data analysis
Killzone Shadow Fall: Threading the Entity Update on PS4
On the PS3 the SPUs were too limited for doing typical entity logic (e.g. humanoids walking around and shooting), so all of our entity logic was running on a single thread. For the PS4 we adopted the "entity as a job" approach and developed a set of rules that entities have to adhere to. We also developed a way to validate these rules at runtime so that no race conditions occur. This presentation will also show how we spread out entity updates across frames. The resulting system has proven itself on the PS4 -- and there have been surprisingly few race conditions that were not easily caught.
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache Flink, presented at the Apache Flink and Neo4j meetup, March 2016, Berlin
What's hot (20)
Tanny Ng, Nadeem Syed [WP Engine] | How WP Engine Transformed Monitoring Into...
Tanny Ng, Nadeem Syed [WP Engine] | How WP Engine Transformed Monitoring Into...
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
Our journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scale
What we've learned from running a PostgreSQL managed service on Kubernetes
What we've learned from running a PostgreSQL managed service on Kubernetes
How azeti Monitors PLC and SCADA Systems Using MQTT and InfluxDB
How azeti Monitors PLC and SCADA Systems Using MQTT and InfluxDB
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
Killzone Shadow Fall: Threading the Entity Update on PS4
Killzone Shadow Fall: Threading the Entity Update on PS4
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache Flink
Similar to How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood
Challenges of monitoring distributed systems
Back in the days, you had a single machine and you could scroll down the single log file to figure out what is going on. In this Big Data world you need to combine a lot of logs together to figure out what is going on. Data is coming in huge volumes, with high speed so choosing important information and getting rid of noise becomes real challenge. There is a need for a centralized monitoring platform which will aid the engineers operating the systems, and serve the right information at the right time.
This talk will try to help you understand all the challenges and you will get an idea which tools and technology stacks are good fit to successfully monitor Big Data systems. The focus will be on open source and free solutions. The problem can be separated in two domains which both are the subject of this talk: metrics stack to gather simple metrics on central place and log stack to aggregate logs from different machines to central place. We will finish up with a combined stack and ideas how it can be improved even further with alerting and automated failover scenarios.
Monitoring & alerting presentation sabin&mustafa
Monitoring & alerting presentation sabin&mustafa
Gaza Sky Geeks
Challenges of monitoring distributed systems
Back in the days, you had a single machine and you could scroll down the single log file to figure out what is going on. In this Big Data world you need to combine a lot of logs together to figure out what is going on. Data is coming in huge volumes, with high speed so choosing important information and getting rid of noise becomes real challenge. There is a need for a centralized monitoring platform which will aid the engineers operating the systems, and serve the right information at the right time. This talk will focus on monitoring stack we like to use including Riemann, InfluxDB, ELK and Grafana. Cassandra will be used as an example of distributed system. Problem will be separated in two domains: metric collection and log collection and we will finish with example how you can combine both to pinpoint issues.
Statistical Process Control
Can you put a price on your annual wastage? QI Analyst helps you to improve quality and product variability by improving the stability of your process!
Webinar Content
What is Statistical Process Control (SPC)?
How can you use Statistical Process Control to improve your processes?
Tools you need to drive SPC
Ways this can interface with Wonderware
A success story
Short Demo
Q & A
This webinar is hosted by our Senior Consultant Chris Titley. Chris has worked at SolutionsPT for 13 years standing and has a masters Degree in Control Engineering and Optimisation. Previously he worked in a user capacity in the electronics industry. He now looks after the Terminal Services solutions and specifically managers ACP ThinManager offerings. Chris also has wide experience of InTouch, QI Analyst, ScadaAlarm, SPC Pro, Historian and Active Factory.
Cloud-based Stream Analytics VS InfuxDB Time-series analytics
Slides of comparisons between the Cloud-based Stream Analytics and InfuxDB Time-series analytics
Mastering AIOps with Deep Learning
In planet-scale deployments, the Operation and Maintenance (O&M) of cloud platforms cannot be done any longer manually or simply with off-the-shelf solutions. It requires self-developed automated systems, ideally exploiting the use of AI to provide tools for autonomous cloud operations. This talk will explain how deep learning, distributed traces, and time-series analysis (sequence analysis) can be used to effectively detect anomalous cloud infrastructure behaviors during operations to reduce the workload of human operators. The iForesight system is being used to evaluate this new O&M approach. iForesight 2.0 is the result of 2 years of research with the goal to provide an intelligent new tool aimed at SRE cloud maintenance teams. It enables them to quickly detect and predict anomalies thanks to the use of artificial intelligence when cloud services are slow or unresponsive.
INSECS: Intelligent networks security system
Presentation about an Intrusion Detection system designed for modern distributed systems like IoT and Cloud systems.
Mathworks CAE simulation suite – case in point from automotive and aerospace.
Mathworks CAE simulation suite – case in point from automotive and aerospace.WMG centre High Value Manufacturing Catapult
Bulat Khusainov, Technical Specialist, Mathworks
Plant Maintenance & Condition Monitoring
Condition monitoring (or CM) is the process of monitoring a parameter of condition in machinery (vibration, temperature etc.) and identify a significant change which is indicative of a developing fault.
The use of condition monitoring allows maintenance to be scheduled, or other actions to be taken to prevent failure and avoid its consequences.
Anomaly Detection for Information Systems
The presentation from poster session for Machine Learning course, Lviv IT School
Vissec2014
This is the presentation I gave at VizSec 2014 on our information-theoretic method for anomaly detection. The conference was held in Paris in November 2014.
Intelligent Monitoring
This presentation describes a intelligent IT monitoring solution that uses Nagios as source of information, Esper as the CEP engine and a PCA algorithm.
Webinar Monitoring in era of cloud computing
The webinar "Monitoring in era of cloud computing" covers the topics:
1. What is Monitoring
2. Ceilometer
- Architecture
- Agents (Compute/Central)
- Storage & API
- Quick Demo
3. Monasca
- Architecture
- Events/Messages
- Storage & API
- Quick Demo
Similar to How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood (20)
Monitoring & alerting presentation sabin&mustafa
Monitoring & alerting presentation sabin&mustafa
Cloud-based Stream Analytics VS InfuxDB Time-series analytics
Cloud-based Stream Analytics VS InfuxDB Time-series analytics
Mathworks CAE simulation suite – case in point from automotive and aerospace.
Mathworks CAE simulation suite – case in point from automotive and aerospace.
More from InfluxData
Announcing InfluxDB Clustered
InfluxData is excited to announce InfluxDB Clustered, the self-managed version of InfluxDB 3.0 with unparalleled flexibility, speed, performance, and scale. The evolution of InfluxDB Enterprise, InfluxDB Clustered is delivered as a collection of Kubernetes-based containers and services, which enables you to run and operate InfluxDB 3.0 where you need it, whether that's on-premises or in a private cloud environment. With this new enterprise offering, we’re excited to provide our customers with real-time queries, low-cost object storage, unlimited cardinality, and SQL language support – all with improved data access, support, and security! The newest version of InfluxDB was built on Apache Arrow, and through the open source ecosystem and integrations, extends the value of your time-stamped data.
Join this webinar to learn more about InfluxDB Clustered, and how to manage your large mission-critical workloads in the highly available database service offering!
In this webinar, Balaji Palani and Gunnar Aasen will dive into:
Key features of the new InfluxDB Clustered solution
Use cases for using the newest version of the purpose-built time series database
Live demo
During this 1-hour technical webinar, you’ll also get a chance to ask your questions live.
Best Practices for Leveraging the Apache Arrow Ecosystem
Apache Arrow is an open source project intended to provide a standardized columnar memory format for flat and hierarchical data. It enables more efficient analytics workloads for modern CPU and GPU hardware, which makes working with large data sets easier and cheaper.
InfluxData and Dremio are both members of the Apache Software Foundation (ASF). Dremio is a data lakehouse management service known for its scalability and capacity for direct querying across diverse data sources. InfluxDB is the purpose-built time series database, and InfluxDB 3.0 has a new columnar storage engine and uses the Arrow format for representing data and moving data to and from Parquet. Discover how InfluxDB and Dremio have advanced their solutions by relying on the Apache Arrow framework.
Join this live panel as Alex Merced and Anais Dotis-Georgiou dive into:
Advantages to utilizing the Apache Arrow ecosystem
Tips and tricks for implementing the columnar data structure
How developers can best utilize the ASF to innovate and contribute to new industry standards
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
Bevi are the creators of smart water dispensers which empower people to choose their desired beverage — flat or sparkling, their desired flavor and temperature. Since 2014, Bevi users have saved more than 350 million bottles and cans. Their "smart" water coolers have prevented the extraction of 1.4 trillion oz of oil from Earth and have saved 21.7 billion grams of CO2 from the atmosphere.
Discover how Bevi uses a time series database to enable better predictive maintenance and alerting of their entire ecosystem — including the hardware and software. They are using InfluxDB to collect sensor data in real-time remotely from their internet-connected machines about their status and activity — i.e., flavor and CO2 levels, water temp, filter status, etc. They a7re using these metrics to improve their customer experience and continuously improve their sustainability practices. Gain tips and tricks on how to best utilize InfluxDB's schema-less design.
Join this webinar as Spencer Gagnon dives into:
Bevi's approach to reducing organizations' carbon footprint — they are saving 50K+ bottles and cans annually
Their entire system architecture — including InfluxDB Cloud, Grafana, Kafka, and DigitalOcean
The importance of using time-stamped data to extend the life of their machines
Power Your Predictive Analytics with InfluxDB
If you're using InfluxDB to store and manage your time series data, you're already off to a great start. But why stop there? In our upcoming webinar, we'll show you how to take your data analysis to the next level by building predictive analytics using a variety of tools and techniques.
We will demonstrate how to use Quix to create custom dashboards and visualizations that allow you to monitor your data in real-time. We'll also introduce you to Hugging Face, a powerful tool for building models that can predict future trends and identify anomalies. With these tools at your disposal, you'll be able to extract valuable insights from your data and make more informed decisions about the future. Don't miss out on this opportunity to improve your data analysis skills and take your business to the next level!
What you will learn:
Use InfluxDB to store and manage time series data
Utilize Quix and Hugging Face to build models, visualize trends, and identify anomalies
Extract valuable insights from your data
Improve your data analysis skills to make informed decision
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
Are you considering replacing your legacy data historian and moving your OT data to the cloud? Join this technical webinar to learn how to adopt InfluxDB and IO Base - a digital platform used to improve operational efficiencies!
Teréga Solutions are the creators of digital solutions used to improve energy efficiencies and to address decarbonization challenges. Their network includes 5,000+ km of gas pipelines within France; they aim to help France attain carbon neutrality by 2050. With these impressive goals in mind, Teréga has created IO-Base — the digital platform to improve industrial performance, and increase profitability. Creating digital twins for their clients allows them to collect data from all production sites and view it in real time, from anywhere and at any time.
Discover how Teréga uses InfluxDB, Docker, and AWS to monitor its gas and hydrogen pipeline infrastructure. They chose to replace their legacy data historian with InfluxDB — the purpose built time series database. They are collecting more than 100K different metrics at various frequencies — some are collected every 5 seconds to only every 1-2 minutes. THey have reduced overall IT spend by 50% and collect 2x the amount of data at 20x frequency! By using various industrial protocols (Modbus, OPC-UA, etc.), Teréga improved output, reduced the TCO, and is now able to create added-value services: forecast, monitoring, predictive maintenance.
Join this webinar as Thomas Delquié dives into:
Teréga's approach to modernizing fossil fuel pipelines IT systems while improving yields and safety
Their centralized methodology to collecting sensor, hardware, and network metrics
The importance of time series data and why they chose InfluxDB
Build an Edge-to-Cloud Solution with the MING Stack
FlowForge enables organizations to reliably deliver Node-RED applications in a continuous, collaborative, and secure manner. Node-RED is the popular, low-code programming solution that makes it easy to connect different services using a visual programming environment. InfluxData is the creator of InfluxDB, the purpose-built time series database run by developers at scale and in any environment in the cloud, on-premises, or at the edge.
Jump-start monitoring your industrial IoT devices and discover how to build an edge-to-cloud solution with the MING stack. The MING stack includes Mosquitto/MQTT, InfluxDB, Node-RED, and Grafana. This solution can be used to improve fleet management, enable predictive maintenance of industrial machines and power generation equipment (i.e. turbines and generators) and increase safety practices (i.e. buildings, construction sites). Join this webinar to learn best practices from industrial IoT SME's.
In this webinar, Robert Marcer and Jay Clifford dive into:
Best practices for monitoring sensor data collected by everyone — from the edge to the factory
Tips and tricks for using Node-RED and InfluxDB together
Demo — see Node-RED and InfluxDB live
Meet the Founders: An Open Discussion About Rewriting Using Rust
Rust is a systems programming language designed for high performance, type safety, and concurrency. According to Stack Overflow’s annual survey in 2022, Rust is the most loved language with 87% of developers saying they want to continue using it. The same survey also reported that nearly 20% of developers aren’t currently using Rust, but want to start developing using it.
Ockam’s suite of programming libraries, command line tools, and managed cloud services enable developers to orchestrate end-to-end encryption. InfluxDB is the purpose-built time series database developed to handle time series data for IoT, monitoring, and real-time analytics. Ockam was originally developed using C, and InfluxDB was originally written using Go; both solutions have been completely rewritten in Rust. Discover why two founders decided to rewrite their developer tools using Rust, and gain insight into the strategy beforehand and the entire process.
Join this live panel as Mrinal Wadhwa and Paul Dix dive into:
Their approach to rewriting a project in Rust
How to build and train engineering teams
Tips and tricks learned along the way - pitfalls to look out for!
Join this webinar as there will be a live discussion with Q&A
Introducing InfluxDB Cloud Dedicated
InfluxData is excited to announce the general availability of InfluxDB Cloud Dedicated! It is a fully managed time series database service running on cloud infrastructure resources that are dedicated to a single tenant. With this new offering, we’re excited to provide our customers with additional security options, and more custom configuration options to best suit customers’ workload requirements. Join this webinar to learn more about InfluxDB Cloud, and the new dedicated database service offering!
In this webinar, Balaji Palani and Gary Fowler will dive into:
Key features of the new InfluxDB Cloud Dedicated solution
Use cases for using the newest version of the purpose-built time series database
Live demo
During this 1-hour technical webinar, you’ll also get a chance to ask your questions live.
Gain Better Observability with OpenTelemetry and InfluxDB
Many developers and DevOps engineers have become aware of using their observability data to gain greater insights into their infrastructure systems. InfluxDB is the purpose-built time series database used to collect metrics and gain observability into apps, servers, containers, and networks. Developers use InfluxDB to improve the quality and efficiency of their CI/CD pipelines. Start using InfluxDB to aggregate infrastructure and application performance monitoring metrics to enable better anomaly detection, root-cause analysis, and alerting.
This session will demonstrate how to record metrics, logs, and traces with one library — OpenTelemetry — and store them in one open source time series database — InfluxDB. Zoe will demonstrate how easy it is to set up the OpenTelemetry Operator for Kubernetes and to store and analyze your data in InfluxDB.
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
American Metal Processing Company ("AMP") is the US' largest commercial rotary heat treat facility with customers in the automotive, construction, military, and agriculture industries. They use their atmosphere-protected rotary retort furnaces to provide their clients with three primary hardening services: neutral hardening (quench and temper), carburizing, and carbonitriding.
This furnace style ensures consistent, uniform heat treatment process vs. traditional batch-or-belt-style furnaces; excels at processing high volumes of smaller parts with tight tolerances; and improves the strength and toughness of plain carbon steels. Discover why AMP’s use of Telegraf, InfluxDB, Node-RED, and Grafana allows them to gain 24/7 insights into their plant operations and metallurgical results. Learn how they use time-stamped data to gain accurate metrics about their consumables usage, furnace profiles, and machine status.
Join this webinar as Grant Pinkos dives into:
American Metal Processing's approach to heat treating in a digitized environment through connected systems
Their approach to collecting and measuring sensor data to enable predictive maintenance and improve product quality
Why they need a time series database for managing and analyzing vast amounts of time-stamped data
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
Delft University is the oldest and largest technical university in the Netherlands with 25,000+ students. Since 1999, they have had a team of students (undergraduate and graduate) designing, building, and racing cars, as part of the Formula Student worldwide competition. The competition has grown to include teams from 1K+ universities in 20+ countries. Students are responsible for all aspects of car manufacturing (research, construction, testing, developing, marketing, management, and fundraising). Delft University's team includes 90 students across disciplines.
Discover how Delft University's team uses Marple and InfluxDB to collect telemetry and sensor metrics while they develop, test, and race their electrics cars. They collect sensor data about their EV's control systems using a time series platform. During races, they are collecting IoT data about their batteries, accelerometer, gyroscope, tires, etc. The engineers are able to share important car stats during races which help the drivers tweak their driving decisions — all with the goal of winning. After races, the entire team are able to analyze data in Marple to understand what to do better next time. By using Marple + InfluxDB, their team are able to collect, share and analyze high frequency car data used to make their car faster at competitions.
Join this webinar as Robbin Baauw and Nero Vanbiervliet dive into:
Marple's approach to empowering engineers to organize, analyze, and visualize their data
Delft University's collaborative methodology to building and racing their Formula-style race car
How InfluxDB is crucial to their collaborative engineering and racing process
Introducing InfluxDB’s New Time Series Database Storage Engine
InfluxData is excited to announce the general availability of InfluxDB Cloud's new storage engine! It is a cloud-native, real-time, columnar database optimized for time series data. InfluxDB's rebuilt core was coded in Rust and sits on top of Apache Arrow and DataFusion. InfluxData's team picked Apache Parquet as the persistent format. In this webinar, Paul Dix and Balaji Palani will demonstrate key product features including the removal of cardinality limits!
They will dive into:
The next phase of the InfluxDB platform
How using Apache Arrow's ecosystem has improved InfluxDB's performance and scalability
Key features of InfluxDB Cloud's new core — including SQL native support
Understanding InfluxDB’s New Storage Engine
Learn more about InfluxDB’s new storage engine! The team developed a cloud-native, real-time, columnar database optimized for time series data. We built it all in Rust and it sits on top of Apache Arrow and DataFusion. We chose Apache Parquet as the persistent format, which is an open source columnar data file format. This new storage engine provides InfluxDB Cloud users with new functionality, including the removal of cardinality limits, so developers can bring in massive amounts of time series data at scale.
In this webinar, Anais Dotis-Georgiou will dive into:
Requirements for rebuilding InfluxDB’s core
Key product features and timeline
How Apache Arrow’s ecosystem is used to meet those requirements
Stick around for a demo and live Q&A
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
RudderStack — the creators of the leading open source Customer Data Platform (CDP) — needed a scalable way to collect and store metrics related to customer events and processing times (down to the nanosecond). They provide their clients with data pipelines that simplify data collection from applications, websites, and SaaS platforms. RudderStack's solution enables clients to stream customer data in real time — they quickly deploy flexible data pipelines that send the data to the customer's entire stack without engineering headaches. Customers are able to stream data from any tool using their 16+ SDK's, and they are able to transform the data in-transit using JavaScript or Python. How does RudderStack use a time series platform to provide their customers with real-time analytics?
Join this webinar as Ryan McCrary dives into:
RudderStack's approach to streamlining data pipelines with their 180+ out-of-the-box integrations
Their data architecture including Kapacitor for alerting and Grafana for customized dashboards
Why using InfluxDB was crucial for them for fast data collection and providing single-sources of truths for their customers
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Customers using ThingWorx and the Manufacturing Solutions often need to store property data longer than the Solutions default to. These customers are recommended to use InfluxDB, and this presentation will cover the key considerations for moving to InfluxDB vs the standard ThingWorx value streams. Join this session as Ward highlights ThingWorx’s solution and its easy implementation process.
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Two new features are coming to Flux that add flexibility
and functionality to your data workflow—polymorphic
labels and dynamic types. This session walks through
these new features and shows how they work.
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Closing thoughts on day 2 of InfluxDays.
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Welcome to day 2 of InfluxDays!
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Closing thoughts on day 1.
More from InfluxData (20)
Best Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow Ecosystem
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
Build an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING Stack
Meet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using Rust
Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
Introducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage Engine
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
Recently uploaded
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
DevOps and Testing slides at DASA Connect
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Epistemic Interaction - tuning interfaces to provide information for AI support
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
The Art of the Pitch: WordPress Relationships and Sales
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
By Design, not by Accident - Agile Venture Bolzano 2024
As presented at the Agile Venture Bolzano, 4.06.2024
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentationsRecently uploaded (20)
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
By Design, not by Accident - Agile Venture Bolzano 2024
By Design, not by Accident - Agile Venture Bolzano 2024
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
How Robinhood Built a Real-Time Anomaly Detection System to Monitor and Mitigate Risk | Allison Wang | Robinhood
- 1. Real-time Anomaly Detection with InfluxDB Allison Wang - Software Engineer, Robinhood
- 3. Overview ● Why anomaly detection ● Algorithm for anomaly detection ● Choosing a time-series database ● Building an end-to-end anomaly detection system
- 4. We want intelligent alerts on critical metrics without human staring at the dashboards 24/7 Why Building An Anomaly Detection System
- 5. ● Work well with simple time-series ● Fail to account for seasonality and trend Threshold-based Alerting
- 6. Normal Distribution ● 1σ - 68.27% ● 2σ - 95.45% ● 3σ - 99.73%
- 7. Bucketing Data Points ● Aggregate data points in a small time interval, e.g every minute) ● Construct normal distribution for every minute in the day for the past 30 days
- 8. Anomaly Detection Check if the incoming data point (aggregated) is in the range (μ - 3σ, μ + 3σ)
- 9. System Requirements ● A database for fast time-series data ingestion and aggregation ● A system for querying and computing anomaly in real-time ● Visualization and alerting
- 10. Time Series Databases ● InfluxDB ● Prometheus ● Elasticsearch ● OpenTSDB ● Postgres ● ...
- 11. ● Lightweight: OpenTSDB ● Schemaless: Postgres ● Allow indexing via a specific field in the data: Prometheus ● Fast data ingestion (write) and aggregation (read): Elasticsearch ● High availability (enterprise version) ● InfluxData Stack (Kapacitor, Chronograf, Telegraf) ● Incredible community Why InfluxDB
- 12. Real-time Stream Processing Need a system that can send queries and run anomaly detection algorithm.
- 13. Why Faust ● Performant (Asyncio) ● Fault-tolerant (RocksDB) ● Scalable (Kafka)
- 14. Combine Faust with InfluxDB
- 17. Boundary Visualization in Grafana
- 19. High Level System Architecture
- 20. More Anomaly Detection Algorithms ● Kapacitor anomaly detection library ● InfluxDB’s Holt-Winters Function ● scikit-learn, numpy, pandas ● Deep learning, LSTM ● ...
- 21. Conclusion ● Lightweight ● Extensible ● Horizontally scalable
- 22. Thank You!