Near real-time statistical modeling and anomaly detection using Flink!

•Download as PPTX, PDF•

0 likes•244 views

Flink Forward San Francisco 2022. At ThousandEyes we receive billions of events every day that allow us to monitor the internet; the most important aspect of our platform is to detect outages and anomalies that have a potential to cause serious impact to customer applications and user experience. Automatic detection of such events at lowest latency and highest accuracy is extremely important for our customers and their business. After launching several resilient and low latency data pipelines in production using Flink we decided to take it up a notch; we leveraged Flink to build statistical models in near real-time and apply them on incoming stream of events to detect anomalies! In this session we will deep dive into the design as well as discuss pitfalls and learnings while developing our real-time platform that leverages Debezium, Kafka, Flink, ElasticCache and DynamoDB to process events at scale! by Kunal Umrigar & Balint Kurnasz

Technology

Near real-time statistical
modeling and anomaly detection
using Flink!
Kunal Umrigar
Balint Kurnasz

2
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Customer
Digital Experience
See, understand and
improve connected
experiences everywhere
Customers:
Employee
Digital Experience
Ready your network and
successfully embrace
multi-cloud and SaaS
Customers:
End to End Network Visibility
WAN
Experience
Deploy modern hybrid and
SD-WANs with confidence
Customers:

3
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Visualize Experience from Every Vantage Point
Data
Center
Data
Center
SaaS
IaaS
CDN
DNS
DNS Provider
Security Provider
API
Customer
Cloud Endpoint
Enterprise

4
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Data
Center
Scale @ ThousandEyes
Branch Office
Branch Office
Branch Office
Branch Office
Siebel
HRMS
Flow
PCAP
SNMP
Exchange
Data
Center
Data
Center
Data
Center
Data
Center
SaaS
IaaS
CDN
DNS
DNS Provider
Security Provider
API
Customer
2B+
Raw Events per day
36K+
agents across
cloud and enterprise
accounts
~150K+
Alerts sent out per
day

5
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
ThousandEyes Alerting Overview
Alerter Controller
● Ingests change capture events using Debezium
● Enriches raw events with metadata to generate
control events
Alerter Engine
● Ingests and merges control events and raw data
points
● Enrich, evaluate and aggregate data
● Generates alert events
Alert Persistence and Notification
● Persists alert events to alerts datastore
● Notifies the notification subsystem
● Notification system delivers alerts to Customers using
Email, Webhooks, Slack, PagerDuty, ServiceNow,
MS-Teams, many third party integrations…

6
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Alert Evaluation
● Rule expressions have specific grammar
and features
● Generated AST from the expression
● Variety of supported data types

7
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
From static to dynamic thresholds
● Static evaluation challenges:
○ Static thresholds can not adapt to seasonal changes
○ Static values require finding the right thresholds
● Moving to dynamic thresholds:
○ Establish baseline based on past 24Hr mean
○ Establish Dynamic thresholds based on
■ absolute value above baseline
■ percentage delta over baseline
■ N standard deviations over baseline
○ Evaluation becomes a feedback loop

8
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Calculations
● Moving Average - 24h sliding window
● Moving Standard Deviation - 3hr sliding window
● Perfect fit for Flink AggregateFunction

9
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Streaming statistical computations
● Statistical computations in
a separate Flink Job
● Results lookup using a
GRPC service call from an
AsyncFunction
● Advantages:
○ independent scalability
○ async processing
○ eventually consistent

10
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Conclusion & Next Steps
● Simple and robust Flink job for statistical computations
● Low maintenance
● Easy to add support for more metrics using configuration
● Extremely low latency evaluations
Next Steps:
● Optimize rate of false positives
● Computing statistics over larger durations
● Leveraging normalized standard deviation
● Machine Learning to detect trends and anomalies.

11
© 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes
Q & A
We are hiring!
Check out: thousandeyes.com/careers/engineering

Near real-time statistical modeling and anomaly detection using Flink!

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Flink powered stream processing platform at Pinterest

Flink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Evening out the uneven: dealing with skew in Flink

Flink Forward

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flink Forward

Flink Forward San Francisco 2022. Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way. by Jeff Chao

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

Where is my bottleneck? Performance troubleshooting in Flink

Flink Forward

Flinkn Forward San Francisco 2022. In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times. by Piotr Nowojski

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Practical learnings from running thousands of Flink jobs

Flink Forward

Flink Forward San Francisco 2022. Task Managers constantly running out of memory? Flink job keeps restarting from cryptic Akka exceptions? Flink job running but doesn’t seem to be processing any records? We share practical learnings from running thousands of Flink Jobs for different use-cases and take a look at common challenges they have experienced such as out-of-memory errors, timeouts and job stability. We will cover memory tuning, S3 and Akka configurations to address common pitfalls and the approaches that we take on automating health monitoring and management of Flink jobs at scale. by Hong Teoh & Usamah Jassat

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...

Flink Forward

Getting data in and out of Flink is by far the most important aspect, and an everyday typical requirement of building Flink applications. Doing so in an end-to-end exactly-once manner, however, can be tricky. Being able to reliably consume data from the outside world without any duplicate processing and guaranteeing consistent distributed state, and at the same time provide computed results back to the outside world also without introducing duplicates, is crucial for the consistency and correctness of applications built upon stream processors. In this talk, we will talk about how end-to-end exactly-once guarantees can be achieved with Apache Flink. We will talk about Flink’s checkpointing mechanism, and how exactly to leverage it when consuming and producing data from your Flink streaming pipelines. In particular, we will be having a detailed review on how our supported connectors do so, with the aim to provide reference implementations for your own custom consumers and sinks.

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Stream processing with Apache Flink (Timo Walther - Ververica)

KafkaZone

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Apache Flink in the Cloud-Native Era

Flink Forward

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Autoscaling Flink with Reactive Mode

Flink Forward

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Introduction to Apache Flink - Fast and reliable big data processing

Till Rohrmann

This presentation introduces Apache Flink, a massively parallel data processing engine which currently undergoes the incubation process at the Apache Software Foundation. Flink's programming primitives are presented and it is shown how easily a distributed PageRank algorithm can be implemented with Flink. Intriguing features such as dedicated memory management, Hadoop compatibility, streaming and automatic optimisation make it an unique system in the world of Big Data processing.

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Unified Stream and Batch Processing with Apache Flink

DataWorks Summit/Hadoop Summit

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

Introduction to KSQL: Streaming SQL for Apache Kafka®

confluent

Join Tom Green, Solution Engineer at Confluent for this Lunch and Learn talk covering KSQL. Confluent KSQL is the streaming SQL engine that enables real-time data processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python. KSQL is scalable, elastic, fault-tolerant, and it supports a wide range of streaming operations, including data filtering, transformations, aggregations, joins, windowing, and sessionization. By attending one of these sessions, you will learn: -How to query streams, using SQL, without writing code. -How KSQL provides automated scalability and out-of-the-box high availability for streaming queries -How KSQL can be used to join streams of data from different sources -The differences between Streams and Tables in Apache Kafka

Apache Flink: API, runtime, and project roadmap

Kostas Tzoumas

Extending Flink SQL for stream processing use cases

Flink Forward

Flink Forward San Francisco 2022. Apache Flink is a powerful stream processing platform that enables users to build complex real time applications. Flink SQL provides a SQL interface that implements standard SQL. While the standard SQL provides a perfect interface for batch processing, in stream processing context, it can result is ambiguity and complex syntax. As an example, consider these three types of streams: Append-only stream, Retract stream and Upsert stream. Using standard SQL, we would represent all of these streams as Table along with the Table concept in batch processing. Such overloading of concepts can result in ambiguity in SQL statements in streaming context. In this talk, we will present extensions to the Flink SQL that simplify SQL statements in the context of stream processing. We will show how such extensions work in the context of a Flink application using different use cases. These extensions are only sugar syntax and users should be able to use Flink SQL as is if they desire. by Hojjat Jafarpour

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

HostedbyConfluent

Apache Kafka is used as the primary message bus for propagating events and logs across Uber. In particular, it pairs with Apache Pinot, a real-time distributed OLAP datastore, to deliver real-time insights seconds after the messages produced to Kafka. One challenge we faced was to update existing data in Pinot with the changelog in Kafka, and deliver an accurate view in the real-time analytical results. For example, the financial dashboard can report gross booking with the corrected Ride fares. And restaurant owners can analyze the UberEats orders with their latest delivery status. Implementing upserts in an immutable real-time OLAP store like Pinot is nontrivial. We need to make architectural changes in how data is distributed via Kafka amongst the server nodes, how it's indexed and queried in a distributed fashion. In this talk I will discuss how we leveraged Kafka's partition-by-key feature to this end and how we added this ability in Pinot without any performance degradation.

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Databricks

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache

Dremio Corporation

From DataEngConf 2017 - Everybody wants to get to data faster. As we move from more general solution to specific optimization techniques, the level of performance impact grows. This talk will discuss how layering in-memory caching, columnar storage and relational caching can combine to provide a substantial improvement in overall data science and analytical workloads. It will include a detailed overview of how you can use Apache Arrow, Calcite and Parquet to achieve multiple magnitudes improvement in performance over what is currently possible.

Getting Started With ThousandEyes Proof of Concepts: End User Digital Experience

ThousandEyes

Getting Started with ThousandEyes Proof of Concepts

ThousandEyes

What's hot

Processing Semantically-Ordered Streams in Financial Services

Flink Forward

Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...

Flink Forward

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Stream processing with Apache Flink (Timo Walther - Ververica)

KafkaZone

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Apache Flink in the Cloud-Native Era

Flink Forward

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Autoscaling Flink with Reactive Mode

Flink Forward

Introduction to Apache Flink - Fast and reliable big data processing

Till Rohrmann

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Unified Stream and Batch Processing with Apache Flink

DataWorks Summit/Hadoop Summit

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Introduction to KSQL: Streaming SQL for Apache Kafka®

confluent

Apache Flink: API, runtime, and project roadmap

Kostas Tzoumas

Extending Flink SQL for stream processing use cases

Flink Forward

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

HostedbyConfluent

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Databricks

Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache

Dremio Corporation

What's hot (20)

Processing Semantically-Ordered Streams in Financial Services

Flink Forward Berlin 2017: Piotr Nowojski - "Hit me, baby, just one time" - B...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Stream processing with Apache Flink (Timo Walther - Ververica)

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Dynamic Rule-based Real-time Market Data Alerts

Apache Flink in the Cloud-Native Era

Building a fully managed stream processing platform on Flink at scale for Lin...

Autoscaling Flink with Reactive Mode

Introduction to Apache Flink - Fast and reliable big data processing

Tame the small files problem and optimize data layout for streaming ingestion...

Unified Stream and Batch Processing with Apache Flink

Stephan Ewen - Experiences running Flink at Very Large Scale

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Introduction to KSQL: Streaming SQL for Apache Kafka®

Apache Flink: API, runtime, and project roadmap

Extending Flink SQL for stream processing use cases

Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache

Similar to Near real-time statistical modeling and anomaly detection using Flink!

Getting Started With ThousandEyes Proof of Concepts: End User Digital Experience

ThousandEyes

Getting Started with ThousandEyes Proof of Concepts

ThousandEyes

ciscothousandeyesusecase

RENJITHKNAIR5

Getting Started with ThousandEyes Proof of Concepts

ThousandEyes

0328apjcintrotothousandeyeswebinar-230328233735-4df10d7f.pdf

Saurabh Chauhan

Introduction To ThousandEyes

ThousandEyes

Mindsphere: an open cloud-based IoT operating system for Industry

IIoTWorld

Getting Started with ThousandEyes Proof of Concepts

ThousandEyes

Getting Started with ThousandEyes Proof of Concepts

ThousandEyes

Check Point and Cisco: Securing the Private Cloud

Check Point Software Technologies

Security as an Accelerator for Cloud Adoption

MarketingArrowECS_CZ

Who Broke My Cloud? SaaS Monitoring Best Practices

ThousandEyes

Getting Started with ThousandEyes Proof of Concepts

ThousandEyes

CloudSmart Webinarnloek

Graph-Based Network Topology Analysis for Telecom Operators

Neo4j

Getting Demo & POV Ready

ThousandEyes

A proof of concept is an excellent way to showcase how a technology will provide immediate business value for your customer. To conduct a successful proof of concept using ThousandEyes, it's important to qualify the opportunity and outline what success looks like to that customer. In this webinar, we will walk you through: - What you need to know to run successful ThousandEyes proof of concepts focusing on the Enterprise Digital Experience use case - A demo of how to capture interesting events within the platform during a proof of concept

Building Performance & Sustainability through Digitalization

IGBC Green Building Congress

TENDENCIAS DE SEGURIDAD PARA AMBIENTES EN LA NUBE

Cristian Garcia G.

OPS Executive insights Webinar - Tanzu Slides

VMware Tanzu

API First or Events First: Is it a Binary Choice?

Rohit Kelapure

When do you use API-first or events-first architecture? Is this a binary choice? This is a false dichotomy! A mental model is needed to frame the architecture, packaging, and programming choices for modern applications. Varying degrees of combination of events and APIs can be used to design a system. Event notifications-based systems require an API callback to the source. CQRS and event-sourcing patterns, on the other hand, are on the complex end of the event-driven spectrum. APIs also have a maturity model, evolving with the adoption of reactive paradigms. In this session, we’ll look at heuristics such as cost, latency, security, and external integrations that will influence implementation. Architects will learn actionable fitness functions to strike a balance between APIs and events to build sustainable architectures.

Similar to Near real-time statistical modeling and anomaly detection using Flink! (20)

Getting Started With ThousandEyes Proof of Concepts: End User Digital Experience

Getting Started with ThousandEyes Proof of Concepts

ciscothousandeyesusecase

Getting Started with ThousandEyes Proof of Concepts

0328apjcintrotothousandeyeswebinar-230328233735-4df10d7f.pdf

Introduction To ThousandEyes

Mindsphere: an open cloud-based IoT operating system for Industry

Getting Started with ThousandEyes Proof of Concepts

Check Point and Cisco: Securing the Private Cloud

Security as an Accelerator for Cloud Adoption

Who Broke My Cloud? SaaS Monitoring Best Practices

Getting Started with ThousandEyes Proof of Concepts

CloudSmart Webinar

Graph-Based Network Topology Analysis for Telecom Operators

Getting Demo & POV Ready

Building Performance & Sustainability through Digitalization

TENDENCIAS DE SEGURIDAD PARA AMBIENTES EN LA NUBE

OPS Executive insights Webinar - Tanzu Slides

API First or Events First: Is it a Binary Choice?

More from Flink Forward

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

One sink to rule them all: Introducing the new Async Sink

Flink Forward

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

The Current State of Table API in 2022

Flink Forward

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

Flink SQL on Pulsar made easy

Flink Forward

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

Welcome to the Flink Community!

Flink Forward

Flink Forward San Francisco 2022. At Flink Forward, we get to hear creative, unique use cases, often on the bleeding edge of some of the most exciting current technologies. This talk will give you a chance to get to open up the hood on our driven and innovative Open Source community. I will cover what our community has been working on this past year, and how this work relates to our (Ververica's) exciting new Flink engineering roadmap! I will also go through some best practices and upcoming opportunities for getting involved in this community! by Caito Scherr

The top 3 challenges running multi-tenant Flink at scale

Flink Forward

Apache Flink is the foundation for Decodable's real-time SaaS data platform. Flink runs critical data processing jobs with strong security requirements. In addition, Decodable has to scale to thousands of tenants, power various use cases, provide an intuitive user experience and maintain cost-efficiency. We've learned a lot of lessons while building and maintaining the platform. In this talk, I'll share the top 3 toughest challenges building and operating this platform with Flink, and how we solved them.

Using Queryable State for Fun and Profit

Flink Forward

Flink Forward San Francisco 2022. A particular feature in our system relies on a streaming 90-minute trailing window of 1-minute samples - implemented as a lookaside cache - to speed up a particular query, allowing our customers to rapidly see an overview of their estate. Across our entire customer base, there is a substantial amount of data flowing into this cache - ~1,000,000 entries/second, with the entire cache requiring ~600GB of RAM. The current implementation is simplistic but expensive. In this talk I describe a replacement implementation as a stateful streaming Flink application leveraging Queryable State. This Flink application reduces the net cost by ~90%. In this session, the implementation is described in detail, including windowing considerations, a sliding-window state buffer that avoids the sliding window replication penalty, and a comparison of queryable state and Redis queries. The talk concludes with a frank discussion of when this distinctive approach is, and is not, appropriate. by Ron Crocker

Changelog Stream Processing with Apache Flink

Flink Forward

Flink Forward San Francisco 2022. The world is constantly changing. Data is continuously produced and thus should be consumed in a similar fashion by enterprise systems. Only this enables real-time decisions at scale. Message logs such as Apache Kafka can be found in almost every architecture, while databases and other batch systems still provide the foundation. Change Data Capture (CDC) propagates changes downstream. In this talk, we will highlight what it means to be a general data processor and how Flink can act as an integration hub. We present the current state of Flink and how it can power various use cases on both finite and infinite streams. We demonstrate Flink's SQL engine as a changelog processor that is shipped with an ecosystem tailored to process CDC data and maintain materialized views. We will use Kafka as an upsert log, Debezium for connecting to databases, and enrich streams of various sources. Finally, we will combine Flink's Table API with DataStream API for event-driven applications beyond SQL. by Timo Walther

Large Scale Real Time Fraudulent Web Behavior Detection

Flink Forward

Flink Forward San Francisco 2022. Neuro-ID analyzes web behavior at a large scale to determine visitors' intent on web pages, specifically in the online lending industry. When users interact with an online loan application, our software analyzes their behavior to determine if the applicant may be potentially fraudulent. Lenders can then request various scores describing the applicant's intentions in real-time to use to make decisions during the application flow. Flink gives our product the ability to observe behavior in a stateful manner. As an applicant interacts with an online loan application, a Flink application is used to compare earlier actions to later actions. This processing in Flink can determine the applicant's intent throughout the process of the application. by Jeff Niemann & Randy Hanak

Building Reliable Lakehouses with Apache Flink and Delta Lake

Flink Forward

Flink Forward San Francisco 2022. Apache Flink and Delta Lake together allow you to build the foundation for your data lakehouses by ensuring the reliability of your concurrent streams from processing to the underlying cloud object-store. Together, the Flink/Delta Connector enables you to store data in Delta tables such that you harness Delta’s reliability by providing ACID transactions and scalability while maintaining Flink’s end-to-end exactly-once processing. This ensures that the data from Flink is written to Delta Tables in an idempotent manner such that even if the Flink pipeline is restarted from its checkpoint information, the pipeline will guarantee no data is lost or duplicated thus preserving the exactly-once semantics of Flink. by Scott Sandre & Denny Lee

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Flink Forward

Flink Forward San Francisco 2022. With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi. by Ethan Guo & Kyle Weller

More from Flink Forward (12)

Introducing the Apache Flink Kubernetes Operator

One sink to rule them all: Introducing the new Async Sink

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Batch Processing at Scale with Flink & Iceberg

Welcome to the Flink Community!

The top 3 challenges running multi-tenant Flink at scale

Using Queryable State for Fun and Profit

Changelog Stream Processing with Apache Flink

Large Scale Real Time Fraudulent Web Behavior Detection

Building Reliable Lakehouses with Apache Flink and Delta Lake

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Knowledge engineering: from people to machines and back

Elena Simperl

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

The Future of Platform Engineering

Jemma Hussein Allen

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 3

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

PCI PIN Basics Webinar from the Controlcase Team

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Securing your Kubernetes cluster_ a step-by-step guide to success !

Leading Change strategies and insights for effective change management pdf 1.pdf

Mission to Decommission: Importance of Decommissioning Products to Increase E...

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Connector Corner: Automate dynamic content and events by pushing a button

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Neuro-symbolic is not enough, we need neuro-*semantic*

Elevating Tactical DDD Patterns Through Object Calisthenics

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Knowledge engineering: from people to machines and back

Epistemic Interaction - tuning interfaces to provide information for AI support

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

The Future of Platform Engineering

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Near real-time statistical modeling and anomaly detection using Flink!

1. Near real-time statistical modeling and anomaly detection using Flink! Kunal Umrigar Balint Kurnasz

2. 2 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Customer Digital Experience See, understand and improve connected experiences everywhere Customers: Employee Digital Experience Ready your network and successfully embrace multi-cloud and SaaS Customers: End to End Network Visibility WAN Experience Deploy modern hybrid and SD-WANs with confidence Customers:

3. 3 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Visualize Experience from Every Vantage Point Data Center Data Center SaaS IaaS CDN DNS DNS Provider Security Provider API Customer Cloud Endpoint Enterprise

4. 4 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Data Center Scale @ ThousandEyes Branch Office Branch Office Branch Office Branch Office Siebel HRMS Flow PCAP SNMP Exchange Data Center Data Center Data Center Data Center SaaS IaaS CDN DNS DNS Provider Security Provider API Customer 2B+ Raw Events per day 36K+ agents across cloud and enterprise accounts ~150K+ Alerts sent out per day

5. 5 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes ThousandEyes Alerting Overview Alerter Controller ● Ingests change capture events using Debezium ● Enriches raw events with metadata to generate control events Alerter Engine ● Ingests and merges control events and raw data points ● Enrich, evaluate and aggregate data ● Generates alert events Alert Persistence and Notification ● Persists alert events to alerts datastore ● Notifies the notification subsystem ● Notification system delivers alerts to Customers using Email, Webhooks, Slack, PagerDuty, ServiceNow, MS-Teams, many third party integrations…

6. 6 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Alert Evaluation ● Rule expressions have specific grammar and features ● Generated AST from the expression ● Variety of supported data types

7. 7 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes From static to dynamic thresholds ● Static evaluation challenges: ○ Static thresholds can not adapt to seasonal changes ○ Static values require finding the right thresholds ● Moving to dynamic thresholds: ○ Establish baseline based on past 24Hr mean ○ Establish Dynamic thresholds based on ■ absolute value above baseline ■ percentage delta over baseline ■ N standard deviations over baseline ○ Evaluation becomes a feedback loop

8. 8 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Calculations ● Moving Average - 24h sliding window ● Moving Standard Deviation - 3hr sliding window ● Perfect fit for Flink AggregateFunction

9. 9 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Streaming statistical computations ● Statistical computations in a separate Flink Job ● Results lookup using a GRPC service call from an AsyncFunction ● Advantages: ○ independent scalability ○ async processing ○ eventually consistent

10. 10 © 1992–2020 Cisco Systems, Inc. All rights reserved. @ThousandEyes Conclusion & Next Steps ● Simple and robust Flink job for statistical computations ● Low maintenance ● Easy to add support for more metrics using configuration ● Extremely low latency evaluations Next Steps: ● Optimize rate of false positives ● Computing statistics over larger durations ● Leveraging normalized standard deviation ● Machine Learning to detect trends and anomalies.

Near real-time statistical modeling and anomaly detection using Flink!

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Near real-time statistical modeling and anomaly detection using Flink!

Similar to Near real-time statistical modeling and anomaly detection using Flink! (20)

More from Flink Forward

More from Flink Forward (12)

Recently uploaded

Recently uploaded (20)

Near real-time statistical modeling and anomaly detection using Flink!