Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu

•

1 like•300 views

Uber operates in the complex physical world. One of the challenges of providing a reliable service is detecting highly geolocalized and dynamic scenarios in real-time, such as spatial hotspot, demand/supply imbalanced neighborhoods and etc. The problem is hard because the global scale of Uber is massive, neighborhoods and traffic characteristics are localized and the time to detection needs to be under low latency to be actionable. To solve this problem, Uber engineers have built the situation detection platform powered by Apache Flink and CEP library. In this talk, I will cover i) How we evolve our end-to-end solution to this date by leveraging Apache Flink, Uber’s hexagonal spatial indexing system H3 and large scale clustering algorithms. ii) How we aggregate billions of events across the globe and derive geospatial semantics through CEP pattern matching. iii) Challenges involved in scaling the platform and the various techniques we employed.

Geospatial Situation Detection through
FlinkCEP @Uber
Niel Hu
Engineer | Uber
niel.hu@uber.com

Uber | Geospatial Situation Detection through FlinkCEP
● Marketplace
● Observability Problem
● Large-Scale Clustering
● Situation Detection through Pattern Matching
● Tips, Tricks and Lessons Learned
Outline

Uber | Geospatial Situation Detection through FlinkCEP
Marketplace
● Modeling the physical world
● Global Logistics Network
● Real-Time Decision Engine

Uber | Geospatial Situation Detection through FlinkCEP
Marketplace
Dynamic Pricing
Forecasting
Driver Positioning
Intelligent Dispatch Marketplace Health
Marketplace Platform & Data
Fares
Driver / Rider Pricing

Uber | Geospatial Situation Detection through FlinkCEP
Photo: Jessica Christian / The Chronicle
Observability Problem

Uber | Geospatial Situation Detection through FlinkCEP
Source: Giphy
[https://giphy.com/gifs/FmNXeuoadNTpe]
Scaling Observability
● 700+ Cities
● Local Heterogeneity
● Space and Time Dimensions
● Real-Time Constraints

Uber | Geospatial Situation Detection through FlinkCEP
Problem
EDGE ZOOM IN PATTERN MATCH OBSERVER

Uber | Geospatial Situation Detection through FlinkCEP
Detecting the Region
● Similar Characteristics
● Connected Region
● Arbitrary Shape
● Cheap in Computation

Uber | Geospatial Situation Detection through FlinkCEP
Detecting the Region through clustering
● K-means ?
● Density-based clustering ?

Uber | Geospatial Situation Detection through FlinkCEP
Density-based Spatial Clustering with Noise
● Epsilon Ball Rule
● Worst Case
○ O(n²) !
○ O(n log n)
(auxiliary data structure)

Uber | Geospatial Situation Detection through FlinkCEP
How can we do better ?

Uber | Geospatial Situation Detection through FlinkCEP
H3 : Hexagonify the World !
https://h3geo.org/#/

Uber | Geospatial Situation Detection through FlinkCEP
Credited to Nick Rabinowitz

Uber | Geospatial Situation Detection through FlinkCEP
Credited to Nick Rabinowitz
Uniform Adjacency

Uber | Geospatial Situation Detection through FlinkCEP
Credited to Nick Rabinowitz
DBSCAN in Hexagons

Uber | Geospatial Situation Detection through FlinkCEP
Data SIO, NOAA, U.S. Navy, NGA, GEBCO,
Image Landsat / Copernicus
Image IBCAO
Credited to Nick Rabinowitz

Uber | Geospatial Situation Detection through FlinkCEP
Junior, M.R., Souza, B.J., & Endler, M. (2019). DG2CEP: a near
real-time on-line algorithm for detecting spatial clusters large
data streams through complex event processing. Journal of
Internet Services and Applications, 10, 1-28.
Low-Latency Clustering on Streams

Uber | Geospatial Situation Detection through FlinkCEP
Low-Latency Clustering on Streams

Uber | Geospatial Situation Detection through FlinkCEP
Clustering on streaming data
● Create
● Update
● Merge (expensive!)

Uber | Geospatial Situation Detection through FlinkCEP
Clustering on streaming data
● Disperse
● Split (expensive!)

Uber | Geospatial Situation Detection through FlinkCEP
Static clusters are not good enough to capture
marketplace dynamics

Uber | Geospatial Situation Detection through FlinkCEP
Situation
CLUSTERS IN MOTION
t1 t2 t3

Uber | Geospatial Situation Detection through FlinkCEP
Complex Event Processing

Uber | Geospatial Situation Detection through FlinkCEP
CEP : Cluster as Primitive
Overlap / Expand
Overlap / Shrink
S1
Time
S2

Uber | Geospatial Situation Detection through FlinkCEP
Time
S+ followed by S
CEP : Situation as Primitive

Uber | Geospatial Situation Detection through FlinkCEP
Human in the loop

Uber | Geospatial Situation Detection through FlinkCEP
Dynamic Rule Conﬁguration

Uber | Geospatial Situation Detection through FlinkCEP
Tips, Tricks and Lessons learned

Uber | Geospatial Situation Detection through FlinkCEP
Staggering Window
● Thundering Herd Workload
● Low utilization of compute resource
● Massive Fan-Out

Uber | Geospatial Situation Detection through FlinkCEP
Staggering Window
FLINK-12855

Uber | Geospatial Situation Detection through FlinkCEP
Continuous Learning on Streams
● Non-Stationarity !
● Overfitting to the Noise
● Combine Offline/Online Models

Uber | Geospatial Situation Detection through FlinkCEP
Thank you !
niel.hu@uber.com
https://www.linkedin.com/in/tenghu/

Uber | Geospatial Situation Detection through FlinkCEP
Appendix

Uber | Geospatial Situation Detection through FlinkCEP
Observe, Orient, Decide, Act

Planet has the ambitious goal of imaging everywhere on earth once per day with a fleet of small satellites. Now with over 100 operational satellites, Planet is collecting over a hundred million square kilometers of remote sensing data every day and for the first time we are able to take actions based on the daily changes that we observe. In addition to this unique data set, Planet has taken an 'API-first' approach to distributing data, allowing our users to build their own applications or integrations directly on our platform services. Safe Software's own Planet transformer is a great example of this kind of integration, giving FME users easy access to Planet's growing archive of satellite imagery.

UROP Symposium PosterNeil Schwartz

Effect of Weather on Uber Ridership_rev1 (1)Anusha Mamillapalli

2015 on Spatial Development (Personal Projects)

David Eliseo Martinez Castellanos

android.location - a tool for developers or GNSS researchers?

Lukasz Kosma Bonenberg

GOOGLE EARTH INTRODUCTION

Akshat Singh

A Day in the Life of King County GIS

Greg Babinski

Driving behaviors for adas and autonomous driving XII

Yu Huang

Koober Preduction IO Presentation

Salesforce Engineering

Koober Machine Learning

James Ward

TRAFFIC MANAGEMENT THROUGH SATELLITE IMAGING -- Part 1

NanubalaDhruvan

20210226 esa-science-coffee-v2.0

Advanced-Concepts-Team

The Impact of Always-on Connectivity for Geospatial Applications and Analysis

SingleStore

mzyautonavigationsystem.pptx

mzy201030

Putting Your Data on a Map

Shawn Day

Mapping your data can help to provide new insights on your research findings. However, many scholars are put off by the steep learning curve demanded by Geographic Information Systems (GIS) such as ArcGIS from ESRI. New and simple tools have become available that offer sophisticated output without extensive training. In fact, tools such as Google Maps, Google Earth, Open Street Map among others can offer immediate returns in a matter of hours where tasks in the past required, weeks, months and even years of training.

최근의 공간정보 동향과 시사점 - 한국역학회 특강

SANGHEE SHIN

Surveying Areas in Developing Regions Through Context Aware Drone Mobility

Alessandro Montanari

Developing regions are often characterized by large areas that are poorly reachable or explored. The mapping of these regions and the census of roaming populations in these areas are often difficult and sporadic. In this paper we put forward an approach to aid area surveying which relies on autonomous drone mobility. In particular we illustrate the two main components of the approach. An efficient on device object detection component, built on Convolutional Neural Networks, capable of detecting human settlements and animals on the ground with acceptable performance (latency and accuracy) and a path planning component, informed by the object identification module, which exploits Artificial Potential Fields to dynamically adapt the flight in order to gather useful information of the environment, while keeping optimal flight paths. We report some initial performance results of the on board visual perception module and describe our experimental platform based on a fixed-wing aircraft.

Transit 2.0 - World Intelligent Transportation Systems Congress

Aaron Antrim

Lesson3 esa summer_school_brovelli

Maria Antonia Brovelli

jardinSundeep Bajikar

Transit sketch planning (webinar slides)

garrjacobs

This is the presentation from a webinar focused on communicating useful knowledge and context for sketch planning and other planning initiatives within transit. The topics covered included basic philosophies in transit planning, common considerations that can be anticipated when planning for and kicking off a transit project, and helpful transit planning data resources. The Transitmix team provided demonstrations throughout the presentation on how their app can help local governments and communities work together to achieve transit planning goals.

Swarm.Robotics Research Report IEEE

Asad Masood

This project aims a swarm behaviour of multiple robots operating indoor in a closed formation. An on-board computation is used for localization and to get linear control of the robots. The entire task is performed indoor, performing distinct trajectories assigned to individual or group of robots. In our project we showed several estimation techniques which increase the robust tracking of trajectories. External localization is obtained to get the position of robots with the help of webcam. MILP algorithm or convex optimization is used to generate a complex trajectory that is performed by the distinct robots. We performed various trajectories, as an example to show the agility of micro robots tracking complex paths and avoiding stationary obstacles.

What's hot

Planet: Imaging Earth Every Day

Safe Software

UROP Symposium PosterNeil Schwartz

Effect of Weather on Uber Ridership_rev1 (1)Anusha Mamillapalli

2015 on Spatial Development (Personal Projects)

David Eliseo Martinez Castellanos

android.location - a tool for developers or GNSS researchers?

Lukasz Kosma Bonenberg

GOOGLE EARTH INTRODUCTION

Akshat Singh

A Day in the Life of King County GIS

Greg Babinski

What's hot (7)

Planet: Imaging Earth Every Day

UROP Symposium Poster

Effect of Weather on Uber Ridership_rev1 (1)

2015 on Spatial Development (Personal Projects)

android.location - a tool for developers or GNSS researchers?

GOOGLE EARTH INTRODUCTION

A Day in the Life of King County GIS

Similar to Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu

Driving behaviors for adas and autonomous driving XII

Yu Huang

Koober Preduction IO Presentation

Salesforce Engineering

Koober Machine Learning

James Ward

TRAFFIC MANAGEMENT THROUGH SATELLITE IMAGING -- Part 1

NanubalaDhruvan

20210226 esa-science-coffee-v2.0

Advanced-Concepts-Team

The Impact of Always-on Connectivity for Geospatial Applications and Analysis

SingleStore

mzyautonavigationsystem.pptx

mzy201030

Putting Your Data on a Map

Shawn Day

최근의 공간정보 동향과 시사점 - 한국역학회 특강

SANGHEE SHIN

Surveying Areas in Developing Regions Through Context Aware Drone Mobility

Alessandro Montanari

Transit 2.0 - World Intelligent Transportation Systems Congress

Aaron Antrim

Lesson3 esa summer_school_brovelli

Maria Antonia Brovelli

jardinSundeep Bajikar

Transit sketch planning (webinar slides)

garrjacobs

Swarm.Robotics Research Report IEEE

Asad Masood

Similar to Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu (15)

Driving behaviors for adas and autonomous driving XII

Koober Preduction IO Presentation

Koober Machine Learning

TRAFFIC MANAGEMENT THROUGH SATELLITE IMAGING -- Part 1

20210226 esa-science-coffee-v2.0

The Impact of Always-on Connectivity for Geospatial Applications and Analysis

mzyautonavigationsystem.pptx

Putting Your Data on a Map

최근의 공간정보 동향과 시사점 - 한국역학회 특강

Surveying Areas in Developing Regions Through Context Aware Drone Mobility

Transit 2.0 - World Intelligent Transportation Systems Congress

Lesson3 esa summer_school_brovelli

jardin

Transit sketch planning (webinar slides)

Swarm.Robotics Research Report IEEE

More from Flink Forward

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Evening out the uneven: dealing with skew in Flink

Flink Forward

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Autoscaling Flink with Reactive Mode

Flink Forward

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

One sink to rule them all: Introducing the new Async Sink

Flink Forward

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Flink powered stream processing platform at Pinterest

Flink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Apache Flink in the Cloud-Native Era

Flink Forward

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Where is my bottleneck? Performance troubleshooting in Flink

Flink Forward

Flinkn Forward San Francisco 2022. In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times. by Piotr Nowojski

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

The Current State of Table API in 2022

Flink Forward

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

Flink SQL on Pulsar made easy

Flink Forward

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Processing Semantically-Ordered Streams in Financial Services

Flink Forward

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

More from Flink Forward (20)

Building a fully managed stream processing platform on Flink at scale for Lin...

Evening out the uneven: dealing with skew in Flink

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Introducing the Apache Flink Kubernetes Operator

Autoscaling Flink with Reactive Mode

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

One sink to rule them all: Introducing the new Async Sink

Tuning Apache Kafka Connectors for Flink.pptx

Flink powered stream processing platform at Pinterest

Apache Flink in the Cloud-Native Era

Where is my bottleneck? Performance troubleshooting in Flink

Using the New Apache Flink Kubernetes Operator in a Production Deployment

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Dynamic Rule-based Real-time Market Data Alerts

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Processing Semantically-Ordered Streams in Financial Services

Tame the small files problem and optimize data layout for streaming ingestion...

Batch Processing at Scale with Flink & Iceberg

Recently uploaded

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

By Design, not by Accident - Agile Venture Bolzano 2024

Pierluigi Pugliese

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Peter Spielvogel

Building better applications for business users with SAP Fiori. • What is SAP Fiori and why it matters to you • How a better user experience drives measurable business benefits • How to get started with SAP Fiori today • How SAP Fiori elements accelerates application development • How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities • How SAP Fiori paves the way for using AI in SAP apps

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

UiPath Community Day Dubai: AI at Work..

UiPathCommunity

Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking. 📕 Curious on our agenda? Wait no more! 10:00 Welcome note - UiPath Community in Dubai Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank 10:20 A UiPath cross-region MEA overview Ashraf El Zarka, VP and Managing Director MEA, UiPath 10:35: Customer Success Journey Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank 11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more Boris Krumrey, Global VP, Automation Innovation, UiPath 12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services. Brendan Lingam, Director of Sales and Business Development, Marc Ellis

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

UiPathCommunity

In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni. 📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath: Autopilot per Studio Web Autopilot per Studio Autopilot per Apps Clipboard AI GenAI applicata alla Document Understanding 👨‍🏫👨‍💻 Speakers: Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath Andrei Tasca, RPA Solutions Team Lead @NTT Data

The Future of Platform Engineering

Jemma Hussein Allen

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

Enhancing Performance with Globus and the Science DMZ

Globus

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Recently uploaded (20)

Elizabeth Buie - Older adults: Are we really designing for our future selves?

By Design, not by Accident - Agile Venture Bolzano 2024

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

FIDO Alliance Osaka Seminar: Overview.pdf

PCI PIN Basics Webinar from the Controlcase Team

PHP Frameworks: I want to break free (IPC Berlin 2024)

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

UiPath Community Day Dubai: AI at Work..

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

Essentials of Automations: The Art of Triggers and Actions in FME

Removing Uninteresting Bytes in Software Fuzzing

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

Epistemic Interaction - tuning interfaces to provide information for AI support

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

The Future of Platform Engineering

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Enhancing Performance with Globus and the Science DMZ

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu

1. Geospatial Situation Detection through FlinkCEP @Uber Niel Hu Engineer | Uber niel.hu@uber.com

2. Uber | Geospatial Situation Detection through FlinkCEP ● Marketplace ● Observability Problem ● Large-Scale Clustering ● Situation Detection through Pattern Matching ● Tips, Tricks and Lessons Learned Outline

3. Uber | Geospatial Situation Detection through FlinkCEP Marketplace ● Modeling the physical world ● Global Logistics Network ● Real-Time Decision Engine

4. Uber | Geospatial Situation Detection through FlinkCEP Marketplace Dynamic Pricing Forecasting Driver Positioning Intelligent Dispatch Marketplace Health Marketplace Platform & Data Fares Driver / Rider Pricing

5. Uber | Geospatial Situation Detection through FlinkCEP Photo: Jessica Christian / The Chronicle Observability Problem

6. Uber | Geospatial Situation Detection through FlinkCEP Source: Giphy [https://giphy.com/gifs/FmNXeuoadNTpe] Scaling Observability ● 700+ Cities ● Local Heterogeneity ● Space and Time Dimensions ● Real-Time Constraints

7. Uber | Geospatial Situation Detection through FlinkCEP Problem EDGE ZOOM IN PATTERN MATCH OBSERVER

8. Uber | Geospatial Situation Detection through FlinkCEP Detecting the Region ● Similar Characteristics ● Connected Region ● Arbitrary Shape ● Cheap in Computation

9. Uber | Geospatial Situation Detection through FlinkCEP Detecting the Region through clustering ● K-means ? ● Density-based clustering ?

10. Uber | Geospatial Situation Detection through FlinkCEP Density-based Spatial Clustering with Noise ● Epsilon Ball Rule ● Worst Case ○ O(n²) ! ○ O(n log n) (auxiliary data structure)

11. Uber | Geospatial Situation Detection through FlinkCEP How can we do better ?

12. Uber | Geospatial Situation Detection through FlinkCEP H3 : Hexagonify the World ! https://h3geo.org/#/

13. Uber | Geospatial Situation Detection through FlinkCEP Credited to Nick Rabinowitz

14. Uber | Geospatial Situation Detection through FlinkCEP Credited to Nick Rabinowitz Uniform Adjacency

15. Uber | Geospatial Situation Detection through FlinkCEP Credited to Nick Rabinowitz DBSCAN in Hexagons

16. Uber | Geospatial Situation Detection through FlinkCEP Data SIO, NOAA, U.S. Navy, NGA, GEBCO, Image Landsat / Copernicus Image IBCAO Credited to Nick Rabinowitz

17. Uber | Geospatial Situation Detection through FlinkCEP Junior, M.R., Souza, B.J., & Endler, M. (2019). DG2CEP: a near real-time on-line algorithm for detecting spatial clusters large data streams through complex event processing. Journal of Internet Services and Applications, 10, 1-28. Low-Latency Clustering on Streams

18. Uber | Geospatial Situation Detection through FlinkCEP Low-Latency Clustering on Streams

19. Uber | Geospatial Situation Detection through FlinkCEP Clustering on streaming data ● Create ● Update ● Merge (expensive!)

20. Uber | Geospatial Situation Detection through FlinkCEP Clustering on streaming data ● Disperse ● Split (expensive!)

21. Uber | Geospatial Situation Detection through FlinkCEP Static clusters are not good enough to capture marketplace dynamics

22. Uber | Geospatial Situation Detection through FlinkCEP Situation CLUSTERS IN MOTION t1 t2 t3

23. Uber | Geospatial Situation Detection through FlinkCEP Complex Event Processing

24. Uber | Geospatial Situation Detection through FlinkCEP CEP : Cluster as Primitive Overlap / Expand Overlap / Shrink S1 Time S2

25. Uber | Geospatial Situation Detection through FlinkCEP Time S+ followed by S CEP : Situation as Primitive

26. Uber | Geospatial Situation Detection through FlinkCEP Human in the loop

27. Uber | Geospatial Situation Detection through FlinkCEP Dynamic Rule Conﬁguration

28. Uber | Geospatial Situation Detection through FlinkCEP Tips, Tricks and Lessons learned

29. Uber | Geospatial Situation Detection through FlinkCEP Staggering Window ● Thundering Herd Workload ● Low utilization of compute resource ● Massive Fan-Out

30. Uber | Geospatial Situation Detection through FlinkCEP Staggering Window FLINK-12855

31. Uber | Geospatial Situation Detection through FlinkCEP Staggering Window FLINK-12855

32. Uber | Geospatial Situation Detection through FlinkCEP Continuous Learning on Streams ● Non-Stationarity ! ● Overfitting to the Noise ● Combine Offline/Online Models

33. Uber | Geospatial Situation Detection through FlinkCEP Thank you ! niel.hu@uber.com https://www.linkedin.com/in/tenghu/

34. Uber | Geospatial Situation Detection through FlinkCEP Appendix

35. Uber | Geospatial Situation Detection through FlinkCEP Observe, Orient, Decide, Act

Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Similar to Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu

Similar to Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu (15)

More from Flink Forward

More from Flink Forward (20)

Recently uploaded

Recently uploaded (20)

Virtual Flink Forward 2020: Geospatial Situation Detection through FlinkCEP @Uber - Teng (Niel) Hu