This document summarizes a presentation about how United Airlines could use streaming data and Apache Flink to improve customer experience by analyzing social media posts. Key points include: - United has large scale operations that generate vast amounts of streaming data from aircraft, passengers, employees, etc. - Goals are to improve customer experience, revenue, operations through personalized offers, issue tracking, and predictive analytics. - Apache Flink is proposed as a stream processing engine to classify, enrich, and route social media posts to the appropriate support channels in real-time. - Techniques discussed include asynchronous calls, windowed joins, state management, and control streams to dynamically update machine learning models without interrupting the data stream

1. Streaming and
Social Media
Joe Olson
Senior Manager, Big Data Analytics
Apache Road Show Chicago - May 2019

2. Agenda
United and the Airline Industry
How Streaming Model Presents
Opportunity
Apache Flink
4 Q & A

3. 2
About United Airlines…..
 1,348 aircraft (779 mainline, 569 regional) with 250+ on order (supply chain)
 158M passengers in 2018
(public facing web site, mobile app, time / geospatial based inventory, loyalty program, surveys, ancillary sales)
 4900 daily departures (scheduling, operations, weather, route planning)
 355 airports served, in 48 countries (baggage claim, check-ins)
 88,000 employees worldwide (scheduling, pay)
 Constantly in motion! Future (and past) always changing.
 A data scientist / data engineer dream.
Source: https://hub.united.com/corporate-fact-sheet/

4. 3
Business Goals
 Improve Customer Experience
- How can we reduce friction when booking a reservation? Maneuvering through an airport?
- How can we deliver a consistent message across all channels? (mobile app, web site, social media etc)
 Improve Employee Experience
- How can we keep employees better informed of the current situation so they can relay it to the customers?
- What are we learning from our surveys about what the customer bases says is / isn’t working?
 Revenue Generation
- What personalized offers can we make to our customers?
- Are our offers competitive with the rest of the industry?
 Improve Operational Reliability
- How can we better prepare for weather or other operational interruptions?
- How can we manage the fleet better and insure spare parts are where they need to be?

5. 4
Industry Ideas – Customer Experience

6. 5
Use Case – Improve Customer Experience Via Social Media
 Social media represents a unique opportunity for any service company
- Connect with customers in a familiar environment.
- Consistent messaging and brand management.
- Build community and advocacy.
- Direct issues to appropriate channels so they can be handled expediently.

7. 6
Use Case – Customer Experience
 Can we use social media as a giant issue tracking database?
 Obstacles:
- Who am I talking to?
- Is there an issue? If so, what is the issue?
- What is the current state of the issue? How did it get there?
- Are there any recommendations on how to handle the issue?
- Who is best equipped to handle this issue?
All of these need to be overcome within a few seconds of receiving a notification…

8. 7
Use Case – Customer Experience
 Actions
- Identification (Who am I talking to?)
- Classification, prioritization (Is there an issue? What is it? How important is it?)
- State determination (What is the current state of the issue? How did it get there?)
- Recommendation, clustering (Are there any recommendations on how to handle the issue?)
- Routing (Who is best equipped to handle this issue?)
Conclusion: several enrichments + state lookup
Other needs: low latency, fault tolerance, high availability, elasticity…

9. 8
Stream Processing Engine
 Apache Flink - Stateful Computations over Data Streams
What about enrichment?

10. 9
Stream Processing Engine - Enrichment
 Enrichment options:
- Option 1: Data lives in an external database or service using a map
- Option 2: Data arrives as a second stream
Option #1:
Social Media Messages
Social Media Messages
Source
Source
Map
(keyBy)
Map
(keyBy)
Map
Map

11. 10
Stream Processing Engine - Enrichment
 Option #1 Issues
- Synchronous requests are slow and prone to error, jamming up the pipeline
- Wasted resources while waiting for the service to respond
 What about asynchronous?
- AsyncFunction in DataStream API since Flink 1.2
• A queue of promises
• Emitter on a different thread
- Client needs to support async requests

12. 11
Stream Processing Engine - Enrichment
 Async call:
DataStream<Tuple2<String, String>> result =
AsyncDataStream.(un)orderedWait(stream,
new MyAsyncFunction(),
1000, TimeUnit.MILLISECONDS, 100)
– our asycFunction
– a timeout: max time until considered failed
– capacity: max number of queued up requests
– unorderedWait: emit results in order of completion
– orderedWait: emit results in order of arrival
Timeout: Exception thrown. Can override exception handler.
Capacity exceeded: back pressure.

13. 12
Stream Processing Engine - Enrichment
 Option #2 - joining streams
Social Media Messages Source Map
(keyBy)
Social Media Messages Source Map
(keyBy)
Events Source Map
(keyBy)
Join

14. 13
Stream Processing Engine - Joining
 Window join
- Only elements within the same window can be joined
• Tumbling window
• Sliding window
• Session window
- Interval Join
• Common key and where elements of stream B have event timestamps that lie in a relative
time interval to event timestamps of elements in stream A

15. 14
Stream Processing Engine - State
 Managing state
- Ability to store and retrieve information about a key.
VS.
Client - Server Stateful Streaming

16. 15
Stream Processing Engine - State
 Operate on a key-value pull on a keyed stream
 Several possible back ends, all easily configurable at cluster create time:
- Memory (very small state)
- File on disk
- RocksDB (very large state)
Keyed Stream
<Key> <Value>

17. 16
Stream Processing Engine - State
 Types of state
- ValueState<T> - use this when the state is a single value
- ListState<T> - use this when the state is a list of items
- ReducingState<T> - single value that represents an aggregation of all values added to state
- AggregatingState<IN, OUT> - similar to ReducingState, the aggregation function can change
based on different inputs types.
- MapState<UK, UV> - mapping. Can use put(UK, UV) or get(UK). Also iterable.

18. 17
Stream Processing Engine – Queryable State
 Ability to query state from outside a Flink cluster via an API:
Flink Compute Cluster
Keyed Stream <K>
<V>

19. 18
State - Other Issues
 Fault tolerance and high availability: Savepointing
HDFS / S3, etc

20. 19
Stream Processing - Other Issues
 Elasticity:
- Flink Active (Flink controls resource allocation) / Reactive (external entity controls resource
allocation) mode.
- FLIP-6
- Idea: cluster manager creates and destroys task managers based on demand.
- Flink Forward San Francisco 2019: Future of Apache Flink Deployments: Containers,
Kubernetes and More - Till Rohrmann

21. 20
Use Case – Customer Experience
 Actions
- Identification (Who am I talking to?)
- Classification, prioritization (Is there an issue? What is it?)
- State determination (What is the current state of the issue, and how did it get there?)
- Recommendation, clustering (Are there any recommendations on how to handle the issue?)
- Routing (Who is best equipped to handle this issue?)
 Some of these are machine learning / model type applications.
 How to switch model versions without interrupting the stream?
- Control Stream!

22. 21
Stream Processing Engine – Interacting With Models
 Control Stream:
Social Media Messages Source
Model A
List State
Make sure the output stream contains which model version was used!
Map
(KeyBy)
Control Stream Source
Connect CoFlatMap
Model B
Map
(KeyBy)

23. 22
Apache Communities
 Twitter: @ApacheFlink
 Mailing Lists
- news@flink.apache.org
- community@flink.apache.org
- user@flink.apache.org
- dev@flink.apache.org
- issues@flink.apache.org
 Stack Overflow: apache-flink tag
 Github
- https://github.com/apache/flink
Apache Flink

24. Thank You!
We’re hiring!
- Data Engineers
- Data Scientists