Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL

Stream Processing Airport Data
Sönke Liebau – Co-Founder and Partner @ OpenCore
January 10th
2019
Serving the Real-Time Data Needs of an Airport with Kafa
Streams and KSQL

Who Am I?
• Partner & Co-Founder at
• Small consultng company with a Big Data & Open Source focus
• First producton Kafa deployment in 2014
Website: www.opencore.com
soenke.liebau@opencore.com
htps:/p/pwww.linkedin.com/pin/psoenkeliebau/p
@soenkeliebau

Source: htps:/p/pkafa.apache.org/p20/pdocumentaton/pstreams/p 13
What Is Kafka Streams?
“The easiest way to write mission-critcal real-tme applicatons and
microservices”
“Kafa Streams is a client library for building applicatons and microservices, where the input and output
data are stored in Kafa clusters. “

What Is KSQL?
Confluent S QL is the open source,
streaming QL engine that enables
real-tme data processing against
Apache Safaa
Source: htps:/p/pwww.coniuent.io/pproduct/pksql/p 14

© 2019 OpenCore GmbH & Co. KG 15
Kafka Streams & KSQL In The
Ecosystem
Sources KafaConnect
KafaConnect
Destnatons
Kafa
Streams
Jobs

The Big Diference

20
Using Kafka Streams
final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();

KStream<String, String> textLines = builder.stream("streams-plaintext-input",
    Consumed.with(stringSerde, stringSerde);

KTable<String, Long> wordCounts = textLines
    .fatMapValues(value -> Arrays.asList(value.toLowerCase().split("W+")))
     .groupBy((key, value) -> value)
     .count()

wordCounts.toStream().to("streams-wordcount-output", Produced.with(Serdes.String(), Serdes.Long()));

Source: htps:/p/pkafa.apache.org/p20/pdocumentaton/pstreams/pquickstart

21
Using Kafka Streams
final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();

KStream<String, String> textLines = builder.stream("streams-plaintext-
input",
    Consumed.with(stringSerde, stringSerde);

KTable<String, Long> wordCounts = textLines
    .fatMapValues(value ->
Arrays.asList(value.toLowerCase().split("W+")))
     .groupBy((key, value) -> value)
     .count()

wordCounts.toStream().to("streams-wordcount-output",
Produced.with(Serdes.String(), Serdes.Long()));

Source: htps:/p/pkafa.apache.org/p20/pdocumentaton/pstreams/pquickstart
…

Using KSQL
RestInterface
CLI
Rest
Client
SELECT *
FROM security_in
WHERE status=’success’
AND terminal=’t1’;

The Competition

Safa treams S QL
When To Use Which?
• Ofers lower level access
• More data formats supported
• Queryable state
• Problems that cannot be expressed in
SQL
• Easier for people used to SQL
• No need for additonal orchestraton
• Data exploraton

Our Airport

A Few Facts Up Front
• A lot of independent data sources
• Airline tcketng
• Baggage transport system
• Passenger countng
• Retail
• Radar
• Weather
• …
• Spread over multple companies
• Many legacy interfaces

Integrations
Operatons
Database
External
System
External
System
External
System
External
System
External
System
External
System

Isolated Islands Of Data
• A lot of isolated data stores to provide data for necessary solutons
• Spiderweb of integratons
• Operatonal DB needs to push data to a lot of systems
• Many diferent formats

The Dream
…
Weird
binary
source
XML
Source
Destnaton
Destnaton
Destnaton
Raw Source Processed
RestStream Processing

Ingest Transformation - Kafka Streams
StreamsBuilder builder = new StreamsBuilder();
Serde<ProprietaryObject> weirdFormatSerde = new ProprietaryWeirdFormatSerde();
Serde<ProprietaryObject> avroSerde = new ProprietaryAvroSerde();

builder.stream(“proprietary_input_topic",
    Consumed.with(
        Serdes.String(),
        weirdFormatSerde))
    .to("avro_output_topic",
        Produced.with(
            Serdes.String(),
            avroSerde));

33
Ingest Transformation - KSQL
ksql> CREATE STREAM source (uid INT, name VARCHAR) WITH (KAFKA_TOPIC='mysql_users',
VALUE_FORMAT='JSON‘);
ksql> CREATE STREAM target_avro WITH (VALUE_FORMAT='AVRO', KAFKA_TOPIC='mysql_users_avro')
AS SELECT * FROM source;
Source: htps:/p/pgist.github.com/prmof/p165b05e4554c41719b71f1a47ee7b113

Stream Processing
• Stream processing jobs read converted avro topics and create enriched
topics/palerts/p… by
• Joining streams
• Aggregatng streams
• Filtering or alertng on streams
• …

DISCLAIMER

Gate Changes
• Gate changes can be based on diferent informaton
• Delays of the incoming iight
• Changes on other outgoing iights
• …
• Join relevant streams and publish change events that are consumed by
• Apps
• Gate monitors
• Departure boards
• …

Passenger Count
• Join stream of tckets scanned before line to security check and camera
count of passengers leaving security check to estmate number of waitng
passengers
• Change routng of passengers (physical: signs change & digital: diferent routng in
app)
• Also consumed by
• Monitors to display predicted waitng tme
• App to display predicted wait tme
• Prediciton systems to feed models for capacity planning
• Models to predict if a passenger might miss his iight -> reroute to priority lane

Wait Time
• Calculate how long a passenger took to clear the security checkpoint by
joining when he scanned his boarding pass and when he is frst spoted by
an iBeacon beyond security
• Push ofers based on wait tme and iight tme
• Long wait, lot of tme tll take-of -> free cofee or sandwich
• Long wait, short tme tll take-of -> duty free voucher
• …

Baggage Notifcation
• Baggage containers are scanned when they are loaded/punloaded
• By joining this with data from the baggage sorter passengers could receive
push notfcatons when their luggage is loaded/punloaded into/pfrom the
plane

Arrival At Gate
• There are complex models running to estmate when the plane will arrive at
the gate afer it has landed
• Based on ground radar data
• Can be used to
• Predict whether the following iight might be delayed
• Coordinate cleaning crews
• Coordinate refueling
• Feed into gate change decisions

An Example Flow
{"boardingpass_id":"123",
"passenger“:"smith",
"flight_number":"LH454“,
“checked_bags”:1}
"security_area":"t1_2",
"status":"success"}
{"security_area":"t1_2",
"count":"1"}
{"passenger":"smith",
"beacon_id":"t1_b123"}
"item_group":"cigarettes"}
"status":"success"}
{"flight_no":"LH454",
“runway":“1north"}
{“old_gate":“a12”,
“new_gate":“e50"}

Check-In Event
"passenger":"smith",
"flight_number":"LH454“,
“terminal”:“terminal1” }
check_in_count
CREATE TABLE check_in_count
AS SELECT terminal, count(terminal)
FROM security_in
WINDOW TUMBLING (SIZE 24 hour)
GROUP BY terminal;
check_in
What is it good for?
• Early warning for security capacity
• „Don‘t dawdle“ warning based on
security queues

Passenger Enters Security Area
"security_area":"t1_2",
"status" : "success"}
security_in_count
CREATE TABLE security_in_count
AS SELECT
  security_area,
  count(security_area)
FROM security_in
WHERE status='success'
GROUP BY security_area;
security_in
• Monitor for failed atempts
• Passenger routng to security
• Unload baggage of late passengers
• …
tme_to_security
SELECT
  s.boardingpass_id, c.rowtime - s.rowtime
as time_to_security
FROM security_in s
LEFT JOIN check_in c WITHIN 1 HOUR
ON s.boardingpass_id=c.boardingpass_id;

Passenger Leaves Security Area
{"security_area":"t1_2",
"count“:"1"}
security_out security_out_count
CREATE TABLE security_out_count
AS SELECT security_area, sum(count)
FROM security_out
GROUP BY security_area;
security_in_count
current_count
• Capacity planning
• Wait tme predicton
• Passenger routng (apps & physical)
• Alertng on late passengers checking in
• …
SELECT
  i.terminal AS terminal,
  i.KSQL_COL_1 AS entry,
  o.KSQL_COL_1 AS exit,
  i.KSQL_COL_1 - o.KSQL_COL_1 AS
waiting
FROM security_in_count i
INNER JOIN security_out_count o
ON i.terminal=o.terminal;

Passenger Located Via iBeacon
{"passenger":"smith",
"beacon_id":"t1_b123"}
security_duraton
security_in
dutyfree_joined
CREATE STREAM dutyfree_joined
AS SELECT c.boardingpass_id, d.passenger
FROM dutyfree_in d
LEFT JOIN security_in s WITHIN 1 HOURS
ON s.passenger=d.passenger;
dutyfree_in
SELECT
  d.boardingpass_id,
  d.d_passenger,
  d.rowtime - s.rowtime as
time_in_security
FROM dutyfree_in_with_bc d
LEFT JOIN security_in s WITHIN 1 HOUR
ON d.boardingpass_id=s.boardingpass_id;
• Refning wait tme predicton
• Targeted questonaire (fnd reasons for
outliers)
• Vouchers for huge delays
• …

Purchase Event
"item_group":"cigarettes"}
iight_informaton
check_in
dutyfree_joined
• Retail models
• Route to smoking area nearest to gate
• Advise of walk tme if tme is tght
• …
dutyfree_purchase
CREATE STREAM dutyfree_joined
AS SELECT
  c.boardingpass_id,
  c.passenger,
  p.purchase_type,
  f.gate
FROM dutyfree_purchase p
LEFT JOIN check_in c WITHIN 1 HOURS
ON c.passenger=p.passenger
LEFT JOIN flight_information f
WITHIN 1 HOURS
ON f.flight_number = c.flight_number;

Gate Change
expected_gate_arrival
notfcatons
expected_gate_departure
CREATE STREAM gate_wait_time
AS SELECT
  a.flight,
  d.departure_time - a.arrival_time as wait_time
FROM expected_gate_arrival a
INNER JOIN expected_gate_departure d WITHIN 1 HOURS
ON a.gate=d.gate;
gate_wait_tme
gate_change
CREATE STREAM gate_change
AS SELECT
  flight
FROM gate_wait_time
WHERE wait_time > 600000;
CREATE STREAM notifications
AS SELECT f.passenger
FROM gate_change g
LEFT JOIN flight_information f
WITHIN 1 HOURS ON f.gate=g.gate;
iight_informaton

Passenger Boards Plane
"status":"success"}
gate_in
• Alert on bags without matching passengers
• Trigger unloading based on related events
• Gate closed
• Time based
• …
bags_joined
check_in baggage_loaded
CREATE STREAM bag_join
AS SELECT
c.passenger,
c.bags
FROM gate_in g
LEFT JOIN check_in c
WITHIN 1 HOURS
ON c.boardingpass_id=g.boardingpass_id
LEFT JOIN baggage_loaded b
WITHIN 1 HOURS
ON b.bag_id = c.bag_id;

Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL

Similar to Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL (20)

Recently uploaded

Recently uploaded (20)

Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL