Support de présentation : The end of airline Data revenue management as we know it ?

1. Strictly
CONFIDENTIAL
The end of Airline Revenue Management
as we know it?
(Deep) Reinforcement Learning for Revenue Management
Sophia Conf
July 2018
Speaker: Quan NGUYEN
©AmadeusITGroupanditsaffiliatesandsubsidiaries

2. Revenue Management System (RMS)
Introduction
− RMS: automated system
− Goal: maximize flight revenue
− All seats
− Whole reservation period (1 year)
− What the systems does:
− Set the right fares at the right time

3. Motivation – limitations of RMS
RMS assumptions
− Rely of historical observations – may represent the future or not
− Collection of historical observations
− The future is similar to the past (e.g. schedules, fares, volume, WTP, arrival rate, etc.)
− Rely on a model representation – may be right or wrong
− Requires a demand model for volume and WTP
− Requires an optimization model (e.g. EMSR or Dynamic Programming)
− No exploration – no validation of assumptions
− RMS always takes the “optimal” decision
− Assumes monopoly – in reality there is almost always competition
− Average policies over all competitors states and actions

4. Strictly
CONFIDENTIAL
Introduction to
Reinforcement Learning
©AmadeusITGroupanditsaffiliatesandsubsidiaries

5. 5
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Reinforcement Learning
Introduction
Agent
Environment
Action
𝑎 𝑡
Reward 𝑟𝑡
State 𝑠𝑡
𝑟𝑡+1
𝑠𝑡+1
𝐸 ෍
𝑡=1
𝑇
𝛾 𝑡
𝑟𝑡
Objective: maximize
long term reward

6. 6
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Reinforcement Learning
Applied to Revenue Management
Airline
RMS
Customers
Action
𝑎 𝑡 = 𝑓
Reward 𝑟𝑡
State 𝑠𝑡
𝑟𝑡+1
𝑠𝑡+1
Bookings (x)Empty Full
Depar-
ture
{t,x}
{t+1,x} {t+1,x+1}
State 𝑠𝑡
No
Sale
Sale
Time(t)

7. Strictly
CONFIDENTIAL
QL and DQL - Monopoly
©AmadeusITGroupanditsaffiliatesandsubsidiaries

8. Performance of QL in Monopoly
70%
80%
90%
100%
110%
0 50 100 150 200 250
RevenueIndex
Years!
QL
RMS optimal=100%
QL ~99.3%
Base Reinforcement Learning in a Monopoly
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 7 6 4 9 1 6 3 2 0 3 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 2 2 7 5 8 5 7 7 6 9 5 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 2 2 2 5 10 8 10 2 1 7 0 5 9 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 2 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 2 0 1 1 0 2 2 3 3 3 10 9 6 4 8 2 5 2 7 3 0 6 6 1 4 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
1 1 2 0 1 0 0 0 1 0 2 0 0 0 0 0 0 0 0 0 0 0 0 2 1 0 0 1 1 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 1 1 0 1 0 0 0 0 0 0 1 0 0 1 1 0 1 0 0 0 1 0 1 0 0 0 1 1 2 3 2 2 4 9 8 8 10 4 10 7 3 10 3 6 6 0 8 7 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
5 1 1 0 2 1 1 1 1 1 4 1 2 1 1 1 0 3 1 0 1 0 1 0 0 1 0 0 2 2 0 0 2 2 0 1 1 0 0 0 1 0 0 0 0 2 3 1 2 0 1 0 2 1 0 0 1 1 1 0 0 2 0 1 1 2 1 0 0 0 2 0 1 1 4 1 1 1 1 0 1 1 1 1 1 0 1 1 2 0 1 1 1 1 1 2 6 6 7 6 3 7 4 1 5 5 1 7 1 4 3 3 3 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
5 7 10 7 0 7 1 0 0 1 1 3 2 0 1 3 2 3 4 1 1 2 4 0 3 2 2 1 0 0 1 2 1 0 2 2 1 0 1 0 0 1 1 0 0 2 0 2 0 0 0 2 1 0 2 1 0 0 2 0 2 0 4 0 1 0 1 2 0 1 3 1 0 2 0 2 1 2 3 0 0 2 0 0 1 1 1 0 0 0 1 1 2 1 4 0 1 1 1 1 0 0 1 1 2 2 1 3 0 4 6 2 4 6 6 1 7 5 5 2 7 4 3 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
1 5 6 5 5 7 1 3 1 3 3 1 2 0 1 1 0 2 2 1 1 1 2 1 2 1 0 1 1 1 1 1 3 2 0 2 0 2 1 0 1 3 1 0 1 1 0 1 0 1 0 0 2 1 0 1 1 3 1 1 3 1 0 0 2 0 1 0 1 3 0 3 0 2 1 1 0 1 3 0 3 3 1 2 1 1 0 0 2 0 1 1 0 0 2 1 1 1 1 0 0 1 1 1 3 1 1 0 1 1 1 1 1 1 0 1 2 0 5 6 2 4 2 4 2 3 4 0 2 3 1 3 3 3 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
2 1 2 4 4 2 5 7 3 1 1 2 2 0 1 1 2 2 1 0 0 0 2 3 2 2 0 2 0 0 2 0 1 1 0 1 0 1 3 1 2 2 3 2 1 2 2 3 0 0 0 1 2 3 1 2 1 1 0 1 2 0 3 1 1 2 1 2 1 1 2 3 1 2 2 1 3 1 3 1 2 1 0 2 0 3 2 1 0 0 0 0 1 1 2 2 0 2 1 1 1 3 0 2 0 1 2 3 1 1 1 0 0 0 2 1 0 1 1 0 2 2 1 0 0 0 2 1 1 2 5 1 6 1 4 3 5 3 2 4 3 2 3 1 3 6 6 6 6 5 2 6 6 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
3 1 4 0 2 4 3 2 1 4 1 4 3 0 0 3 5 1 2 2 2 1 1 0 0 1 1 3 1 1 0 0 0 1 0 1 1 2 2 2 2 1 1 2 0 1 1 0 4 0 1 3 1 1 0 1 1 3 1 2 1 0 1 1 0 1 1 0 1 0 4 3 2 0 1 1 2 1 3 1 0 1 1 1 1 0 0 2 2 1 1 2 1 0 0 2 1 0 2 1 0 1 1 0 2 1 1 0 0 0 1 2 2 1 0 1 1 1 0 0 0 1 1 1 1 1 1 2 0 0 1 1 0 1 0 0 0 1 2 4 1 2 3 4 6 2 3 4 3 5 0 1 3 3 5 6 7 0 2 7 1 7 7 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
0 1 0 3 1 3 1 2 4 1 2 4 3 3 1 2 3 2 4 3 3 1 3 3 2 1 0 1 3 0 0 1 2 0 2 0 1 1 3 1 4 1 2 1 2 1 0 0 1 4 1 0 2 1 1 2 1 1 1 4 0 3 3 0 2 0 2 2 1 1 0 0 1 1 2 1 1 0 3 1 3 1 1 1 0 3 3 1 1 1 0 3 1 1 0 0 1 1 0 3 1 1 0 1 1 1 2 2 0 0 0 2 2 2 0 0 2 1 1 1 1 0 1 0 3 0 2 1 2 1 0 2 1 1 1 1 2 0 1 1 0 0 1 1 0 2 2 2 1 1 2 4 1 3 2 3 3 2 2 3 0 1 7 0 1 6 4 8 1 4 1 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
2 2 4 5 1 0 3 3 1 4 3 3 4 1 0 0 3 0 0 3 2 1 3 1 3 3 3 1 1 1 2 1 1 3 0 1 2 2 1 2 1 1 0 1 2 2 0 1 2 1 2 1 1 0 2 1 0 1 0 0 0 1 1 0 0 1 1 1 0 1 1 1 1 1 1 0 0 0 1 0 0 0 1 1 2 3 0 1 1 0 1 3 0 1 1 1 1 0 1 1 0 1 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 2 0 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 1 1 0 0 1 1 1 1 0 1 1 1 1 1 1 0 1 2 0 0 1 0 1 3 2 1 2 2 2 1 1 1 5 6 6 1 5 4 1 1 1 8 9 1 9 4 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
3 5 5 5 4 3 0 3 1 2 2 4 4 3 3 3 2 0 3 0 3 1 2 2 3 2 2 2 2 1 0 0 1 1 3 1 1 0 1 1 0 0 2 0 0 0 1 1 2 2 1 2 1 1 1 0 1 1 1 0 0 1 4 0 2 1 0 2 1 0 1 0 0 1 1 1 0 0 0 1 1 2 0 1 1 1 1 1 1 2 1 0 1 0 1 0 1 2 0 0 1 1 0 0 0 1 0 0 1 1 0 0 0 1 1 0 0 1 2 0 2 1 0 3 2 2 0 1 1 1 1 2 0 0 0 1 1 0 2 1 0 1 1 1 2 1 1 0 1 1 2 0 1 0 0 0 1 0 1 1 0 0 1 0 1 2 1 2 3 2 2 2 1 2 8 2 3 1 0 5 7 1 5 0 2 9 1 0 3 5 3 1 9 9 9 9 9 9 9 9
6 6 6 6 5 1 0 3 3 2 2 6 0 6 1 0 0 6 3 5 2 2 0 1 0 1 1 1 0 2 4 2 1 0 1 4 0 4 1 0 2 1 1 0 0 1 1 1 0 2 1 0 1 0 1 1 0 1 1 0 1 1 0 0 1 1 1 0 1 1 1 1 0 0 1 3 0 1 0 1 1 1 1 1 0 0 1 1 2 0 1 1 0 1 1 1 1 1 2 0 2 1 0 1 1 0 1 0 1 1 0 1 0 0 1 1 0 1 1 0 1 0 1 1 1 1 0 1 0 1 1 4 0 1 1 2 0 0 0 1 1 0 0 0 1 1 1 1 1 1 1 1 0 2 1 1 3 0 1 2 0 0 0 1 0 1 0 0 0 0 1 0 0 2 0 1 6 3 7 0 0 1 1 7 1 0 8 6 9 8 9 9 8 3 0 3 1 5 6 9
6 6 6 6 1 6 6 5 1 4 6 0 6 3 5 2 6 1 1 2 2 2 3 2 1 1 2 1 1 0 1 0 0 3 2 1 1 1 2 0 1 2 0 2 2 0 1 0 1 0 0 1 1 0 2 0 2 0 0 0 1 0 0 0 1 1 2 0 1 1 1 1 0 1 1 1 1 0 0 0 1 0 0 2 0 0 0 0 0 1 0 3 1 0 0 1 1 0 0 2 1 0 0 0 0 0 2 1 0 1 0 0 1 1 3 0 1 1 1 1 0 1 0 0 0 1 0 1 1 1 0 2 0 1 1 1 1 1 1 0 1 0 1 0 0 2 0 1 0 1 1 1 0 0 1 1 0 1 1 0 2 1 1 0 1 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 0 0 1 2 0 1 1 1 0 0 5 0 8 4 4 5 3 3 3 0
7 7 7 7 7 7 7 7 5 0 3 7 4 4 2 1 6 7 7 4 5 1 6 4 5 6 6 2 1 2 0 3 1 0 0 0 0 3 3 2 0 1 0 2 1 0 2 1 1 0 2 1 1 0 0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 0 1 1 0 1 0 0 0 1 1 1 1 1 0 1 1 0 1 1 1 1 0 1 1 1 2 0 1 0 0 0 0 0 1 1 0 1 1 0 2 1 1 3 2 0 1 1 0 2 0 0 0 2 1 0 1 0 2 0 1 1 0 1 0 0 3 1 1 0 1 0 0 0 0 0 1 0 1 1 1 0 0 0 1 0 1 0 0 0 0 1 2 0 0 0 0 1 2 0 1 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 2 2 2 2 1 4
7 7 7 7 7 7 7 7 7 7 7 4 7 7 7 2 4 4 0 1 2 6 7 2 7 6 7 5 3 6 7 2 2 1 4 2 1 2 0 1 0 0 0 2 1 2 1 2 2 1 1 1 0 0 1 0 0 1 1 2 0 1 1 0 2 1 1 1 1 0 1 1 1 0 1 0 1 0 1 0 1 1 0 2 1 1 0 1 2 2 1 0 1 1 1 2 0 1 0 1 0 1 2 1 0 1 1 1 0 0 0 1 2 1 0 2 0 0 1 3 1 1 1 1 1 2 0 0 0 0 1 1 1 0 0 0 2 0 1 0 1 2 1 0 1 1 2 0 1 1 2 2 0 0 1 0 1 1 0 2 0 0 1 0 2 1 1 2 1 0 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 0 0
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 0 0 1 2 1 4 2 7 8 6 2 7 4 3 5 3 3 0 3 1 0 3 1 0 1 0 1 1 1 0 1 1 0 0 1 1 0 1 1 1 0 0 1 0 0 1 0 0 0 1 1 0 1 2 1 0 0 0 0 1 1 0 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 1 2 0 1 2 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 0 0 0 1 1 0 2 1 1 0 1 0 0 1 0 1 0 1 1 0 0 0 0 0 1 0 0 0 1 0 2 1 0 1 0 1 2 0 1 1 1 1 1 0 0 1 1 1 0 0 0 1 1 1 1 1 0 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 2 8 2 4 4 6 2 8 1 1 2 3 1 2 2 2 3 1 0 1 1 1 1 0 2 0 0 1 1 1 1 1 0 1 1 0 1 0 0 1 2 1 0 0 1 3 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 0 0 0 1 1 0 1 1 1 0 0 0 0 0 1 1 1 0 1 1 0 0 0 1 1 0 1 1 3 0 0 0 1 1 2 1 1 0 0 0 1 1 2 0 0 0 2 1 2 2 0 1 1 1 0 2 1 0 0 1 0 0 0 0 1 0 1 0 0 0 1 1 0 0 0 0 1 0 1 0 1 1 0 0 1 0 0 0 1 0 0 0 1 1 0 1 2 0 0 1 0 1 0 1 0 0 0 0 1 1 1 0 0 0 1 0 0 0 0
BookingsEmpty Full
Timetodeparture
QL Fares vs. Optimal Fares
Toy scenario
Fenceless Fare Structure
10 seats
3 price points

9. 9
Deep Reinforcement Learning
Q-Learning
each observation
Deep Q-Learning
Deep Neural Network
(3; 100, 100, 100; 1)
𝑥𝑓
𝑡
each observation
𝑄 𝑥, 𝑡; 𝑓 ← 𝛼 𝑓 + 𝑀𝑎𝑥 𝑓′ 𝑄 𝑥′, 𝑡 + 1; 𝑓′ + 1 − 𝛼 𝑄 𝑥, 𝑡; 𝑓
𝐿 𝜃 = 𝑓 + 𝑀𝑎𝑥𝑓′ 𝑄 𝑥′, 𝑡 + 1; 𝑓′; 𝜃 − 𝑄 𝑥, 𝑡; 𝑓; 𝜃
2
𝑥
𝑡
𝑓
𝑄(𝑥, 𝑡; 𝑓; 𝜃)
𝛼

10. 10
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Deep QL
5-13Y
of
flight data
~
99%
of
opt. rev
Normal QL
165kY
of
flight data
~
97%
of
opt. rev
Compare performance of QL vs. DQL in Monopoly
RMS optimal=100%
QL ~ 97.0%
80%
90%
100%
110%
20 30 40 50 60
Revenue
Departures (millions)
80%
90%
100%
110%
1000 2000 3000 4000 5000
Revenue
Departures
RMS optimal=100%
DQL ~ 99.3%
Scenario
Fenceless Fare Structure
200 seats
10 price points
12.000 times faster

11. Strictly
CONFIDENTIAL
Competition
©AmadeusITGroupanditsaffiliatesandsubsidiaries

12. ©AmadeusITGroupanditsaffiliatesandsubsidiaries
Simulation set-up - network
AL1
AAA BBB
AL2
AAA BBB
• Fenceless fare structure
• 10 price points
• CAP=50
• High demand (Volume/Cap=1.5)
• Two customer segments: Business and Leisure pax (mix=50/50)
• Input frat5 curve
• AL1: RMS(DP), DQL
• AL2: RMS(DP), DQL

13. 13
©AmadeusITGroupanditsaffiliatesandsubsidiaries
DQL calibration
Initialization with historical data
1000 departure dates - DQL300 departure dates - RMS
Poor Quality!
Warm-up Benchmarking
DQL Initialization

14. ©AmadeusITGroupanditsaffiliatesandsubsidiaries
DQL calibration
Initialization with RMS Bid Prices
Supervised
Learning
1000 departure dates - DQL
RMS 𝑩𝑷(𝒙, 𝒕)
table
𝑸(𝒙, 𝒕; 𝒇)
table
DQL Policy
˜ RMS Policy
Benchmarking
300 departure dates - RMS
Warm-up
DQL Initialization

15. Performance of RMS vs. RMS
99,8
100,1
Revenue index
DCP
Class
Booking Class Mix
Price evolution
DCP
Booking evolution
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10
RMS RMS
0
10
20
30
40
50
0 5 10 15 20
RMS RMS
50
100
150
200
0 5 10 15 20
RMS RMS
70
80
90
100
110
120
0 200 400 600 800 1000
RMS RMS Depdates
PaxPax

16. 0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
DQL RMS
Performance of DQL
84.32
105.36
Revenue index
Class
Booking Class Mix
Depdates
DQL vs. RMS
Pax
70
80
90
100
110
120
0 200 400 600 800 1000
DQL RMS
Pax
DQL vs. DQL
Booking Class Mix
Revenue index
Class
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10
DQL DQL
92.18
Depdates
91.18
70
80
90
100
110
120
0 200 400 600 800 1000
DQL DQL

17. Performance of RMS vs. RMS; DQL vs. RMS; DQL vs. DQL
Revenue wrt baseline
Booking Class Mix
Price evolution
5,36%
-8,32%
-15%
-10%
-5%
0%
5%
10%
DQL vs. RMS DQL vs. DQL
50
100
150
200
0 5 10 15 20
RMS vs RMS DQL vs. RMS DQL vs. DQL
0
5
10
15
1 2 3 4 5 6 7 8 9 10
RMS vs RMS
0
5
10
15
1 2 3 4 5 6 7 8 9 10
DQL vs. RMS
0
5
10
15
1 2 3 4 5 6 7 8 9 10
DQL vs. DQL
RMS vs. RMS
DCP
Class
Pax

18. 18
Conclusion
− Reinforcement Learning (RL) opens the door to a radical new approach
• Model free - no forecasting and no optimization
• Leans by direct price testing
− Shown that QL and DQL converges to RMS optimal solution in monopoly
− Why does DQL vs. DQL underperform?
• Hypothesis: Both ALs agree that WTP is increasing in traditional RMS?
− Next steps
• Apply to networks
• Add more information to the state (e.g., competitors and market)
*) Nash equilibrium
RMS DQL
RMS 100; 100 84.2; 105.4
DQL 105.4; 84.2 91,7*; 91.7*