Approximate Dynamic Programming Jong Min Lee Chemical and Materials Engineering University of Alberta A New Paradigm for Process Control & Optimization

How does a process industry run? Feedstock Purchase Plant / Unit Operation Inventory Control Supply Chain Management

What decisions do we make in process industries? Regulatory Control Real Time Optimizer Production Planning Strategic Planning Customer Plant Scheduling Advaced Process Control $ $ $ $ sec min ~ day week ~ month month ~ year

Ethylene Plant Furnaces Primary Fractionator Quench Tower Charge Gas Compressor Chilling Demethanizer Deethanizer Ethylene Fractionator Debutanizer Propylene Fractionator Depropanizer Fuel Oil Hydrogen Methane Ethylene Ethane Propylene Propane B - B Gasoline Light H-C Naphtha Feedstock

Regulatory Control LC LC FC FC Feed Keep flow rates, levels, .. @ specified values Decisions: Valve opening [sec] Uncertainties: Valve dynamics, resolutions

Scheduling and Planning Demands Inventories Ethylene Plant Feedstock Market Blending Daily ~ Monthly Maximize CSL and Profit Decisions: Purchase / Blending / Unit Maintenance / Inventories / Distributions Uncertainties: Market Prices / Raw Mat. Properties / Unit Failures / Demands… ? ? ? ? ETY PPY ETA BBP GSL

All the decision-making problems are fundamentally SAME We are concerned with future performance Future Time Profit

Conventional Tools Observer Decision Feedforward New Information Real outcome Optimizer Model Constraints Objective Function max  t = k+ 1 k+p performance Real World Future Past k k+ 1 k+p time

What are the issues of conventional tools? 1. They ignore UNCERTAINTIES. - Can yield wrong decisions 2. They put too much efforts ONLINE. - Can be late for timely decision

Analogy to Chess Me Opponent (Plant) Model Predictive Control Mixed Integer Programming h g f e d c b a 1 2 3 4 5 6 7 8 Opponent’s Move New Piece Position Exponential Explosion

Model Predictive Control

Mixed Integer Programming

Unbeatable Chess Player – Dynamic Programming Score (Value) for every feasible position Pick up the action giving the best “score” (position: mine & the opponent’s) Already calculated (offline) before we start a game h g f e d c b a 1 2 3 4 5 6 7 8 Expected Optimal Value Set of Next Piece Positions Decision u1 x1 45 u2 x2 55

How do we find the “scores”? Discretization of entire state & action space INFEASIBLE = J  ( x ) min u  ( x , u )  J  ( x ’ ) + E x 1 x 2 x 3 u 1 u 2 u 3

Can we find the scores “approximately”? Converged Value Fcn On-line Implementation Simulations w/ initial policies Value Function Approximation Iterative Improvement Off-line

Advantages of Approximate Dynamic Programming Manageable online computation Applicable to practical systems Stochastic systems as well as deterministic system All about simulation! Improved policy

Manageable online computation

Applicable to practical systems

Stochastic systems as well as deterministic system

All about simulation!

Improved policy

Key to Success of ADP Store – Search – Averaging e.g.) nearest neighbor Convergence of Off-line Learning

Resource-Constrained Project Scheduling J. Choi, et al. Computers and Chemical Engineering , 28 (2004)

Drug Discovery / Development Discovery Development Market Drug 1 Drug 2 Drug n Phase 1 Phase 2 a/b Phase 3 Submission & Approval 0.5 – 2 yrs 1 – 2 yrs 1.5 – 3.5 yrs 2.5 – 4 yrs 0.5 – 2 yrs $2-4 MM $1-3 MM $5-25 MM $50-250 MM $5-20 MM Pre-clinical Development R&D takes 6.5 – 13.5 years 60 – 300 million $

Problem Complexity I 1 I 2 P 1 I 3 I 4 P 2 I 5 I 6 I 7 P 3 I 8 I 9 I 10 P 4 I 11 I 12 P 5 Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Success/Failure, Duration, Cost 1.2 x 10 9 scenarios 5 3 6 6 5 3 7 4 5 4 6 3 3 8 4 3 5

Simulations X = [s 1 , s 2 , s 3 , s 4 , s 5 , z 1 , z 2 , z 3 , z 4 , z 5 , L 1 , L 2 , t] Which task is performed? Result of the most recent task Duration 230 billion points Simulations (150000) 1. High Success Probability Task First 2. Short Duration Task First 3. High Reward Project First Sampled X 3.7 x 10 5 probabilistic description

ADP improved on the starting policies 10000 realizations 0 4000 8000 12000 H1 H2 H3 ADP

Stochastic Optimal Control

If you ignore uncertainties… y(k+1) = y(k) + b u(k) + e(k+1) parameter change noise enters

ADP “actively” handles uncertainties Output & Input Parameter Estimate & Variance Active probing at t=t b ( 10 ) : t e =15 Decrease of parameter uncertainty t=10: parameter changes, t=15: exogenous noise enters

Summary ADP is a computationally feasible approach to large-scale and uncertain systems and provides an improved solution “ ”