This document discusses stochastic dynamic programming and its applications. It covers Bellman's principle of optimality, solving stochastic dynamic programming problems using value function iteration, and applying these concepts to agroforestry and livestock herd dynamics models. It also discusses estimating intertemporal preferences using dynamic models that relax the assumption of time-additive separability and allow for risk aversion. Examples are provided of solving a resource management problem numerically using value iteration over continuous state and control variables.
International Conference on Monte Carlo techniques
Closing conference of thematic cycle
Paris July 5-8th 2016
Campus les cordeliers
Chris Sherlock's slides
International Conference on Monte Carlo techniques
Closing conference of thematic cycle
Paris July 5-8th 2016
Campus les cordeliers
Chris Sherlock's slides
Wigner Quasi-probability Distribution of the Cosecant-squared Potential WellJohn Ray Martinez
This is a thesis under Theoretical Physics in journal article format. The paper was accepted for oral presentation and published in SPVM (Samahang Pisika ng Visayas at Mindanao) publication. This was presented during 10th SPVM National Physics Conference and Workshop and won the Best Paper Presentor award.
A brief discussion of cyclostationary processes.
Prof. H.Amindavar complementary notes for the first session of "Advanced communications theory" course, Spring 2021
Stochastic Approximation and Simulated AnnealingSSA KPI
AACIMP 2010 Summer School lecture by Leonidas Sakalauskas. "Applied Mathematics" stream. "Stochastic Programming and Applications" course. Part 8.
More info at http://summerschool.ssa.org.ua
A brief discussion of Multivariate Gaussin, Rayleigh & Rician distributions
Prof. H.Amindavar complementary notes for the first session of "Advanced communications theory" course, Spring 2021
We present a class of continuous, bounded, finite-time stabilizing controllers for the tranlational and double integrator based on Bhat and Bernstein's work of IEEE Transactions on Automatic Control, Vol. 43, No. 5, May 1998
Wigner Quasi-probability Distribution of the Cosecant-squared Potential WellJohn Ray Martinez
This is a thesis under Theoretical Physics in journal article format. The paper was accepted for oral presentation and published in SPVM (Samahang Pisika ng Visayas at Mindanao) publication. This was presented during 10th SPVM National Physics Conference and Workshop and won the Best Paper Presentor award.
A brief discussion of cyclostationary processes.
Prof. H.Amindavar complementary notes for the first session of "Advanced communications theory" course, Spring 2021
Stochastic Approximation and Simulated AnnealingSSA KPI
AACIMP 2010 Summer School lecture by Leonidas Sakalauskas. "Applied Mathematics" stream. "Stochastic Programming and Applications" course. Part 8.
More info at http://summerschool.ssa.org.ua
A brief discussion of Multivariate Gaussin, Rayleigh & Rician distributions
Prof. H.Amindavar complementary notes for the first session of "Advanced communications theory" course, Spring 2021
We present a class of continuous, bounded, finite-time stabilizing controllers for the tranlational and double integrator based on Bhat and Bernstein's work of IEEE Transactions on Automatic Control, Vol. 43, No. 5, May 1998
Kazushi Okamoto: Families of Triangular Norm Based Kernel Function and Its Application to Kernel k-means, Joint 8th International Conference on Soft Computing and Intelligent Systems and 17th International Symposium on Advanced Intelligent Systems (SCIS-ISIS2016), 2016.08.25
Mx/G(a,b)/1 With Modified Vacation, Variant Arrival Rate With Restricted Admi...IJRES Journal
In this paper, a bulk arrival general bulk service queuing system with modified M-vacation policy, variant arrival rate under a restricted admissibility policy of arriving batches and close down time is considered. During the server is in non- vacation, the arrivals are admitted with probability with ' α ' whereas, with probability 'β' they are admitted when the server is in vacation. The server starts the service only if at least ‘a’ customers are waiting in the queue, and renders the service according to the general bulk service rule with minimum of ‘a’ customers and maximum of ‘b’ customers. At the completion of service, if the number of waiting customers in the queue is less than ‘𝑎’ then the server performs closedown work , then the server will avail of multiple vacations till the queue length reaches a consecutively avail of M number of vacations, After completing the Mth vacation, if the queue length is still less than a then the server remains idle till it reaches a. The server starts the service only if the queue length b ≥ a. It is considered that the variant arrival rate dependent on the state of the server.
MVPA with SpaceNet: sparse structured priorsElvis DOHMATOB
The GraphNet (aka S-Lasso), as well as other “sparsity + structure” priors like TV (Total-Variation), TV-L1, etc., are not easily applicable to brain data because of technical problems
relating to the selection of the regularization parameters. Also, in
their own right, such models lead to challenging high-dimensional optimization problems. In this manuscript, we present some heuristics for speeding up the overall optimization process: (a) Early-stopping, whereby one halts the optimization process when the test score (performance on leftout data) for the internal cross-validation for model-selection stops improving, and (b) univariate feature-screening, whereby irrelevant (non-predictive) voxels are detected and eliminated before the optimization problem is entered, thus reducing the size of the problem. Empirical results with GraphNet on real MRI (Magnetic Resonance Imaging) datasets indicate that these heuristics are a win-win strategy, as they add speed without sacrificing the quality of the predictions. We expect the proposed heuristics to work on other models like TV-L1, etc.
An Exponential Observer Design for a Class of Chaotic Systems with Exponentia...ijtsrd
In this paper, a class of generalized chaotic systems with exponential nonlinearity is studied and the state observation problem of such systems is explored. Using differential inequality with time domain analysis, a practical state observer for such generalized chaotic systems is constructed to ensure the global exponential stability of the resulting error system. Besides, the guaranteed exponential decay rate can be correctly estimated. Finally, several numerical simulations are given to demonstrate the validity, effectiveness, and correctness of the obtained result. Yeong-Jeu Sun "An Exponential Observer Design for a Class of Chaotic Systems with Exponential Nonlinearity" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-5 | Issue-1 , December 2020, URL: https://www.ijtsrd.com/papers/ijtsrd38233.pdf Paper URL : https://www.ijtsrd.com/engineering/electrical-engineering/38233/an-exponential-observer-design-for-a-class-of-chaotic-systems-with-exponential-nonlinearity/yeongjeu-sun
Chaos Suppression and Stabilization of Generalized Liu Chaotic Control Systemijtsrd
In this paper, the concept of generalized stabilization for nonlinear systems is introduced and the stabilization of the generalized Liu chaotic control system is explored. Based on the time-domain approach with differential inequalities, a suitable control is presented such that the generalized stabilization for a class of Liu chaotic system can be achieved. Meanwhile, not only the guaranteed exponential convergence rate can be arbitrarily pre-specified but also the critical time can be correctly estimated. Finally, some numerical simulations are given to demonstrate the feasibility and effectiveness of the obtained results. Yeong-Jeu Sun | Jer-Guang Hsieh "Chaos Suppression and Stabilization of Generalized Liu Chaotic Control System" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-1 , December 2018, URL: http://www.ijtsrd.com/papers/ijtsrd20195.pdf
http://www.ijtsrd.com/engineering/electrical-engineering/20195/chaos-suppression-and-stabilization-of-generalized-liu-chaotic-control-system/yeong-jeu-sun
Avionics 738 Adaptive Filtering at Air University PAC Campus by Dr. Bilal A. Siddiqui in Spring 2018. This lecture covers background material for the course.
In this work, we propose to apply trust region optimization to deep reinforcement
learning using a recently proposed Kronecker-factored approximation to
the curvature. We extend the framework of natural policy gradient and propose
to optimize both the actor and the critic using Kronecker-factored approximate
curvature (K-FAC) with trust region; hence we call our method Actor Critic using
Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this
is the first scalable trust region natural gradient method for actor-critic methods.
It is also a method that learns non-trivial tasks in continuous control as well as
discrete control policies directly from raw pixel inputs. We tested our approach
across discrete domains in Atari games as well as continuous domains in the MuJoCo
environment. With the proposed methods, we are able to achieve higher
rewards and a 2- to 3-fold improvement in sample efficiency on average, compared
to previous state-of-the-art on-policy actor-critic methods. Code is available at
https://github.com/openai/baselines.
On the principle of optimality for linear stochastic dynamic systemijfcstjournal
In this work, processes represented by linear stochastic dynamic system are investigated and by
considering optimal control problem, principle of optimality is proven. Also, for existence of optimal
control and corresponding optimal trajectory, proofs of theorems of necessity and sufficiency condition are
attained.
Importance sampling has been widely used to improve the efficiency of deterministic computer simulations where the simulation output is uniquely determined, given a fixed input. To represent complex system behavior more realistically, however, stochastic computer models are gaining popularity. Unlike deterministic computer simulations, stochastic simulations produce different outputs even at the same input. This extra degree of stochasticity presents a challenge for reliability assessment in engineering system designs. Our study tackles this challenge by providing a computationally efficient method to estimate a system's reliability. Specifically, we derive the optimal importance sampling density and allocation procedure that minimize the variance of a reliability estimator. The application of our method to a computationally intensive, aeroelastic wind turbine simulator demonstrates the benefits of the proposed approaches.
Low Emissions Development Strategies (Colombia Feb 20, 2014)IFPRI-EPTD
FROM GLOBAL TO LOCAL:MODELING LOW EMISSIONS DEVELOPMENT STRATEGIES IN COLOMBIA
Globally, agriculture is responsible for 10 – 14% of GHG emissions and largest source of no-CO2 GHG emissions. Countries can choose among technologies with different emission characteristics and we believe it's less costly to avoid high-emissions lock-in than replace them, so EFFORT TO ENCOURAGE LEDS is key.
Low Emissions Development Strategies (LEDS) Training Sept 9, 2013IFPRI-EPTD
Globally, agriculture is responsible for 10 – 14% of GHG emissions and largest source of no-CO2 GHG emissions. Countries can choose among a portfolio of growth-inducing technologies with different emission characteristics. We believe that is less costly to avoid high-emissions lock-in than replace high-emissions technologies. There's a need to encourage Low Emission Development Strategies.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdfTechSoup
In this webinar you will learn how your organization can access TechSoup's wide variety of product discount and donation programs. From hardware to software, we'll give you a tour of the tools available to help your nonprofit with productivity, collaboration, financial management, donor tracking, security, and more.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
The Art Pastor's Guide to Sabbath | Steve ThomasonSteve Thomason
What is the purpose of the Sabbath Law in the Torah. It is interesting to compare how the context of the law shifts from Exodus to Deuteronomy. Who gets to rest, and why?
We all have good and bad thoughts from time to time and situation to situation. We are bombarded daily with spiraling thoughts(both negative and positive) creating all-consuming feel , making us difficult to manage with associated suffering. Good thoughts are like our Mob Signal (Positive thought) amidst noise(negative thought) in the atmosphere. Negative thoughts like noise outweigh positive thoughts. These thoughts often create unwanted confusion, trouble, stress and frustration in our mind as well as chaos in our physical world. Negative thoughts are also known as “distorted thinking”.
Ethnobotany and Ethnopharmacology:
Ethnobotany in herbal drug evaluation,
Impact of Ethnobotany in traditional medicine,
New development in herbals,
Bio-prospecting tools for drug discovery,
Role of Ethnopharmacology in drug evaluation,
Reverse Pharmacology.
How to Make a Field invisible in Odoo 17Celine George
It is possible to hide or invisible some fields in odoo. Commonly using “invisible” attribute in the field definition to invisible the fields. This slide will show how to make a field invisible in odoo 17.
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
2. Understand Bellman’s Principle of Optimality
and the basic Stochastic Dynamic
programming problem
Solve the SDP with value function iteration
Apply the concepts of models to agro-
forestry and livestock herd dynamics
Make changes to the SDP and simulate the
corresponding change in optimal solution
Day 4 NotesHowitt and Msangi 2
3. Re-cap on rangeland stocking model….
Introduction to Stochastic Dynamic Programming
◦ Extend DP framework to include stochastic state variables
and apply to herd and agro forestry management
Stochastic Cake Eating
Multi-State Models
◦ Function Approximation
Agro-Forestry Application
◦ Input Data and State Space
◦ Simulation
Herd Dynamics Application
◦ Input Data
◦ Simulation
Day 4 NotesHowitt and Msangi 3
4. An Application to Reservoir
Management
Day 3 NotesHowitt and Msangi 4
5. “Estimating Intertemporal Preferences for
Resource Allocation” AJAE, 87(4): 969-983.
(Howitt RE, S Msangi, A Reynaud, KC Knapp)
What started out as a calibration exercise –
ended up as a research project (with some
interesting research discoveries)
Day 3 NotesHowitt and Msangi 5
6. Many of the Important Policy Questions in
Natural Resource Management Revolve
Around How to Deal with Uncertainty over
Time (Global Climate Change, Extreme
Weather Events, Invasive Species
Encroachment, Disease Outbreak, etc. )
Policy Makers look to Economic Models to
Provide them with Guidance on Best
Management Practices
7. Economic Policy Models Have Typically
Downplayed the Role of Risk in the
Preferences of the Decision-maker
Few Studies Have Ever Tried to Measure the
Degree to Which Risk Aversion Matters in
Resource Management Problems
Time-Additive Separability in Dynamic
Models Imposes Severe Constraints on
Intertemporal Preferences
8. In order to Address this Gap in the Natural
Resources literature….
We Applied Dynamic Estimation Methods to
an Example of Reservoir Management
We Relaxed the Assumption of Time-Additive
Separability of the Decision-Maker’s Utility
We Tested with Alternative Utility Forms to
Determine the Importance of Risk Aversion
9. Koopmans (1960) laid the foundation for
eliminating deficiencies of TAS with
recursive preferences.
Recursive Utility is a class of functionals
designed to offer a generality to time
preferences while still maintaining time
consistency in behavior.
Allows for the potential smoothing of
consumption by allowing complementarity
between time periods.
10. ( )W
( )1( ) ( ), ( )U W u c U S=c c
States the weak separability of the future from present
where
is an aggregator function
For TAS, the aggregator is simply
( )( ), ( )W u c x u c xβ= +
11. ( ) ( )
1
( ), 1 ( )W u c x u c x
ρρ ρ
β β = − ⋅ + ⋅
1
( )
1
EIS σ
ρ
=
−
So we choose our aggregator to be
and the implied elasticity (“resistance”) to inter-temporal substitution
is given by
where ( ),0 (0,1]ρ ∈ −∞ ∪
12. Time Additive Separable Utility
Using Bellman’s recursive relationship:
{ }
{ }
{ }
{ }
1
2
1 2
1
1 1 2
2 2 3
2 3
1 2 3
, ,
1
1, 2
( ) max ( ) ( )
( ) max ( ) ( )
( ) max ( ) ( )
:
( ) max ( ) ( ) ( ) ( )
( )
( )
t
t
t
t t t
t t t
c
t t t
c
t t t
c
t t t t t
c c c
t
t t
V x U c V x
V x U c V x
V x U c V x
Substituting and simplifying
V x U c U c U c V x
u c
Note that MRS c
β
β
β
β β β
β
+
+
+ +
+
+ + +
+ + +
+ + +
+
+ +
+
+
+
= + + +
′
=
2( )tu c +
′
13. Iso-Elastic Recursive Utility
A utility function with a CES across time periods.
1
2
1
1
1
1
1 1 2
1
2 2 3
( 1) 1
,
( ) max (1 ) ( ) ( )
( ) max (1 ) ( ) ( )
( ) max (1 ) ( ) ( )
:
( ) max (1 ) ( ) (1 ) (
t
t
t
t t
t t t
c
t t t
c
t t t
c
t At t t t
c c
V x U c V x
V x U c V x
V x U c V x
Substituting and simplifying
V x U c U c
ρ ρ ρ
ρ ρ ρ
ρ ρ ρ
ρ
β β
β β
β β
β β β
+
+
+
+
+ + +
+ + +
+ +
= − +
= − +
= − +
= − + −
1
1 2
1 1
2
2
1
2 3
( 2) 1 2 3
, ,
1 2
1 1 2
1
2
2
) ( )
( ) max (1 ) ( ) (1 ) ( ) (1 ) ( ) ( )
( )
(1 ) ( ) (1 ) ( ) (1 ) ( ) ( )
( )
(1 )
t t t
t
t At t t t t t
c c c
t
t t t t
t
t
t
V x
V x U c U c U c V x
V x
u c U c U c V x
c
V x
c
ρ
ρ
ρ ρ
ρ ρ ρ ρ ρ
ρ ρ ρ ρ
β
β β β β β β
β β β β β β
β β
+ +
−
+
+ + + +
−
+ + +
+
+
+
= − + − + − +
∂
′ = − − + − + ∂
∂
′= −
∂
1 1
1 2 3
2 1 2 3( ) (1 ) ( ) (1 ) ( ) (1 ) ( ) ( )t t t t tu c U c U c U c V x
ρ
ρ ρ ρ ρ ρ
β β β β β β
−
−
+ + + +
− + − + − +
14. With Recursive Utility All Periods Enter into
MRS
1
1 1
2
1 1 2
1, 2 2 3
2 1 2 3
( ) (1 ) ( ) (1 ) ( ) ( )1
( ) (1 ) ( ) (1 ) ( ) (1 ) ( ) ( )
t t t t
t t
t t t t t
u c U c U c V x
MRS
u c U c U c U c V x
ρ ρ ρ ρ ρ
ρ ρ ρ ρ
β β β β
β β β β β β β
− −
+ + +
+ +
+ + + +
′ − + − +
= ′ − + − + − +
In micro-economics we have an appreciation of the difference
between linear and CES utility in static consumer theory
The same intuition applies here in a dynamic context….
15. The previous equations show that the marginal rate
of substitution across time is path dependent.
Timing is now an explicit economic control variable
We no longer assume that “The marginal rate of
substitution between lunch and dinner is
independent of the amount of breakfast” (Henry
Wan).
A smaller elasticity of intertemporal substitution
flattens out the optimal time path of resource use-
yielding a time consistent sustainable result.
16. Stochastic Equations of Motion link Stocks
and Flows
Randomness in the equations of motion or
exogenous random shocks change the
system evolution
The current state and future distributions
are usually known to decision makers
Management decisions inherently optimize
a stochastic dynamic path of resource use
and consequently maximize dynamic
stochastic utility
17. A Simple Resource Network with a
Single State Variable
te1
~
te2
~
Demand
tS
tw
18. ( ) ( )2 1
1
1Max (1 ) E ( ) Et e t t e t
w
U W q U
ρ ρ
ρ α α
β β +
= − ⋅ +
.
≥
≤
≥
+=
−+=
+
+
+
0
~
~
1
1
2
11
t
t
t
ttt
tttt
w
SS
SS
ewq
weSS
The Optimization
Problem
19. ( )
1
2 2 1 1
0
, Max (1 ) ( )d ( , )dt /
w
V S e W w e Φ V S e Φ
ρ ρ
ρ α α
β β
≥
= − ⋅ + +
∫ ∫
Which can be re-stated in
terms of Bellman’s Recurrence
Relationship…
..and which we solve by numerically with
Continuous-valued State and Control
Variables
20. Solving for the Expected Value Function
Initialize with
Intermediate Value Function W(Xt , ut )
Nodes for: State Evaluation and Stochastic Inflow values
Probabilities of Inflow over k Stochastic node values
Define the State Variable on
[ -1, 1] Interval for each polynomial node j
Value Iteration Loop (n steps)
n = n+1
Error =
If Error > 0.1 e-7
Stop
Value Function Chebychev Polynomial Coefficients
jXaVpuXWPVNB
k i
kj
ti
n
iktt
n
j ∀
+= ∑ ∑ +
−
)(),(max ,
1
1
φβ
( )∑ −
−
i
n
i
n
i aa
21
( )
( ) ( )∑
∑
++
+
=
j
nj
ti
nj
ti
j
nj
ti
n
j
n
i
XX
XPVNB
a ,
1
,
1
,
1
φφ
φ
( )j
x
21.
22.
23. ( ) 32
0067.045.0150 qqqqW ⋅+⋅−⋅=
ttt
ttttt
capee
eecapesp
⋅⋅⋅+
⋅+⋅=
1
3
1
2
111
~0.02305-~0.000993
~0.005024~0.095382),~(
Current Profit Function
Spill Function
24. Net Benefit Function for Water
0
1
2
3
4
5
6
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56
q, (MAF)
W(q),1000M$US
25. We employ a nested procedure to solve the
SDP problem with value iteration, while we
systematically change the parameter values
of the objective function to maximize a
likelihood function.
We employ a derivative-free ( Nelder Meade)
search algorithm to implement the ‘hill-
climbing’ procedure that searches for the
likelihood-maximizing values of preference
parameters
26. EIS value
( )1 1 ρ−
Coeff. of Risk
Aversion
1 α−
ρ
α
These parameters were calculated with a fixed discount rate of β
Parameter
Estimated
Value
Standard Error
-9.000 4.60 0.100
-0.440 0.23 1.440
Log
Likelihood
-10.257
=0.95.
Standard errors are based on 500 bootstrap repetitions
27. 1,set estimateα ρ=
For Risk-Neutral Recursive model (RNR)
For Risk-Neutral (non-Recursive) model (RN)
For Non-Recursive model (with Risk) use CRRA
1set ρ α= =
( )0.95fix β =
( )0.95fix β =
)(
1 1
)1(
+
−
+
−
= t
t
t UE
W
U βα
α
,estimate α β
33. Clearly a non-recursive model that ignores
risk fares the worst, when compared to actual
storage and releases
Adding risk, but not recursivity of
preferences, gets you closer to actual
values…but not quite….
A Recursive Specification outperforms both of
these, with or without risk aversion
34. Estimation of the Fully-Recursive model is
robust to Discount Values and the
Parameter Estimates appear to be Stationary
over the Study Period
Once we allow Intertemporal Preferences to
be recursive, the role of Risk in explaining
Resource Management Behavior is Reduced
Imposing Time-Additive Separability on
Dynamic Models may have more severe
implications for behavior than most
researchers realize…..
36. Extend DP framework to include stochastic
state variables in the model
Apply the new framework to herd dynamics
and agro-forestry management
Return to cake eating example
Day 4 NotesHowitt and Msangi 36
37. Stochastic Cake Eating
◦ What if I want cake today, but not tomorrow?
Cake Eating Example:
CakeEatingDP_ChebyAprox_Stochastic_Day4.gms
Consider a taste shock , so that utility from
cake consumption is now:
◦ Knows the value of stochastic shock today, but
unknown for future periods.
◦ Agent should factor in the potential future shocks
in today’s consumption decision
Day 4 NotesHowitt and Msangi 37
ε
( , )u c ε
38. Step 1: Define nature of stochastic shock
◦ First-order Markov process: probability of future
shocks is described by current period
◦ Two states: , described by and
◦ The transition between states follows a first-order
Markov process, described by matrix :
◦ An element in the matrix yields the probability of
moving from state i to j in the next period:
Day 4 NotesHowitt and Msangi 38
andl h hε lε
Π
ll lh
hl hh
π π
π π
Π =
( )1Pr |ij t j t iπ ε ε ε ε+≡ = =
39. Agent’s choice of how much cake to eat
depends on:
◦ Size of cake
◦ Realization of the taste shock
With current shock knowledge and expected
transition to future periods, the stochastic
cake-eating problem can be written as:
Day 4 NotesHowitt and Msangi 39
( ) ( ){ }1| 1 1 1( , ) max , , , ~t t
t
t t t t t t t t t
c
V x u c E V x x x c Markovε εε ε β ε ε+ + + +=+ =−
40. Markov process for evolution of taste shock
states that today’s preferences yields the
probability of tomorrow’s preferences
◦ This may not hold if we believe that tomorrow does not
depend on the value today
We can specify any type of random variable in the
SDP problem.
◦ Consider specifying the taste shock as a random variable
◦ Define e points, with known probability, , of a shock
with magnitude , we define the probabilities such
that:
◦
Day 4 NotesHowitt and Msangi 40
epr
eshk
1e
e
pr =∑
41. After defining the known probability and shock of
magnitude, we can re-write the stochastic cake-eating
problem as:
Assume the stochastic shock affects utility multiplicatively:
Simple stochastic process where the distribution of e in
future periods is independent of the current period and
independent of other states and the control.
The contraction mapping theorem holds: there exists a
fixed point of the function equation (Bellman)
◦ Solve for this point using same methods for the deterministic DP
Day 4 NotesHowitt and Msangi 41
( ) ( ){ }1| 1 1 1( , ) max , , , ~t t
t
t t t t e e t t t t t
c
V x e u c e E V x e x x c e RVβ + + + +=+ =−
( )1 1 1 1( , ) max ( ) ( ) ( ) ( ) ,
t
t t t t t t t t t t
c
e
V x e shk e u c pr e shk e V x e x x cβ + + + +
=+ =−
∑
42. SDP and DP framework both extend naturally to
models with several state variables.
◦ Will generally involve multiple states that we need to
simultaneously model
For example: Herd stocking (prices, disease, rainfall, herd
size and population dynamics
◦ In general, we can write for any number of states m:
Computational costs of extending the dynamic
framework to many states
◦ As the number of states increases, so does the number
of points we must evaluate and solve the DP.
◦ “Curse of dimensionality”
Day 4 NotesHowitt and Msangi 42
( ) ( ) ( ){ }1
1
1 1( ) max , ,..., ,..., ,t
t
m m m m m
t t t t t t t t
c
V x f c x x V x x x g x cβ ++ += + =
43. Function Approximation
◦ Extend naturally to multi-state applications
Chebychev approximation approach
◦ Extension to m states
Define the state variables upper and lower bounds:
Map to the [-1,1] interval using the same formula:
Transformation back to the interval can be
calculated as:
Day 4 NotesHowitt and Msangi 43
,m m
L U
2 1
ˆ cos , for 1,...,
2j
m j
x j n
n
π
−
= =
,m m
L U
( )( )ˆ
2
j
j
m m m m m
m
x L U U L
x
+ −
=
44. ◦ Given the mapping back to the interval, we can
now define the Chebychev interpolation matrix using the
recursive formula:
◦ Defined the state space and Chebychev nodes and basis
functions for each state variable m.
◦ We can write the Chebychev approximation to the value
function as:
◦ The value function approximation with multiple state
simply extends the Chebychev polynomials to additional
dimensions to approximate the solution over each state.
Day 4 NotesHowitt and Msangi 44
,m m
L U
1
2
1 2
1
ˆ
ˆ2 3
m
m
m
j j j
x
x j
φ
φ
φ φ φ− −
=
=
= − ∀ ≥
1..1
.... m jj jm
m
mj j
V a φ= ∑ ∑ ∏
45. Agro-Forestry Example:
AgroForestryModel_DP_Day4.gms
◦ Varying degree of age, expected yield and
profitability—how do I manage a fixed amount of
land with new plantings and removals?
Input Data and State Space
◦ 20 year time horizon
◦ Early, mature and old trees
◦ 60% of early tree plantings transition to mature
trees and 30% of mature trees transition to old
trees
Day 4 NotesHowitt and Msangi 45
46. The transition between age profiles are as follows:
Model Data
◦ 100 hectares
◦ Cost to uproot is 20/ha
◦ Cost to replant is 100/ha
◦ 5% discount rate
Key Model Parameters
Day 4 NotesHowitt and Msangi 46
Transition Matrix Early Mature Old
Early 0.4 0.6 0
Mature 0 0.7 0.3
Old 0 0 1
Model Data Early Mature Old
Price per kg 10 10 10
Yield (kg/ha) 0 10 5
Initial profile (plantings) 10 5 4
47. Simulation
◦ Three state variables: early, mature, old
◦ Approximate the solution of the infinite horizon
problem by Chebychev approximation of the value
function
Define m=3, and:
Day 4 NotesHowitt and Msangi 47
1 2 3, ,1 2 3
jj j j
m
mj j j
V a φ= ∑∑∑ ∏
48. Herd Dynamics Example:
HerdDynamics_DP_Day4.gms
◦ Varying degree of age and productivity
◦ Three state variables: juvenile, female adult and
male adult
Productive output: milk and meat
Grazing land: fixed amount and known productivity
Minimum number of livestock for breeding purposes
◦ When do we add to the herd, or sell from the herd,
given market conditions and resource constraints?
Day 4 NotesHowitt and Msangi 48
49. Input Data
◦ 40 year time horizon
◦ 5% discount rate
◦ Other key input assumptions:
◦ Females birth rate = 1.5 juveniles per year
30% juveniles, 30% transition to males, and 40% transition to females
◦ Herd can be fed by grazing on a fixed amount of land, or from off-farm
purchased feed
Different nutrient content and ultimately different animal productivity
Day 4 NotesHowitt and Msangi 49
Input Data
Juven
ile
Adult
Male
Adult
Female
Animal weight 40 300 275
Milk yield (kg/yr/animal) 0 0 50
Initial animals 60 20 30
Birth rate per female
(animal/yr) 1.5 0 0
Transition Matrix Juvenile
Adult
Male
Adult
Female
Juvenile 0.3 0.3 0.4
Adult Male 0 1 0
Adult Female 0 0 1
50. Simulation
◦ Over a 100 year time horizon
◦ Approximates the value function at 3 Chebychev
nodes
◦ Agent to maximize present value of profits by
determining optimal rates of:
Animals sold and purchased
Milk sold
◦ Agent may purchase off-farm feed, and responds to
fixed and known market demand and supply for
inputs and outputs
◦ Herd age evolves endogenously by defined
parameters
Day 4 NotesHowitt and Msangi 50