Recommended
Recommended
More Related Content
Similar to Halftime ppt emilie3
Similar to Halftime ppt emilie3 (20)
Halftime ppt emilie3
- 1. 1 Modeling Learning By Doing in Natural Resource Management Half-time seminar by: Emilie Lindkvist Supervisors: Jon Norberg (SRC), Maja Schlüter (SRC), Örjan Ekeberg (KTH) A centre with:
- 2. Outline • Short introduction • The method – computerized learning agent • Experiment setup, results, and conclusions of paper I and II individally • Key findings • Future research
- 3. Introduction • The uncertainties humans face when dealing with natural resources are increasing as a consequence of global environmental change ? ? IPCC (2007), World Bank(2012), Smith et al.(2011), New et al.(2011), Levin (2003)
- 4. The Solution ? • To deal with uncertainty and surprise use Adaptive (co-)management which emphasize Learning by Doing (LBD) • Account for resource users knowledge Allen et al 2011, Armitage et al 2011, Folke (2004), Fazey et al.(2007), Walter&Holling (1990)
- 5. GAP • Increased understanding of – Individual learning process lacking (focus on social learning) – trade-offs in the individual learning process in phase if change • Exploration vs. exploitation • Value future vs. present • Stick to the past or adapt to present • Learn from the past or trust present
- 6. Aim • Understand – the core of an LBD process – impact of different learning parameters • Study how the LBD process responds to different structures of & changes in resource dynamics
- 7. The Method & basic model setup
- 8. The Social-Ecological system Agent = 1 Fisher (computerized learning agent) Resource = 1 Fish stock
- 9. 1 Update Rate of Mental Model 2 Discount Factor 3 Level of hindsight 4 Exploration Level GOAL Performance = net income $ LEARNING PARAMETERS LEARNING PROBLEM ? ?
- 10. The fisher repeatedly needs fish, goes fishing, harvest, learns, and updates his mental model (Fisher) (Fish stock) (Effort) $ Sutton & Barto (1998), Poggio & Girosi (1989)
- 11. Papers I. Lindkvist, E. and Norberg, J. Modeling Experiential Knowledge: Limitations in Learning Non-Linear Dynamics for Sustainable Renewable Resource Management. Submitted to Ecological Economics II. Lindkvist, E. and Norberg, J. Theoretical Aspects on Learning By Doing: Adapting to Effects of Environmental Change. Manuscript
- 12. Research Questions Paper I • Are there limitations of a LBD approach for “sustainably exploiting” a renewable resource? • What are the effects of update rate of mental model, discount factor, hindsight, exploration level, on management outcome?
- 14. The problems the fisher should learn to manage sustainably Logistic Problem • density dependent • slower regrowth at low and high stocks Threshold Problem • density dependent • slower regrowth at low and high stocks • Threshold: hysteresis effect – if a threshold is trespassed it takes about 20 times longer to regrow IF fishing pressure is kept low Paper I C.W. Clark (1976, 2010 )
- 15. Increased Mortality Rate Every fishing event with a 5% chance 5-95% of the fish stock is removed (out of the fishers control) Paper I
- 16. Update Rate : How fast should the fisher adapt his mental model? Low High Inertia Adapt instantaneously Sticking to past perception of the resource dynamics Over-learning (oscillating effect) Paper I (Update rate: parameter in mental model)
- 17. THRESHOLD LOGISTIC Paper I $ A modest update rate of ones mental model is beneficial for sustainable fisheries
- 18. Discount Factor: How much should the fisher values future outcomes? Low High Greedy behavior Takes future fish stocks into account when learning Optimize now Optimize over time – infinite time horizon (Discount factor: parameter in learning method)
- 19. THRESHOLD NO THRESHOLD Paper I High when logistic problem. Lower when a threshold is enforced out of the fishers control $ LOGISTIC
- 20. Hindsight: How should the fisher evaluate previous fishing efforts, and how led up to the current state? Low High Adapt continuously to new experiences Take all past experiences into account when learning from current fishing event Don’t re-evaluate past experiences Re-evaluate all past experiences (Hindsight : parameter in learning method)
- 21. THRESHOLD NO THRESHOLD Paper IHindsight should not be too high for either problem(but increased performance for threshold up to 0.7) $ LOGISTIC
- 22. Exploration level: How much should the fisher try other fishing efforts than what he perceives as optimal? Low High Always does what is best according to his mental model Never does what he thinks is best according to his mental model Never deviates from his current view of the system Always deviates from his current view of the system (Exploration level: parameter in decision-making model)
- 23. THRESHOLD Paper I $ LOGISTIC Low exploration optimal for threshold problem, but higher optimal for logistic problem
- 24. Conclusions • LDB works well but certain implications • Logistic problem – To best manage the logistic problem a high exploration, high valuation of future outcomes, little hindsight, update mental model at 10 to 80% is optimal. • Threshold problem – LDB ill-advised (R. Biggs et al. 2009) – However, ok if in a planned setting others can learn (C. Walters 2007) – The fisher develops a very precise mental model of how the fish stock behaves dependent on fish stock biomass, and develops a more careful behavior Paper I
- 25. Research Questions Paper II 1. How does a LBD approach respond to changes in the growth rate of a renewable resource? 2. Do key learning parameters differ depending on the type change? (e.g. linear vs. abrupt)
- 26. 1 Update Rate of Mental Model 2 Discount Factor 3 Level of hindsight 4 Exploration Level GOAL Performance = net income $
- 28. Paper II (growth rate change)
- 32. Conclusions • LBD generally good • Effects of environmental change – Tricky if abrupt decreasing (like threshold) – Always better than a non-adaptive • Brown et al 2012. How long can fisheries management delay action in response to ecosystem and climate change? • Niiranen et al 2012 implications for (not)modeling uncertainties in growth rates in baltic sea models
- 33. Future Use the agent in basic networks (Motifs, Bodin Tengö 2012) to study impact of agent interactions (information sharing, trust, knowledge)
- 34. Future Add governance layer, to study impact of different forms of leadership (Gutiérrez et al 2011) Governing Agent
- 35. Supervisors, friends, family, listeners! Subscribe to our newsletter www.stockholmresilience.org/subscribe 35 Thank you, A centre with:
- 36. Extra Slides
- 37. 1. Make decision (on effort) Learning by Doing (LBD) 2. Perform action (fishing effort) 3. Harvest 4. Learn (update mental model of system) Update stock (Fisher) (Fish stock)
- 40. Results paper I & II •Significant •High for decreasing growth •Low for •Significant if thresholds •Continuous adaption •Significant •Value future outcomes at 95% •But less if regime shifts •Significant •Change mental model 20% if regime shift •20-80% if change Mental Model update rate Discount Factor Exploration level Hindsight
- 43. Action sensitivity dep. On Growth Rate Paper I & II
- 44. Paper I & II
Editor's Notes
- Thanks for beinghere at my half-timeseminar! So,I’memilieand Istarted as a phd student a fewyearsago. I have a background from computer science and mathematicswhichyouwillnoticethrough my research.I’m going topresent someof my PhD-work on modelinglearning by doing on renewablenaturalresouceswhich I havebeenworking on togetherwith my supervisor Jon Norberg, and alsowith support from Maja and Örjan from KTH who han beenverysupportive on the methodoloissues.
- I will start with an introduction and background to my work behind this half-time, Introduction – adapt to changeAim – understand the LBD processMethod – computerized learning agentPaper I – sustainable exploitation & LBDPaper II – How LDB responds to changeFuture paths – Network of agents & leadershipgo through the model, the two papers, and then results, discusison, conclusions, and touch upon future paths and other work.
- So as to establish some common ground I will just take a minute share the motivation behind this work. The uncertainties humans face when dealing with natural resources are increasing as a consequence of global environmental change.This poses additional challenges to management on how we can adapt efficientlyREMOVED:Ecosystems are inherently unpredictable and evermore so in phase of global warming which seems to increase the frequency of extreme weather events and have unpredictable effects on different species traits.To cope with these changing dynamics, improved management is necessary. Ecosystems are complex adaptive systems and thus will always be subject to uncertainty, unknown events and unknowable phenomena. Particularly now when the impact of human activity is increasing the frequency of these changes through increased air and seawater temperature, and extreme weather events.The uncertainties we face when dealing with natural resources are increasing as a consequence of global environmental change, as a championed solution comes adaptive management which incorporates learning by doing, in order to capture learning and knowledge generation processes, both crucial for sustainable resource use in presence of uncertainty and sudden fluctuation. And also we can rarely fully understand the complex dynamics going on in ecosystems, thus continous exploration will help finding best solutions.Commonly social learning is the focus of environmental scholars, However, in these papers we seek to understand the underlying components at an individual level, as a prerequisite for understanding the broader context of social learning and the need to treat social-ecological systems as complex adaptive systems. Exploiting a renewable natural resource sustainably has two fundamental constrains. First, the difficulty of agreeing on appropriate actions common pool dilemmas, secondly, the limitations in that we can never fully understanding the dynamics of a resource system. And in these papers I willfocus on the latter. In future common pool dilemmas.
- Adaptive management has been championed as an advantageous approach for management of natural resources.
- The main aims with this work has been to understand the LBD process and its implications when trying to manage different types of feedback structures within renewable resource. And while many results might not be extremely surprising we can actually parameterize some learning features and show their influence dependent on the resource feedbacks.Parameters chosen in accordance with REMOVED:Update rate of mental model Discounting (valuing future outcomes)Hindsight (learning from past experiences)Exploration vs. exploitation
- How do I tend to investigate these issues?This is a theoretical model – but to make it comrehenciple I will use this fisher and his fish-stock as an example through out the presentation. Formy social ecologial system I use one agent and one resource. It is an agent based model although with one agent, a fisher, trying to learn how to best manage a fish stock . The agent is built on artificial intelligence, and the resource a classic population growth model commonly used within modeling renewable resources, and for example our colleagues at the Beijer Institute use the same fish stock model in their lab experiments.REMOVED:I nthe two papers then, we play around a bit with the dynamics in the resource to se how the agent and thus the LBD approach is able to manage the different changes in feedbacks.
- Although this is a strictly theoretical experiment, this is a way to try to convey the simulation setup. We have the fisher and the fish stock.The goal is to get as mush fish over time as possible. The learning problem is to manage the fish stock sustainably. And how the fisher does this is by learning from his fishing experiences.-Update rate of Mental Model – the fisher puts out 5 nets every for 1 year and get 10kg of fish each time, then one Tuesday he only get 5 kgs. What is true? His old way still works or should he adapt his mental model and expect less-Discount factor: value future outcomes. Take everything to day or save some to earn more in the future. Keep some so they can reproduce.-Level of hindsight: also contributes to updating the fishers mental model. But it goes back in time to reevaluate past experiences.How did my previous fishing efforts lead up to the state of the stock today? When last week I put out one net I got little catch but today I actually got a lot! Is it due to that?-Exploration level: usually I put out 5 nets, but today I will try something else, 6 nets or 4 commonly. But randomly bold and put out 10 nets.Although you see one fisher here, this could also correspond to a strong leader, or a fully cooperative fishing unit with in a common pool resource system. (a group of agreeing fishers, or top down management)Because these parameters are valid in management dilemmas of renewable resources.
- Conceptual model:AGENT: has to have a goal – maximize harvest over time, learn, store experiences and build up a mental model, make decision based on experiences, take action. From Montague in Nature: RL has opened up new quantitative fields, ripe for attack by young scholarsResource: In the two papers I use different versions of this function and change it over time to see the effects on the LDB approach.The core of the work has been to understand the model, how the learning method, mental model and decision model work together.The learning method: I use a state-of-the-art methods called reinforcement learning inspired by psychology and used within artificial intelligence to study various forms of learning within a multitude of disciplines. To store knowledge I use an artificial neural network (I wont go into details on this one), and for the decision model a method called softmax.
- Just to let you digest what the work is about, these are the papers that I aim to discuss, first Paper 1, In this paper I introduce the model and apply it to two different archetypical population growth functions, both having logistic growth but where one resource has a threshold or tipping point in it. And if you cross this threshold the if it were a fish stock the stock would crash and take many years to rebuild. And how does a learning by doing (trial and error) approach work then?Second paper: I take the same one-agent-one resource model but now I introduce temporal change into the growth rate of the fish stock, to mimic effects of environmental change.In paper 1: We test the agent on a system with out a regime shift and one with a regime shift to see how the LBD responds.Paper 2: Effects on for example marine systems during climate change are concerning, and growth rate change is a fact that we need to deal with… (less known)
- To operationalize this, we probe the following re- search questions: Sustainably exploiting is just to emphasize here to day that it is about harvesting sustainably over time but also to gain as much out of the resource as possible as to not have any harvest go to waist.In this paper we studied an artificially intelligent agents ability to learn sustainable management of a renewable resource (with or without threshold dynamics). Background: regime-shift are increasing and could have very negative impacts on human well being thus need more research. (well known)
- In both problems we add increased mortailty rate ie take away some fish
- For the last ten years the fisher has put out 5 nets and gotten 5kgs of fish, but thus Tuesday he only got 2 kg. How should he change his mental model now of the system? Was it factors like weather, or has the system actually changed so that he must adapt his behavior?
- For simplicity we have fixed three of the learning parameters and varied one.EXPLAINAXES:Go for what you have learned previously (threshold – learning more important).On y axis: The y-axis show the outcome in relation to the optimal outcome, so perfect behavior means 1, = 100%. On the x-axis the parameter value of how fast to update your mental model is depicted. Straight lines: shows the fixed value, where the parameters value is fixed when varying each other learning parameter separately. 1) Grafencisar, yaxel, xaxel, På den orange Jag harfixerat 3 ochhar en rörlig.Upprepa , performance index är net income ochatt update rate of mental model etc.How much should I update my mental model according to the different feed backs I get from the system? Should I trust my own previous knowledge or adapt to new?Changing too fast not good, static not good.No threshold: 10-60%Threshold: 25%
- Here instead we have different values of the discount factor on the x-axis. (Crepin)If a threshold is going to be encountered it is better to take what is there, compared to no thresholdI.e. don’t value future outcomes cause will not be there
- Hindsight quite important for threshold, need to learn not to hit threshold again.For no threshold, not important, learn through constant adaption is enough.
- This graph shows the performance of the agent for different levels of exploration. Exploration on the x-axis for threshold problem translates to about 1-5% and for logistic about 20% in relation to what the agent perceives what is optimal to fish.If you don’t explore you will get stuck when the stock goes down?So how should you know what to do? If you don’t know what system you are in?
- In paper two look at how an LDB approah responds to changes in growthrate of a renewable resource as a consequence of environmental change.Background: Paper 2: Effects on for example marine systems during climate change are concerning, and growth rate change is a fact that we need to deal with… (less known)SusaNiiranen and collegues here have been looking at management recommendations from different models of baltic sea and conclude that is you include uncertainty on stock growth rates, the recommendations commonly lead to resource collapse.
- SAME fisher, same learning parameters but Use the logistic function and change the growth rate according to different scenarios
- To operationalize this, we probe the following re- search questions: Sustainably exploiting is just to emphasize here to day that it is about harvesting sustainably over time but also to gain as much out of the resource as possible as to not have any harvest go to waist.In this paper we studied an artificially intelligent agents ability to learn sustainable management of a renewable resource (with or without threshold dynamics).
- The different scenarios. The blue lines show the intrinsic growth rate that we change over time, so first there is no change and after 50 time steps we induce change. The best action to take here is to follow the growth rate (K-c)/2. So what you see is what is best to act as well. Ie for this cyclic one need to change between 2 and 8 nets depending on the stock.
- In this paper we take two different learning agents and contrast against a static non-learning agent. The learning agent has different learning parameters for each scenario, and the general agent has the exact same parameter setup independent of scenario.The static agent fishes with the optimal action for the first 50 steps all the way through. So you can see the effect of doing this in all different scenarios.
- In this paper we take two different learning agents and contrast against a static non-learning agent. The learning agent has different learning parameters for each scenario, and the general agent has the exact same parameter setup independent of scenario.The static agent fishes with the optimal action for the first 50 steps all the way through. So you can see the effect of doing this in all different scenarios.
- In this paper we take two different learning agents and contrast against a static non-learning agent. The learning agent has different learning parameters for each scenario, and the general agent has the exact same parameter setup independent of scenario.The static agent fishes with the optimal action for the first 50 steps all the way through. So you can see the effect of doing this in all different scenarios.
- All parameters important but depends on type of changeStatic or increasing easy (Brown) foodweb more importantThreshold hardest to learn through this modelGenerally trial and error methods quite bad when threshold (Biggs)Future: multi agent model, learn from each other, inertia, not adapt every time step.Leadership – influence by best performer, strong leaderNiiranen, S., Blenckner, T., Hjerne, O., Tomczak, M.T., 2012. Uncertainties in a baltic sea food-web model reveal challenges for future projections.
- All parameters important but depends on type of changeStatic or increasing easy (Brown) foodweb more importantThreshold hardest to learn through this modelGenerally trial and error methods quite bad when threshold (Biggs)Future: multi agent model, learn from each other, inertia, not adapt every time step.Leadership – influence by best performer, strong leader
- All parameters important but depends on type of changeStatic or increasing easy (Brown) foodweb more importantThreshold hardest to learn through this modelGenerally trial and error methods quite bad when threshold (Biggs)Future: multi agent model, learn from each other, inertia, not adapt every time step.Leadership – influence by best performer, strong leaderGutierrez: We identified strong leadership as the most important attribute contributing to success, followed by individual or community quotas, social cohesion and protected areas. Less important conditions included enforcement mechanisms, long-term management policies and life history of the resources.
- Thanks to my supervisors, all my friends and colleagues here at work, family and to all of you for listening
- LBDin its very basic form, one time step in the simulation corresponds to one fishing event. From the start the fisher hasno a priori understanding of the resource dynamics deciding on a harvest effort (how much to fish)applying the harvest effort (go fish)receive harvest (net income)learn (update mental model of the system, with help of the first three learning parameters)This approach is taken from tarditional psychology which also involves these four steps.DON’T SAY: Compared to classic LBD from psychology which also involves these four steps is 1) concrete experience (harvest) 2) reflective observation (learn) 3) abstract conceptualization (make decision) and 4) active experimentation (do action) [KOLB 1984]Agent controls or gets info according to 1-4. Update stock happens outside of his control.
- Maybe remove?The dark red is where the agent believes the optimum action to take in each state. So how many nets to put out, depending on state. Below 3.5 the best thing to do is to stop fishing.The green cross shows the maximum economic yield, and the yellow line the optimal thing to do if you have deviated from the optimal stock.You can see here that the agent has a more careful behavior as not to fall over the tipping point.
- Maybe remove?The dark red is where the agent believes the optimum action to take in each state. So how many nets to put out, depending on state. Below 3.5 the best thing to do is to stop fishing.The green cross shows the maximum economic yield, and the yellow line the optimal thing to do if you have deviated from the optimal stock.You can see here that the agent has a more careful behavior as not to fall over the tipping point.