Motivated Machine Learning for  Water Resource Management   Janusz Starzyk School of Electrical Engineering and Computer S...
<ul><li>Challenges in Water Management </li></ul><ul><li>Embodied Intelligence (EI) </li></ul><ul><li>Embodiment of Mind <...
<ul><li>Water management is challenging for various reasons: </li></ul><ul><li>Strategies in water management are develope...
<ul><li>Decision makers must answer important questions: </li></ul><ul><li>How do we make water use sustainable?  </li></u...
<ul><li>Why development of integrated modeling to support decision making is important ? </li></ul><ul><li>Computerized mo...
<ul><li>What are deficiencies of machine created models? </li></ul><ul><li>Artificial neural networks, fuzzy logic, and ge...
<ul><li>Embodied intelligence may  support decision making: </li></ul><ul><li>EI mimics biological intelligent systems,  e...
<ul><li>Use the motivated learning scheme to integrate modelling and decision making: </li></ul><ul><li>It is suggested to...
<ul><li>“… Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of conscio...
Traditional AI  Embodied Intelligence <ul><li>Abstract intelligence </li></ul><ul><ul><li>attempt to simulate “highest” hu...
Design principles of intelligent systems from Rolf Pfeifer “Understanding of Intelligence”, 1999 <ul><li>Interaction with ...
Embodied Intelligence  <ul><ul><li>Definition </li></ul></ul><ul><ul><li>Embodied Intelligence (EI) is a  mechanism that l...
Embodiment of a Mind <ul><li>Embodiment contains intelligence core and sensory motor interfaces under its control to inter...
<ul><li>Brain learns own body’s dynamic </li></ul><ul><li>Self-awareness is a result of identification with own embodiment...
INPUT OUTPUT Simulation or Real-World System Task Environment Agent Architecture Long-term Memory Short-term Memory Reason...
How to Motivate a Machine  ?   The fundamental question is how to motivate a machine to do anything, in particular to incr...
How to Motivate a Machine  ?  <ul><li>I suggest that hostility of environment motivates us .  </li></ul><ul><ul><li>It is ...
Pain-center and Goal Creation <ul><li>Simple Mechanism </li></ul><ul><li>Creates hierarchy of values </li></ul><ul><li>Lea...
Primitive Goal Creation - + Pain Dry soil Primitive  level open tank sit on  garbage refill faucet w. can water Dual pain
Abstract Goal Creation <ul><li>The goal  is to reduce the primitive pain level </li></ul><ul><li>Abstract goals  are creat...
Abstract Goal Hierarchy <ul><li>A hierarchy of abstract goals   is created - they satisfy  </li></ul><ul><li>the lower lev...
GCS vs. Reinforcement Learning RL Actor-critic design Goal creation system Case study: “How can  Wall-E  water his plants ...
Goal Creation Experiment Sensory-motor pairs and their effect on the environment - lake water fall rain 29 lake water rese...
Results from GCS scheme 0 100 200 300 400 500 600 0 2 4 pain Dry soil 0 100 200 300 400 500 600 0 1 2 pain No   water in c...
GCS vs. Reinforcement Learning Averaged performance over 10 trials: GCS: RL: Machine using GCS learns to control all abstr...
Goal Creation Experiment Action scatters in 5 CGS simulations
Goal Creation Experiment The average pain signals in 100 CGS simulations  0 100 200 300 400 500 600 0 0.5 Primitive pain –...
Compare RL (TDF) and GCS Mean primitive pain Pp value as a function of the number of iterations. Dashed lines indicate mom...
<ul><li>Comparison of execution time on log-log scale </li></ul><ul><ul><li>TD-Falcon green </li></ul></ul><ul><ul><li>GCS...
Promises of embodied intelligence <ul><li>To society </li></ul><ul><ul><li>Advanced use of technology </li></ul></ul><ul><...
2002 2010 2020 2030 Biomimetics and Bio-inspired Systems Impact on Space Transportation, Space Science and Earth Science E...
Sounds like science fiction <ul><li>If you’re trying to look far ahead, and what you see seems like science fiction, it mi...
Questions?
Upcoming SlideShare
Loading in …5
×

Motivated Machine Learning for Water Resource Management

441 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
441
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 04/26/10
  • At first, the only pain that machine receives is the primitive pain. Once machine learns that eating food reduces the primitive pain, the lack of food becomes an abstract pain. As there is less and less food in the environment, the primitive pain increases again (since the machine cannot get the food) and the machine must learn how to get the food (buy the grocery). Once it learns this, a new pain source is created and so on. Notice that the primitive pain is maintained under control eventually in spite of changing environment conditions. In this presented trial, the machine can learn to create, develop and solve all abstract pains in this experiment within 300 iterations. In this experiment, school opportunity is designed as always available. Therefore, it is noted in Figure 6.18 that the abstract pain for “lack of school opportunity”, although was created when solving lower level pains, were never activated and stayed zero.
  • 04/26/10
  • 04/26/10
  • 04/26/10
  • Motivated Machine Learning for Water Resource Management

    1. 1. Motivated Machine Learning for Water Resource Management Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA www.ent.ohiou.edu/~starzyk UNESCO Workshop on Integrated Modeling Approaches to Support Water Resource Decision Making: Crossing the Chasm
    2. 2. <ul><li>Challenges in Water Management </li></ul><ul><li>Embodied Intelligence (EI) </li></ul><ul><li>Embodiment of Mind </li></ul><ul><li>EI Interaction with Environment </li></ul><ul><li>How to Motivate a Machine </li></ul><ul><li>Goal Creation Hierarchy </li></ul><ul><li>GCS Experiment </li></ul><ul><li>Promises of EI </li></ul><ul><ul><li>To economy </li></ul></ul><ul><ul><li>To society </li></ul></ul>Outline
    3. 3. <ul><li>Water management is challenging for various reasons: </li></ul><ul><li>Strategies in water management are developed mostly on national level </li></ul><ul><li>There is a growing competition between countries for water </li></ul><ul><li>Water policy making effects environment and society, health and development, and economy </li></ul><ul><li>Growing demands of countries’ populations for water </li></ul><ul><ul><li>Leads to hydrological nationalism </li></ul></ul><ul><ul><li>Creates a need to integrate water sciences and policy making </li></ul></ul><ul><li>There is an acute need for legitimate scientific data </li></ul><ul><li>Decision making in water-related health, food and energy systems are critical to economical development and security </li></ul>Challenges in Water Management
    4. 4. <ul><li>Decision makers must answer important questions: </li></ul><ul><li>How do we make water use sustainable? </li></ul><ul><li>Who owns the water? </li></ul><ul><li>What policies, institutional and legal framework can promote sustainable use of water? </li></ul><ul><li>How to protect water resources from overuse and contamination? </li></ul><ul><li>Water problems became too complex, interconnected and large to be handled by any one institution or by one group of professionals </li></ul><ul><li>It is a challenge to evolve strategies and institutional frameworks for quick policy changes towards an acceptable water use </li></ul><ul><li>It is necessary to create assessment and modeling tools to improve policy making and facilitate interaction. </li></ul>Challenges in Water Management
    5. 5. <ul><li>Why development of integrated modeling to support decision making is important ? </li></ul><ul><li>Computerized models were used for many years to support water related decisions. </li></ul><ul><li>Models often simplify dynamics of economic, social and environmental interactions and lead to inappropriate policy making and management decisions. </li></ul><ul><li>This note proposes models to emerge from interaction with real dynamically changing environments with all of their complexities and societal dependencies. </li></ul><ul><li>The main objective is to suggest an integrated modeling framework that may assist with the tasks of water related decision making. </li></ul>Challenges in Water Management
    6. 6. <ul><li>What are deficiencies of machine created models? </li></ul><ul><li>Artificial neural networks, fuzzy logic, and genetic algorithms have all been used to model the hydrological cycle </li></ul><ul><li>However, it is still difficult to apply these tools in making real-life water decisions as the related parameters are not explicitly known </li></ul><ul><li>What may be needed is a motivated machine learning for characterizing the data and making predictions about their dynamic changes </li></ul><ul><ul><li>It could support intelligent decision making in dynamically changing environment </li></ul></ul><ul><ul><li>It could be used to observe impacts of alternative water management policies </li></ul></ul><ul><ul><li>It may consider social, cultural, political, economic and institutional elements that influence decision making </li></ul></ul><ul><li>This strategic note presents a goal creation approach in embodied intelligence (EI) that motivates machine to develop into a useful research toll. </li></ul>Challenges in Water Management
    7. 7. <ul><li>Embodied intelligence may support decision making: </li></ul><ul><li>EI mimics biological intelligent systems, extracting general principles of intelligent behavior and applying them to design intelligent agents </li></ul><ul><li>It uses emerging, self-organizing, goal creation (GC) system that motivates embodied intelligence to learn how to efficiently interact with the environment </li></ul><ul><ul><li>Knowledge is not entered into such systems, but rather is a result of their successful use in a given environment. </li></ul></ul><ul><ul><li>This knowledge is validated through active interaction with the environment. </li></ul></ul><ul><li>Motivated intelligent systems adapt to unpredictable and dynamic situations in the environment by learning, which gives them a high degree of autonomy </li></ul><ul><li>Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory </li></ul>Challenges in Water Management
    8. 8. <ul><li>Use the motivated learning scheme to integrate modelling and decision making: </li></ul><ul><li>It is suggested to apply ML approach to water management in changing environments where the existing methods fail or work with difficulty. </li></ul><ul><ul><li>For instance, using classical machine learning to predict the future for physical processes works only under the assumption that same processes will repeat. </li></ul></ul><ul><ul><li>However, if a process changes beyond certain limits, the prediction will not be correct. </li></ul></ul><ul><ul><li>GC systems may overcome this difficulty and such systems can be trained to advice humans. </li></ul></ul><ul><li>Design concepts, computational mechanisms, and architectural organization of embodied intelligence are presented in this talk </li></ul><ul><li>The talk will explain internal motivation mechanism that leads to effective goal oriented learning </li></ul><ul><li>In addition, a goal creation mechanism and goal driven learning will be described. </li></ul>Challenges in Water Management
    9. 9. <ul><li>“… Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al. </li></ul><ul><ul><li>E. R. Kandel won Nobel Price in 2000 for his work on physiological basis of memory storage in neurons. </li></ul></ul><ul><li>“… The question of intelligence is the last great terrestrial frontier of science...” from Jeff Hawkins On Intelligence. </li></ul><ul><ul><li>Jeff Hawkins founded the Redwood Neuroscience Institute devoted to brain research </li></ul></ul>Intelligence AI’s holy grail From Pattie Maes MIT Media Lab
    10. 10. Traditional AI Embodied Intelligence <ul><li>Abstract intelligence </li></ul><ul><ul><li>attempt to simulate “highest” human faculties: </li></ul></ul><ul><ul><ul><li>language, discursive reason, mathematics, abstract problem solving </li></ul></ul></ul><ul><li>Environment model </li></ul><ul><ul><li>Condition for problem solving in abstract way </li></ul></ul><ul><ul><li>“ brain in a vat” </li></ul></ul><ul><li>Embodiment </li></ul><ul><ul><li>knowledge is implicit in the fact that we have a body </li></ul></ul><ul><ul><ul><li>embodiment supports brain development </li></ul></ul></ul><ul><li>Intelligence develops through interaction with environment </li></ul><ul><ul><li>Situated in environment </li></ul></ul><ul><ul><li>Environment is its best model </li></ul></ul>
    11. 11. Design principles of intelligent systems from Rolf Pfeifer “Understanding of Intelligence”, 1999 <ul><li>Interaction with complex environment </li></ul><ul><li>cheap design </li></ul><ul><li>ecological balance </li></ul><ul><li>redundancy principle </li></ul><ul><li>parallel, loosely coupled processes </li></ul><ul><li>asynchronous </li></ul><ul><li>sensory-motor </li></ul><ul><li>coordination </li></ul><ul><li>value principle </li></ul>Agent Drawing by Ciarán O’Leary- Dublin Institute of Technology
    12. 12. Embodied Intelligence <ul><ul><li>Definition </li></ul></ul><ul><ul><li>Embodied Intelligence (EI) is a mechanism that learns how to survive in a hostile environment </li></ul></ul><ul><ul><ul><li>Mechanism: biological, mechanical or virtual agent </li></ul></ul></ul><ul><ul><ul><li>with embodied sensors and actuators </li></ul></ul></ul><ul><ul><ul><li>EI acts on environment and perceives its actions </li></ul></ul></ul><ul><ul><ul><li>Environment hostility is persistent and stimulates EI to act </li></ul></ul></ul><ul><ul><ul><li>Hostility: direct aggression, pain, scarce resources, etc </li></ul></ul></ul><ul><ul><ul><li>EI learns so it must have associative self-organizing memory </li></ul></ul></ul><ul><ul><ul><li>Knowledge is acquired by EI </li></ul></ul></ul>
    13. 13. Embodiment of a Mind <ul><li>Embodiment contains intelligence core and sensory motor interfaces under its control to interact with environment </li></ul><ul><li>Necessary for development of intelligence </li></ul><ul><li>Not necessarily constant or in the form of a physical body </li></ul><ul><li>Boundary transforms modifying brain’s self-determination </li></ul>
    14. 14. <ul><li>Brain learns own body’s dynamic </li></ul><ul><li>Self-awareness is a result of identification with own embodiment </li></ul><ul><li>Embodiment can be extended by using tools and machines </li></ul><ul><li>Successful operation is a function of correct perception of environment and own embodiment </li></ul>Embodiment of a Mind
    15. 15. INPUT OUTPUT Simulation or Real-World System Task Environment Agent Architecture Long-term Memory Short-term Memory Reason Act Perceive RETRIEVAL LEARNING EI Interaction with Environment From Randolph M. Jones, P : www.soartech.com
    16. 16. How to Motivate a Machine ? The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity? How to motivate it to explore the environment and learn how to effectively work in this environment? Can a machine that only implements externally given goals be intelligent? If not how these goals can be created ?
    17. 17. How to Motivate a Machine ? <ul><li>I suggest that hostility of environment motivates us . </li></ul><ul><ul><li>It is the pain that moves us. </li></ul></ul><ul><ul><li>Our intelligence that tries to minimize this pain motivates our actions, learning and development </li></ul></ul><ul><li>We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain </li></ul><ul><li>In this work I propose, based on the pain, mechanism that motivates the machine to act, learn and develop. </li></ul><ul><ul><li>So the pain is good . </li></ul></ul><ul><ul><li>Without the pain there will be no intelligence . </li></ul></ul><ul><ul><li>Without the pain there will be no motivation to develop . </li></ul></ul>
    18. 18. Pain-center and Goal Creation <ul><li>Simple Mechanism </li></ul><ul><li>Creates hierarchy of values </li></ul><ul><li>Leads to formulation of complex goals </li></ul><ul><li>Reinforcement : </li></ul><ul><ul><li>Pain increase </li></ul></ul><ul><ul><li>Pain decrease </li></ul></ul><ul><li>Forces exploration </li></ul>Wall-E’s goal is to keep his plants from dying + - Environment Sensor Motor Pain level Dual pain level Pain increase Pain decrease (-) (+) Excitation (-) (-) (+) (+)
    19. 19. Primitive Goal Creation - + Pain Dry soil Primitive level open tank sit on garbage refill faucet w. can water Dual pain
    20. 20. Abstract Goal Creation <ul><li>The goal is to reduce the primitive pain level </li></ul><ul><li>Abstract goals are created to reduce abstract pains in order to satisfy the primitive goals </li></ul><ul><li>Abstract pain center </li></ul>- + Pain Dual pain + Dry soil Abstract pain “ water can” – sensory input to abstract pain center Sensory pathway (perception, sense) Motor pathway (action, reaction) Primitive Level Level I Level II faucet - w. can open water Activation Stimulation Inhibition Reinforcement Echo Need Expectation
    21. 21. Abstract Goal Hierarchy <ul><li>A hierarchy of abstract goals is created - they satisfy </li></ul><ul><li>the lower level goals </li></ul>Activation Stimulation Inhibition Reinforcement Echo Need Expectation - + + Dry soil Primitive Level Level I Level II faucet - w. can open water + Sensory pathway (perception, sense) Motor pathway (action, reaction) Level III tank - refill
    22. 22. GCS vs. Reinforcement Learning RL Actor-critic design Goal creation system Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?” Sensory pathway Motor pathway GCS Environment Pain States Gate control Desired action &state Action decision Action
    23. 23. Goal Creation Experiment Sensory-motor pairs and their effect on the environment - lake water fall rain 29 lake water reservoir water open pipe 22 reservoir water water in tank refill tank 15 water in tank water in can open faucet 8 water in can moisture water the plant water can 1 DECREASES INCREASES MOTOR SENSORY PAIR #
    24. 24. Results from GCS scheme 0 100 200 300 400 500 600 0 2 4 pain Dry soil 0 100 200 300 400 500 600 0 1 2 pain No water in can 0 100 200 300 400 500 600 0 1 2 pain No water in tank 0 100 200 300 400 500 600 0 0.5 1 pain No water in reservoir 0 100 200 300 400 500 600 0 2 4 pain No water in lake
    25. 25. GCS vs. Reinforcement Learning Averaged performance over 10 trials: GCS: RL: Machine using GCS learns to control all abstract pains and maintains the primitive pain signal on a low level in demanding environment conditions. 0 100 200 300 400 500 600 0 10 20 30
    26. 26. Goal Creation Experiment Action scatters in 5 CGS simulations
    27. 27. Goal Creation Experiment The average pain signals in 100 CGS simulations 0 100 200 300 400 500 600 0 0.5 Primitive pain – dry soil Pain 0 100 200 300 400 500 600 0 0.1 0.2 Lack of water in can Pain 0 100 200 300 400 500 600 0 0.1 0.2 Lack of water in tank Pain 0 100 200 300 400 500 600 0 0.1 0.2 Lack of water in reservoir Pain 0 100 200 300 400 500 600 0 0.05 0.1 Lack of water in lake Pain Discrete time
    28. 28. Compare RL (TDF) and GCS Mean primitive pain Pp value as a function of the number of iterations. Dashed lines indicate moment when Pp is getting stable - green line for TDF - blue line for GCS.
    29. 29. <ul><li>Comparison of execution time on log-log scale </li></ul><ul><ul><li>TD-Falcon green </li></ul></ul><ul><ul><li>GCS blue </li></ul></ul><ul><li>Combined efficiency of GCS 1000 better than TDF </li></ul>Compare RL (TDF) and GCS Conclusion: embodied intelligence, with motivated learning based on goal creation system, effectively integrates environment m odeling and decision making – thus it is poised to cross the chasm Problem solved
    30. 30. Promises of embodied intelligence <ul><li>To society </li></ul><ul><ul><li>Advanced use of technology </li></ul></ul><ul><ul><ul><li>Robots </li></ul></ul></ul><ul><ul><ul><li>Tutors </li></ul></ul></ul><ul><ul><ul><li>Intelligent gadgets </li></ul></ul></ul><ul><ul><li>Intelligence age follows </li></ul></ul><ul><ul><ul><li>Industrial age </li></ul></ul></ul><ul><ul><ul><li>Technological age </li></ul></ul></ul><ul><ul><ul><li>Information age </li></ul></ul></ul><ul><ul><li>Society of minds </li></ul></ul><ul><ul><ul><li>Superhuman intelligence </li></ul></ul></ul><ul><ul><ul><li>Progress in science </li></ul></ul></ul><ul><ul><ul><li>Solution to societies’ ills </li></ul></ul></ul><ul><li>To industry </li></ul><ul><ul><li>Technological development </li></ul></ul><ul><ul><li>New markets </li></ul></ul><ul><ul><li>Economical growth </li></ul></ul>ISAC, a Two-Armed Humanoid Robot Vanderbilt University
    31. 31. 2002 2010 2020 2030 Biomimetics and Bio-inspired Systems Impact on Space Transportation, Space Science and Earth Science Embryonics Extremophiles DNA Computing Brain-like computing Self Assembled Array Artificial nanopore high resolution Mars in situ life detector Sensor Web Skin and Bone Self healing structure and thermal protection systems Biologically inspired aero-space systems Space Transportation Memristors Biological nanopore low resolution Mission Complexity Biological Mimicking
    32. 32. Sounds like science fiction <ul><li>If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong . </li></ul><ul><li>But if it doesn’t seem like science fiction, it’s definitely wrong. </li></ul><ul><li>From presentation by Foresight Institute </li></ul>
    33. 33. Questions?

    ×