The document discusses reinforcement learning. It defines reinforcement learning as learning via interactions with an environment where an agent receives rewards or penalties for its actions without being told which actions are correct. The document outlines different types of learning including supervised learning and reinforcement learning. It also discusses key concepts in reinforcement learning including the reinforcement learning model, model-based vs model-free approaches, passive vs active learning, exploration problems, and using generalization techniques like function approximation to deal with large state spaces.
An efficient use of temporal difference technique in Computer Game LearningPrabhu Kumar
A computer game using temporal difference algorithm of Machine learning which improves the ability of the computer to learn and also explore the best next move for the game by greedy movement techniques and exploration method techniques for the future states of the game.
Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning.
An efficient use of temporal difference technique in Computer Game LearningPrabhu Kumar
A computer game using temporal difference algorithm of Machine learning which improves the ability of the computer to learn and also explore the best next move for the game by greedy movement techniques and exploration method techniques for the future states of the game.
Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning.
Reinforcement learning is the training of machine learning models to make a sequence of decisions. The agent learns to achieve a goal in an uncertain, potentially complex environment. In reinforcement learning, an artificial intelligence faces a game-like situation.
AUTOMATIC TRANSFER RATE ADJUSTMENT FOR TRANSFER REINFORCEMENT LEARNINGgerogepatton
This paper proposes a novel parameter for transfer reinforcement learning to avoid over-fitting when an
agent uses a transferred policy from a source task. Learning robot systems have recently been studied for
many applications, such as home robots, communication robots, and warehouse robots. However, if the
agent reuses the knowledge that has been sufficiently learned in the source task, deadlock may occur and
appropriate transfer learning may not be realized. In the previous work, a parameter called transfer rate
was proposed to adjust the ratio of transfer, and its contribution include avoiding dead lock in the target
task. However, adjusting the parameter depends on human intuition and experiences. Furthermore, the
method for deciding transfer rate has not discussed. Therefore, an automatic method for adjusting the
transfer rate is proposed in this paper using a sigmoid function. Further, computer simulations are used to
evaluate the effectiveness of the proposed method to improve the environmental adaptation performance in
a target task, which refers to the situation of reusing knowledge.
Reinforcement Learning Guide For Beginnersgokulprasath06
Reinforcement Learning Guide:
Land in multiple job interviews by joining our Data Science certification course.
Data Science course content designed uniquely, which helps you start learning Data Science from basics to advanced data science concepts.
Content: http://bit.ly/2Mub6xP
Any Queries, Call us@ +91 9884412301 / 9600112302
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017MLconf
Deep Reinforcement Learning with Shallow Trees:
In this talk, I present Concept Network Reinforcement Learning (CNRL), developed at Bonsai. It is an industrially applicable approach to solving complex tasks using reinforcement learning, which facilitates problem decomposition, allows component reuse, and simplifies reward functions. Inspired by Sutton’s options framework, we introduce the notion of “Concept Networks” which are tree-like structures in which leaves are “sub-concepts” (sub-tasks), representing policies on a subset of state space. The parent (non-leaf) nodes are “Selectors”, containing policies on which sub-concept to choose from the child nodes, at each time during an episode. There will be a high-level overview on the reinforcement learning fundamentals at the beginning of the talk.
Bio: Matineh Shaker is an Artificial Intelligence Scientist at Bonsai in Berkeley, CA, where she builds machine learning, reinforcement learning, and deep learning tools and algorithms for general purpose intelligent systems. She was previously a Machine Learning Researcher at Geometric Intelligence, Data Science Fellow at Insight Data Science, Predoctoral Fellow at Harvard Medical School. She received her PhD from Northeastern University with a dissertation in geometry-inspired manifold learning.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Reinforcement learning is the training of machine learning models to make a sequence of decisions. The agent learns to achieve a goal in an uncertain, potentially complex environment. In reinforcement learning, an artificial intelligence faces a game-like situation.
AUTOMATIC TRANSFER RATE ADJUSTMENT FOR TRANSFER REINFORCEMENT LEARNINGgerogepatton
This paper proposes a novel parameter for transfer reinforcement learning to avoid over-fitting when an
agent uses a transferred policy from a source task. Learning robot systems have recently been studied for
many applications, such as home robots, communication robots, and warehouse robots. However, if the
agent reuses the knowledge that has been sufficiently learned in the source task, deadlock may occur and
appropriate transfer learning may not be realized. In the previous work, a parameter called transfer rate
was proposed to adjust the ratio of transfer, and its contribution include avoiding dead lock in the target
task. However, adjusting the parameter depends on human intuition and experiences. Furthermore, the
method for deciding transfer rate has not discussed. Therefore, an automatic method for adjusting the
transfer rate is proposed in this paper using a sigmoid function. Further, computer simulations are used to
evaluate the effectiveness of the proposed method to improve the environmental adaptation performance in
a target task, which refers to the situation of reusing knowledge.
Reinforcement Learning Guide For Beginnersgokulprasath06
Reinforcement Learning Guide:
Land in multiple job interviews by joining our Data Science certification course.
Data Science course content designed uniquely, which helps you start learning Data Science from basics to advanced data science concepts.
Content: http://bit.ly/2Mub6xP
Any Queries, Call us@ +91 9884412301 / 9600112302
Matineh Shaker, Artificial Intelligence Scientist, Bonsai at MLconf SF 2017MLconf
Deep Reinforcement Learning with Shallow Trees:
In this talk, I present Concept Network Reinforcement Learning (CNRL), developed at Bonsai. It is an industrially applicable approach to solving complex tasks using reinforcement learning, which facilitates problem decomposition, allows component reuse, and simplifies reward functions. Inspired by Sutton’s options framework, we introduce the notion of “Concept Networks” which are tree-like structures in which leaves are “sub-concepts” (sub-tasks), representing policies on a subset of state space. The parent (non-leaf) nodes are “Selectors”, containing policies on which sub-concept to choose from the child nodes, at each time during an episode. There will be a high-level overview on the reinforcement learning fundamentals at the beginning of the talk.
Bio: Matineh Shaker is an Artificial Intelligence Scientist at Bonsai in Berkeley, CA, where she builds machine learning, reinforcement learning, and deep learning tools and algorithms for general purpose intelligent systems. She was previously a Machine Learning Researcher at Geometric Intelligence, Data Science Fellow at Insight Data Science, Predoctoral Fellow at Harvard Medical School. She received her PhD from Northeastern University with a dissertation in geometry-inspired manifold learning.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
2. What is learning?
Learning takes place as a result of interaction
between an agent and the world, the idea
behind learning is that
Percepts received by an agent should be used not
only for acting, but also for improving the agent’s
ability to behave optimally in the future to achieve
the goal.
3. Learning types
Learning types
Supervised learning:
a situation in which sample (input, output) pairs of the
function to be learned can be perceived or are given
You can think it as if there is a kind teacher
Reinforcement learning:
in the case of the agent acts on its environment, it
receives some evaluation of its action (reinforcement),
but is not told of which action is the correct one to
achieve its goal
4. Reinforcement learning
Task
Learn how to behave successfully to achieve a
goal while interacting with an external
environment
Learn via experiences!
Examples
Game playing: player knows whether it win or lose,
but not know how to move at each step
Control: a traffic system can measure the delay of
cars, but not know how to decrease it.
6. RL model
Each percept(e) is enough to determine the
State(the state is accessible)
The agent can decompose the Reward component
from a percept.
The agent task: to find a optimal policy, mapping
states to actions, that maximize long-run measure
of the reinforcement
Think of reinforcement as reward
Can be modeled as MDP model!
7. Review of MDP model
MDP model <S,T,A,R>
Agent
Environment
State
Reward
Action
s0
r0
a0
s1
a1
r1
s2
a2
r2
s3
• S– set of states
• A– set of actions
• T(s,a,s’) = P(s’|s,a)– the
probability of transition from
s to s’ given action a
• R(s,a)– the expected reward
for taking action a in state s
'
'
)
'
,
,
(
)
'
,
,
(
)
,
(
)
'
,
,
(
)
,
|
'
(
)
,
(
s
s
s
a
s
r
s
a
s
T
a
s
R
s
a
s
r
a
s
s
P
a
s
R
8. Model based v.s.Model free
approaches
But, we don’t know anything about the environment
model—the transition function T(s,a,s’)
Here comes two approaches
Model based approach RL:
learn the model, and use it to derive the optimal policy.
e.g Adaptive dynamic learning(ADP) approach
Model free approach RL:
derive the optimal policy without learning the model.
e.g LMS and Temporal difference approach
Which one is better?
9. Passive learning v.s. Active
learning
Passive learning
The agent imply watches the world going by and
tries to learn the utilities of being in various states
Active learning
The agent not simply watches, but also acts
11. Passive learning scenario
The agent see the the sequences of state
transitions and associate rewards
The environment generates state transitions and the
agent perceive them
e.g (1,1) (1,2) (1,3) (2,3) (3,3) (4,3)[+1]
(1,1)(1,2) (1,3) (1,2) (1,3) (1,2) (1,1) (2,1)
(3,1) (4,1) (4,2)[-1]
Key idea: updating the utility value using the
given training sequences.
13. LMS updating
Reward to go of a state
the sum of the rewards from that state until a
terminal state is reached
Key: use observed reward to go of the state as
the direct evidence of the actual expected utility
of that state
Learning utility function directly from sequence
example
14. LMS updating
function LMS-UPDATE (U, e, percepts, M, N ) return an updated U
if TERMINAL?[e] then
{ reward-to-go 0
for each ei in percepts (starting from end) do
s = STATE[ei]
reward-to-go reward-to-go + REWARS[ei]
U[s] = RUNNING-AVERAGE (U[s], reward-to-go, N[s])
end
}
function RUNNING-AVERAGE (U[s], reward-to-go, N[s] )
U[s] = [ U[s] * (N[s] – 1) + reward-to-go ] / N[s]
15. LMS updating algorithm in
passive learning
Drawback:
The actual utility of a state is constrained to be probability- weighted
average of its successor’s utilities.
Converge very slowly to correct utilities values (requires a lot of sequences)
for our example, >1000!
16. Temporal difference method in
passive learning
TD(0) key idea:
adjust the estimated utility value of the current state based on its
immediately reward and the estimated value of the next state.
The updating rule
is the learning rate parameter
Only when is a function that decreases as the number of times
a state has been visited increased, then can U(s)converge to the
correct value.
))
(
)
'
(
)
(
(
)
(
)
( s
U
s
U
s
R
s
U
s
U
18. Adaptive dynamic programming(ADP)
in passive learning
Different with LMS and TD method(model free
approaches)
ADP is a model based approach!
The updating rule for passive learning
However, in an unknown environment, T is not given,
the agent must learn T itself by experiences with the
environment.
How to learn T?
))
'
(
)
'
,
(
(
)
'
,
(
)
(
'
s
U
s
s
r
s
s
T
s
U
s
20. Active learning
An active agent must consider
what actions to take?
what their outcomes maybe(both on learning and receiving the
rewards in the long run)?
Update utility equation
Rule to chose action
))
'
(
)
'
,
,
(
)
,
(
(
max
arg
'
s
U
s
a
s
T
a
s
R
a
s
a
))
'
(
)
'
,
,
(
)
,
(
(
max
)
(
'
s
U
s
a
s
T
a
s
R
s
U
s
a
21. Active ADP algorithm
For each s, initialize U(s) , T(s,a,s’) and R(s,a)
Initialize s to current state that is perceived
Loop forever
{
Select an action a and execute it (using current model R and T) using
Receive immediate reward r and observe the new state s’
Using the transition tuple <s,a,s’,r> to update model R and T (see further)
For all the sate s, update U(s) using the updating rule
s = s’
}
))
'
(
)
'
,
,
(
)
,
(
(
max
arg
'
s
U
s
a
s
T
a
s
R
a
s
a
))
'
(
)
'
,
,
(
)
,
(
(
max
)
(
'
s
U
s
a
s
T
a
s
R
s
U
s
a
22. How to learn model?
Use the transition tuple <s, a, s’, r> to learn T(s,a,s’) and
R(s,a). That’s supervised learning!
Since the agent can get every transition (s, a, s’,r) directly, so take
(s,a)/s’ as an input/output example of the transition probability
function T.
Different techniques in the supervised learning(see further reading
for detail)
Use r and T(s,a,s’) to learn R(s,a)
'
)
'
,
,
(
)
,
(
s
r
s
a
s
T
a
s
R
23. ADP approach pros and cons
Pros:
ADP algorithm converges far faster than LMS and Temporal
learning. That is because it use the information from the the model
of the environment.
Cons:
Intractable for large state space
In each step, update U for all states
Improve this by prioritized-sweeping (see further reading for detail)
24. Another model free method–
TD-Q learning
Define Q-value function
Q-value function updating rule
<*>
Key idea of TD-Q learning
Combined with temporal difference approach
The updating rule
Rule to chose the action to take
)
,
(
max
)
( a
s
Q
s
U
a
))
'
(
)
'
,
,
(
)
,
(
(
max
)
(
'
s
U
s
a
s
T
a
s
R
s
U
s
a
)
'
(
)
'
,
,
(
)
,
(
)
,
(
'
s
U
s
a
s
T
a
s
R
a
s
Q
s
)
'
,
'
(
max
)
'
,
,
(
)
,
(
)
,
(
'
'
a
s
Q
s
a
s
T
a
s
R
a
s
Q
s
a
))
,
(
)
'
,
'
(
max
(
)
,
(
)
,
(
'
a
s
Q
a
s
Q
r
a
s
Q
a
s
Q
a
)
,
(
max
arg a
s
Q
a
a
25. TD-Q learning agent algorithm
For each pair (s, a), initialize Q(s,a)
Observe the current state s
Loop forever
{
Select an action a and execute it
Receive immediate reward r and observe the new state s’
Update Q(s,a)
s=s’
}
)
,
(
max
arg a
s
Q
a
a
))
,
(
)
'
,
'
(
max
(
)
,
(
)
,
(
'
a
s
Q
a
s
Q
r
a
s
Q
a
s
Q
a
26. An action has two kinds of outcome
Gain rewards on the current experience
tuple (s,a,s’)
Affect the percepts received, and hence
the ability of the agent to learn
Exploration problem in Active
learning
27. Exploration problem in Active
learning
A trade off when choosing action between
its immediately good(reflected in its current utility estimates using the
what we have learned)
its long term good(exploring more about the environment help it to
behave optimally in the long run)
Two extreme approaches
“wacky”approach: acts randomly, in the hope that it will eventually
explore the entire environment.
“greedy”approach: acts to maximize its utility using current model
estimate
See Figure 20.10
Just like human in the real world! People need to decide between
Continuing in a comfortable existence
Or striking out into the unknown in the hopes of discovering a new
and better life
28. Exploration problem in Active
learning
One kind of solution: the agent should be more wacky when it has
little idea of the environment, and more greedy when it has a
model that is close to being correct
In a given state, the agent should give some weight to actions that it
has not tried very often.
While tend to avoid actions that are believed to be of low utility
Implemented by exploration function f(u,n):
assigning a higher utility estimate to relatively unexplored action state
pairs
Chang the updating rule of value function to
U+ denote the optimistic estimate of the utility
))
,
(
),
'
(
)
'
,
,
(
(
)
,
(
(
max
)
(
'
s
a
N
s
U
s
a
s
T
f
a
s
r
s
U
s
a
29. Exploration problem in Active
learning
One kind of definition of f(u,n)
if n< Ne
u otherwise
is an optimistic estimate of the best possible reward
obtainable in any state
The agent will try each action-state pair(s,a) at least Ne times
The agent will behave initially as if there were wonderful rewards
scattered all over around– optimistic .
)
,
( n
u
f
R
R
30. Generalization in
Reinforcement Learning
So far we assumed that all the functions learned by
the agent are (U, T, R,Q) are tabular forms—
i.e.. It is possible to enumerate state and action
spaces.
Use generalization techniques to deal with large state
or action space.
Function approximation techniques
31. Genetic algorithm and Evolutionary
programming
Start with a set of individuals
Apply selection and reproduction operators to “evolve” an individual that is
successful (measured by a fitness function)
32. Genetic algorithm and Evolutionary
programming
Imagine the individuals as agent functions
Fitness function as performance measure or reward
function
No attempt made to learn the relationship the
rewards and actions taken by an agent
Simply searches directly in the individual space to
find one that maximizes the fitness functions
33. Genetic algorithm and Evolutionary
programming
Represent an individual as a binary string(each bit of the string is called a gene)
Selection works like this: if individual X scores twice as high as Y on the fitness
function, then X is twice likely to be selected for reproduction than Y is
Reproduction is accomplished by cross-over and mutation