Connection between Bellman equation and Markov Decision Processes

In this slide, we investigate the relationship between Bellman equation and Markov decision processes (MDPs). While the principle of optimality directly gives us the relationships, we derive this connection by solving the KKT conditions of infinite horizon optimal control problems.