Abbeel coatesquigleyng nips2006_poster
•Download as PPT, PDF•
0 likes•98 views
The document discusses using apprenticeship learning algorithms to learn control policies for complex physical systems like helicopters from expert demonstrations, avoiding the need for explicit exploration or specifying a reward function. Experimental results show the algorithms effectively learned autonomous helicopter flight skills from human pilot data, completing tasks like stationary flips that are difficult to specify rewards for.
![Apprenticeship Learning for the Dynamics Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],S T A N F O R D An Application of Reinforcement Helicopter Pieter Abbeel, Adam Coates, ,[object Object],[object Object],[object Object],[object Object],[object Object],Expert human pilot flight ( a 1 , s 1 , a 2 , s 2 , a 3 , s 3 , ….) Learn P sa ( a 1 , s 1 , a 2 , s 2 , a 3 , s 3 , ….) Autonomous flight Learn P sa Dynamics Model P sa Reward Function R Reinforcement Learning Control policy Take away message: In the apprenticeship learning setting, i.e., when we have an expert demonstration, we do not need explicit exploration to perform as well as the expert. Theorem. Assuming we have a polynomial number of teacher demonstrations, then the apprenticeship learning algorithm will return a policy that performs as well as the teacher within a polynomial number of iterations. [Abbeel & Ng, 2005 for more details.] Dynamics Model P sa Reward Function R Reinforcement Learning Control policy ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Reinforcement Learning and Apprenticeship Learning Data collection: aggressive exploration is dangerous Have good model of dynamics? NO “ Explore” YES “ Exploit”](https://image.slidesharecdn.com/abbeelcoatesquigleyngnips2006poster-110117223134-phpapp02/85/Abbeel-coatesquigleyng-nips2006_poster-1-320.jpg)
![Stationary Rolls Stationary Flips S T A N F O R D Tail-In Funnels Nose-In Funnels Learning to Autonomous Flight Morgan Quigley and Andrew Y. Ng Experimental Results Video available. ,[object Object],[object Object],[object Object],[object Object],ADDITIONAL REFERENCES (SPECIFIC TO AUTONOMOUS HELICOPTER FLIGHT) [1] J. Bagnell and J. Schneider. Autonomous helicopter control using reinforcement learning policy search methods. In International Conference on Robotics and Automation. IEEE, 2001. [2] V. Gavrilets, I. Martinos, B. Mettler, and E. Feron. Control logic for automated aerobatic flight of miniature helicopter. In AIAA Guidance, Navigation and Control Conference , 2002. [3] M. La Civita. Integrated Modeling and Robust Control for Full-Envelope Flight of Robotic Helicopters . PhD thesis, Carnegie Mellon University, Pittsburgh, PA, 2003. [4] M. La Civita, G. Papgeorgiou, W. C. Messner, and T. Kanade. Design and flight testing of a high-bandwidth H-infinity loop shaping controller for a robotic helicopter. Journal of Guidance, Control, and Dynamics , 29(2):485-494, March-April 2006. [5] B. Mettler, M. Tischler, and T. Kanade. System identification of small-size unmanned helicopter dynamics. In American Helicopter Society, 55 th Forum , 1999. [6] Jonathan M. Roberts, Peter I. Corke, and Gregg Buskey. Low-cost flight control system for a small autonomous helicopter. In IEEE Int’l Conf. On Robotics and Automation , 2003. [7] S. Saripalli, J. Montgomery, and G. Sukhatme. Visually-guided landing of an unmanned aerial vehicle, 2003. ,[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)