[RakutenTechConf2013] [D-2] RPaaS DevOps: Lessons from using Cloudfoundry in Production


Published on

Rakuten Technology Conference 2013
"RPaaS DevOps: Lessons from using Cloudfoundry in Production"
Waldemar Quevedo (Rakuten)

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

[RakutenTechConf2013] [D-2] RPaaS DevOps: Lessons from using Cloudfoundry in Production

  1. 1. RPaaS DevOps: Lessons from using Cloudfoundry in Production Oct/26/2013 Waldemar Quevedo PaaS DevOps team. Rakuten, Inc.
  2. 2. About me • Waldemar Quevedo • From Mexico • Joined Rakuten in 2010 as an App developer • Joined the RPaaS team in 2012 in its earlier stages • Github: https://github.com/wallyqs 2
  3. 3. WARNING • No funny pictures in these slides! Sorry! 3
  4. 4. What is RPaaS? • Internal PaaS at use in Rakuten • Built on Cloudfoundry • Started to service production applications since mid 2012 • Derek Collison, its creator is a technical fellow of our team 4
  5. 5. Example site: tech.rakuten.co.jp 5
  6. 6. 6
  7. 7. Why have an application Platform as a Service? • Natural evolution of how to develop, release and maintain applications. • Improve productivity by automating the provisioning of resources on demand. • Scale horizontally (more instances) • Scale vertically (more cpus, memory, etc..) • Guarantees high availability 7
  8. 8. Example use case: Scaling How to go from this, aaa.bbb.ccc.ddd:80 tech.rakuten.co.jp to this, aaa.bbb.ccc.ddd:9393 aaa.bbb.ccc.ddd:9395 aaa.bbb.ccc.ddd:9291 aaa.bbb.ccc.ddd:8888 tech.rakuten.co.jp tech.rakuten.co.jp tech.rakuten.co.jp tech.rakuten.co.jp • …in a matter of seconds? 8
  9. 9. Increasing instances 9
  10. 10. Increasing instances 10
  11. 11. Benefits from an application PaaS • Improves agility of the application team • Application team can focus on the application not maintaining servers. • Simplified operations 11
  12. 12. Simplified operations 12
  13. 13. Benefits from an application PaaS • Enforces automation and better practices for developers. • Reduces the amount of workflows required to get things done. • Integration with internal environment. • And many more… 13
  14. 14. Topic of the day: How to adopt an application PaaS • An application PaaS is all goodness for developers, since they can stop caring about servers. • Instead, they can rely on the RPaaS team! 14
  15. 15. Usual DevOps metaphor: Vampires vs. Werewolves • Vampires (吸血鬼) app developers provide value • Werewolves (オオカミ人間) sys admins protect value 15
  16. 16. Vampires vs. Werewolves • A PaaS admin is somewhere in the middle. • We take care of the servers on behalf of users but in the end we also care about: • How the user is actually using the runtimes. • Demand from application runtimes available. • Support the users when they run into problems with one of the runtimes. 16
  17. 17. Different metaphor with PaaS: Villagers and Samurai • Villagers (村人) Application team can survive without PaaS but has lots of issues (-agility, -flexibility, -availability) • Samurai (侍) Solves the issues of the application team by providing a better environment. 17
  18. 18. In order to deliver to users… • RPaaS team approach: • “you should know everything” 1. Know your stack (Cloudfoundry, Nginx, Services) 2. Know your environment (infrastructure, the network…) 3. Know your users (runtimes: Ruby, Java, support) 4. Know your capacity (metrics & trends) 18
  19. 19. Know your stack • We needed to learn the internals of Cloudfoundry (OpenSource software FTW) • • • • • CloudController DEA HealthManager Router NATS 19
  20. 20. Cloudfoundry 20
  21. 21. Know your stack • Specially need to be prepared for failure scenarios: • How long does it take for the HM to execute the application failover when a DEA goes down? • What happens when there is a NATS failure? • How long does it take for the Router to stop balancing an application when the app goes down? 21
  22. 22. Know your environment • Remember the „Fallacies of distributed computing‟ http://en.wikipedia.org/wiki/Fallacies_of_Distribut ed_Computing • Need to be familiar with your infrastructure, the network etc… • Don‟t be afraid of tcpdump • Share your findings about the environment with the users • Each datacenter is a different beast 22
  23. 23. Know your environment • Again, need to be prepared for trouble scenarios: • Storage will fail • Physical hosts will fail • That VM that you expected to never to restart, will restart (hint: NATS) 23
  24. 24. Know your users • Insight: When introducing a PaaS into your company, you need an extra effort during the adoption phase. • „Do things that Don‟t Scale‟: http://paulgraham.com/ds.html 24
  25. 25. Know your users • At the time we started Cloudfoundry was relatively new technology • Roadmap mismatch: Private PaaS vs Public PaaS • Must features for our users were different: • Logging, monitoring, billing, team based deployment • Need highly available services • These were blockers we need to comply with so that teams would start adoption of the PaaS 25
  26. 26. Know your capacity • re: „Auto-scale‟: • You usually don‟t want to do it • You do need to automate provisioning of resources though • Monitor the trends and provision according to them: • DEAs (app resources) • Disks • Services (cache, dbs) • Traffic (increase max qps limit) 26
  27. 27. Managing user expectations • “Under promise and over deliver” • Need to make it very clear for users what is possible and what is just not feasible according to the current limitations of the system. • The concept of the cloud is cause of confusion sometimes • “it should be elastic” • “it should auto scale” • “it should self heal” 27
  28. 28. Managing user expectations • “The RPaaS team will fix it for us” • It helps becoming familiar with popular libraries for different runtimes to give good advice to the users on how to solve their issues in the environment you provide. • Learn to say no sometimes • Ideally: Make PaaS users collaborators • Provide a straightforward way for users to have a single node setup of the PaaS. 28
  29. 29. Wrapping up • The PaaS field continues to evolve • The application PaaS we run is still first,second generation technology • We already sort of see traces of the next evolution 29
  30. 30. Wrapping up • Keep up with the trend • When we started it seemed Ruby was the language from the cloud https://twitter.com/mart_brooks/status/27311502 8008341506 30
  31. 31. Wrapping up • Nowadays, it seems that Go is looking out to displace it little by little. • Biggest gains seem to be performance, and it being so practical to deploy. • Also, docker. https://twitter.com/derekcollison/status/245522124666716160 http://www.slideshare.net/derekcollison/go-conference-japan 31
  32. 32. Thanks! • Questions? Comments? 32