1. RPaaS DevOps:
Lessons from using
Cloudfoundry in Production
PaaS DevOps team. Rakuten, Inc.
2. About me
• Waldemar Quevedo
• From Mexico
• Joined Rakuten in 2010 as an App developer
• Joined the RPaaS team in 2012 in its earlier stages
• Github: https://github.com/wallyqs
• No funny pictures in these slides! Sorry!
4. What is RPaaS?
• Internal PaaS at use in Rakuten
• Built on Cloudfoundry
• Started to service production applications since
• Derek Collison, its creator is a technical fellow of
5. Example site: tech.rakuten.co.jp
7. Why have an application Platform as a Service?
• Natural evolution of how to develop, release and
• Improve productivity by automating the
provisioning of resources on demand.
• Scale horizontally (more instances)
• Scale vertically (more cpus, memory, etc..)
• Guarantees high availability
8. Example use case: Scaling
How to go from this,
• …in a matter of seconds?
9. Increasing instances
10. Increasing instances
11. Benefits from an application PaaS
• Improves agility of the application team
• Application team can focus on the application
not maintaining servers.
• Simplified operations
12. Simplified operations
13. Benefits from an application PaaS
• Enforces automation and better practices for
• Reduces the amount of workflows required to get
• Integration with internal environment.
• And many more…
14. Topic of the day: How to adopt an application PaaS
• An application PaaS is all goodness for
developers, since they can stop caring about
• Instead, they can rely on the RPaaS team!
15. Usual DevOps metaphor: Vampires vs. Werewolves
• Vampires (吸血鬼)
• Werewolves (オオカミ人間)
16. Vampires vs. Werewolves
• A PaaS admin is somewhere in the middle.
• We take care of the servers on behalf of users but
in the end we also care about:
• How the user is actually using the runtimes.
• Demand from application runtimes available.
• Support the users when they run into problems
with one of the runtimes.
17. Different metaphor with PaaS:
Villagers and Samurai
• Villagers (村人)
Application team can survive without PaaS but
has lots of issues (-agility, -flexibility, -availability)
• Samurai (侍)
Solves the issues of the application team by
providing a better environment.
18. In order to deliver to users…
• RPaaS team approach:
• “you should know everything”
1. Know your stack
(Cloudfoundry, Nginx, Services)
2. Know your environment (infrastructure, the network…)
3. Know your users
(runtimes: Ruby, Java, support)
4. Know your capacity
(metrics & trends)
19. Know your stack
• We needed to learn the internals of Cloudfoundry
(OpenSource software FTW)
21. Know your stack
• Specially need to be prepared for failure scenarios:
• How long does it take for the HM to execute the
application failover when a DEA goes down?
• What happens when there is a NATS failure?
• How long does it take for the Router to stop
balancing an application when the app goes
22. Know your environment
• Remember the „Fallacies of distributed
• Need to be familiar with your infrastructure, the
• Don‟t be afraid of tcpdump
• Share your findings about the environment with
• Each datacenter is a different beast
23. Know your environment
• Again, need to be prepared for trouble scenarios:
• Storage will fail
• Physical hosts will fail
• That VM that you expected to never to restart,
will restart (hint: NATS)
24. Know your users
• Insight: When introducing a PaaS into your
company, you need an extra effort during the
• „Do things that Don‟t Scale‟:
25. Know your users
• At the time we started Cloudfoundry was
relatively new technology
• Roadmap mismatch: Private PaaS vs Public
• Must features for our users were different:
• Logging, monitoring, billing, team based
• Need highly available services
• These were blockers we need to comply with
so that teams would start adoption of the PaaS
26. Know your capacity
• re: „Auto-scale‟:
• You usually don‟t want to do it
• You do need to automate provisioning of
• Monitor the trends and provision according to
• DEAs (app resources)
• Services (cache, dbs)
• Traffic (increase max qps limit)
27. Managing user expectations
• “Under promise and over deliver”
• Need to make it very clear for users what is
possible and what is just not feasible according
to the current limitations of the system.
• The concept of the cloud is cause of confusion
• “it should be elastic”
• “it should auto scale”
• “it should self heal”
28. Managing user expectations
• “The RPaaS team will fix it for us”
• It helps becoming familiar with popular libraries
for different runtimes to give good advice to the
users on how to solve their issues in the
environment you provide.
• Learn to say no sometimes
• Ideally: Make PaaS users collaborators
• Provide a straightforward way for users to
have a single node setup of the PaaS.
29. Wrapping up
• The PaaS field continues to evolve
• The application PaaS we run is still
first,second generation technology
• We already sort of see traces of the next
30. Wrapping up
• Keep up with the trend
• When we started it seemed Ruby was the
language from the cloud
31. Wrapping up
• Nowadays, it seems that Go is looking out to
displace it little by little.
• Biggest gains seem to be performance, and it
being so practical to deploy.
• Also, docker.