Rakuten has been running the open-source version of Cloud Foundry internally for over 5 years. In this talk we will discuss our experience on three important topics: how we integrated Cloud Foundry with our internal systems, what are the most common issues users face when migrating their apps to Cloud Foundry and how to work with your users to make them advocates for the platform.

1. Carlo Alberto Ferraris, Ronak Banka | Rakuten, Inc.
Everyday life with CF
in a big organization

2. 2
5 years of Cloud Foundry at Rakuten
https://www.youtube.com/watch?v=CwJJyQQUsV4

3. Integrating with company systems
Porting existing applications
Turning users into advocates

4. Integrating with company systems

5. 5
RPaaS API and plugins
• API for Rakuten-specific tasks
– Automated organization creation
– Billing system integration
• Operates with admin privileges on the CF API on behalf
of regular users
• Runs as Cloud Foundry application

6. 6
RPaaS API and plugins
• User-facing features exposed via CF CLI plugins
– Org administration (including demo orgs)
• Sign up can be done fully via CLI
– Billing report
• Report resource usage
– Manifest generation
• Rakuten-specifics aware
• Helps new users onboarding

7. 7
RPaaS API and plugins
• Benefits
– Vanilla CF API
– Our API is outside the critical path
– Easy/low risk to experiment with
• Limitations
– Can’t be used for “policy enforcement”

8. 8
Multiple envs and the “stack hack”
• Rakuten has multiple networks (e.g. prod/non-prod)
• Small team delivering a prod-level platform using the
open-source version of CF
–Minimizing human operation work is important
• Placement pools Elastic clusters Isolation segments
Rainbows and unicorns were (and still are) not ready

9. 9
Multiple envs and the “stack hack”
• Solution: using the CF stack mechanism to create
different zones
–Use the standard cflinuxfs2 stack but give it different
names on different “zones”
–Concourse pipeline patch the buildpacks to disable
the stack name check
–Plugin helps users select buildpack and stack name

10. 10
Multiple envs and the “stack hack”
http://slides.com/cafxx/the-stack-hack

11. Porting existing applications

12. 12
Can I use NFS?
• Why
– Lots of legacy apps depends on NFS for data exchange
• Possible solution
– Using FUSE NFS with cf apps
• Challenges
– Security over NFS mounts
– Customizations required to support system calls during app
startup
– Reliability from production application point of view

13. 13
How can I know what my application is doing?
• Why
– Metrics which are provided on cli output are not enough to
understand system behavior
– Many system utils can’t be used with default user on container
– Metrics like latency, I/O, swap, RPS per instance are not
available for users.
• Possible solution
– Something which can correlate data between routers & app
instances and stream them on logging pipeline

14. 14
Can I restrict some of app operations in my space?
• Why
– RBACLs too coarse, space developers can do all the
operations
– L1 support don’t need the ability to push application but may
need to restart an instance
– Configurations (credentials) are visible to all space users
• Possible solution
– Support for operation based role creation (e.g. RPaaS API)

15. 15
My application is not able to access a file?
• Why
– Hardcoded paths can create issues because of the way
buildpacks configure the app directory
– Hardcoded configurations are again a big issue, when porting
applications to different PaaS environments.
• Possible solution
– Symlinks can only do so much
– Go with docker images, lose part of the “PaaS experience”

16. 16
Can I run my app with PHP 5.4?
• Issue
– There are lots of applications out there running on unsupported
versions of runtimes
– Custom buildpacks and docker images make this pretty painful
– As a operator I want to have visibility of runtimes which people
are using from security perspective
• Possible solutions
– Version check on cloud controller can help with hardening
– Give cf files-like access to an auditing system

17. Turning users into advocates

18. 18
Users and advocates
Rakuten doesn’t centrally mandate the technology to use
+
In a company with a “long” history many ways of doing
things are deeply ingrained in people
=
Without a corporate champion for the platform getting new
users turns into a house-by-house battle

19. 19
Supporting our users
Users and advocates
How we spend time in our team
Extending the platform Operating the platform

20. 20
Users and advocates
You don’t need to convince users that the platform is
better than what they have now
You need to convince them that it is
SO FRIGGIN’ AWESOME THAT
OMG I HAVE TO TELL MY BUDDIES IN OTHER TEAMS

21. 21
Users and advocates
Keep all channels open
Be transparent
Be (with) the user

22. 22
Users and advocates
https://www.youtube.com/watch?v=1o3LcxkAuNM

23. 23
Users and advocates
Screencasts
Introduction sessions
Architectural support
Operational support
Documentation
Samples

24. 24
Allies
CF summits and cf-dev
are great for exchanging ideas and solutions
(with some caveats)

25. 25
Allies
Holding sessions with other “private” CF operators is very
fertile ground for inspiration and knowledge sharing

26. 26
Q&A
Now or during the networking section after the talks
(BTW, we’re hiring!)