The document discusses Yahoo Japan's adoption of Cloud Foundry to enhance their service offerings, focusing on streamlining development and improving productivity. It outlines the company's structure, the challenges faced during implementation, and a case study of their initial rollout with key learnings. Future plans include expanding Cloud Foundry integration while addressing security and operational improvements.

1. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
2017年6月21日
The Road to "JYU-BAI"
- Adopting Cloud Foundry at Yahoo! JAPAN -
2017年6月20日

2. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
About me
2
Software Engineer
Manager
Yahoo! JAPAN
Yasuhiko Kubono

3. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
3
• Introducing Cloud Foundry into our
services
-Yasuhiko Kubono
• How do we Actually Operate
-Yusuke Kondo

4. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Introducing Cloud Foundry
into our services

5. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Agenda
5
• About Yahoo! JAPAN
• Why we use Cloud Foundry?
• Introducing Cloud Foundry into our services
• Case study

6. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
About
Yahoo! JAPAN

7. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Outline
Yahoo Japan Corporation
(SoftBank Group)
Businesses: Internet Advertising - e-Commerce
- Member services - Others
Headquarters: Tokyo Japan
Founded: January 31, 1996
# of Employees: 5,826 (As of March 31, 2017)
7

8. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
# of Engineers & Designers
2,500

9. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
More than 100
Web Services

10. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
39,89MActive User IDs
1Month
※2017年1-3月の平均
67,4B Page Views
Total requests

11. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Why we use
Cloud Foundry?

12. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Why we use Cloud Foundry？
Speed up
development time
JYU-BAI
increase productivity by 10 times
12

13. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Adoption Plan
13
2016
Initial introduction to
a few services
2017
Full-scale
implementation
2018
Expand
implementation
Here

14. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Introducing
Cloud Foundry
into our Services

15. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
More than 100
Web Services

16. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Programming Languages
16
C, Perl, C++,
PHP, Node.js, Java...

17. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Architecture Differs by Web Service
17
e.g. Travel tips e.g. Yahoo! Auction
Small-scale web services
list
cart
logic
logic
logic
Large-scale web services
Search
API
Gate
way

18. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Obstacles
18
Same architecture does not fit
in each web services

19. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Solutions
Enroll CF Coach in each web services
around 20 staff / 15 services
Coaches role:
Promote cloud design methods that suit for
each web services
19

20. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Role map
20
Core Team
CF Coach
For
Shopping
Shopping
engineer
Shopping
engineer
CF Coach
For
Auction
Auction
engineer
CF Coach
For
Media
Media engineer
…

21. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Case study

22. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Where we started from

23. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
List Necessary Functions
23
Service A Service B Service C Service D Service E Service F Service G
MySQL ● ● ● ●
Oracle ● ● ●
KVS
ObjectStrage ●
C/C++ ●
PHP ● ● ● ●
Node.js ● ● ● ● ●
Java ● ● ●
advertisement ● ● ● ● ●
beacon ● ●

24. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Challenges we encountered
Functions that can’t be used in the cloud because of
complicated dependency
Internal security polices are not suited for cloud
environment
Most of our web services were stateful design
24

25. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
How we started
We selected one web service, and started
by preparing the necessary functions for
that service
Resolve issues each time they occur
25

26. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
So, which web service
did we start with?

27. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Criteria for the web service
1. Simplicity
• Service with limited functions and external
PF that can be used
2. Actively developed
• Web services that actively developed
so that the effectiveness of introducing CF
can be measured
27

28. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
First target : CS tool
• Constructed with few servers in OpenStack environment
• WebServer: apache
• Apache Traffic Server (ATS) : Reverse Proxy
28
 Characteristics
• Language: PHP
• Framework： cakephp
• Uses REST API
• MySQL
 Server Configuration
ATS
CS tool
(apache)
API
ATS
MySQL
HTTPS
HTTP MySQL
CS tool
(apache)
(our auction service)

29. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Partial release using ATS (entry points)
Partially diverted entry point to CF apps using ATS:
• So that CF or OpenStack can be switched in entry points
29
ATS
CS tool
(apache)
CS tool
(apache)
API
HTTPS
HTTP
Some
entry
points
ATS
CS tool
(CF)

30. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Lessons learned from the first target
• How to Implement in Production
• Development method based on OSS
• How to make service stateless on CF
30

31. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Adopting & Expanding to other services
31
Decide
target
Investigate
issues
Adopting
knowledge
Solve issues
Knowhow
accumulated

32. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Next Presentation,
How do we Actually Operate
Photo by: Aflo

33. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Hello CF Summit 2017!
Yusuke Kondo or @konfoo
Responsible to...
• operating Cloud Foundry & Concourse on IaaS
• increasing engineers’ productivity by providing
tools and best practices around CI/CD

34. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Overview of Yahoo! JAPAN proprietary Infrastructure
More than four DCs in Japan
More than 90,000 VM running on OpenStack
34

35. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Cluster Spec
35
dev production
Load Balancer Software Hardware x2
IaaS Openstack Openstack
Hypervisor # 40 40

36. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Current Status (As of Jun. 9, 2017)
36
dev production
Cluster # 1 1
Cell # per Cluster 40 30
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000

37. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Future Plan (As of Jun. 9, 2017)
37
dev production
Cluster # 1 1 => 6
Cell # per Cluster 40 30 => 100
Org # 136 38
App Instance # approx. 2,000 approx. 400
Rps at peak time N/A approx. 2,000

38. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration
with
Backend Services

39. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
App Role Based
ACL
MQ
Service
RDB
Object
Storage
Key Value
Store
Cache
Service
FaaS
Existent Platforms
39

40. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration with Existent Platforms
• Cookie off-loading Route Service
• On-demand MySQL (OpenStack Trove API)
• Distributed pubsub service (Pulsar)
40

41. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Marketplace Dashboard
41
Goal: Providing all PFs in CF Marketplace

42. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Issues we faced
Platform ACL is based on IP address or hostname
=> Requesting for exceptional permission for
accessing via IP Range with a limited term.
=> Migrating from Host-based ACL to Role-
Based ACL in the long term
42

43. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration
with
Logs and Metrics

44. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
What we already have
In-house Monitoring & Alerting PF based on
Apache Kafka, Hbase,
Splunk, an enterprise log analytics platform
44

45. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
User-side Logs and Metrics
45
PCF Cluster-1
VM
APP
VM
App
VM
VM
APP
App
Loggregator
Splunk
Monitoring PF
App App
App
APP
APP
APP
APP
No action is needed for App developers

46. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
What we prepared
Firehose Nozzle and Relay Server
• Nozzle filters and formats the App
logs streamed by Firehose
• Relay Server forwards the log stream
to specific index
46

47. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Issue we faced
High log traffic. 900 lines per sec !
(as of Jun. 8, 2017)
=> Provided users with CF friendly logger
47

48. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Operator-side logs and Metrics
Splunk
• Platform logs such as CF component syslog
Prometheus
• Bosh metrics, VM metrics, Firehose metrics
• Emitting alerts to our smartphone
48

49. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
49

50. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration
with
other System

51. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration with package monitoring tool
51
Application Source Code
Dependent packages
Runtime Buildpack version
Vulnerable
Package
Monitoring Tool
Track the buildpack version which the App are staged with and report outdated apps.

52. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration with package monitoring tool
52
Application Source Code
Dependent packages
Runtime
Scan package version
Scan whole source code

53. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Integration with Concourse
We use Concourse for
• deploying new Cloud Foundry release
• updating buildpacks
• syncing employee accounts with UAA
• backup database to object storage
• ...
53

54. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
Lessons learned

55. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.
We are still on the way to change mind
Changing your organization mind is the most
essential part.
• Educate not only users, but also platform
division where you belong.
• Work closely with your security paranoid
team. Involve them to update the policy
55

56. Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved.