Gree is a gaming company with the distinction of launching the world’s first mobile social game in 2007 and later went on to develop a rich portfolio of original games. With an active acquisition and development strategy, Gree is transforming the way games are built. Learn how Gree leveraged Amazon VPCs and Aviatrix Enterprise cloud-defined networking in AWS Marketplace to rapidly integrate acquisitions and develop new games to maintain competitive differentiation in this fast moving segment. Hear about the use case, requirements and lessons learned. Learn how you can transitioned to Amazon VPCs in the middle of the development process.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloud Based Game Development
David Pippenger, Director DevOps, GREE
NET308
How GREE Launched New Games Faster and
More Securely with AWS Marketplace and
Amazon VPC

2. Company Overview
GREE International
Entertainment, Inc.
San Francisco, CA
~200 People, 2011
GREE Inc.
Tokyo, Japan
~2000 People, 2004
Started using AWS 2012
Evolved toward
immutable infrastructure
to run games
Now continuing to
evolve toward
decentralized model

3. Business Model: Publishing Engine
Buy a studio,
operate it as a
programming
division
First Party
Commission a
game to GREE’s
specifications, pay
development costs
Second Party
Buy an existing
game
Third Party

4. Game Publishing Engine
Marketing Analytics LiveOps
in-game
sales events
TechOps
AWS Infrastructure

5. Games have a lifecycle. Nothing lasts forever.
New
Game
Creation
Long tail games with loyal users
BUY GAME SELL GAME
TOP 10
REVENUE
GAMES

6. Gree started in
colos until the
Funzio acquistion
in 2012
Funzio was a
small mobile
gaming company
with 3 successful
games

7. Dedicated jobs and cron hosts
Deploying tarballs
Hadoop
EC2 Based Relays
EC2 Based Vertica
Management
Amazon EC2 memcache clusters
EC2 MySQL
EC2 Zeus load balancers
Provisioning hosts using Runbooks
Amazon ElastiCache
Amazon RDS
Elastic Load Balancing
Puppet
Jenkins for code deployments
Deploying packages (RPM, Docker)
Amazon Elastic Map Reduce (EMR)
Amazon Kinesis
Amazon Redshift
AWS

8. SSH Jump hosts and OpenSwan
Networking
ELK logging on EC2
Nagios on EC2
SumoLogic
DataDog
SaaS

9. Technical
Stack
EC2 AutoscalingCloudformation S3 ELB
Kinesis RDS Elasticache DynamoDB

10. 1 Leverage SaaS
2 Avoid building things you can buy
3 Infrastructure as code
1 Leverage AWS services
2 Automate everything
3 Immutable infrastucture
4 Servers are cattle not pets
DevOps
Principles

11. GAME STACK
DevOps at GREE – Architecture High Level Design
Amazon
S3
Amazon
Route 53
AWS
Directory
Service
AWS
Code
Commit
Active
Directory
GitHub
Enterprise
AWS
VPC

12. DevOps at GREE – Architecture Low Level Design
Service 1 Service 2Amazon
DynamoDB
ILB
ELB
GAME
Puppet
Jenkins
Aviatrix
Gateway
Auto Scaling
Auto Scaling Auto Scaling
VPC

13. Secure Remote User Access
GREE AWS Account
Region 1 Region 2
IPSEC
Aviatrix
Gateway
Aviatrix
Gateway
AWSAWS
Dragon Soul AWS Account
VPC VPC
Remote Users
(dev, devops, admins)
SSL VPN

14. Patterns of Enterprise Application Architecture
inspired by Martin Fowler’s Books

15. Codebase
• One codebase tracked in
revision control, many deploys
• Use Git
I

16. Dependencies
• Explicitly declare and isolate
dependencies
• Use Puppet
II

17. Config
• Store config in the environment
• Use Puppet with Hieradata
III

18. Backing Services
• Treat backing services as
attached resources
• Use Puppet with Hieradata to
capture endpoints
IV

19. Build,
release, run
• Strictly separate build and run
stages
• Use Jenkins to produce
package artifacts
• Use Jenkins pipeline to
orchestrate deployment
V

20. Processes
• Execute the app as one or
more stateless processes
• Use ELB
• STONITH
• https://en.wikipedia.org/wiki/ST
ONITH
VI

21. Port Binding
• Export services via port binding
• Use ILB
VII

22. Concurrency
• Scale out via the process
model
• Use ELB
VIII

23. Disposability
• Maximize robustness with fast
startup and graceful shutdown
• Use Auto Scaling
• Use custom AMIs
• Use ElastiCache or DynamoDB
• STONITH
IX

24. Dev/Prod
Parity
• Keep development, staging
and production as
similar as possible
• Use AWS CloudFormation
• Use Puppet
• Use Docker
X

25. Logs
• Treat logs as event streams
• Use Sumo logic
XI

26. Admin Processes
• Run admin/management tasks
as one-off processes
• Use Jenkins
XII

27. 14 DevOps engineers
Takeaways
Hundreds of monitoring pages
6
10

28. VPC History at GREE
VPC I
• AWS Classic
• EU-west-1 VPC only
• 6 week deadline,
things were rushed
• Poor choices and
bad shortcuts
VPC II
• Second attempt at
VPC: hire an expert
• Company mandate to
move from AWS Classic
to VPC
• 6 months of design, lots
of over engineering
VPC III
• Initial setup is less
than an hour
• Clean and simple
model

29. Demo video: Bringing a cloud controller online

30. Aviatrix - New VPC with AGW

31. GREE purchased the
DragonSoul game from PerBlue
During the due diligence phase,
Aviatrix was used to link
GREE to the VPC
Use Case

33. W1 W2 W3 W4 W10
RDS
Aviatrix
Gateway
World
Game
Backend
Systems
Game
Admin
VPC
Payment ADS
Original Deployment

34. Dragon Soul Deployment 1
Region 1 Region 2
IPSEC
Cross Account / Region Peering
Aviatrix Gateway Aviatrix Gateway
AWSAWS
Dragon Soul Deployment 2
VPC VPC

35. W11
Wn
RDS
11
ELB11
Aviatrix
Gateway
VPC ELB12 ELBn
W12
Auto Scaling
Group
Auto Scaling
Group
Auto Scaling
Group
RDS
12
RDS
n
Second Deployment

36. Challenges
Single points of failure
RDS db.r3.8xlarge at capacity
World servers are stateful and unwinding the Java ORM
will take time.
Goals
Grow the game fast: double the
number of players
Don’t rock the boat: maintain continuity in existing
game service

37. In Summary:
DevOps is not just technology, it’s people
using technology.
Docker is great
but it has some
sharp edges
Games have a
lifecycle,
nothing lasts
forever
CloudFormation
is great but it
is hard to get
started
Use dedicated
AWS sub-
accounts
Use Aviatrix
cloud
networking
People leave,
institutional
knowledge is lost,
so capture
everything in
automation
People take
shortcuts, magic
happens, so
enforce
automation

38. Thank you!