GAM305_Automating Mother Nature

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Automating Mother Nature
Sumin Byeon @NEXON
G A M 3 0 5
N o v e m b e r 2 7 , 2 0 1 7

The Beginning

(Unknown title) by Ivan T is licensed under CC-BY

“Ace bet business card 262333” by Pixabay is licensed under CC0

AUTOMATING MOTHER NATURE
ECS
S3 SQS
ECR
Game Servers

NEXON
• Headquartered in Seoul, South Korea
• Global offices in Japan, North America, Europe
• Specializes in online games for PC and mobile
• Services +100 titles in +190 countries

TODO: Insert the intro video

DURANGO
• Dinosaurs!
• An open-world MMORPG
• Survival, hunting, craftsmanship, farming, and collecting

We want to make our ecosystem
as realistic as possible

REALISTIC ECOSYSTEM
• Forests get destroyed by players
• Slowly replenished over time

GROWING FOREST
(Screenshot of evolving ecosystem)

FORMING VEGETATION
Factors to be considered:
• Temperature
• Humidity
• Fertility of soil
• Distance from water and other plants
• Biome type
• (and more)
“The distribution of vegetation types as a function of mean annual
temperature and precipitation” by Navarras is licensed under CC0

ECOSYSTEM SIMULATOR
Determines the type and the location of plants

WHY NOT PLANT TREES MANUALLY?
Islands
10,000
Estimated number of trees,
shrubs, rocks, mud pits, and so on
180M
Potential slots
to put something
+2.1B
And still growing…

WHY USE A DEDICATED SIMULATOR?
• A lot of matrix operations and other calculations
• Cannot be handled by a single node; multiple compute nodes are employed
• Independently scale in/out Ecosystem Simulator
• Game server written in Python
(Not an ideal language for CPU-bound tasks)

REPLENISHING WORKFLOW
Ecosystem
Simulator
Current Garden
State
Next Garden State
Game Server

GARDEN STATE
Tile coordinate Type of plant
Mango tree
Reed
Heliconia
x y entity_type
325 262 12852
462 53 13360
… … …
128 509 11559

Tile
284, 136

THE ARCHITECTURE
ECS
S3 SQS
ECR
Game Servers

PRODUCTION ENVIRONMENT
• Multiple machines running Ecosystem Simulator
• Software deployment can be tricky sometimes
• Operating system, libraries, and frameworks versions

NEED FOR SOLID BASE SYSTEM
(Some reliable base system)
Ecosystem
Simulator
Ecosystem
Simulator
Ecosystem
Simulator

SOLUTION: DOCKER

AMAZON EC2 CONTAINER SERVICE (ECS)
• Highly scalable, high performance container management service
• Managed cluster of Amazon EC2 instances
• Container deployment, scheduling, monitoring
• Cluster configuration, management
• Requires provisioning of underlying compute resources

PULLING FROM AMAZON ECR
ECS ECR
I need an Ecosystem
Simulator Docker image
Okay

AMAZON EC2 CONTAINER REGISTRY (ECR)
• Fully-managed Docker container registry
• Similar to Docker Hub
• IAM based access control
• No additional cost; pay for S3 storage and data transfer usage

REGIONAL OFFLOADING
• Initially thought the Amazon ECR must be in the same region as the Amazon
ECS (That was not the case!)
• Amazon ECR was only available in Virginia when we started working on this
• Game servers in Tokyo, Amazon ECS cluster in Virginia
• Virginia has 22% lower Amazon EC2 prices than Tokyo
• Ended up saving some cost in computing resources

COST SAVINGS
With on-demand instances, beta test in September 2017
$9,100/mo
Tokyo
$7,300/mo
Virginia
20%

INTER-REGIONAL DATA TRANSFER
$90/TB
$90,000/PB
$0.09 for each GB transferred

DATA TRANSFER ESTIMATION
3,510 GB, $193
Tokyo Virginia
Beta test in September 2017 (estimated)

DATA TRANSFER: VIRGINIA ➜ TOKYO
570 GB, $11
Beta test in September 2017

DATA TRANSFER: TOKYO ➜ VIRGINIA
111 TB, $10,000
“ashamed-sad-silhouette-face-hand” by OpenClipart-Vectors is licensed under CC0

C U R R E N TP A S T
ECR & ECS
Virginia
us-east-1
Tokyo
ap-northest-1
ECSECR
Virginia
us-east-1
ECSECR

DATA TRANSFER CHARGES
Virginia
us-east-1
Tokyo
ap-northest-1
ECSECR
Docker image size ≈ 1 GB
$0.09/GB
• Multiple ECS agents
• EC2 instances get launched,
terminated due to auto-scaling
Pulling Docker images 100,000 times? Highly unlikely.

THE REAL CAUSE
Tokyo
ap-northest-1
S3 (Terrains)
Virginia
us-east-1
Game Servers ECS
+3 TB/day
$0.09/GB
+$270/day

AMAZON S3 BUCKET REPLICATION
Tokyo
ap-northest-1
S3
Virginia
us-east-1
S3

AMAZON S3 BUCKET REPLICATION
Tokyo
ap-northest-1
S3
Virginia
us-east-1
S3 ECS
Only once Free

What triggers Amazon ECS tasks?

NO AUDIENCE, NO PLAY
• No observer, no action
(For example, animals pause when no one is looking at them)
• Conserves compute resources
• Ecosystem Simulator does not run on areas with no activity

SELECTIVE SIMULATION
• Vast lands, many islands
• Takes too much time to run Ecosystem Simulator everywhere
• Runs simulator selectively based on player activities

PLAYER ACTIVITIES
Triggers Ecosystem Simulator
at a particular probability

1 chunk = 16x16 tiles

RUN AMAZON ECS TASKS
run_ecs_task()
(Producer) (Consumer)
ECSGame Servers
Python boto3

FLUCTUATING DEMANDS
Computing Demands Computing Capacity

DestroyedReplenished

SOLUTION: AMAZON SQS + AWS LAMBDA
SQS ECSLambdaGame Servers

AMAZON SIMPLE QUEUE SERVICE (SQS)
• Fully managed messaging queue service
• Makes it simple to decouple application components
• Fault tolerant, easy to scale
• Optional server-side encryption

TWO TYPES OF QUEUES
• Standard queue
• At-least-once delivery
• Best-effort ordering
• Maximum throughput
• FIFO (first-in-first-out) queue
• Exactly-once delivery
• Exact ordering
• With limited throughput (300 transactions per second)

TWO TYPES OF QUEUES
• Standard queue
• At-least-once delivery
• Best-effort ordering
• Maximum throughput
• FIFO (first-in-first-out) queue
Exactly-once semantics is not a major concern
• Exactly-once delivery
• Exact ordering
• With limited throughput (300 transactions per second)

VIEW AS A MATH PROBLEM
Problem Solution
Out of order 𝑓 𝑥 ⨀𝑓 𝑦 = 𝑓(𝑦)⨀𝑓(𝑥) Communicative
Duplication 𝑓 𝑥 ⨀𝑓 𝑥 = 𝑓(𝑥) Idempotent

f(x)
f(y)
f(z)

ASYMPTOTIC GROWTH
Iterations
#ofnaturalobjects
lim
𝑖→∞
𝑁𝑖 − 𝑁𝑖−1 = 0
Marginal difference in each iteration

AMAZON SQS + AWS LAMBDA
Solves producer-consumer
speed mismatch issue

AMAZON SQS + AWS LAMBDA
?

AWS LAMBDA
• Serverless compute service
• Runs code without provisioning underlying compute resources
• Runs code in response to events
(API Gateway, Amazon S3, Amazon DynamoDB, and more)
• High availability, automatic scaling
• Pay by CPU-time and memory

INVOKING AWS LAMBDA
Lambda
API, Command
Event Source
or

AWS LAMBDA EVENT SOURCES
• API Gateway
• AWS IoT
• Amazon CloudWatch Events
• Amazon CloudWatch Logs
• AWS CodeCommit
• Amazon Cognito Sync Trigger
• Amazon DynamoDB
• Amazon Kinesis
• Amazon S3
• Amazon SNS
• Amazon SQS is not on here

How do we trigger Lambda from Amazon SQS?

SOLUTION 1
• CloudWatch alarm cron task that runs every minute
• Amazon SQS poller in a Lambda function
SQSLambdaCloudWatch Alarm
Every minute

SOLUTION 2
• CloudWatch alarm on ApproximateNumberOfMessagesVisible >= 1
• The alarm publishes to an SNS topic that triggers a Lambda function
• Amazon SQS poller in a Lambda function
SQSLambdaCloudWatch alarm SNS

AMAZON SIMPLE NOTIFICATION SERVICE (SNS)
• Fully managed pub/sub messaging and notifications service
• Sends messages to various endpoints
(for example, Lambda, SQS, HTTP/S, email, SMS)
• Sends notifications to mobile devices
• Fan out messages to a large number of subscribers

PASSING METADATA
• .yml files containing
• Properties of natural objects
• Point of interests (POIs), terrain specifications
• Current garden state
• Describes where plants are located

LIMITATIONS OF AMAZON SQS
• Retention period of one to 14 days (four days by default)
• Maximum of 256 KB per message
• 120,000 messages in-flight for a standard queue (20,000 for a FIFO queue)

LIMITATIONS OF AMAZON SQS
• Retention period of one to 14 days (four days by default)
• Maximum of 256 KB per message
The total size of metadata and garden state often exceeds this limit
• 120,000 messages in-flight for a standard queue (20,000 for a FIFO queue)

AMAZON S3 AS TEMPORARY STORAGE
ECSS3 SQSGame Servers
Metadata, garden state Replenish requests

PROBLEM: EVER-GROWING S3 BUCKET
Grows indefinitely

AMAZON S3 LIFECYCLE MANAGEMENT
• Transition: Move objects to another storage class
• Standard
• Standard—infrequent access (IA)
• Glacier
• Expiration: Delete objects
• Based on user-defined rules

OBJECT EXPIRATION
• Object expiration policy with prefix matching
• Deletes objects
• Cleans up incomplete multipart uploads
• Runs once a day
• We use Amazon S3 as temporary storage,
delete any object older than a day

OBJECT AUTO-EXPIRATION
Proportional to
player activities

OBJECT TRANSITION
• Move objects to different storage class
• Standard-IA
• 25% lower cost
• Lower availability (99.9% vs 99%)
• Glacier
• 80% lower cost
• First-byte latency ranges from minutes to hours

Getting the results back

CALLBACK
HTTP POST
x y entity_type
325 262 12852
462 53 13360
… … …
128 509 11559
ECS
Garden State
Game Servers

CALLBACK
Virginia
us-east-1
Tokyo
ap-northest-1
HTTP over VPN
ECSGame Servers
No public HTTP
endpoint

CALLBACK
• Every test environment must have an external HTTP endpoint
• Difficult in some cases; virtual machine, corporate network
• Especially game designers and other non-technical people struggle with it
• Openswan instance ➜ potential SPOF

LOCAL DEVELOPMENT ENVIRONMENT
Windows
Ubuntu Linux
CorporateFirewall
• Virtual machine
• Port forwarding
• Security policy
• Manager approval
• Talking to IT department
ECS

POTENTIAL SINGLE POINT OF FAILURE
Virginia
us-east-1
Tokyo
ap-northest-1
Virtual Private
Gateways
Virtual Private
Gateways
EC2 Instance with
Openswan
SPOF!

POLLING MODEL
ECSS3 SQS
Periodically
Metadata, garden state
Game Servers

• No direct communication between the game servers and Amazon ECS cluster
• Game servers and Amazon ECS cluster need not be in the same VPC
• No VPN
• No external HTTP endpoint
• Each component can operate, scale, and fail independently
• No SPOF
POLLING MODEL

ECSS3 SQS
ECSLambdaGame Servers S3 SQS
Replenish Request
Replenish Response
Game Servers

Replenish Request
Replenish Response
ECSS3 SQS
ECSLambdaS3 SQSGame Servers
Game Servers

Dealing with over-provision and under-provision

Life was much simpler when Amazon was just a forest
“Amazon Manaus forest” by Phil P Harris is licensed under CC-BY-SA
(but, perhaps much more difficult)

AMAZON ELASTIC COMPUTE CLOUD (EC2)
• Launch or terminate instances with a few button clicks or an API call
• No up-front cost for purchasing, maintaining on-premises hardware
• Provisioning the right amount of computing resources is still a challenge

AUTO SCALING
• Amazon SQS is good for smoothing out short-term spikes
What if we have long-term computing resource shortage?
What if we provisioned too many Amazon EC2 instances?
• Builds systems that respond to changes in demand
• Launches or terminates AWS resources based on user-defined conditions

AUTO SCALING

Taking cost reduction measures even further

SPOT INSTANCES
• Bidding on spare Amazon EC2 compute capacity
• Customers set the maximum bidding price
• Significant lower price than on-demand instances
• Spot instances launch when the price falls below the bidding price
• Price changes in real time, based on supply and demand

SUDDEN INTERRUPTIONS
• Spot instances terminate when the price goes above the bidding price
• Good for time-flexible and interruption-tolerant tasks
• Your architecture must be able to deal with sudden interruptions
• May not be suitable for mission-critical systems
• We use a combination of on-demand and spot instances
• They may belong to a single ECS cluster
• Guarantee some instances are running all the time

COST SAVED WITH SPOT INSTANCES
78%
$7,300 $1,600

RESERVED INSTANCES
• Instance type, availability zone, platform
(For example, m4.xlarge, us-east-1a, Ubuntu Linux)
• One-year or three-year contract terms
• 30–60% discounted rates
• Various payment plans
(No up-front, partial up-front, full up-front)

Recap

Game Servers
S3 SQS
• Provides a solid infrastructure to run Docker containers
• Fully managed
• Container deployment, scheduling, monitoring
• Cluster configuration, management
• Elastic computing capacity
with Auto Scaling
• Need to provision
underlying compute resources
AMAZON EC2 CONTAINER SERVICE (ECS)
ECS
ECR

ECS
ECR
Game Servers
AMAZON SIMPLE QUEUE SERVICE (SQS)
• Buffer between producer and consumer
• Decouple application components
• Build fault-tolerant, highly scalable systems
S3 SQS

ECS
ECR
Game Servers
• Object lifecycle management to move or delete objects
based on user-defined rules
• Bucket replication
AMAZON SIMPLE STORAGE SERVICE (S3)
S3 SQS

AWS LAMBDA
• Think about your code, nothing else
(Servers, capacity, deployment, scalability, availability, operating systems,
language updates)
• Some limitations apply
(Time, memory usage, languages runtimes)

COST REDUCTION
• Auto Scaling (Prevent over-provisioning)
• Spot instances (Bidding on spare EC2 instances)
• Reserved instances (Long-term contract to get a discounted rate)
• Minimize inter-regional data transfer

COST REDUCTION
• Auto Scaling
• Spot instances
• Reserved instances
• Minimize inter-regional data transfer
“World Map” by molumen is licensed under the conditions of unlimited commercial use

THANK YOU!

GAM305_Automating Mother Nature

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to GAM305_Automating Mother Nature

Similar to GAM305_Automating Mother Nature (20)

More from Amazon Web Services

More from Amazon Web Services (20)

GAM305_Automating Mother Nature