Web Consumables

WEB CONSUMABLES
can burak çilingir @ xxxxxxx——-
and opportunities

YOUR EXPECTATION
{This should happen in..
• INSTANT
• ON CLICK
• IMMEDIATELY
• EVEN IF INTERNET 
IS SLOW!

E
la
s
tic
L
o
a
d
B
a
la
n
c
in
g
A
E
la
s
tic
L
o
a
d
B
a
la
n
c
in
g
B
Amazon
R
o
u
te
)
z
Amazon
C
lo
u
d
F
ro
n
t
Amazon
S
z
A
u
to
S
c
a
lin
g
A
u
to
S
c
a
lin
g
A
m
a
zo
n
E
C
6
A
m
a
zo
n
E
C
6A
u
to
S
c
a
lin
g
A
u
to
S
c
a
lin
g
A
m
a
zo
n
E
C
6
A
m
a
zo
n
E
C
6
Database
Servers
Load
Balancer
Load
Balancer
Web
ServersWeb
Servers
Application
ServersApplication
Servers
Application
ServersApplication
Servers
A
m
a
zo
n
R
D
S
M
u
lti-A
Z
S
ta
n
d
b
yA
m
a
zo
n
R
D
S
M
u
lti-A
Z
S
ta
n
d
b
y
S
y
n
c
h
ro
n
o
u
s
R
e
p
lic
a
tio
n
E
la
s
tic
L
o
a
d
B
a
la
n
c
in
g
A
m
a
zo
n
R
D
S
M
a
s
te
r
A
m
a
zo
n
R
D
S
M
a
s
te
r
Resources and
Static Content
Content
Delivery
Network
DNS Resolution
Web
ServersWeb
Servers
Highly available and scalable web hosting can be complex and
expensiveZ Dense peak periods and wild swings in traffic patterns
result in low utilization of expensive hardwareZ Amazon Web
Services provides the reliableR scalableR secureR and high-
performance infrastructure required for web applications while
enabling an elasticR scale-out and scale-down infrastructure to
match IT costs in real time as customer traffic fluctuatesZ
System
Overview
WEB APPLICATION
HOSTING
A
m
a
zo
n
R
o
u
te
)
z
A
m
a
zo
n
S
z
A
m
a
zo
n
E
C
6
E
la
stic
L
o
a
d
B
a
la
n
cin
g
Amazon CloudFront
AWS
Reference
Architectures
A
u
to
S
c
a
lin
g
A
m
a
zo
n
R
D
S
7
6
z
(
) 7
6
6
6
(
)
z
7
7
7
The userBs DNS requests are served by Amazon Route
)zR a highly available Domain Name System 'DNSN
serviceZ Network traffic is routed to infrastructure running in
Amazon Web ServicesZ
StaticR streamingR and dynamic content is delivered by
Amazon CloudFrontR a global network of edge
locationsZ Requests are automatically routed to the nearest
edge locationR so content is delivered with the best possible
performanceZ
HTTP requests are first handled by Elastic Load
BalancingR which automatically distributes incoming
application traffic among multiple Amazon Elastic Compute
Cloud OEC6P instances across Availability Zones 'AZsNZ It
enables even greater fault tolerance in your applicationsR
seamlessly providing the amount of load balancing capacity
needed in response to incoming application trafficZ
Web servers and application servers are deployed on
Amazon ECj instancesZ Most organizations will select
an Amazon Machine Image OAMIP and then customize it to
their needsZ This custom AMI will then become the starting
point for future web developmentZ
Web servers and application servers are deployed in an
Auto Scaling groupZ Auto Scaling automatically adjusts
your capacity up or down according to conditions you defineZ
With Auto ScalingR you can ensure that the number of
Amazon EC6 instances you’re using increases seamlessly
during demand spikes to maintain performance and
decreases automatically during demand to minimize costsZ
To provide high availabilityR the relational database that
contains applicationBs data is hosted redundantly on a
multi-AZ 'multiple Availability Zones–zones A and B hereN
deployment of Amazon Relational Database Service
'Amazon RDSNZ
Resources and static content used by the web
application are stored on Amazon Simple Storage
Service OSzPR a highly durable storage infrastructure
designed for mission-critical and primary data storageZ

satisfactory player experience. Amazon Web Services provides
different tools and services that can be used for building online
games that scale under high usage traffic patterns.
This document presents a cost-effective online game architecture
featuring automatic capacity adjustment, a highly available and
high-speed database, and a data processing cluster for player
behavior analysis.
System
Overview
ONLINE
GAMES
A
m
a
z
o
n
E
C
2
E
lastic
Load
B
alancing
A
m
azon
D
yn
am
o
D
B
A
m
a
z
o
n
E
M
R
A
u
to
S
c
a
lin
g
AWS
Reference
Architectures
A
m
a
z
o
n
S
3
Online games back-end infrastructures can be challenging to
maintain and operate. Peak usage periods, multiple players, and
high volumes of write operations are some of the most common
problems that operations teams face.
But the most difficult challenge is ensuring flexibility in the scale of
that system. A popular game might suddenly receive millions of
users in a matter of hours, yet it must continue to provide a
______
A
m
a
z
o
n
S
E
S
1 Browser games can be represented as client-server
applications. The client generally consists of static files,
such as images, sounds, flash applications, or Java applets.
Those files are hosted on Amazon Simple Storage Service
(Amazon S3), a highly available and reliable data store.
5 Log files generated by each web server are pushed
back into Amazon S3 for long-term storage.
2 As the user base grows and becomes more
geographically distributed, a high-performance cache
like Amazon CloudFront can provide substantial
improvements in latency, fault tolerance, and cost. By using
Amazon S3 as the origin server for the Amazon CloudFront
distribution, the game infrastructure benefits from fast
network data transfer rates and a simple publishing/caching
workflow.
3 Requests from the game application are distributed by
Elastic Load Balancing to a group of web servers
running on Amazon Elastic Compute Cloud (Amazon EC2)
instances. Auto Scaling automatically adjusts the size of this
group, depending on rules like network load, CPU usage, and
so on.
4 Player data is persisted on Amazon DynamoDB, a
fully managed NoSQL database service. As the player
population grows, Amazon DynamoDB provides predictable
performance with seamless scalability.
A
m
a
z
o
n
R
o
u
te
5
3
6 Managing and analyzing high data volumes produced
by online games platforms can be challenging. Amazon
Elastic MapReduce (Amazon EMR) is a service that
processes vast amounts of data easily. Input data can be
retrieved from web server logs stored on Amazon S3 or from
player data stored in Amazon DynamoDB tables to run
analytics on player behavior, usage patterns, etc. Those
results can be stored again on Amazon S3, or inserted in a
relational database for further analysis with classic business
intelligence tools.
7 Based on the needs of the game, Amazon Simple
Email Service (Amazon SES) can be used to send
email to players in a cost-effective and scalable way.
A
m
azon
C
lo
u
d
F
ro
n
t
w
w
w
.m
ygam
e
.co
m
A
m
azo
n
R
o
u
te
5
3
DNS
Resolution
A
m
a
z
o
n
D
y
n
a
m
o
D
B
G
am
e
in
te
ractio
n
(statu
s,JSO
N
,...)
A
u
to
Scalin
g
A
u
to
Scalin
g
Elastic
Lo
ad
B
alan
cin
g
Web
Servers
A
m
azo
n
C
lo
u
d
Fro
n
t
Content
Delivery
Network
A
m
azo
n
S3
G
am
e
file
s
(flash
,ap
p
let,...)
Files
Repository
Game
Database
A
m
azo
n
Elastic
M
ap
R
e
d
u
ce
Game
Analysis
log files
4
5
logfiles
Game
clientfiles
7
A
m
azo
n
SES
●
6
Players
Email
Emitter
logfiles
2
1
3

Customers want to find the products they are interested in quickly,
and they expect pages to load quickly. Worldwide customers want
to be able to make purchases at any time, so the website should
be highly available. Meeting these challenges becomes harder as
your catalog and customer base grow.
With the tools that AWS provides, you can build a compelling,
scalable website with a searchable product catalog that is
accessible with very low latency.
System
Overview
E-COMMERCE
WEB SITE
PART 1: WEB FRONT-END
A
m
azon
R
o
u
te
53
A
m
azon
D
yn
am
o
D
B
A
m
azon
E
lastiC
ach
e
A
W
S
E
lastic
B
eanstalk
AWS
Reference
Architectures
A
m
azon
S
3
With Amazon Web Services, you can build a highly available e-
commerce website with a flexible product catalog that scales with
your business.
Maintaining an e-commerce website with a large product catalog
and global customer base can be challenging. The catalog should
be searchable, and individual product pages should contain a rich
information set that includes, for example, images, a PDF manual,
and customer reviews.
A
m
azon
C
lo
u
d
F
ro
n
t
1 DNS requests to the e-commerce website are handled
by Amazon Route 53, a highly available Domain Name
System (DNS) service.
5 Amazon DynamoDB is a fully-managed, high
performance, NoSQL database service that is easy to
set up, operate, and scale. It is used both as a session store
for persistent session data, such as the shopping cart, and as
the product database. Because DynamoDB does not have a
schema, we have a great deal of flexibility in adding new
product categories and attributes to the catalog.
2 Amazon CloudFront is a content distribution network
(CDN) with edge locations around the globe. It can
cache static and streaming content and deliver dynamic
content with low latency from locations close to the customer.
3 The e-commerce application is deployed by AWS
Elastic Beanstalk, which automatically handles the
details of capacity provisioning, load balancing, auto scaling,
and application health monitoring.
4 Amazon Simple Storage Service (Amazon S3) stores
all static catalog content, such as product images,
manuals, and videos, as well as all log files and clickstream
information from Amazon CloudFront and the e-commerce
application.
6 Amazon ElastiCache is used as a session store for
volatile data and as a caching layer for the product
catalog to reduce I/O (and cost) on DynamoDB.
7 Product catalog data is loaded into Amazon
CloudSearch, a fully managed search service that
provides fast and highly scalable search functionality.
8 When customers check out their products, they are
redirected to an SSL-encrypted checkout service.
9 A marketing and recommendation service consumes
log data stored on Amazon S3 to provide the customer
with product recommendations.
A
m
azon
C
lo
u
d
S
earch
Custom
er
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
m
a
z
o
n
C
lo
u
d
F
ro
n
t
A
m
a
z
o
n
R
o
u
te
5
3
1
6
A
m
a
z
o
n
E
la
s
tiC
a
c
h
e
5
9
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
m
a
z
o
n
C
lo
u
d
S
e
a
rc
h
A
m
a
z
o
n
S
3
A
m
a
z
o
n
D
y
n
a
m
o
D
B
LOGS
MARKETING AND
RECOMMENDATION
SERVICE
Part
3
CHECKOUT
SERVICE
Part
2
E-commerce
Application
Recommendation
Web Service
Recommendation
Web Service
Catalog Cache &
Transient Session
Store
Search
Engine
Product Catalog &
Persistent Session
Store
Checkout
Application
Checkout
Application
Log File Repository &
Static Catalog Content
DNS
2
3
4
7
8
Secure
Connection
Secure
Connection

Customers expect their private data, such as their purchase
history and their credit card information, to be managed on a
secure infrastructure and application stack. AWS has achieved
multiple security certifications relevant to e-commerce business,
including the Payment Cards Industry (PCI) Data Security
Standard (DSS).
With the tools that AWS provides, you can build a secure checkout
service that manages the purchasing workflow from order to
fulfillment.
System
Overview
A
m
a
z
o
n
V
P
C
A
m
azon
S
E
S
Amazon EC2
E
la
s
tic
B
e
a
n
s
ta
lk
AWS
Reference
Architectures
A
m
a
z
o
n
R
D
S
With Amazon Web Services, you can build a secure and highly
available checkout service for your e-commerce website that
scales with your business. Managing the checkout process
involves many steps, which have to be coordinated. Some steps,
such as credit card transactions, are subject to specific regulatory
requirements. Other parts of the process involve manual labor,
such as picking, packing, and shipping items from a warehouse.
Amazon SW
F
1 The e-commerce web front end redirects the customer
to an SSL-encrypted checkout application to
authenticate the customer and execute a purchase.
5 SWF Workers are deployed on Amazon EC2
instances within a private subnet. The EC2 instances
are part of an Auto Scaling group, which can scale in and
out according to demand. The Workers manage the different
steps of the checkout pipeline, such as validating the order,
reserving and charging the credit card, and triggering the
sending of order and shipping confirmation emails.
2 The checkout application, which is deployed by AWS
Elastic Beanstalk, uses Amazon Simple Workflow
Service (Amazon SWF) to authenticate the customer and
trigger a new order workflow.
3 Amazon SWF coordinates all running order workflows
by using SWF Deciders and SWF Workers.
4 The SWF Decider implements the workflow logic. It
runs on an Amazon Elastic Compute Cloud (Amazon
EC2) instance within a private subnet that is isolated from the
public Internet.
6 SWF Workers can also be implemented on mobile
devices, such as tablets or smartphones, in order to
integrate pick, pack, and ship steps into the overall order
workflow.
7 Amazon Simple Email Service (Amazon SES) is used
to send transactional email, such as order and shipping
confirmations, to the customer.
8 To provide high availability, the customer and orders
databases are hosted redundantly on a multi-AZ (multi
Availability Zone) deployment of Amazon Relational
Database Service (Amazon RDS)within private subnets that
are isolated from the public Internet.
E-COMMERCE
WEB SITE
PART 2: CHECKOUT SERVICE
●
7
2
1
5
4
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
WEB FRONT-END
Part
1 E-Commerce
Application
E-Commerce
Application
Custom
er
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
m
a
z
o
n
S
E
S
Checkout
Application
Email
Service
Customers & Orders
Database
Mobile Workers
(in warehouse)
A
u
to
S
c
a
lin
g
A
u
to
S
c
a
lin
g
A
m
a
z
o
n
S
W
F
A
m
a
z
o
n
R
D
S
M
a
s
te
r
A
m
a
z
o
n
R
D
S
M
u
lti-A
Z
S
ta
n
d
b
y
Workers
Workers
Decider
Decider
Order Emails
6
8
Workflow
Service 3

CHECKOUT
SERVICE
Part
2
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
●
A
m
a
z
o
n
S
E
S
7
5
3
A
m
a
z
o
n
R
D
S
R
e
a
d
R
e
p
lic
a
A
m
a
z
o
n
E
la
s
tic
M
a
p
R
e
d
u
c
e
A
m
a
z
o
n
D
y
n
a
m
o
D
B
A
m
a
z
o
n
S
3A
m
a
z
o
n
S
3
1
Email
Service
Marketing
Mgmt App
User
Profiles
A
m
a
z
o
n
R
D
S
M
a
s
te
r
A
m
a
z
o
n
R
D
S
M
a
s
te
r
Customer
& Orders DB
Customer
& Orders DB
Log File
Repository
Log File
Repository
Recommendation
Web Service
Customer
& Orders DB
Read Replica
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
A
W
S
E
la
s
tic
B
e
a
n
s
ta
lk
WEB FRONT-END
Part
1
E-commerce
Application
E-commerce
Application
Marketing
Emails
Marketing
Manager
Custom
ers
46
2
The insights that you gain about your customers can also be used
to manage personalized marketing campaigns targeted at specific
customer segments.
With the tools that AWS provides, you can build highly scalable
recommendation services that can be consumed by different
channels, such as dynamic product recommendations on the
e - commerce website or targeted email campaigns for your
customers.
System
Overview
E-COMMERCE
WEBSITE
PART 3: MARKETING & RECOMMENDATIONS
A
m
a
z
o
n
E
M
R
A
m
a
z
o
n
S
E
S
A
W
S
E
lastic
B
ean
stalk
A
W
S
E
lastic
B
eanstalk
AWS
Reference
Architectures
A
m
a
z
o
n
R
D
S
With Amazon Web Services, you can build a recommendation and
marketing service to manage targeted marketing campaigns and
offer personalized product recommendations to customers who
are browsing your e-commerce site.
In order to build such a service, you have to process very large
amounts of data from multiple data sources. The resulting user
profile information has to be available to deliver real-time product
recommendations on your e-commerce website.
A
m
a
z
o
n
S
3
1 Amazon Elastic MapReduce (Amazon EMR) is a
hosted Hadoop framework that runs on Amazon Elastic
Compute Cloud (Amazon EC2) instances. It aggregates and
processes user data from server log files and from the
customer´s purchase history.
5 A recommendation web service used by the web front
end is deployed by AWS Elastic Beanstalk. This
service uses the profile information stored on Amazon
DynamoDB to provide personalized recommendations to be
mm
shown on the e-commerce web front end.
2 An Amazon Relational Database Services (Amazon
RDS) Read Replica of customer and order databases is
used by Amazon EMR to compute user profiles and by
Amazon Simple Email Service (Amazon SES) to send
targeted marketing emails to customers.
3 Log files produced by the e-commerce web front end
have been stored on Amazon Simple Storage Service
(Amazon S3) and are consumed by the Amazon EMR cluster
to compute user profiles.
4 User profile information generated by the Amazon EMR
cluster is stored in Amazon DynamoDB, a scalable,
high-performance managed NoSQL database that can serve
recommendations with low latency.
6 A marketing administration application deployed by
AWS Elastic Beanstalk is being used by marketing
managers to send targeted email campaigns to customers
with specific user profiles. The application reads customer
email addresses from an Amazon RDS Read Replica of the
customer database.
7 Amazon SES is used to send marketing emails to
customers. Amazon SES is based on the scalable
technology used by Amazon web sites around the world to
send billions of messages a year.
A
m
azon
D
yn
am
o
D
B

A
W
S
D
ata P
ip
elin
e
This elasticity is achieved by using Auto Scaling groups for ingest
processing, AWS Data Pipeline for scheduled Amazon Elastic
MapReduce jobs, AWS Data Pipeline for intersystem data
orchestration, and Amazon Redshift for potentially massive-scale
analysis. Key architectural throttle points involving Amazon SQS
for sensor message buffering and less frequent AWS Data
Pipeline scheduling keep the overall solution costs predictable and
controlled.
System
Overview
TIME SERIES
PROCESSING
A
m
azon
E
C
2
A
m
azon
E
M
R
A
m
azon
D
yn
am
o
D
B
A
W
S
D
a
ta
P
ip
e
lin
e
A
u
to
S
c
a
lin
g
AWS
Reference
Architectures
A
m
azon
S
3
When data arrives as a succession of regular measurements, it is
known as time series information. Processing of time series
information poses systems scaling challenges that the elasticity of
AWS services is uniquely positioned to address.
A
m
azon
S
Q
S
A
m
azon
E
C
2
S
p
o
t
2 Send messages to an Amazon Simple Queue Service
queue for processing into Amazon DynamoDB using
autoscaled Amazon EC2 workers. Or, if the sensor source
can do so, post sensor samples directly to Amazon
DynamoDB. Try starting with a DynamoDB table that is a
week-oriented, time-based table structure.
2
1
6
3
3 If a Supervisory Control and Data Acquisition (SCADA)
system exists, create a flow of samples to or from
Amazon DynamoDB to support additional cloud processing
or other existing systems, respectively.
4 Using AWS Data Pipeline, create a pipeline with a
regular Amazon Elastic MapReduce job that both
calculates expensive sample processing and delivers
samples and results.
4
7
7 The pipeline also optionally exports results in a
format custom applications can accept.
Corporate
Data Center
A
m
a
z
o
n
S
Q
S
A
m
a
z
o
n
D
y
n
a
m
o
D
B
A
u
to
S
c
a
lin
g
Worker
Nodes
S
e
n
s
o
r
S
a
m
p
le
d
D
a
ta
SCADA
AmazonS3
R
e
m
o
te
S
e
n
s
o
r
M
e
s
s
a
g
e
s
A
m
a
z
o
n
Elastic
MapReduce
+EC2 Spot Instances
A
m
a
z
o
n
R
e
d
s
h
ift
5
Custom
Application
5 The pipeline places results into Amazon Redshift for
additional analysis.
8 Amazon Redshift optionally imports historic
samples to reside with calculated results.
9 Using in-house or Amazon partner business
intelligence solutions, Amazon Redshift supports
additional analysis on a potentially massive scale.
1 Remote devices such as power meters, mobile clients,
ad-network clients, industrial meters, satellites, and
environmental meters measure the world around them and
send sampled sensor data as messages via HTTP(S) for
processing.
6 The pipeline exports historical week-oriented
sample tables, from Amazon DynamoDB to
Amazon Simple Storage Service (Amazon S3)
Business
Intelligence
User
8
9
A
m
a
z
o
n
E
C
2

PROBLEMS SUMMARISED
• So many types of clients
• So many users
• Low latency expectance
• So many servers
• Many data-centers
• So many services
• Inter-service communication
• Inter-dc communication
• Service consumer to dc
routing
• Low latency expectance!
W
H
AT
ABO
UT
D
EPLO
YM
EN
T

DEVELOPMENT
• Developers develop
• They need to develop together
• They need to see how their code works together
• Customers need to see what is happening
• Staging, testing

DEPLOYMENT
Developed code must be deployed to…

DEPLOYMENT
• Rolling to a subset of servers
• Testing new features by selectively enabling trafﬁc
• Pulling back
• Tools!

POSTGRESQL
• SQL
• Stores data

ELASTICSEARCH
• stores structured data:
• {a: 1, b:2, c:3}
• output: immediate reply to
searches
• all documents where a>1

JENKINS
• deploys code
• runs code when code
changes

DOCKER
• packages the application
• highest level of dependancy
management
• eases deployment/scales
dractically

GIT - GITHUB
• code management
• discussion on bugs
• project management

DJANGO
• framework
• building web services
• web apps

NAEMON
• monitoring anything
• even temperature
• graphing
• spotting trends
• why mornings dc is hot?

PUPPET
• mass server management
• ensures clocks are ticking
• services are running

GRAYLOG
• applications log internal
data
• analyzes, monitors logs

REDIS
• stores data structures
• provides immediate access
• no searching
• caching

AMAZON WEB SERVICES
• datacenter on cloud
• services on cloud
• everything on cloud

SOME NUMBERS
6 digits ($) hosting invoice
~200k RPM
more than 500 servers
You can compare us with others:
https://aws.amazon.com/solutions/case-studies/all/
ik@metglobal.com

CAN BURAK ÇİLİNGİR
can@canb.net

Web Consumables

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Web Consumables

Similar to Web Consumables (20)

Recently uploaded

Recently uploaded (20)

Web Consumables