What if you could devote 30% more of
your resources to your customers?
IT Map - Traditional IT
E-mail, Productivity,
Collaboration, HR,
Finance, ERP
Desktop Support, Device
Management, Telephony,
IT Support
Information Security, CISO
Encryption, Key Management, Identity
Management, Firewalls, IDS, DDoS
Business Applications
Digital Products, Brand
Websites, Mobile
Applications, Point of Sale
Systems, Commerce
Corporate Applications End User Computing
Infrastructure Servers, Storage, Networking, Databases,
Data Warehousing, Data Centers
IT Map - Traditional IT with AWS
Information Security, CISO
Corporate Applications End User Computing
Infrastructure
Business Applications
AWS Elastic Beanstalk,
AWS Lambda, Amazon SQS,
Amazon SNS, Amazon
Mobile Analytics, Amazon
CloudFront
Amazon WorkMail,
Amazon WorkDocs, AWS
Marketplace, AWS
Directory Service, SaaS
Amazon WorkSpaces,
Amazon AppStream, AWS
Marketplace, AWS Mobile
Services, SaaS
AWS Identity and Access Management
(IAM), AWS CloudHSM, AWS Key
Management Service (AWS KMS),
Security Groups, AWS Marketplace
Amazon EC2, Amazon S3, Amazon RDS, Amazon VPC,
Amazon Direct Connect, Directory Service, IAM, AWS
Service Catalog
What are containers?
• OS virtualization
• Process isolation
• Automation
• ImagesServer
Guest OS
Bins/Libs Bins/Libs
App2App1
Common Customer Challenges/Desires
• Cluster Management
• Configuration Management
• Availability
• Scalability (application and repository)
• Scheduling
• Monitoring
• AWS integration
– VPC, ELB, Auto Scaling, CloudWatch, etc.
Amazon EC2 Container Service
• Building Block Service
• Cluster Management Made
Easy
• Flexible Scheduling
• Performance at Scale
• Security
• Extensible
Yes, you can do with EC2 instances…
Amazon S3 Bucket Events
Original image Thumbnailed image
1
2
3
Application
Monitoring
Security
Deploy
High
Availability Scalability
What is AWS Lambda?
AWS Lambda is a compute service that runs your code in
response to events such as image uploads, in-app activity,
website clicks, or outputs from connected devices.
Data Triggers: Amazon S3
Amazon S3 Bucket Events AWS Lambda
Original image Thumbnailed image
1
2
3
Three types of data-driven development
Retrospective
analysis and
reporting
Here-and-now
real-time processing
and dashboards
Predictions
to enable smart
applications
Amazon Kinesis
Amazon EC2
AWS Lambda
Amazon Redshift,
Amazon RDS
Amazon S3
Amazon EMR
Machine learning and smart applications
Machine learning is the technology that
automatically finds patterns in your data
and uses them to make predictions for new
data points as they become available
Your data + machine learning = smart applications
Building smart applications – a counter-pattern
Dear Thomas,
This awesome quadcopter is on sale
for just $49.99!
Smart applications by counter-example
SELECT c.ID
FROM customers c
LEFT JOIN orders o
ON c.ID = o.customer
GROUP BY c.ID
HAVING o.date > GETDATE() – 30
We can start by
sending the offer to
all customers who
placed an order in
the last 30 days
Smart applications by counter-example
SELECT c.ID
FROM customers c
LEFT JOIN orders o
ON c.ID = o.customer
GROUP BY c.ID
HAVING
AND o.date > GETDATE() – 30
… let’s narrow it
down to just
customers who
bought toys
Smart applications by counter-example
SELECT c.ID
FROM customers c
LEFT JOIN orders o
ON c.ID = o.customer
GROUP BY c.ID
HAVING o.category = ‘toys’
AND
(COUNT(*) > 2
AND
)
… and expand the
query to customers
who purchased other
toy helicopters
recently
Smart applications by counter-example
SELECT c.ID
FROM customers c
LEFT JOIN orders o
ON c.ID = o.customer
GROUP BY c.ID
HAVING o.category = ‘toys’
AND
(COUNT(*) > 2
AND SUM(o.price) > 200
AND o.date > GETDATE() – 30)
)
Use machine learning
technology to learn
your business rules
from data!
Why aren’t there more smart applications?
1. Machine learning expertise is rare
2. Building and scaling machine learning
technology is hard
3. Closing the gap between models and
applications is time-consuming and
expensive
Introducing Amazon Machine Learning
Easy to use, managed machine learning
service built for developers
Robust, powerful machine learning
technology based on Amazon’s internal
systems
Create models using your data already
stored in the AWS cloud
Deploy models to production in seconds
Batch predictions with Amazon Redshift
Structured data
In Amazon Redshift
Load predictions into
Amazon Redshift
-or-
Read prediction results
directly from S3
Predictions
in S3
Query for predictions with
Amazon ML batch API
Your application
Real-time predictions for interactive applications
Your application
Query for predictions with
Amazon ML real-time API
Unconstrained Big Data Growth
• IT/Application server logs
IT Infrastructure logs, Metering, Audit logs,
Change logs
• Websites/Mobile apps/Ads
Clickstream, User Engagement
• Sensor data/IoT
Weather, Smart Grids, Wearables
• Social media, user content
450MM+ Tweets/day
GB
TB
PB
ZB
EB
Current DB architectures are monolithic
Multiple layers of
functionality all on a
single box
SQL
Transactions
Caching
Logging
Current DB architectures are monolithic
Even when you scale
it out, you’re still
replicating the same
stack
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Application
Current DB architectures are monolithic
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Storage
Application Even when you scale
it out, you’re still
replicating the same
stack
This is a problem.
For cost. For flexibility. And for availability.
Reimagining the relational database
What if you were inventing the database today?
You wouldn’t design it the way we did in 1970. At least not entirely.
You’d build something that can scale out, that is self-healing, and that
leverages existing AWS services.
Aurora storage
• Highly available by default
– 6-way replication across 3 AZs
– 4 of 6 write quorum
• Automatic fallback to
3 of 4 if an AZ is unavailable
– 3 of 6 read quorum
• SSD, scale-out, multi-tenant storage
– Seamless storage scalability
– Up to 64 TB database size
– Only pay for what you use
• Log-structured storage
– Many small segments, each with
their own redo logs
– Log pages used to generate data pages
– Eliminates chatter between database and
storage
SQL
Transactions
AZ 1 AZ 2 AZ 3
Caching
Amazon S3
Self-healing, fault-tolerant
• Lose two copies or an AZ failure without read or write availability impact
• Lose three copies without read availability impact
• Automatic detection, replication, and repair
SQL
Transactio
n
AZ 1 AZ 2 AZ 3
Caching
SQL
Transactio
n
AZ 1 AZ 2 AZ 3
Caching
Read and write availabilityRead availability
Survivable caches
• We moved the cache out of
the database process
• Cache remains warm in the
event of a database restart
• Lets you resume fully loaded
operations much faster
• Instant crash recovery +
survivable cache = quick and
easy recovery from DB
failures
SQL
Transactions
Caching
SQL
Transactions
Caching
SQL
Transactions
Caching
Caching process is outside the DB process
and remains warm across a database restart
Simulate failures using SQL
• To cause the failure of a component at the database node:
ALTER SYSTEM CRASH [{INSTANCE | DISPATCHER | NODE}]
• To simulate the failure of disks:
ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN
[DISK index | NODE index] FOR INTERVAL interval
• To simulate the failure of networking:
ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type
[TO {ALL | read_replica | availability_zone}] FOR INTERVAL interval
Operating shared file storage today is a pain
Application owner
or developer
IT administrator
Business owner
• Estimate demand
• Procure hardware
• Set aside physical space
• Set up and maintain hardware (and network)
• Manage access and security
• Provide demand forecasts/business case
• Add lead times and extra coordination to your schedule
• Limit your flexibility and agility
• Make up-front capital investments, over-buy, stay on a
constant upgrade/refresh cycle
• Sacrifice business agility
• Distract your people from your business’s mission
We focused on changing the game
EFS is
simple
EFS is
elastic
EFS is
scalable
1 2 3
EFS is simple
• Fully managed
– No hardware, network, file layer
– Create a scalable file system in seconds!
• Seamless integration with existing
tools and apps
– NFS v4—widespread, open
– Standard file system semantics
– Works with standard OS file system APIs
• Simple pricing = simple forecasting
1
EFS is elastic
• File systems grow and shrink
automatically as you add and remove
files
• No need to provision storage capacity
or performance
• You pay only for the storage space you
use, with no minimum fee
2
• File systems can grow to petabyte
scale
• Throughput and IOPS scale
automatically as file systems grow
• Consistent low latencies regardless
of file system size
• Support for thousands of concurrent
NFS connections
EFS is scalable3