This document discusses how Amazon Web Services (AWS) can be used for data-driven innovation. It provides an overview of AWS computing, storage, database and analytics services that can be used to collect, compute and collaborate on data. Specific services highlighted include S3, DynamoDB, EMR and EC2. Use cases discussed include log analysis, risk analysis, fraud prevention and market trend analysis. It also covers how AWS services allow for scalable, flexible and cost-effective infrastructure.

1. Data Driven
Innovation with
Amazon Web Services
Dr. Matt Wood
DATA I N T E N S I V E & H I G H P E R F O R M A N C E C O M P U T I N G

2. Hello.

3. Thank you

4. Infrastructure
building blocks

5. Compute
Storage Databases
Infrastructure
building blocks
Tools and Support
frameworks

6. The data timeline

7. Collect Compute Collaborate

8. Collect Compute Collaborate
Secure platform

10. Data stays local
Retain ownership and access control

11. Availability zones
Design for durability

12. Shared responsibility

13. Service Organisation
Controls 1 ISO 27001
Shared responsibility
FISMA PCI DSS
Moderate Level 1

14. Collect Compute Collaborate

15. Collect Compute Collaborate

16. In-situ: historical transactions, time series, logs
Collect Compute Collaborate
In-flight: recent market data, risk models

17. In-situ data
Immutable. Time series. Historical value.
Read-only profile. Gets really big, really quickly.

18. Amazon S3
Object store. Utility pricing.

19. 6 years young
Amazon S3 launched on March, 2006

20. Objects in S3
Billions of objects
762B
800.000
600.000
400.000
200.000
0
Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q4 2011
500k peak transactions per second

21. 99.999999999%
Data durability

22. Data movement
Upload. Physical media. Direct Connect.

23. Transaction history
Integrate with customer profiles for insight and
product recommendations

24. Historical market data
Parameterise or test financial risk models

25. Time series data
Find signal in the noise.
Fourier transform parameterisation

26. Service logs
Collected and collated to identify hot spots in
application and customer behaviour

27. In-flight data
Unstructured. High or variable throughput.
Complex. Read/write profile. Many sources.

28. Any database on Amazon EC2
MySQL, DB2, Oracle, PostgreSQL...

29. Relational Database Service
Managed MySQL and Oracle databases

30. Rapid High
provisioning availability
Scalable Scalable
storage compute
Relational Database Service
Managed MySQL and Oracle databases

31. High performance databases
Increase throughput
Increase availability
Reduce latency

32. High performance databases
Read replicas
Push-button
scaling Increase throughput
ElastiCache
Increase availability
Reduce latency

33. High performance databases
Increase throughput
Multi-AZ Increase availability
Reduce latency

34. High performance databases
Increase throughput
Increase availability
Reduce latency ElastiCache

35. Problem
Performance decreases at scale

36. Performance
Predictable, consistent
Scale

37. Performance
Predictable, consistent
Degraded performance
with scale
Scale

38. Performance
Predictable, consistent
Degraded performance
with scale
Scale

39. = more problems

40. Data caching
Provisioning!
Data sharding
= more problems
Cluster management
Fault management

41. Undifferentiated
heavy lifting

42. DynamoDB

43. Fully managed
NoSQL database
service

44. Extremely fast
performance

45. Consistently fast
performance

46. Low latency
Single digit millisecond

47. SSD backed
Low latency
Single digit millisecond
< 5 ms reads < 10 ms writes

48. Seamless scalability
No table size limits. Unlimited storage.
Live repartitioning. Zero admin.

49. Durable and available
Consistent, disk-only writes.

50. Predictable performance
Provisioned throughput

51. Reserve required IOPS
Per table. Set at creation. Scale via API.

52. Scale at any time
No downtime

53. Flexible data model
Unstructured. Multiple sources.

54. High throughput.
High scale.
Zero admin.

55. Integrated analytics
with Elastic MapReduce

56. Storage and archive

57. Input

58. Collect Compute Collaborate

59. Scale out systems
Collect Compute Collaborate
Tightly coupled systems

60. Utility computing
Elastic Compute Cloud: EC2

61. On-demand
Utility computing
Elastic Compute Cloud: EC2
Pay for what Flexible
you use

66. On-demand
Utility computing
Elastic Compute Cloud: EC2
Pay for what Flexible
you use

67. On-demand
Utility supercomputing
Pay for what Flexible
you use

69. EC2 instance sizes
Range of compute, storage and memory
Metered by Elastic Compute Unit

70. Standard High memory High CPU
m1 m2 c1

71. 1 ECU, 1.7Gb, 160Gb
Standard High memory High CPU
m1 m2 c1

72. Standard High memory High CPU
m1 m2 c1
26 ECU, 68 Gb, 1.7Tb

73. 20 ECU, 7 Gb, 1.7Tb
Standard High memory High CPU
m1 m2 c1

74. 1 instance for 100 hours
=
100 instances for 1 hour
You choose where to balance cost against time.

75. Batch processing
Processing pipelines at any scale.
Orchestration with Simple Workflow Service.

76. “Our 40-instance cluster can scan, filter,
and aggregate 1 billion rows in 950
milliseconds.”
Mike Driscoll, Metamarkets

77. Hadoop
High scale. Data-driven. Flourishing ecosystem.
Complex to configure and maintain.

78. Elastic MapReduce
Managed Hadoop service.
Built for data. Designed for humans.

79. Batch processing
Elastic MapReduce
Managed Hadoop service.
Built for data. Designed for humans.
Complex analytics

80. Undifferentiated
heavy lifting

81. S3 + DynamoDB
Input data

82. S3 + DynamoDB
Input data
Code Elastic
MapReduce

83. S3 + DynamoDB
Input data
Code Elastic Name
MapReduce node

84. S3 + DynamoDB
Input data
Code Elastic Name
MapReduce node
Elastic
cluster

85. S3 + DynamoDB
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

86. S3 + DynamoDB
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

87. S3 + DynamoDB
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

88. S3 + DynamoDB
Input data
Output
S3 + SimpleDB

89. Log analysis
Internal service behaviour and performance.
User interactions. Better insight.

90. Total risk analysis
Integrate existing products with market data.
Financial pulse.

91. Fraud prevention
Identify and profile movement patterns. Strategic insights.

92. Historical evaluation
Test and refine models against historical “in-situ” data.
Rapid iteration. Low barrier to entry.

93. Market trends
Evaluate risk models against “in-flight” and “in-situ” data.
Integrate disparate data sources.

94. Scale out systems
Collect Compute Collaborate
Tightly coupled systems

95. Parallel computation

96. Risk analysis

97. Monte Carlo simulations

98. 23 hours to 20 minutes

99. Cluster compute

100. Placement groups
Cluster compute
10 gig E, non blocking network

101. 42nd
240 TFLOPS. World’s fastest ethernet supercomputer.

102. GPU
2 x NVIDIA Tesla. 400 cores each.
Hedging. Risk modeling.

103. On-demand
Utility supercomputing
Pay for what Flexible
you use

104. Economies of scale

105. 19 price drops
Committed to passing savings to customers

106. Utilisation

107. Achieving economies of scale
100%
Time

108. Achieving economies of scale
100%
Reserved capacity

109. Achieving economies of scale
100%
On-demand
Reserved capacity

110. Achieving economies of scale
100%
On-demand
Reserved capacity

111. Spot market
Choose your own price for compute

112. Fine grained
time/cost benefit

113. Build model on spot
Adjust with “in-flight” data via complex event processing

114. 30,472 cores
$1,279 per hour

115. Continual risk assessment
Driven by data, powered by IT.

116. Towards real time
Data-centric decisions

117. Up to date traders
Data-centric decisions

118. High frequency
trade models

119. Remove constraints
Increase capacity without increasing
fleet size, power or capital spend

120. Collect Compute Collaborate

121. Canonical source
Decouple storage from compute.
Replicated storage. Reduce number of data sources.

122. Bring compute
to the data

123. Treat your data as a
royal garden
Compute is the mulch.
Interchangeable and cheap.

124. Empower users
Rapid reporting and ETL. Scale to support.

125. Empower users
Automation and orchestration

126. Empower users
Remove constraints.

127. Empower users
Flexible, response, agile, IT.

128. Integrate

129. Virtual Private
Cloud

130. Network control Public and private
subnets
Virtual Private
Cloud
Bring your own IP Extend VPN to EC2
addressing

131. Public facing service
Public subnet

132. Multi-tier applications
Public subnet
Network ACLs + security groups
Private subnet

133. Extend your data centre
Public subnet
Private subnet
IPsec VPN + Direct Connect
On-premise

134. Extend your data centre
Private subnet
IPsec VPN + Direct Connect
On-premise

135. Multi-AZ
Create a VPC that spans availability zones

137. Elastic MapReduce
Launch job flows within VPC

138. AWS Storage Gateway
Backup to AWS.
Restore on-premise or in the cloud.

140. Collect Compute Collaborate

141. Collect Compute Collaborate

142. Data as a
competitive advantage

143. Flexible IT to power
competitive advantage

144. Collect Compute Collaborate
Secure platform

145. aws.amazon.com/security

146. Think Big Analytics
thinkbiganalytics.com

147. Free tier to get started
aws.amazon.com/free

148. Join us for a coffee
with the Amazon business development and
technical staff

149. Thank you!

150. Q&A
matthew@amazon.com