This document discusses how data science platforms can be built on cloud computing infrastructure like Amazon Web Services (AWS). It highlights how AWS provides scalable, on-demand computing and storage resources that allow data and compute needs to scale rapidly. Example applications and customer case studies are presented to show how various organizations are using AWS for large-scale data analysis, including genomics, computational fluid dynamics, and more. The document argues that distributed, programmable cloud infrastructure can support new types of data-driven science by providing massive, rapidly scaling resources.

1. There is no magic
There is only awesome
A platform for data science
D e e p a k S i n g h

2. bioinformatics
image: Ethan Hein

3. 3

4. collection

5. curation

6. analysis

7. what’s the big deal?

9. Source: http://www.nature.com/news/specials/bigdata/index.html

10. Image:Yael Fitzpatrick (AAAS)

11. Image:Yael Fitzpatrick (AAAS)

12. lots of data

13. lots of people

14. lots of places

15. to make data effective

16. versioning

17. provenance

18. filter

19. aggregate

20. extend

21. mashup

22. human interfaces

24. hard problem

25. really hard problem

27. change how we think
about compute

28. change how we think
about data

29. change how we think
about science

30. information
platforms

31. Image: Drew Conway

32. dataspaces
Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data

33. the unreasonable
effectiveness of data
Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)

34. accept all data
formats

35. evolve APIs

36. beyond the
database
and the data
warehouse

37. data as a
programmable
resource

38. data as a royal garden

39. compute as a fungible
commodity

40. which brings us to ...

41. amazon web services

44. common characteristics

45. on demand

46. pay as you go

47. secure

48. elastic

49. 3000 CPU’s for one firm’s risk management application
3444JJ'
!"#$%&'()'*+,'-./01.2%/'
344'+567/'(.'
8%%9%.:/'
344'JJ'
I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1='
>?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

50. programmable

51. “infrastructure as
code”

53. include_recipe "packages"
include_recipe "ruby"
include_recipe "apache2"
if platform?("centos","redhat")
if dist_only?
# just the gem, we'll install the apache module within apache2
package "rubygem-passenger"
return
else
package "httpd-devel"
end
else
%w{ apache2-prefork-dev libapr1-dev }.each do |pkg|
package pkg do
action :upgrade
end
end
end
gem_package "passenger" do
version node[:passenger][:version]
end
execute "passenger_module" do
command 'echo -en "nnnn" | passenger-install-apache2-module'
creates node[:passenger][:module_path]
end

54. import boto
import boto.emr
from boto.emr.step import StreamingStep
Connect to Elastic MapReduce
from boto.emr.bootstrap_action import BootstrapAction
import time
# set your aws keys and S3 bucket, e.g. from environment or .boto
AWSKEY=
SECRETKEY=
S3_BUCKET=
NUM_INSTANCES = 1
conn = boto.connect_emr(AWSKEY,SECRETKEY)
bootstrap_step = BootstrapAction("download.tst",
"s3://elasticmapreduce/bootstrap-actions/download.sh",None)
Install packages
step = StreamingStep(name='Wordcount',
mapper='s3n://elasticmapreduce/samples/wordcount/wordSplitter.py',
cache_files = ["s3n://" + S3_BUCKET + "/boto.mod#boto.mod"],
reducer='aggregate',
input='s3n://elasticmapreduce/samples/wordcount/input',
output='s3n://' + S3_BUCKET + '/output/wordcount_output')
Set up mappers &
jobid = conn.run_jobflow(
name="testbootstrap",
reduces
log_uri="s3://" + S3_BUCKET + "/logs",
steps = [step],
bootstrap_actions=[bootstrap_step],
num_instances=NUM_INSTANCES)
print "finished spawning job (note: starting still takes time)"
state = conn.describe_jobflow(jobid).state
print "job state = ", state
print "job id = ", jobid
while state != u'COMPLETED':
print time.localtime() job state
time.sleep(30)
state = conn.describe_jobflow(jobid).state
print "job state = ", state
print "job id = ", jobid
print "final output can be found in s3://" + S3_BUCKET + "/output" + TIMESTAMP
print "try: $ s3cmd sync s3://" + S3_BUCKET + "/output" + TIMESTAMP + " ."

56. “I terminate the
instance and
relaunch it. Thats
my error handling”
Source: @jtimberman on Twitter

57. compute is a fungible
commodity

58. emphasis on
productivity

59. you can get a lot of
awesome

60. dive in

61. just a little

62. S3
Simple Storage
Service

63. highly durable

64. 99.999999999%

65. Highly scalable

66. EC2
Elastic Compute Cloud

67. dynamic

68. autoscaling

69. EC2 instance types

70. s
pe
ty
ce
an
standard “m1”
st
in
2
EC
high cpu “c1”
high memory “m2”
http://aws.amazon.com/ec2/instance-types/

71. cluster compute instances
text

72. cluster GPU instances

73. s
pe
ty
ce
an
st
in
cluster compute “cc1”
2
EC
cluster GPU “cg1”
http://aws.amazon.com/ec2/instance-types/

74. 10gbps

75. Placement
Group

76. full bisection
bandwidth

77. Linpack benchmark
880-instance CC1 cluster
Performance: 41.82 TFlops*
*#231 in Nov 2010 Top 500 rankings

78. Credit: K. Jorissen, F. D.Villa, and J. J. Rehr
WIEN2k Parallel Performance (U. Washington)
KS for huge system
at 1 k-point
H size 56,000 (25GB)
Runtime (16x8 processors)
Local (Infiniband) 3h:48
Cloud (10Gbps) 1h:30 ($40)
VERY DEMANDING
network performance
•1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k

79. cost and use models

80. Ideal Effective Utilization
Spot Utilization
% Utilization
On Demand Utilization
Reserved Utilization
time

81. making things easier

84. Elastic Beanstalk

86. Heroku

87. Ideal Effective Utilization
Spot Utilization
% Utilization
On Demand Utilization
Reserved Utilization
time

88. data at scale

89. some practical
considerations

90. everything fails all the
time

91. compute needs vary

92. new data/compute
paradigms

94. Amazon Elastic MapReduce

96. doing stuff

100. Customer Case Study: cyclopic energy
OpenFOAM®
http://aws.amazon.com/solutions/case-studies/cyclopic-energy/

101. NASA JPL

102. Credit: Angel Pizzaro, U. Penn

103. http://aws.amazon.com/solutions/case-studies/numerate/

104. Bioproximity
http://aws.amazon.com/solutions/case-studies/bioproximity/

106. http://usegalaxy.org/cloud

107. mapreduce for
genomics
http://bowtie-bio.sourceforge.net/crossbow/index.shtml
http://contrail-bio.sourceforge.net
http://bowtie-bio.sourceforge.net/myrna/index.shtml

108. http://cloudbiolinux.org/

109. in summary

110. large scale data
requires a rethink

111. data architecture

112. compute architecture

113. in infrastructure

114. the cloud

115. distributed,
programmable
infrastructure

116. rapid, massive, scaling

117. architecture evolved
with the internet

118. can we build data
science platforms?

119. there is no magic
there is only awesome

120. two more things

121. 10 minutes

122. http://aws.amazon.com/about-aws/build-a-cluster-in-under-10/

123. http://aws.amazon.com/education

124. deesingh@amazon.com
Twitter:@mndoci
http://slideshare.net/mndoci
http://mndoci.com
Inspiration and ideas from
Matt Wood& Larry Lessig
Credit” Oberazzi under a CC-BY-NC-SA license