AWS hosts a variety of public data sets that anyone can access for free. Previously, large datasets such as satellite imagery or genomic data have required hours or days to locate, download, customize, and analyze. When data is made publicly available on AWS, anyone can analyze any volume of data without needing to download or store it themselves. The AWS Open Data Team will share tips and tricks, patterns and anti-patterns and tools to help you most effectively stage your data for analysis in the cloud.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jed Sundwall
Global Open Data Lead, Amazon Web Services
Dave Rocamora
Solutions Architect, Amazon Web Services
194328
AWS Public Datasets: Learnings from
Staging Petabytes of Data for Analysis
in AWS

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why does AWS care about Open Data?
Many AWS customers supply data
to the public to accelerate research
and product development.
Sharing data on AWS makes it accessible to a large and growing community of
researchers, entrepreneurs, and enterprises who use the AWS Cloud.
Many AWS customers use data
shared on AWS to create new
products and services.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The AWS Open Data program expands
access to data by staging it for
analysis in the cloud.
https://opendata.aws

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Public Datasets
https://registry.opendata.aws

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Earth Observation Life Sciences &
Genomics
Machine Learning
https://registry.opendata.aws

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traditional Data Acquisition
“…data must be organized, well-documented, consistently formatted,
and error free. Cleaning the data is often the most taxing part of data
science, and is frequently 80% of the work.”
— Data Driven by DJ Patil and Hilary Mason

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Undifferentiated Heavy Lifting

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Acquisition in the Cloud
Amazon S3
Amazon EC2
Amazon Athena Amazon EMR
Amazon Redshift
AWS Glue AWS Data Pipeline

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Advantages of sharing data in the cloud
Global community of users
Faster pace of research Lower cost of research
New services and tools

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What makes a dataset
successful?

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What makes a dataset
successful?
It is treated like a product.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What makes a dataset
successful?
It is treated like a product.
It is optimized for the cloud.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Highly processedRaw
Userbase

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributing Data
Not accessed
Necessary
Unnecessary (wasted)

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributing Data - Traditional
Not accessed
Necessary
Unnecessary (wasted)

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributing Data – Prepared data on S3
Not accessed
Necessary
Unnecessary (wasted)

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Staging data for analysis
Amazon S3 allows programmatic
and precise access to data at
planetary scale.
Landsat on AWS uses Cloud
Optimized GeoTIFFs that allow
users to get only the data they
need when they need it.
cogeo.org
AWS Lambda.tarUSGS
Cloud Optimized GeoTIFFs
s3://landsat-pds

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Graph by Drew Bollinger (@drewbo19) at Development Seed
Landsat on AWS

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Patterns
S3 Key Index External Index Internal Index

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: GOES-16 Key Naming
s3://noaa-goes16/ABI-L1b-RadF/2018/149/14/
OR_
ABI-L1b-RadF-M3C14_
G16_
s20181491430465_
e20181491441232_
c20181491441300.nc

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: IRS 990 CSV as External Index

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What makes a dataset
successful?
It is treated like a product.
It is optimized for the cloud.
There is a community around it.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Raw Accessible
Documented
Trustworthy
Userbase

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Landsat on AWS community
http://cogeo.org

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
landsat-tiler from Mapbox
https://viewer.remotepixel.ca/

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What makes a dataset
successful?
It is treated like a product.
It is optimized for the cloud.
There is a community around it.

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Please complete the session survey in
the summit mobile app.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank You!
jed@amazon.de
rocamora@amazon.com