Earth Observation in the Cloud
•Download as PPTX, PDF•
3 likes•4,812 views
This document summarizes an event promoting open data on AWS. It introduces open data initiatives like Landsat and NEXRAD imagery available on S3. The document highlights how AWS tools like S3, EC2, and EMR can be used to access, analyze and visualize open data at scale. It discusses how AWS aims to eliminate the "undifferentiated heavy lifting" of data preparation by pre-processing datasets and notifying subscribers of updates.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Earth Observation in the Cloud
Similar to Earth Observation in the Cloud (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Earth Observation in the Cloud
- 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Earth Observation in the Cloud Jed Sundwall, AWS Open Data Global Lead 10 November 2015 Demo Day
- 2. Agenda • Welcome! 🌍🌎🌏 • Open data on AWS • NEXRAD on AWS • Landsat on AWS 2
- 6. Sponsor Sponsor Sponsor Venue Sponsor Thank you to our sponsors!
- 7. Our goals for this event • Show off amazing work being done by our customers • Provide opportunities for you to network • Highlight the diversity of work made possible by Earth observation data • Learn about your priorities and needs
- 8. New whitepaper This Amazon Web Services, Inc. (AWS) package is provided for informational purposes only. The services discussed in this package are standard commercial services. This package may include a set of suggested solutions for this opportunity that are based on our limited information, and should not be construed as a binding offer from AWS. For current prices for AWS services, please refer to the AWS website at www.aws.amazon.com. This package includes Amazon Web Services, Inc. commercial, financial, or trade secret data that includes confidential, and/or trade secret information. AWS Whitepaper: Minimizing Variable Costs for Shared Data November 2015 Amazon Web Services, Inc. 410 Terry Avenue North Seattle, WA 98109-5210 Cage Code: 66EB1 DUNS Number: 965048981 NAICS: 518210 We have just published a new AWS Whitepaper on Minimizing Variable Costs for Shared Data. Download it at: http://bit.ly/s3-requester-pays-open-data
- 9. Open data on AWS 9
- 10. Why does AWS care about open data? Open data is data that can be used by anyone for any purpose for free. Many of our customers rely on quality open data as much as they rely on our computing, storage, and other web services. 10
- 11. Data on AWS Amazon Web Services provides a comprehensive toolkit for gathering, storing, analyzing, and working with data at any scale. Amazon Elastic MapReduce (Amazon EMR) provides the Apache Hadoop analytics framework as an easy-to-use managed service. Amazon S3 lets you store and retrieve any amount of data, at any time, from anywhere on the web. Amazon DynamoDB is a fully-managed NoSQL database service that makes it cost-effective to store and retrieve any amount of data. 11
- 12. 1-click deployment to launch, on multiple regions around the world Pay-as-you-go pricing Advanced AnalyticsData Integration Analysis & Visualization http://bit.ly/awsAnalytics 12
- 13. The power of open data in the cloud Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources. Amazon S3 Bucket Amazon EMR Amazon EC2 AWS Lambda Amazon Redshift Amazon DynamoDB 13
- 14. Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources. Amazon S3 Bucket Amazon EMR Amazon EC2 AWS Lambda Amazon Redshift Amazon DynamoDB The power of open data in the cloud 14
- 15. AWS Partners Focused on Public Sector 15
- 16. History of Innovation AWS has been continually expanding its services to support virtually any cloud workload, now offering more than 40 services. Amazon S3 Amazon SQS Amazon EC2 Amazon SimpleDB Amazon EBS Amazon CloudFront Elastic Load Balancing Auto Scaling Amazon VPC Amazon RDS Amazon SNS AWS Identity and Access Management Amazon Route 53 Amazon SES AWS Elastic Beanstalk AWS CloudFormation Amazon ElastiCache AWS Direct Connect AWS GovCloud AWS Storage Gateway Amazon DynamoDB Amazon CloudSearch Amazon SWF Amazon Glacier Amazon Redshift AWS Data Pipeline Amazon Elastic Transcoder AWS OpsWorks AWS CloudHSM Amazon AppStream AWS CloudTrail Amazon WorkSpaces Amazon Kinesis Amazon ECS Amazon Lambda AWS Config AWS CodeDeploy Amazon RDS for Aurora AWS KMS Amazon Cognito Amazon WorkDocs AWS Directory Service Amazon Mobile Analytics 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Amazon EFS Amazon WorkMail Amazon Machine Learning 16
- 17. AWS has announced price reductions 49* times since our inception in 2006. Recent price drops included… Amazon ElastiCache reduces prices for cache nodes by an average of 34% March 26, 2014 34% Amazon S3 reduces prices for Standard and Reduced Redundancy Storage, by an average of 51% March 26, 2014 51% Amazon Route 53 lowers prices for both standard queries and latency-based routing queries by 20% July 31, 2014 20% 17 * As of June 2015
- 18. Open data as a platform 18 Data Enrichment Sensemaking Data Creation Data at Rest (Object storage) Basic APIs Complex APIs Consumer applications Algorithmic policy Data-driven journalism Data Catalogs Focused data dashboards Predictive modeling Visualizations Lower cost of knowledge
- 19. Data Enrichment Sensemaking Amazon Kinesis Amazon EC2 Amazon EC2 AWS Data Pipeline Amazon S3 Amazon RDS Amazon EMR Amazon Redshift Amazon DynamoDB AWS Lambda Open data as a platform 19
- 20. An Amazonian approach to open data Two ideas that inform how we approach public data sets: • Work backwards from the customer • Eliminate undifferentiated heavy lifting 20
- 21. Working Backwards • Think of data sets as products • Seek out valuable data by listening to customer needs • Consider real-world use cases for the data • Consider the size of the user community or market opportunity 21
- 22. Undifferentiated heavy lifting “…data must be organized, well-documented, consistently formatted, and error free. Cleaning the data is often the most taxing part of data science, and is frequently 80% of the work.” — Data Driven by DJ Patil and Hilary Mason 22
- 23. Undifferentiated heavy lifting “…data must be organized, well-documented, consistently formatted, and error free. Cleaning the data is often the most taxing part of data science, and is frequently 80% of the work.” — Data Driven by DJ Patil and Hilary Mason We ask: How can we get rid of that 80%? 23
- 24. Public datasets on AWS To enable more innovation, AWS hosts a selection of datasets that anyone can access for free. Data in our public datasets is available for rapid access to our flexible and low-cost computing resources. Earth Science Landsat on AWS Life Sciences 1000 Genomes Project Internet Science Common Crawl Corpus 24
- 25. NEXRAD on AWS 25
- 26. NEXRAD on AWS The Next Generation Weather Radar (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in 5 minute intervals from each site. It has traditionally been time consuming and expensive to acquire, store, and analyze NEXRAD data. Accessing the full historical archive has been impossible. 26
- 27. NEXRAD on AWS NEXRAD Sites Public Amazon S3 Bucket Amazon EC2 Public Amazon S3 Bucket Real-time data chunks Volume scan file assembly Continuously updated archive With NEXRAD on AWS, we provide an archive of individual volume scan files and real-time chunks as objects in Amazon S3. This allows the data to be accessed programmatically via a RESTful interface and quickly deployed to any of our products for analysis and processing. 27
- 28. NEXRAD on AWS Our collaborators, including Unidata, The Weather Company, NOAA, Climate Corporation, and CartoDB, have provided early use cases and tutorials on how to use this data in the cloud. A wide range of users are interested in using NEXRAD on AWS for longitudinal analysis, to study and visualize specific weather events, and develop new products. More info at http://aws.amazon.com/public-data-sets/nexrad 28
- 30. Landsat on AWS We have committed to make up to 1 petabyte of Landsat imagery readily available as objects on Amazon S3. All Landsat 8 scenes from 2015 are available, along with a selection of cloud-free scenes from 2013 and 2014. All new Landsat 8 scenes are made available each day (~700 per day), often within hours of production. 30
- 31. The Traditional Approach Data is most commonly accessed via a web interface and downloaded on premises before being loaded into a web server. All bands are downloaded in a .tar archive, even if you only need a few bands. Data acquisition is time consuming and inherently redundant. Analysis is limited by user’s access to bandwidth, storage, memory, and processing power. 31
- 32. Landsat on AWS Landsat on AWS makes each band of each scene readily available as objects on Amazon S3. Data can be accessed programmatically via HTTP and quickly deployed to any of our products for analysis and processing. Users do not need to worry about local storage and have access to virtually unlimited computing power on demand. 32 Amazon EC2 s3://landsat-pds .tarUSGS .tiff
- 33. Undifferentiated heavy lifting We use GDAL to add “internal tiling” on each Landsat on AWS tiff, which allows developers to use HTTP range gets to access specific portions of each scene. This allows people to only access the data they need when they need it. Standard tiff object Internal tiled tiff object 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 33
- 34. RGB Visible light Infrared Vegetation Shortwave infrared Urban areas Wellington, New Zealand – Made on Snapsat.org https://landsat-pds.s3.amazonaws.com/L8/072/089/
- 35. RGB Visible light Infrared Vegetation Shortwave infrared Urban areas Wellington, New Zealand – Made on Snapsat.org https://landsat-pds.s3.amazonaws.com/L8/072/089/
- 36. Landsat on AWS In the first 150 days (19 Mar – 16 Aug 2015) • Over 200,000 scenes available • Over 500 million hits globally Image shows frequency of scene requests by path/row. White: ~100 requests Orange: >300k requests 36 Visualization by Drew Bollinger Development Seed
- 37. Landsat on AWS as a platform 37
- 38. New SNS topic for Landsat on AWS 38 arn:aws:sns:us-west-2:274514004127:OpenObjectAddL8 You can now subscribe to a publicly available Amazon Simple Notification Service (Amazon SNS) topic to be notified whenever a new batch of Landsat scenes are available at s3://landsat-pds.
Editor's Notes
- HELLO! **WELCOME!** I’m Jed Sundwall, and I lead AWS’s open data program. Each of you has received an email from me about this event, so it’s nice to meet you in person. I can’t express how excited we are to have you all here. This is the first big event the AWS Open Data team has put on and it’s very cool to see the response.
- I want to take this time to tell you about why we’re here today, explain our open data program, and give you a few updates on our NEXRAD and Landsat Public Data Sets. Let’s go.
- I love emoji.
- 6. First off, let’s thank our sponsors for making today possible. There are plenty of folks from Esri, DigitalGlobe, and Development Seed here, so please seek them out and thank them if you can. And I want to give a special thanks to Chief for sharing their venue with us. I’ve been a big fan of the team at Chief for years. They are fully committed to helping government take full advantage of technology, which is why they were the first people I reached out to when planning this event.
- So why are we here? First off, we wanted to show off the state of the art of Earth data analysis, which is being done by our customers who you’ll hear from today. We also wanted to provide a venue for leaders at the Department of Interior and NOAA to connect with those of you who use their data. We also want this to be a networking event for all of you. You’ll notice a lot of breaks throughout the day. Those are there to give you ample opportunities to chat between sessions. We’ve also told all of the speakers that they get 30 minutes to present *and* take questions. We want this to be a very conversational event. One of the neat things about AWS is that we have over a million customers who use our services to do a wide variety of work. Hopefully you’ll get to learn about how someone outside of your field is working with Earth observation today. And last, but not least, we want to learn from you, particularly about what series AWS could provide to make your lives easier. Our product roadmap is largely based upon customer feedback, so please give us feedback! You should have a survey, so please fill that out, and feel free to cast your vote for your most wanted data on the window throughout the day.
- 8. Hot off the presses! In addition to this being our first event, we’ve also recently published our first Whitepaper, which is about understanding and mitigating the costs of using Amazon S3 to share data. Just email me for a copy or you can download it at this link.
- OK. Let’s talk about why AWS has an open data program.
- We use a very basic definition of open data. The beauty of open data – if it’s good – is that it gives people something to do with our computing and storage resources. We have very significant customers who rely on open data for their work. Their products wouldn’t exist or wouldn’t be very useful without the availability of high quality, license-free, government information. Open data matters to our customers, so it matters to us, and our goal, as a program, is to continually make more data more available to more people.
- AWS provides a comprehensive toolkit for working with data at any scale. I’m not going to get into describing our products today, but I simply want to make the point that our platform is designed to store and and work with virtually any amount of data using our on-demand computing resources.
- Through the AWS Marketplace we also provide a catalog of data analytics, integration, and visualization applications that you can quickly deploy in the cloud with pay-as-you-go pricing.
- The value of open data on AWS is very basic: once the data is on S3, it becomes easier to work with using any of our computing and database resources. This reduces the cost – in terms of dollars and time – of product development, of analysis, and of scientific discovery. This is tremendously empowering for researchers, product developers, and analysts because it allows them to take their algorithms to the data rather than spending time and money to find, download, and store data in their own computing environments. When data is made available in the cloud, people can pay only for the computing resources they need to analyze the data and don’t have to worry about downloading and paying to store the data themselves.
- The value of open data on AWS is very basic: once the data is on S3, it becomes easier to work with using any of our computing and database resources. This reduces the cost – in terms of dollars and time – of product development, of analysis, and of scientific discovery.
- We have over a million customers. This includes many many partners focused on the public sector. These are just part of a large and growing ecosystem of users who are advancing the state of the art of what can be done with data in the cloud. Of course, you’re going to hear from them today. They are the focus of this event.
- Benefits are also realized as we continue to develop new services and features of our platform. We now offer more than 40 services to support virtually any kind of application in the cloud. As we add services, new applications of data become possible.
- And finally, new possibilities are created as we lower prices. As of June this year, we’ve dropped our prices 49 times since AWS got started in 2006. The on-demand and pay-as-you-go nature of the cloud removes barriers for people to get started, and we continually pass whatever savings we can on to our customers to make it easier for them to access the IT resources they need.
- Talk about temperature data. Talk about lowering the cost of knowledge. Talk about the great grain robbery. 10MM tons of grain sold to Russia in 1972.
- Naturally, we have products that can help. LOLOLOLOL. Seriously, we do have products that help with creation of data, strong it, sharing it, analyzing it, and making it available through applications. Where the open data team focuses is in the bottom left there, which is the point from which data is shared. What can we do to ensure that shared data can realize its value? There are two very Amazonian ideas that guide us.
- These concepts inform the way we think about a lot of our products and they can be applied to open data as well.
- Working backwards from the customer is how we determine what to focus our energies on. A common practice throughout Amazon is to write a hypothetical press release as a first step when considering a new product. It challenges us to write in few words, and in plain language, why what we’re working on would matter and who it would matter to. If we can’t write a plausible press release, our efforts may be better spent elsewhere, or we need to try harder. The same thing can apply when prioritizing data sets. There is a lot of data out there, and it’s hard to tell which is most valuable. The working backwards approach can help with that.
- Undifferentiated heavy lifting is all of the chores that keep you from focusing on what you’re good at and what your mission is. You’re not better at it than anyone else and it distracts from what you’ve actually set out to do. It’s one of the core benefits of the cloud, in that you can access as much storage and computing power as you need without needing to worry about ordering gear, setting it up, securing it, cooling it and worrying about running out of capacity. You just get what you need when you need it. The same thing applies to data. I love this quote from Data Driven (which is free on the Kindle, btw) by DJ Patil and Hilary Mason.
- What we ask ourselves is what we can do to eliminate all of the work that gets in the way of actually analyzing data. That might mean making it analysis ready, but we’ve learned that we can make a huge difference by simply eliminating the need for people to download and store copies of it.
- Our Public Data Sets Program is our lab where we can test out these theories and identify best practices for sharing data in the cloud. We select data sets by listening to our customers and when we foresee opportunities to make an impact on emerging sectors, as we did when we started hosting the 1,000 Genomes data set several years ago. Many of you are here today because of our Landsat and NEXRAD Public Data Sets.
- We launched NEXRAD on AWS last month.
- My DC-based counterpart, Ariel Gold, has managed the NEXRAD project throughout the year and she deserves a round of applause for this groundbreaking project – I don’t think it’s possible to overstate how hard she worked on this. The Next Generation Weather Radar (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in approximately 5 minute intervals from each site. NEXRAD enables severe storm prediction and is used by researchers and commercial enterprises to study and address the impact of weather across multiple sectors. Traditionally, there has been no full historical archive available on demand to NOAA or external users and you have to wait 12 to 24 hours for new volume scan files.
- A real-time feed of NEXRAD data and a full historical archive of original resolution (Level II) NEXRAD data, from June 1991 to present, is now freely available on Amazon S3 for anyone to use. This is the first time the full NEXRAD Level II archive has been accessible to the public on demand. Now anyone can use the data on-demand in the cloud without worrying about storage costs and download time. We are making NEXRAD data available as part of our research agreement with the US National Oceanic and Atmospheric Administration (NOAA) to enable new product development and analysis. Jeff De La Beujardier is going to talk about NOAA’s work to make its data more widely available later today.
- NEXRAD on AWS is new, but we already know of one customer who was able to complete a project *two weeks* in advance because of the easy access to the data in the cloud. If you go to this URL, you’ll find documentation on how to access the data along with tutorials from Climate Corporation and CartoDB.
- Landsat!
- Just under a year ago, we announced that we would make up to 1PB of Landsat data available on AWS over a two year period as a contribution to the White House’s Climate Data Initiative. We launched Landsat on AWS on March 19th this year.
- We had heard from multiple customers that we should look harder at Landsat, so we did. We spoke with a lot of customers who use Landsat data and learned that there was a lot we could do to help people work with it. One of the first things we learned was that acquiring data could be incredibly time consuming, particularly when trying to work with a lot of data. One big reason for this is that scenes are made available at 1GB .tar archives. To work with a scene, a researcher needs to download all 12 bands of that scene’s data even if they only need three or four scenes. That means that 60-70% of bandwidth (and therefore time spent downloading) is wasted
- We subscribe to a feed of Landsat scenes from USGS to acquire all new scenes as they’re created. We then unpack the tarballs and use GDAL to apply some lossless compression and data optimization for each GeoTIFF of each band, which is then made available as a stand-alone object on S3. This means that researchers can access only the data they need when they need it.
- Internal tiling!
- By lowering the cost of accessing Landsat data, we’ve enabled more people to work with it in unexpected ways. I made the visualizations you see here on snapsat.org, which is a web app created by a group of students just a few weeks after we launched Landsat on AWS. It’s a remarkable web app that provides a very novel interface to browse the planet and create Landsat composites in seconds. I love it.
- The Snapsat students were able to build it because they were able to access Landsat data without needing to download or copy it, which they wouldn’t have had the time or resources to do. If you have data to share, share it via URLs!
- Landsat on AWS gets used a lot. Within the first 150 days, data was requested from the bucket over 500MM times. We don’t know how good that is because we’ve never released a comparable data set like this, but we know that a lot of people are accessing a lot of the data on purpose. As you’ll learn today, it’s being used for real work. It exceeded all of our expectations and it’s been amazing to learn about what people are doing with it.
- Remember this thing? This is what it looks like in action. One copy of the data, made available near computing resources, and optimized for analysis. In very little time we’ve seen a lot of activity come out of it, and we couldn’t be happier about it. We’re here today to discover what’s next.
- There’s one more thing. We now have a public SNS topic that you can use to subscribe to notifications whenever a new batch of scenes are available. You can use this to automatically trigger analysis of data from within AWS. I don’t expect you to copy it down, so just email me to get it.