Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS re:Invent 2016: Earth on AWS—Next-Generation Open Data Platforms (STG203)

1,099 views

Published on

Making earth observation data available by using Amazon S3 is accelerating scientific discovery and enabling the creation of new products. Attend and learn how the scale and performance of Amazon S3 lets earth scientists, researchers, startups, and GIS professionals gather and analyze planetary-scale data without worrying about limitations of bandwidth, storage, memory, or processing power. Learn how AWS is being used to combine satellite imagery, social data, and telemetry data to produce new products and services. Learn also how Amazon S3 provides much more than storage, and how an open geospatial data lake on Amazon S3 can be used as the basis for planetary-scale applications built with Amazon EMR, Amazon API Gateway, and AWS Lambda. As part of this talk, AWS customer Digital Globe demonstrates how they use open data stored in S3 to distribute high-resolution satellite imagery to their customers around the world.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

AWS re:Invent 2016: Earth on AWS—Next-Generation Open Data Platforms (STG203)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Jed Sundwall, Open Data Global Lead November 29, 2016 Earth on AWS Working with Planetary-scale Data in the Cloud
  2. 2. What to Expect from the Session • About open data on AWS • Advantages of Amazon S3 for sharing data • How E&J Gallo and DigitalGlobe use AWS to work with geospatial data
  3. 3. Why Does AWS Care About Open Data? Sharing data on AWS makes it accessible to a large and growing community of researchers, entrepreneurs, and enterprises who use the AWS cloud.
  4. 4. “…data must be organized, well- documented, consistently formatted, and error free. Cleaning the data is often the most taxing part of data science, and is frequently 80% of the work.” - Data Driven by DJ Patil and Hilary Mason Undifferentiated Heavy Lifting
  5. 5. Traditional Data Acquisition Data Acquisition in the Cloud
  6. 6. When data is shared in the cloud, anyone can analyze any volume of data without needing to download or store it themselves.
  7. 7. Landsat on AWS
  8. 8. Landsat Landsat 8 satellite Raster data
  9. 9. RGB Visible light Infrared Vegetation Shortwave infrared Urban areas Wellington, New Zealand
  10. 10. Landsat on AWS Amazon EC2 s3://landsat-pds .tarUSGS .tiff
  11. 11. Landsat on AWS Graph by Drew Bollinger (@drewbo19) at Development Seed
  12. 12. landsatonaws.com Serverless browser interface to Landsat on AWS
  13. 13. New open source tools • GDAL – http://www.gdal.org/ • Rasterio – https://mapbox.github.io/rasterio/ • sat-utils suite – https://github.com/sat-utils
  14. 14. Elevation Models Aerial Imagery Climate Models Satellite Imagery High-resolution Radar aws.amazon.com/earth
  15. 15. AWS Cloud Credits for Research provide promotional AWS cloud credits for anyone to conduct research using Earth Observation data. aws.amazon.com/earth/research-credits
  16. 16. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. John Webb, Manager, Information Technology , E & J Gallo Winery Wine and Grape Supply Data Lake E & J Gallo
  17. 17. E&J GALLO WINERY Background
  18. 18. E&J Gallo Winery Largest Winery in the world • Established in 1933 and headquartered in Modesto, California, E. & J. Gallo Winery remains a privately-held and ever-growing company that employs 6,000 people worldwide. • Largest land owner in the State of California Products and Distribution • With products available in more than 90 countries, E. & J. Gallo Winery is the largest exporter of California wine, and imports wines from Argentina, Italy, New Zealand and Spain • WE hold 90 brands and include table and sparkling wines, beverage products, dessert wines and distilled spirits
  19. 19. Gallo Portfolio Popular Premium Ultra Premium
  20. 20. WINEGROWING Perennial v. Row Crop
  21. 21. Wine and Grape Supply Grower Relations • Manage external grape purchases • Inform growers on best management practices Gallo Vineyards Inc. • Gallo owned vineyards • Irrigation management • Yield and Quality estimation • Best management practices for vineyard management • Operations planning and scheduling Viticulture Chemistry and Enology • Internal research division • Quality and yield enhancement • Variable rate irrigation Winemaking
  22. 22. Complexities of Wine Growing Not a Biomass Product • Manage more than just nitrogen and water • A number of activities go against a given vine Perennial Crop • Average vineyard lifecycle of 24 years • Vine management is an on-going activity • Canopy Management • Shoot Thinning • Vine Balancing • Trellis Management • Nutrients • Pesticides High Value Crop Management • Direct correlation between ranch management plan and quality • Yield impact associated with canopy management activities
  23. 23. DATA DRIVEN INSIGHTS Data Lake Modeling
  24. 24. Why Data Matters We operate in an ‘Activity’ based agricultural environment from which we can drive insights • Events or Transactions help us manage our business • These events are both structured and unstructured Structured Events such as… • Tons of grapes delivered to a specific winery • Distribution of soil across a vineyard • Ranch management transactions applied (i.e. shoot thinning, leafing, dropping fruit etc.) Unstructured Events such as… • Imagery • Soil Moisture Nodes • Yield Monitors • Mechanized Asset Control
  25. 25. Data Driven Value Having a robust data and analytics platform allows us to drive insights • Grape to Bottle data management • Improve Quality • Increase Yield • Detect and prevent early on-set disease • Maintain a competitive supply position through predictive yield estimation
  26. 26. Data Lake Model EC2 Elastic Load Balancing S3 Redshift SQL on DB Instance EMR Customer Facing Product Layer EDW Reports Data Processing, Analysis, R&D Layer Data Collection Layer Dashboard Insights D a t a L a y e r
  27. 27. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Shay Har-Noy, VP & General Manager, Platform, DigitalGlobe Getting Data Out of Jail DigitalGlobe’s Geospatial Big Data Platform
  28. 28. @iheartcrowds
  29. 29. 3,500,000 km2 collected EVERY DAY 13,200,000,000,000 pixels
  30. 30. Last 3 Days
  31. 31. Last 14 Days
  32. 32. Last 3 Months
  33. 33. With frequent coverage of high interest areas
  34. 34. Goal: Large scale information extraction
  35. 35. Three trends working in our favor 1.
  36. 36. Three trends working in our favor 2. hidden layer 1 hidden layer 2 hidden layer 3
  37. 37. Three trends working in our favor 3.
  38. 38. Open up Geo to world of innovators working on Deep Learning Solve a new class of problems to impact business, government, and people
  39. 39. strategic approach: Leverage the ecosystem Crowdsourcing
  40. 40. – Data out of jail! Storage Compute GBD Partner Tools Tools MapsAPI Catalog API Workflow API
  41. 41. Scale humans until machines get smarter Use humans to create training data Crowdsourcing
  42. 42. Training and testing set to allow benchmarking and iteration
  43. 43. We have common frameworks to accelerate prototyping We need a common data set to evaluate performance
  44. 44. - 50cm WV-2 image w/8 spectral bands - Each image 200m2 - Covering Rio de Janeiro, Brazil - 7000 Geotiff images Available for your computing pleasure on
  45. 45. Rio Olympics – Use Case • Governments around the world were preparing for the upcoming Olympics in Rio De Janeiro • Thorough understanding of the security and safety of the athletes is front of mind • Knowledge of high risk areas, traffic patterns, line-of-sight and overall structure distribution is critical to ensure a secure games
  46. 46. What you can expect 1) Satellite imagery analytics will be way more common in your industry 2) Computer vision tasks are going to change the way we see the world 3) We are going to learn a lot
  47. 47. Try it yourself aws.amazon.com/public-data-sets/spacenet/ developer.digitalglobe.com
  48. 48. Thank you!
  49. 49. Remember to complete your evaluations!
  50. 50. Related Sessions

×