This document outlines a presentation on analyzing large raster data in a Jupyter notebook with GeoPySpark on AWS. The presentation covers introductory material, exercises on working with land cover and Landsat imagery data, combining data layers to detect crop cycles, and combining different data types to create maps. It discusses where the notebooks are running, data sources, and GeoPySpark capabilities like working with space-time raster data. Attendees are encouraged to tweet maps created during the exercises.

1. Rob Emanuele @lossyrob
ANALYZING LARGE RASTER DATA
IN A JUPYTER NOTEBOOK
WITH GEOPYSPARK
ON AWS

2. Connect to the WIFI
Network: Harvard University
http://getonline.harvard.edu
Click “I am a guest”
Credentials:
U: foss4g2017@gmail.com
P: 7RFQU3rm
FIRST:
Find your Jupyter Notebook URL
https://git.io/v77lh
(lowercase L)
visit the URL next to your name
Log in to the Jupyter Hub
U: hadoop
P: hadoop

3. OUTLINE
8:00 - 8:30 Intro and Background
8:30 - 9:10 Section 1: Land Cover data
9:10 - 10:00 Section 2: Landsat 8 data
10:00 - 10:10 BREAK
10:10 - 10:30 Deployment and Ingestion
10:30 - 11:10 Section 3: Combining data layers
11:10 - 12:00 Section 4: Making Cool Maps

4. NOW:
A MOTIVATING EXAMPLE

8. BY

10. rdd.map(lambda x: x + 1)
Source: http://silverpond.com.au/2016/10/06/balancing-spark.ht

15. (1, 1) (2, 1)(0, 1)
(0, 0) (1, 0) (2, 0)
(1, 2) (2, 2)(0, 2)

16. (1, 1) (2, 1)(0, 1)
(0, 0) (1, 0) (2, 0)
(1, 2) (2, 2)(0, 2)
Node 1
Node 2
Node 3

17. (1, 1) (2, 1)(0, 1)
(0, 0) (1, 0) (2, 0)
(1, 2) (2, 2)(0, 2)
Node 1
Node 2
Node 3

18. (1, 1) (2, 1)(0, 1)
(0, 0) (1, 0) (2, 0)
(1, 2) (2, 2)(0, 2)
Node 1
Node 2
Node 3

19. (1, 1) (2, 1)(0, 1)
Node 1
Node 2
Node 3

20. (1, 1) (2, 1)(0, 1)
Node 1
Node 2
Node 3
rdd.bufferTiles(…)

21. +
+
Interactive and Batch Processing
of large raster data
Web-Speed Processing
of small to medium sized raster data

22. GeoTrellis Ecosystem
Raster Foundry by
Spark SQL and Spark ML support
Raster Frames by
Spark SQL and Spark ML support
GeoPySpark
Python bindings
Vector Pipes
Vector Tiles on Spark
PDAL integration
Point Clouds on Spark

23. GeoPySpark

24. Started December 2016
Follows PySpark’s model of communication
between the JavaVirtual Machine and Python
Access GeoTrellis functionality through Python,
and integrates with your favorite python raster
tools (numpy + friends).
0.2 is released!
GeoPySpark

28. EXERCISE 1:
ANALYZING LAND COVER DATA

29. EXERCISE 2:
WORKING WITH LANDSAT IMAGERY
AND NDVITHROUGHTIME

31. (SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
…
SpaceTimeKey ≈ (col, row, instant)

32. (SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
…
lambda
lambda
lambda
(SpatialKey, (DateTime, Tile))
(SpatialKey, (DateTime, Tile))
(SpatialKey, (DateTime, Tile))
…

33. …
(SpatialKey, [(DateTime, Tile)
(DateTime, Tile)])
(SpatialKey, (DateTime, Tile))
(SpatialKey, (DateTime, Tile))
(SpatialKey, (DateTime, Tile))
(SpatialKey, [(DateTime, Tile)])
…

34. …
(SpatialKey, [(DateTime, Tile)
(DateTime, Tile)])
(SpatialKey, (DateTime, Tile))
(SpatialKey, (DateTime, Tile))
(SpatialKey, (DateTime, Tile))
(SpatialKey, [(DateTime, Tile)])
(Shuffle)
…

35. (SpatialKey, [(DateTime, Tile)
(DateTime, Tile)])
(SpatialKey, [(DateTime, Tile)])
…
mosaic
(SpatialKey, Tile)
(SpatialKey, Tile)
…
mosaic

36. BREAK!

37. WHERE AND HOW ARETHESE
NOTEBOOKS RUNNING?

42. WHERE’STHIS DATA COMING
FROM?

43. Supported Backends

44. EXERCISE 3:
COMBINING LAND COVER AND NDVITO
DETECT CROP CYCLES

46. (SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
…

47. (SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
…
map_to_spatial
(SpatialKey, (STK, Tile))
(SpatialKey, (STK, Tile))
(SpatialKey, (STK, Tile))
…
map_to_spatial
map_to_spatial
STK = SpaceTimeKey

48. (SpatialKey, (STK, Tile))
(SpatialKey, (STK, Tile))
(SpatialKey, (STK, Tile))
…
(SpatialKey, Tile)
(SpatialKey, Tile)
…
ndwi_rdd
nlcd_layer.to_numpy_rdd()
(SpatialKey, ((STK, Tile), Tile))
(SpatialKey, ((STK, Tile), Tile))
(SpatialKey, ((STK, Tile),Tile))
…

49. (SpatialKey, (STK, Tile))
(SpatialKey, (STK, Tile))
(SpatialKey, (STK, Tile))
…
(SpatialKey, Tile)
(SpatialKey, Tile)
…
ndwi_rdd
nlcd_layer.to_numpy_rdd()
(SpatialKey, ((STK, Tile), Tile))
(SpatialKey, ((STK, Tile), Tile))
(SpatialKey, ((STK, Tile),Tile))
…
(Shuffle)

50. mask_ndwi
mask_ndwi
mask_ndwi
(SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
(SpaceTimeKey, Tile)
…
(SpatialKey, ((STK, Tile), Tile))
(SpatialKey, ((STK, Tile), Tile))
(SpatialKey, ((STK, Tile),Tile))
…

51. EXERCISE 4:
COMBINING IMAGERY, ELEVATION AND
LAND COVER DATA
TO MAKE A COOL LOOKING MAP

52. EXERCISE 4:
COMBINING IMAGERY, ELEVATION AND
LAND COVER DATA
TO MAKE A COOL LOOKING MAP
TWEETYOUR SWEET MAP SCREENSHOTS WITH
#GEOPYSPARK #FOSS4G!

53. FINAL QUESTIONS?

54. Thank you!