Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

We trained a neural network on satellite imagery to predict wealth from space.

1,062 views

Published on

Penny is a simple tool to help us understand what wealth and poverty look like to artificial intelligence algorithms. The tool lets you change the landscape of a city, by adding and removing urban features like buildings, parks and freeways to high-resolution satellite imagery. With this interface, you can explore what different kinds of features make a place look wealthy or poor to an AI.

Published in: Data & Analytics

We trained a neural network on satellite imagery to predict wealth from space.

  1. 1. What does money 
 look like from space?
 (here comes the neighborhood)
  2. 2. The Training Model Satellite Imagery From GBDX
  3. 3. The Training Model Census Tracts (2013)
  4. 4. Quartiles The Training Model Quartile 0 < $34,176 Quartile 1 $34,177 - $49,904 Quartile 2 $49,905 - $71,875 Quartile 3 $71,876+
  5. 5. Satellite Imagery & Census Data The Training Model
  6. 6. Centroids The Training Model
  7. 7. Centroid Outlines The Training Model
  8. 8. Satellite Imagery & Census Data The Training Model (Resnet 50)
  9. 9. Satellite Imagery & Census Data Neural Network The Training Model (Resnet 50)
  10. 10. Satellite Imagery & Census Data Output = Neural Network The Training Model (Resnet 50) Q0 = 91%
 Q1 = 5.64%
 Q2 = 2.55%
 Q3 = .41%
  11. 11. Satellite Imagery & Census Data Output = Neural Network The Training Model (Resnet 50) The model carries
 what it has learned and 
 repeats the process. Q0 = 91%
 Q1 = 5.64%
 Q2 = 2.55%
 Q3 = .41%
  12. 12. The Training Model (Resnet 50) Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution Neural Network The data is fed into the model.
  13. 13. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network A number of filters are applied to the image.
  14. 14. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network The resulting new values are normalized to be within learned mean and standard deviations of the dataset.
  15. 15. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network Separates out features from important and non important ones.
  16. 16. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network Repeat.
  17. 17. Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution The Training Model (Resnet 50) Neural Network Results are merged with previous iteration.
  18. 18. The Training Model (Resnet 50) TensorBoard data from CMU
  19. 19. So now you have a Classifier
  20. 20. Centroid Outlines
  21. 21. Classified Approximate
  22. 22. Satellite Imagery & Census Data
  23. 23. What is the classifier seeing?
  24. 24. NYC Areas Classified as Q0
  25. 25. NYC Areas Classified as Q1 & Q2
  26. 26. NYC Areas Classified as Q3
  27. 27. Classification Confidence
 97% Q0
 3% Q1 0% Q2
 0% Q3
  28. 28. Classification Confidence
 92% Q0
 7% Q1 0% Q2
 0% Q3
  29. 29. Classification Confidence
 91% Q0
 5% Q1 2% Q2
 .4% Q3
  30. 30. Classification Confidence
 2% Q0
 2% Q1 6% Q2
 90% Q3
  31. 31. Classification Confidence
 0% Q0
 .1% Q1 0% Q2
 99% Q3
  32. 32. “Despite this encouraging process, there is still little insight into the internal operation and behavior of these complex models, or how they achieve such good performance. From a scientific standpoint, this is deeply unsatisfactory. Without clear understanding of how and why they work, the development of better models is reduced to trial-and-error.” Visualizing and Understanding Convolutional Networks - Matthew D. Zeiler, Dept. of Computer Science, Courant Institute, New York University - Rob Fergus, Dept. of Computer Science, Courant Institute, New York University
  33. 33. Baseball Field Experiment Confidence
 91% Q0
 5% Q1 2% Q2
 .4% Q3 Original Image
  34. 34. Baseball Field Experiment Confidence
 79% Q0 (-12)
 12% Q1 (+7) 7% Q2 (+5)
 .8% Q3 (+.4) Added Trees
  35. 35. Baseball Field Experiment Confidence
 63% Q0 (-28)
 19% Q1 (+14) 13% Q2 (+11)
 3% Q3 (+2.6) Added More Trees
  36. 36. Baseball Field Experiment Confidence
 68% Q0 (-23)
 17% Q1 (+12) 11% Q2 (+9)
 2% Q3 (+1.6) And Added More Trees
  37. 37. Baseball Field Experiment Confidence
 44% Q0 (-47)
 12% Q1 (+7) 23% Q2 (+21)
 18% Q3 (+17.6) Added All The Trees
  38. 38. Tree Experiment Confidence
 79% Q0
 10% Q1 5% Q2
 5% Q3 Original Image
  39. 39. Tree Experiment Confidence
 79% Q0
 10% Q1 5% Q2
 5% Q3
  40. 40. Tree Experiment Confidence
 77% Q0 (-2)
 10% Q1 (0) 6% Q2 (+1)
 5% Q3 (0)
  41. 41. Tree Experiment 2 Confidence
 89% Q0
 10% Q1 0% Q2
 0% Q3 Original Image
  42. 42. Tree Experiment 2 Confidence
 89% Q0
 10% Q1 0% Q2
 0% Q3
  43. 43. Tree Experiment 2 Confidence
 87% Q0 (-2)
 11% Q1 (+1) 0% Q2 (0)
 0% Q3 (0)
  44. 44. Trump Tower Experiment Confidence
 0% Q0
 .1% Q1 0% Q2
 99% Q3 Original Image
  45. 45. Trump Tower Experiment Confidence
 16% Q0 (+16)
 25% Q1 (+24.9) 11% Q2 (+11)
 45% Q3 (-54) Added Grass
  46. 46. Trump Tower Experiment Confidence
 34% Q0 (+34)
 47% Q1 (+46.9) 6% Q2 (+6)
 10% Q3 (-89) Built a Wall
  47. 47. Conclusions: • It's possible to predict income levels from space • Underlying data can provide valuable assistance to complex neural networks • Human-based empirical inquiry has legs • Teasing out why it knows what it knows is interesting
  48. 48. Questions: • What does this thing do when you point it at other cities? • What are the similarities and differences between cities from space? • Can we construct a model to account for seasonal variance? • Can we construct a model to account for architectural difference?
  49. 49. cmu.edu Who stamen.com amantiwari.comgbdx.geobigdata.io Carnegie MellonStamen Design Aman TiwariDigital Globe
  50. 50. Thanks (@stamen rules)

×