Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Session 2Fundamentals of Accuracy Assessment        Raymond L Czaplewski       United States Forest Service     Rocky Moun...
Session 2 Topics• Different sample designs    – Simple Random Sampling (Systematic Sampling)    – Stratified Random Sampli...
Hypothetical “real world”                                  True (reference) populationN   N   N   N   N   ^   N   ^     ^ ...
Hypothetical remotely sensed thematic map model for this “real world”                                              Map #1N...
Remotely sensed thematic Map #1                                                                                           ...
True error matrix parameters                        Reference class                                    Reference class    ...
True error matrix parameters, graphical presentation             True Map Land Cover area                                 ...
True error matrix parameters, graphical presentation                                             36%               18%    ...
True error matrix parameters, graphical presentation                             Users Accuracy                           ...
True error matrix parameters, graphical presentation                             Users Accuracy                           ...
True error matrix parameters, graphical presentation                             Users Accuracy                           ...
In the real world, we do not know the true classification for all 900 pixels                                              ...
In the real world, we do know the true classification for 30 sampled pixels                                               ...
In the real world, we do know the true classification for 30 sampled pixels                                               ...
In the real world, we do not know the true classification for all 900 pixels• Let us leave the real world for the next 30 ...
Comparison of true (unknown) error matrix with (known) sample estimate                True (unknown) error matrix         ...
Examples of random sampling error, simple random sample #1, sample size n=30Area of each Land Cover Type      Sample Map L...
Examples of random sampling error, simple random sample #2, sample size n=30Area of each Land Cover Type      Sample Map L...
Examples of random sampling error, simple random sample #3, sample size n=30Area of each Land Cover Type      Sample Map L...
Examples of random sampling error, simple random sample #4, sample size n=30Area of each Land Cover Type      Sample Map L...
Examples of random sampling error, simple random sample #5, sample size n=30Area of each Land Cover Type      Sample Map L...
But how good is the sample estimate? Example, Producer’s Accuracy UrbanArea of each Land Cover Type      Sample Map Land C...
Example: Producers accuracy for urban                   Area of each Land Cover Type                         Sample Map La...
Sample #1, n=60                                                              Truth = 63%                  Number of sample...
Sample #2, n=60                                                              Truth = 63%                  Number of sample...
Sample #3, n=60                                                              Truth = 63%                  Number of sample...
Sample #4, n=60                                                              Truth = 63%                  Number of sample...
Sample #5, n=60                                                              Truth = 63%                  Number of sample...
Sample #6, n=60                                                              Truth = 63%                  Number of sample...
Sample #7, n=60                                                              Truth = 63%                  Number of sample...
Sample #8, n=60                                                              Truth = 63%                  Number of sample...
Sample #9, n=60                                                              Truth = 63%                  Number of sample...
Sample #10, n=60                                                               Truth = 63%                   Number of sam...
Sample #11, n=60                                                               Truth = 63%                   Number of sam...
Sample #12, n=60                                                               Truth = 63%                   Number of sam...
Sample #13, n=60                                                               Truth = 63%                   Number of sam...
Sample #14, n=60                                                               Truth = 63%                   Number of sam...
Sample #15, n=60                                                               Truth = 63%                   Number of sam...
Sample #16, n=60                                                               Truth = 63%                   Number of sam...
Sample #17, n=60                                                               Truth = 63%                   Number of sam...
Sample #18, n=60                                                               Truth = 63%                   Number of sam...
Sample #19, n=60                                                               Truth = 63%                   Number of sam...
Sample #20, n=60                                                               Truth = 63%                   Number of sam...
Sample #21, n=60                                                               Truth = 63%                   Number of sam...
Sample #22, n=60                                                               Truth = 63%                   Number of sam...
Sample #23, n=60                                                               Truth = 63%                   Number of sam...
Sample #24, n=60                                                               Truth = 63%                   Number of sam...
Sample #25, n=60                                                               Truth = 63%                   Number of sam...
Sample #26, n=60                                                               Truth = 63%                   Number of sam...
Sample #27, n=60                                                               Truth = 63%                   Number of sam...
Sample #28, n=60                                                               Truth = 63%                   Number of sam...
Sample #29, n=60                                                               Truth = 63%                   Number of sam...
Sample #30, n=60                                                               Truth = 63%                   Number of sam...
Sample #31, n=60                                                               Truth = 63%                   Number of sam...
Sample #32, n=60                                                               Truth = 63%                   Number of sam...
Sample #33, n=60                                                               Truth = 63%                   Number of sam...
Sample #34, n=60                                                               Truth = 63%                   Number of sam...
Sample #35, n=60                                                               Truth = 63%                   Number of sam...
Sample #36, n=60                                                               Truth = 63%                   Number of sam...
Sample #37, n=60                                                               Truth = 63%                   Number of sam...
Sample #38, n=60                                                               Truth = 63%                   Number of sam...
Sample #39, n=60                                                               Truth = 63%                   Number of sam...
Sample #40, n=60                                                               Truth = 63%                   Number of sam...
Sample #41, n=60                                                               Truth = 63%                   Number of sam...
Sample #42, n=60                                                               Truth = 63%                   Number of sam...
Sample #43, n=60                                                               Truth = 63%                   Number of sam...
Sample #44, n=60                                                               Truth = 63%                   Number of sam...
Sample #45, n=60                                                               Truth = 63%                   Number of sam...
Sample #46, n=60                                                               Truth = 63%                   Number of sam...
Sample #47, n=60                                                               Truth = 63%                   Number of sam...
Sample #48, n=60                                                               Truth = 63%                   Number of sam...
Sample #49, n=60                                                               Truth = 63%                   Number of sam...
Sample #50, n=60                                                               Truth = 63%                   Number of sam...
Truth = 63%                                     300                 Number of samples                                     ...
In the real world, we do not know the true value                                     300                 Number of samples...
In the real world, we do not know the true value, and we have only 1 sample                 Number of samples             ...
Examples of random sampling error, simple random sample• Any single sample estimate can differ from  true error matrix fro...
Examples of random sampling error, simple random sample• How can we improve reliability of estimate?• What if sample size ...
Examples of random sampling error, simple random sample #51, sample size n=150Area of each Land Cover Type      Sample Map...
Examples of random sampling error, simple random sample #52, sample size n=150Area of each Land Cover Type      Sample Map...
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Fundamentals of accuracy_assessment_session2_czaplewski
Upcoming SlideShare
Loading in …5
×

Fundamentals of accuracy_assessment_session2_czaplewski

118 views

Published on

I presented an introductory workshop at the 9th international Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences, also known as Accuracy2010 . This presentation is second of three. The first was conducted by Dr Steve Stehman, and the third by Dr Giles Foody

Published in: Education, Sports, Technology
  • Be the first to comment

  • Be the first to like this

Fundamentals of accuracy_assessment_session2_czaplewski

  1. 1. Session 2Fundamentals of Accuracy Assessment Raymond L Czaplewski United States Forest Service Rocky Mountain Research Station Fort Collins, Colorado USA 1
  2. 2. Session 2 Topics• Different sample designs – Simple Random Sampling (Systematic Sampling) – Stratified Random Sampling• Different sample survey estimators• Different sample sizes, n=30, 60, 150• How close are estimates to true value?• Example of a 30×30 = 900 pixel world 2
  3. 3. Hypothetical “real world” True (reference) populationN N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ Reference classN N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 30×30 = 900 pixels ^ ^ NaturalN N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ Urban CropN N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^ 3
  4. 4. Hypothetical remotely sensed thematic map model for this “real world” Map #1N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 30×30 = 900 pixels N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^ Map classN N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^ Natural NN N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^ Urban^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^ Crop ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^ 4
  5. 5. Remotely sensed thematic Map #1 True Error Matrix Error matrix presented by Steve Stehman Reference class Natural Urban Crop Total Traditional Analysis: Error Natural 226 27 74 327 Map class (Confusion) Matrix Urban 18 108 36 162 Reference Land Cover Crop 89 36 286 411 Mapped Natural Urban Crop Total Total 333 171 396 900 Natural 0.25 0.03 0.08 0.36 Overall accuracy 69% kappa 51% Urban 0.02 0.12 0.04 0.18 Reference class Crop 0.10 0.04 0.32 0.46 Natural Urban Crop Total Natural 25% 3% 8% 36% Map class Total 0.37 0.19 0.44 Urban 2% 12% 4% 18% Crop 10% 4% 32% 46% Total 37% 19% 44% 100% True (reference) population Map #1 N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^ N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^ 30×30 = 900 pixels N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^ N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^ N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^ N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^ N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ N N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^ N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^ N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^ N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^ N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^ N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^ N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ N N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ N N N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^ N ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^ N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N N N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^ N N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^ ^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^ N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^ N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^ N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ N N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ N N ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N N N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^ ^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ N ^ ^ N N N N N N N N N N N N N N N N N N N N N N N N N N ^ ^ ^ ^ ^ ^ ^ N N ^ ^ ^ N N ^ ^ ^ N N N N N N N N ^ N ^ N ^ ^ N N N ^ N N N N N N ^ N ^ N N N N N N N N N ^ N N ^ N N ^ ^ ^ ^ ^ N ^ N N ^ N ^ N ^ 5
  6. 6. True error matrix parameters Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 25% 3% 8% 36% Map classMap class Urban 18 108 36 162 Urban 2% 12% 4% 18% Crop 89 36 286 411 Crop 10% 4% 32% 46% Total 333 171 396 900 Total 37% 19% 44% 100% Overall accuracy 69% kappa 51% 6
  7. 7. True error matrix parameters, graphical presentation True Map Land Cover area Natural 37% 19% 44% UrbanTrue Reference Land Cover area Crop 0% 50% 100% Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 25% 3% 8% 36% Map class Map class Urban 18 108 36 162 Urban 2% 12% 4% 18% Crop 89 36 286 411 Crop 10% 4% 32% 46% Total 333 171 396 900 Total 37% 19% 44% 100% Overall accuracy 69% kappa 51% 7
  8. 8. True error matrix parameters, graphical presentation 36% 18% 46% True Map Land Cover area Natural UrbanTrue Reference Land Cover area Crop 0% 50% 100% Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 25% 3% 8% 36% Map class Map class Urban 18 108 36 162 Urban 2% 12% 4% 18% Crop 89 36 286 411 Crop 10% 4% 32% 46% Total 333 171 396 900 Total 37% 19% 44% 100% Overall accuracy 69% kappa 51% 8
  9. 9. True error matrix parameters, graphical presentation Users Accuracy Producers Accuracy Reference class Reference class Natural Urban Crop Total Natural Urban Crop Natural 69% 8% 23% 100% Natural 68% 16% 19%Map class Map class Urban 11% 67% 22% 100% Urban 5% 63% 9% Crop 22% 9% 70% 100% Crop 27% 21% 72% Total 100% 100% 100% Overall accuracy 69% kappa 51% User’s Accuracy Producer’s Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Overall accuracy True 0% 50% 100% 9
  10. 10. True error matrix parameters, graphical presentation Users Accuracy Producers Accuracy Reference class Reference class Natural Urban Crop Total Natural Urban Crop Natural 69% 8% 23% 100% Natural 68% 16% 19%Map class Map class Urban 11% 67% 22% 100% Urban 5% 63% 9% Crop 22% 9% 70% 100% Crop 27% 21% 72% Total 100% 100% 100% Overall accuracy 69% kappa 51% User’s Accuracy Producer’s Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Overall accuracy True 0% 50% 100% 10
  11. 11. True error matrix parameters, graphical presentation Users Accuracy Producers Accuracy Reference class Reference class Natural Urban Crop Total Natural Urban Crop Natural 69% 8% 23% 100% Natural 68% 16% 19%Map class Map class Urban 11% 67% 22% 100% Urban 5% 63% 9% Crop 22% 9% 70% 100% Crop 27% 21% 72% Total 100% 100% 100% Overall accuracy 69% kappa 51% User’s Accuracy Producer’s Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Overall accuracy True 0% 50% 100% 11
  12. 12. In the real world, we do not know the true classification for all 900 pixels ? True (reference) population True error matrix ? 900 Reference class Natural Urban Crop Map class Natural N ^ Urban Crop ^ 12
  13. 13. In the real world, we do know the true classification for 30 sampled pixels ? Sample of (reference) population True true (reference) population True error matrix 900 Reference class Natural Urban Crop Map class Natural N ^ Urban Crop ^ 13
  14. 14. In the real world, we do know the true classification for 30 sampled pixels ? Sample of true (reference) population True error matrix 900 Error matrix estimate from sample Reference class Natural Urban Crop Total Natural 8 0 2 10 Map class Urban 0 4 2 6 Crop 2 2 10 14 Total 10 6 14 30 Overall accuracy 73% kappa 58% Reference class Natural Urban Crop Total Map class Natural 27% 0% 7% 33% Urban 0% 13% 7% 20% Crop 7% 7% 33% 47% Total 33% 20% 47% 100% 14
  15. 15. In the real world, we do not know the true classification for all 900 pixels• Let us leave the real world for the next 30 minutes to compare – Known estimate of an error matrix with a sample of 30 pixels – Unknown true error matrix for all 900 pixels 15
  16. 16. Comparison of true (unknown) error matrix with (known) sample estimate True (unknown) error matrix Error matrix estimate from sample Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 8 0 2 10 Map class Map class Urban 18 108 36 162 Urban 0 4 2 6 Crop 89 36 286 411 Crop 2 2 10 14 Total 333 171 396 900 Total 10 6 14 30 Overall accuracy 69% kappa 51% Overall accuracy 73% kappa 58% 16
  17. 17. Examples of random sampling error, simple random sample #1, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 17
  18. 18. Examples of random sampling error, simple random sample #2, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 18
  19. 19. Examples of random sampling error, simple random sample #3, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 19
  20. 20. Examples of random sampling error, simple random sample #4, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 20
  21. 21. Examples of random sampling error, simple random sample #5, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 21
  22. 22. But how good is the sample estimate? Example, Producer’s Accuracy UrbanArea of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 22
  23. 23. Example: Producers accuracy for urban Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% Estimated Producers Accuracy = 80% Crop Urban ` Natural 0% 50% 100% 23
  24. 24. Sample #1, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 80% Crop Urban ` Natural 0% 50% 100% 24
  25. 25. Sample #2, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 71% Crop Urban ` Natural 0% 50% 100% 25
  26. 26. Sample #3, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 60% Crop Urban ` Natural 0% 50% 100% 26
  27. 27. Sample #4, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 53% Crop Urban ` Natural 0% 50% 100% 27
  28. 28. Sample #5, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 70% Crop Urban ` Natural 0% 50% 100% 28
  29. 29. Sample #6, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 73% Crop Urban ` Natural 0% 50% 100% 29
  30. 30. Sample #7, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 88% Crop Urban ` Natural 0% 50% 100% 30
  31. 31. Sample #8, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 60% Crop Urban ` Natural 0% 50% 100% 31
  32. 32. Sample #9, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 32
  33. 33. Sample #10, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 45% Crop Urban ` Natural 0% 50% 100% 33
  34. 34. Sample #11, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 34
  35. 35. Sample #12, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 86% Crop Urban ` Natural 0% 50% 100% 35
  36. 36. Sample #13, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 75% Crop Urban ` Natural 0% 50% 100% 36
  37. 37. Sample #14, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 73% Crop Urban ` Natural 0% 50% 100% 37
  38. 38. Sample #15, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 47% Crop Urban ` Natural 0% 50% 100% 38
  39. 39. Sample #16, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 57% Crop Urban ` Natural 0% 50% 100% 39
  40. 40. Sample #17, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 64% Crop Urban ` Natural 0% 50% 100% 40
  41. 41. Sample #18, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 70% Crop Urban ` Natural 0% 50% 100% 41
  42. 42. Sample #19, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 42
  43. 43. Sample #20, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 77% Crop Urban ` Natural 0% 50% 100% 43
  44. 44. Sample #21, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 44
  45. 45. Sample #22, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 76% Crop Urban ` Natural 0% 50% 100% 45
  46. 46. Sample #23, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 100% Crop Urban ` Natural 0% 50% 100% 46
  47. 47. Sample #24, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 90% Crop Urban ` Natural 0% 50% 100% 47
  48. 48. Sample #25, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 50% Crop Urban ` Natural 0% 50% 100% 48
  49. 49. Sample #26, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 77% Crop Urban ` Natural 0% 50% 100% 49
  50. 50. Sample #27, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 91% Crop Urban ` Natural 0% 50% 100% 50
  51. 51. Sample #28, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 51
  52. 52. Sample #29, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 27% Crop Urban ` Natural 0% 50% 100% 52
  53. 53. Sample #30, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 55% Crop Urban ` Natural 0% 50% 100% 53
  54. 54. Sample #31, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 56% Crop Urban ` Natural 0% 50% 100% 54
  55. 55. Sample #32, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 55
  56. 56. Sample #33, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 56
  57. 57. Sample #34, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 71% Crop Urban ` Natural 0% 50% 100% 57
  58. 58. Sample #35, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 83% Crop Urban ` Natural 0% 50% 100% 58
  59. 59. Sample #36, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 62% Crop Urban ` Natural 0% 50% 100% 59
  60. 60. Sample #37, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 64% Crop Urban ` Natural 0% 50% 100% 60
  61. 61. Sample #38, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 88% Crop Urban ` Natural 0% 50% 100% 61
  62. 62. Sample #39, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 57% Crop Urban ` Natural 0% 50% 100% 62
  63. 63. Sample #40, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 56% Crop Urban ` Natural 0% 50% 100% 63
  64. 64. Sample #41, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 57% Crop Urban ` Natural 0% 50% 100% 64
  65. 65. Sample #42, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 73% Crop Urban ` Natural 0% 50% 100% 65
  66. 66. Sample #43, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 66
  67. 67. Sample #44, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 55% Crop Urban ` Natural 0% 50% 100% 67
  68. 68. Sample #45, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 63% Crop Urban ` Natural 0% 50% 100% 68
  69. 69. Sample #46, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 71% Crop Urban ` Natural 0% 50% 100% 69
  70. 70. Sample #47, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 55% Crop Urban ` Natural 0% 50% 100% 70
  71. 71. Sample #48, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 71
  72. 72. Sample #49, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 77% Crop Urban ` Natural 0% 50% 100% 72
  73. 73. Sample #50, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 89% Crop Urban ` Natural 0% 50% 100% 73
  74. 74. Truth = 63% 300 Number of samples 200 100 0 0 20 40 60 80 100True accuracy = 63% % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 74
  75. 75. In the real world, we do not know the true value 300 Number of samples 200 100 0 0 20 40 60 80 100True accuracy = 63% % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 75
  76. 76. In the real world, we do not know the true value, and we have only 1 sample Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 76
  77. 77. Examples of random sampling error, simple random sample• Any single sample estimate can differ from true error matrix from random sampling error• Given our only sample with n=60, the estimated urban producers accuracy = 78% even though the true value is 63%• However, the sample estimate is expected to equal the true value over all possible samples 77
  78. 78. Examples of random sampling error, simple random sample• How can we improve reliability of estimate?• What if sample size increased from n=60 to n=150? 78
  79. 79. Examples of random sampling error, simple random sample #51, sample size n=150Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 79
  80. 80. Examples of random sampling error, simple random sample #52, sample size n=150Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 80

×