1.
Session 2Fundamentals of Accuracy Assessment Raymond L Czaplewski United States Forest Service Rocky Mountain Research Station Fort Collins, Colorado USA 1

2.
Session 2 Topics• Different sample designs – Simple Random Sampling (Systematic Sampling) – Stratified Random Sampling• Different sample survey estimators• Different sample sizes, n=30, 60, 150• How close are estimates to true value?• Example of a 30×30 = 900 pixel world 2

3.
Hypothetical “real world” True (reference) populationN N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ Reference classN N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 30×30 = 900 pixels ^ ^ NaturalN N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ Urban CropN N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^ 3

4.
Hypothetical remotely sensed thematic map model for this “real world” Map #1N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 30×30 = 900 pixels N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^ Map classN N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^ Natural NN N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^ Urban^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^ Crop ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^ 4

5.
Remotely sensed thematic Map #1 True Error Matrix Error matrix presented by Steve Stehman Reference class Natural Urban Crop Total Traditional Analysis: Error Natural 226 27 74 327 Map class (Confusion) Matrix Urban 18 108 36 162 Reference Land Cover Crop 89 36 286 411 Mapped Natural Urban Crop Total Total 333 171 396 900 Natural 0.25 0.03 0.08 0.36 Overall accuracy 69% kappa 51% Urban 0.02 0.12 0.04 0.18 Reference class Crop 0.10 0.04 0.32 0.46 Natural Urban Crop Total Natural 25% 3% 8% 36% Map class Total 0.37 0.19 0.44 Urban 2% 12% 4% 18% Crop 10% 4% 32% 46% Total 37% 19% 44% 100% True (reference) population Map #1 N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^ N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^ 30×30 = 900 pixels N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^ N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^ N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^ N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^ N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ N N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^ N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^ N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^ N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^ N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^ N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^ N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ N N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ N N N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^ N ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^ N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N N N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^ N N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^ ^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^ N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^ N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^ N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ N N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ N N ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N N N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^ ^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ N ^ ^ N N N N N N N N N N N N N N N N N N N N N N N N N N ^ ^ ^ ^ ^ ^ ^ N N ^ ^ ^ N N ^ ^ ^ N N N N N N N N ^ N ^ N ^ ^ N N N ^ N N N N N N ^ N ^ N N N N N N N N N ^ N N ^ N N ^ ^ ^ ^ ^ N ^ N N ^ N ^ N ^ 5

6.
True error matrix parameters Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 25% 3% 8% 36% Map classMap class Urban 18 108 36 162 Urban 2% 12% 4% 18% Crop 89 36 286 411 Crop 10% 4% 32% 46% Total 333 171 396 900 Total 37% 19% 44% 100% Overall accuracy 69% kappa 51% 6

7.
True error matrix parameters, graphical presentation True Map Land Cover area Natural 37% 19% 44% UrbanTrue Reference Land Cover area Crop 0% 50% 100% Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 25% 3% 8% 36% Map class Map class Urban 18 108 36 162 Urban 2% 12% 4% 18% Crop 89 36 286 411 Crop 10% 4% 32% 46% Total 333 171 396 900 Total 37% 19% 44% 100% Overall accuracy 69% kappa 51% 7

8.
True error matrix parameters, graphical presentation 36% 18% 46% True Map Land Cover area Natural UrbanTrue Reference Land Cover area Crop 0% 50% 100% Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 25% 3% 8% 36% Map class Map class Urban 18 108 36 162 Urban 2% 12% 4% 18% Crop 89 36 286 411 Crop 10% 4% 32% 46% Total 333 171 396 900 Total 37% 19% 44% 100% Overall accuracy 69% kappa 51% 8

9.
True error matrix parameters, graphical presentation Users Accuracy Producers Accuracy Reference class Reference class Natural Urban Crop Total Natural Urban Crop Natural 69% 8% 23% 100% Natural 68% 16% 19%Map class Map class Urban 11% 67% 22% 100% Urban 5% 63% 9% Crop 22% 9% 70% 100% Crop 27% 21% 72% Total 100% 100% 100% Overall accuracy 69% kappa 51% User’s Accuracy Producer’s Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Overall accuracy True 0% 50% 100% 9

10.
True error matrix parameters, graphical presentation Users Accuracy Producers Accuracy Reference class Reference class Natural Urban Crop Total Natural Urban Crop Natural 69% 8% 23% 100% Natural 68% 16% 19%Map class Map class Urban 11% 67% 22% 100% Urban 5% 63% 9% Crop 22% 9% 70% 100% Crop 27% 21% 72% Total 100% 100% 100% Overall accuracy 69% kappa 51% User’s Accuracy Producer’s Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Overall accuracy True 0% 50% 100% 10

11.
True error matrix parameters, graphical presentation Users Accuracy Producers Accuracy Reference class Reference class Natural Urban Crop Total Natural Urban Crop Natural 69% 8% 23% 100% Natural 68% 16% 19%Map class Map class Urban 11% 67% 22% 100% Urban 5% 63% 9% Crop 22% 9% 70% 100% Crop 27% 21% 72% Total 100% 100% 100% Overall accuracy 69% kappa 51% User’s Accuracy Producer’s Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Overall accuracy True 0% 50% 100% 11

12.
In the real world, we do not know the true classification for all 900 pixels ? True (reference) population True error matrix ? 900 Reference class Natural Urban Crop Map class Natural N ^ Urban Crop ^ 12

13.
In the real world, we do know the true classification for 30 sampled pixels ? Sample of (reference) population True true (reference) population True error matrix 900 Reference class Natural Urban Crop Map class Natural N ^ Urban Crop ^ 13

14.
In the real world, we do know the true classification for 30 sampled pixels ? Sample of true (reference) population True error matrix 900 Error matrix estimate from sample Reference class Natural Urban Crop Total Natural 8 0 2 10 Map class Urban 0 4 2 6 Crop 2 2 10 14 Total 10 6 14 30 Overall accuracy 73% kappa 58% Reference class Natural Urban Crop Total Map class Natural 27% 0% 7% 33% Urban 0% 13% 7% 20% Crop 7% 7% 33% 47% Total 33% 20% 47% 100% 14

15.
In the real world, we do not know the true classification for all 900 pixels• Let us leave the real world for the next 30 minutes to compare – Known estimate of an error matrix with a sample of 30 pixels – Unknown true error matrix for all 900 pixels 15

16.
Comparison of true (unknown) error matrix with (known) sample estimate True (unknown) error matrix Error matrix estimate from sample Reference class Reference class Natural Urban Crop Total Natural Urban Crop Total Natural 226 27 74 327 Natural 8 0 2 10 Map class Map class Urban 18 108 36 162 Urban 0 4 2 6 Crop 89 36 286 411 Crop 2 2 10 14 Total 333 171 396 900 Total 10 6 14 30 Overall accuracy 69% kappa 51% Overall accuracy 73% kappa 58% 16

17.
Examples of random sampling error, simple random sample #1, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 17

18.
Examples of random sampling error, simple random sample #2, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 18

19.
Examples of random sampling error, simple random sample #3, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 19

20.
Examples of random sampling error, simple random sample #4, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 20

21.
Examples of random sampling error, simple random sample #5, sample size n=30Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 21

22.
But how good is the sample estimate? Example, Producer’s Accuracy UrbanArea of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 22

23.
Example: Producers accuracy for urban Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% Estimated Producers Accuracy = 80% Crop Urban ` Natural 0% 50% 100% 23

24.
Sample #1, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 80% Crop Urban ` Natural 0% 50% 100% 24

25.
Sample #2, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 71% Crop Urban ` Natural 0% 50% 100% 25

26.
Sample #3, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 60% Crop Urban ` Natural 0% 50% 100% 26

27.
Sample #4, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 53% Crop Urban ` Natural 0% 50% 100% 27

28.
Sample #5, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 70% Crop Urban ` Natural 0% 50% 100% 28

29.
Sample #6, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 73% Crop Urban ` Natural 0% 50% 100% 29

30.
Sample #7, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 88% Crop Urban ` Natural 0% 50% 100% 30

31.
Sample #8, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 60% Crop Urban ` Natural 0% 50% 100% 31

32.
Sample #9, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 32

33.
Sample #10, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 45% Crop Urban ` Natural 0% 50% 100% 33

34.
Sample #11, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 34

35.
Sample #12, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 86% Crop Urban ` Natural 0% 50% 100% 35

36.
Sample #13, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 75% Crop Urban ` Natural 0% 50% 100% 36

37.
Sample #14, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 73% Crop Urban ` Natural 0% 50% 100% 37

38.
Sample #15, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 47% Crop Urban ` Natural 0% 50% 100% 38

39.
Sample #16, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 57% Crop Urban ` Natural 0% 50% 100% 39

40.
Sample #17, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 64% Crop Urban ` Natural 0% 50% 100% 40

41.
Sample #18, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 70% Crop Urban ` Natural 0% 50% 100% 41

42.
Sample #19, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 42

43.
Sample #20, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 77% Crop Urban ` Natural 0% 50% 100% 43

44.
Sample #21, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 44

45.
Sample #22, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 76% Crop Urban ` Natural 0% 50% 100% 45

46.
Sample #23, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 100% Crop Urban ` Natural 0% 50% 100% 46

47.
Sample #24, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 90% Crop Urban ` Natural 0% 50% 100% 47

48.
Sample #25, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 50% Crop Urban ` Natural 0% 50% 100% 48

49.
Sample #26, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 77% Crop Urban ` Natural 0% 50% 100% 49

50.
Sample #27, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 91% Crop Urban ` Natural 0% 50% 100% 50

51.
Sample #28, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 51

52.
Sample #29, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 27% Crop Urban ` Natural 0% 50% 100% 52

53.
Sample #30, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 55% Crop Urban ` Natural 0% 50% 100% 53

54.
Sample #31, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 56% Crop Urban ` Natural 0% 50% 100% 54

55.
Sample #32, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 55

56.
Sample #33, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 56

57.
Sample #34, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 71% Crop Urban ` Natural 0% 50% 100% 57

58.
Sample #35, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 83% Crop Urban ` Natural 0% 50% 100% 58

59.
Sample #36, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 62% Crop Urban ` Natural 0% 50% 100% 59

60.
Sample #37, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 64% Crop Urban ` Natural 0% 50% 100% 60

61.
Sample #38, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 88% Crop Urban ` Natural 0% 50% 100% 61

62.
Sample #39, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 57% Crop Urban ` Natural 0% 50% 100% 62

63.
Sample #40, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 56% Crop Urban ` Natural 0% 50% 100% 63

64.
Sample #41, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 57% Crop Urban ` Natural 0% 50% 100% 64

65.
Sample #42, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 73% Crop Urban ` Natural 0% 50% 100% 65

66.
Sample #43, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 40% Crop Urban ` Natural 0% 50% 100% 66

67.
Sample #44, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 55% Crop Urban ` Natural 0% 50% 100% 67

68.
Sample #45, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 63% Crop Urban ` Natural 0% 50% 100% 68

69.
Sample #46, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 71% Crop Urban ` Natural 0% 50% 100% 69

70.
Sample #47, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 55% Crop Urban ` Natural 0% 50% 100% 70

71.
Sample #48, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 67% Crop Urban ` Natural 0% 50% 100% 71

72.
Sample #49, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 77% Crop Urban ` Natural 0% 50% 100% 72

73.
Sample #50, n=60 Truth = 63% Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 89% Crop Urban ` Natural 0% 50% 100% 73

74.
Truth = 63% 300 Number of samples 200 100 0 0 20 40 60 80 100True accuracy = 63% % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 74

75.
In the real world, we do not know the true value 300 Number of samples 200 100 0 0 20 40 60 80 100True accuracy = 63% % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 75

76.
In the real world, we do not know the true value, and we have only 1 sample Number of samples 10 5 0 0 20 40 60 80 100 % Producers Accuracy Estimated Producers Accuracy = 78% Crop Urban ` Natural 0% 50% 100% 76

77.
Examples of random sampling error, simple random sample• Any single sample estimate can differ from true error matrix from random sampling error• Given our only sample with n=60, the estimated urban producers accuracy = 78% even though the true value is 63%• However, the sample estimate is expected to equal the true value over all possible samples 77

78.
Examples of random sampling error, simple random sample• How can we improve reliability of estimate?• What if sample size increased from n=60 to n=150? 78

79.
Examples of random sampling error, simple random sample #51, sample size n=150Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 79

80.
Examples of random sampling error, simple random sample #52, sample size n=150Area of each Land Cover Type Sample Map Land Cover area True Map Land Cover area Natural Sample Reference Land Cover area Urban True Reference Land Cover area Crop 0% 50% 100% Users Accuracy Producers Accuracy Crop Crop Urban Urban Natural Natural 0% 50% 100% 0% 50% 100% kappa Sample Overall accuracy True 0% 50% 100% 80