SIMS Quant Course Lecture 4

1,943 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,943
On SlideShare
0
From Embeds
0
Number of Embeds
124
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

SIMS Quant Course Lecture 4

  1. 1. Those who don’t know statistics are condemned to reinvent it… David Freedman
  2. 2. All you ever wanted to know about the histogram and more ...
  3. 3. Distribution of No of Graphics on web pages (N=1873) Mean = 17.93 N = 1873 Graphic Count Std. Dev = 17.92 Median = 16.00 1 95.0 90.0 85.0 80.0 75.0 70.0 65.0 60.0 55.0 50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 400 300 200 100 0
  4. 4. Horizontal Scale 2
  5. 5. Distribution of Redundant Link % on web pages (N =1861) Std. Dev = 37.33 Mean = 22.1 N = 1861.00 Median = 14 3 480.0 440.0 400.0 360.0 320.0 280.0 240.0 200.0 160.0 120.0 80.0 40.0 0.0 1000 800 600 400 200 0
  6. 6. Plotting a histogram: endpoint convention, plot frequencies, make equal intervals etc.
  7. 7. Frequency Table convention: include the left endpoint in the class interval 4
  8. 8. Frequency/Probability
  9. 9. No of fonts used on a web-page Frequency /probability 5 0/ 0 200/ .1 400/ .2 600/ .3 800/ .4 1000/ .5 Frequency 110 430 860 280 180 40 20 10 1 3 5 7 9 11 13 15 Probability .06 .22 .45 .15 .09 .02 .01 .01
  10. 10. Cleaning up a histogram: getting rid of outliers
  11. 11. Distribution of word count (N=1903) Std. Dev = 725.24 Mean = 393.2 Maximum = 20,357 Minimum = 0 Median = 223 20000.0 18000.0 16000.0 14000.0 12000.0 10000.0 8000.0 6000.0 4000.0 2000.0 0.0 1600 1400 1200 1000 800 600 400 200 0
  12. 12. Distribution of word count (N=1897) top six removed Std. Dev = 474.04 Mean = 368.0 Maximum = 4132 Minimum = 0 Median = 223 7 4000.0 3600.0 3200.0 2800.0 2400.0 2000.0 1600.0 1200.0 800.0 400.0 0.0 800 600 400 200 0
  13. 13. Distribution of word count (N=1873) Std. Dev = 360.30 Mean = 333.4 Maximum = 4132 Minimum = 0 WORDCNT2 Median = 220 2400.0 2200.0 2000.0 1800.0 1600.0 1400.0 1200.0 1000.0 800.0 600.0 400.0 200.0 0.0 500 400 300 200 100 0
  14. 14. What can histograms tell you
  15. 15. Distribution of link count on good & bad web-pages Good Sites Bad Sites 8 2 8 0 . 0 2 4 0 . 0 2 0 0 . 0 1 6 0 . 0 1 2 0 . 0 8 0 . 0 4 0 . 0 0 . 0 3 0 0 2 0 0 1 0 0 0
  16. 16. Making inferences from histograms: Incidence of riots and temperature 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 0 1 1 0 temperature 9
  17. 17. Mean and Median <ul><li>Mean shifts around, </li></ul><ul><li>Median does not shift much, is more stable </li></ul><ul><li>Computing Median: </li></ul><ul><li>for odd numbered N </li></ul><ul><ul><li>find middle number </li></ul></ul><ul><li>For even numbered N </li></ul><ul><ul><li>interpolate between middle 2, </li></ul></ul><ul><ul><li>e.g. if it is 7 and 9, then 8 is the median </li></ul></ul>Mean is arithmetic average, median is 50% point Mean is point where graph balances
  18. 18. The instability of means and standard deviations
  19. 19. Add two numbers: watch the mean, median, & SD
  20. 20. Add one outlier...
  21. 21. Standard Deviation: a measure of spread
  22. 22. Same mean, different spread S D S D 10
  23. 23. The Standard Deviation
  24. 24. <ul><li>The SD says how far away numbers </li></ul><ul><li>on a list are from their average . </li></ul><ul><li>Most entries on the list will be </li></ul><ul><li>somewhere around one SD away </li></ul><ul><li>from the average. Very few will be </li></ul><ul><li>more than two or three SD’s away. </li></ul>
  25. 25. Understanding the standard deviation <ul><li>Lets start with a list: 1, 2, 2, 3 </li></ul>0% 25% 50% Histogram is symmetric about 2, 2 is mean, and 50% to left of 2, 50% to right
  26. 26. <ul><li>List: 1, 2, 2, 3 </li></ul><ul><li>Average = 2 </li></ul><ul><li>SD = .8 </li></ul>0% 25% 50% 0% 25% 50% List: 1, 2, 2, 5 Average =2.5 SD = 1.73 0% 25% 50% List: 1, 2, 2, 7 Average =3 SD = 2.71
  27. 27. <ul><li>List: 20, 10, 15, 15 </li></ul><ul><li>Average = 15 </li></ul><ul><li>Find deviations from average = </li></ul><ul><li>5, -5, 0, 0 </li></ul><ul><li>Square the deviations: </li></ul><ul><li>(5) 2 (-5) 2 (0) 2 (0) 2 = 50 </li></ul><ul><li>divide it by N-1 = 50/3 = 16.67 </li></ul><ul><li>Square root it =  16.67 = 4.08 </li></ul>Computing the standard deviation
  28. 28. Properties of the standard deviation <ul><li>The standard deviation is in the same units as the mean </li></ul><ul><li>The standard deviation is inversely related to sample size (therefore as a measure of spread it is biased) </li></ul><ul><li>In normally distributed data 68% of the sample lies within 1 SD </li></ul>
  29. 29. Properties of the Normal Probability Curve <ul><li>The graph is symmetric about the mean (the part to the right is a mirror image of the part to the left) </li></ul><ul><li>The total area under the curve equals 100% </li></ul><ul><li>Curve is always above horizontal axis </li></ul><ul><li>Appears to stop after a certain point (the curve gets really low) </li></ul>
  30. 30. <ul><li>The graph is symmetric about the mean = </li></ul><ul><li>The total area under the curve equals 100% </li></ul><ul><li>Mean to 1 SD = +- 68% </li></ul><ul><li>Mean to 2 SD = +- 95% </li></ul><ul><li>Mean to 3 SD = +- 99.7% </li></ul><ul><li>You can disregard rest of curve </li></ul>1 SD= 68% 2 SD = 95% 3 SD= 99.7% 11
  31. 31. Distribution of judges ratings for the Webby Awards Std. Dev = 1.98 Mean = 6.3 N = 1867.00 Skewness = -.43 Kurtosis = -.201 Median = 6.3 12 10.0 9.0 8.0 7.0 6.0 5.0 4.0 3.0 2.0 1.0 500 400 300 200 100 0
  32. 32. It is a remarkable fact that many histograms in real life tend to follow the Normal Curve. For such histograms, the mean and SD are good summary statistics . The average pins down the center, while the SD gives the spread. For histogram which do not follow the normal Curve, the mean and SD are not good summary statistics. What when the histogram is not normal ...
  33. 33. +- 3 SD = (384 * 3) = 1152 Mean - 1152 = about 30% sample had negative number of links Mean = 348.3 Std. Dev = 384.83 Distribution of word count on web pages 13 2800.0 2600.0 2400.0 2200.0 2000.0 1800.0 1600.0 1400.0 1200.0 1000.0 800.0 600.0 400.0 200.0 0.0 500 400 300 200 100 0
  34. 34. Note. A percentile is a score below which a certain % of sample is When SD is influenced by outliers Use inter quartile range 75th percentile - 25th percentile
  35. 35. Measures of Normality <ul><li>Visual examination </li></ul><ul><li>Skewness: measure of symmetry </li></ul>Positively Skewed Negatively Skewed Symmetric 14
  36. 36. Kurtosis: Does it cluster in the middle? Large tail Small tail Normal Tail <ul><li>Kurtosis is based on a distributions tail. </li></ul><ul><ul><li>Distributions with a large tail: leptokurtic </li></ul></ul><ul><ul><li>Distributions with a small tail: platykurtic </li></ul></ul><ul><ul><li>Distributions with a normal tail: mesokurtic </li></ul></ul>15
  37. 37. Positively Skewed and Leptokurtic: Word Count Std. Dev = 725.24 Mean = 393.2 N = 1903.00 Kurtosis = 321.84 Skewness = 13.62 Median = 223 20000.0 18000.0 16000.0 14000.0 12000.0 10000.0 8000.0 6000.0 4000.0 2000.0 0.0 1600 1400 1200 1000 800 600 400 200 0
  38. 38. Distribution of word count (N=1897) top six removed Std. Dev = 474.04 Mean = 368.0 N = 1897.00 Skewness = 3.49 Kurtosis = 16.40 Median = 223 4000.0 3600.0 3200.0 2800.0 2400.0 2000.0 1600.0 1200.0 800.0 400.0 0.0 800 600 400 200 0
  39. 39. Degree of Freedom <ul><li>The number of independent pieces of information remaining after estimating one or more parameters </li></ul><ul><li>Example: List= 1, 2, 3, 4 Average= 2.5 </li></ul><ul><li>For average to remain the same three of the numbers can be anything you want, fourth is fixed </li></ul><ul><li>New List = 1, 5, 2.5, __ Average = 2.5 </li></ul>

×