Wayne gray presentation

586 views

Published on

120 years ago the emergent field of experimental psychology became embroiled in debates as to whether plateaus in performance are real (or not) and if so whether they were due to periods in which league-stepping methods (originally defined as a hierarchy of habits that enabled experts to step leagues while novices were ``bustling over furlongs or inches'') were being acquired (or not). 20 years ago both the human-computer interaction and cognitive science communities were seized with concerns over performance plateaus (i.e., extended periods of stable suboptimal performance) from experts. I briefly review this history with the aim of drawing distinctions between performance asymptotes and performance plateaus, and argue that remediating one is the domain of design while remediating the other is the domain of training.

Published in: Technology
1 Comment
0 Likes
Statistics
Notes
  • Type of expertise and task are important variables. Consider, for example, depth of knowledge of a field, or how several fields intersect. This can continue to grow, and could have performance gains on complex problem-solving tasks that require judgment. Consider case-based learning--is there an asymptote beyond which further cases cannot be processed?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
586
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Wayne gray presentation

  1. 1. Asymptotes, Plateaus, and Limits to Human Performance Wayne D. Gray Cognitive Science Department, Rensselaer Polytechnic Institute Presentation to the IBM Cognitive Systems Institute – 2015.Feb.05 Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 1 / 30
  2. 2. THANKS TO THESE SUPPORTERS! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 2 / 30
  3. 3. The Cognitive Science of The Little Engine that Could Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 3 / 30 A view of human systems (aka people) as attempting to optimize performance under constraints. Where these constraints come from: Bounds on our innate cognitive capacities. Limits to our acquired skill and knowledge. The structure of the external task environment in which we operate. The goals we are trying to achieve. If this human system is chugging along, like the Little Engine that Could, at some given performance ceiling – we can then ask whether this limit this due: to external factors that can be altered, to internal factors that can be altered, or to factors that cannot be altered.
  4. 4. PERFORMANCE GENERALLY IMPROVES WITH PRACTICE. . . BUT WHY? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 4 / 30 W. L. Bryan With practice, performance whether it is with telegraphy, typing, software programs, arithmetic, programming, mnemonics, or video games generally improves. But experts are not simply faster than novices; rather, they develop a hierarchy of habits than enable them to “step leagues” while novices are “bustling over furlongs or inches”. CITATION Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27–53 Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: the acquisition of a hierarchy of habits. Psychological Review, 6(4), 345–375
  5. 5. PERFORMANCE GENERALLY IMPROVES WITH PRACTICE . . . EXCEPT WHEN IT DOESN’T! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 5 / 30 Hypothesized that telegraphic expertise consisted of a hierarchy of habits. Plateaus were periods in which elements at one level of the hierarchy were being combined so as to be used at a higher level. dots and dashes → letters → words → phrases But . . .
  6. 6. THESE PLATEAUS COULD LAST A LONG LONG TIME Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 6 / 30 E. L. Thorndike “I venture to prophesy that the thousand bookkeepers in, say, the grocery stores of New York who have each had a thousand hours of practice at addition, are still, on the average, adding less than two-thirds as rapidly as they could, and making twice as many errors as they would at their limit.” “It appears likely that the majority of teachers make no gain in efficiency after their third year of service, but I am confident that the majority of such teachers could teach very much better than they do.” “It seems to me therefore that mental training in schools, in industry and in morals is characterized, over and over and over again, by spurious limits – by levels or plateaus of efficiency which could be surpassed.” CITATION Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of Learning. NYC: Teachers College, Columbia University
  7. 7. PLAN FOR THIS TALK Skip the remainder of the first 90 years (1897 – 1987) of scientific research on expert performance. Jump to 1987 to Carroll & Rosson’s Paradox of the Active User and Ericsson’s (1993) Deliberate Practice and the view shared by both that expert is not good enough. Three types of performance asymptotes and one type of plateau. Summarize everything in time for questions! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 7 / 30
  8. 8. Outline 1 Mere Expertise is Not Good Enough 2 Plateaus and Asymptotes 3 Outside the Lab: Good → Better → Best! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 8 / 30
  9. 9. OUTSIDE THE LAB – THE PARADOX OF THE ACTIVE USER Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 9 / 30 Jack Carroll Mary Beth Rosson In 1987, Carroll and Rosson coined the term, Paradox of the Active User, to refer to the “suboptimal use of office productivity software” by people who use the systems daily across the course of weeks, months, and years. Carroll and Rosson, who at that time worked for the IBM Watson Research Center, shared the HCI community’s concern that the expected productivity gains of the computer revolution were not occurring.
  10. 10. MERE EXPERTISE IS NOT GOOD ENOUGH Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 10 / 30 Anders Ericsson A few years later, based on his studies of human expertise, Ericsson (1993) concluded that, “the belief that a sufficient amount of experience or practice leads to maximal performance appears incorrect”. CITATION Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406
  11. 11. LIMITS TO EXPERTISE Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 11 / 30 After years of lurking in the background, the plateau had returned to front and center.
  12. 12. RESOLVING THE PARADOX OF THE ACTIVE USER Suboptimal performance can be amazingly stable! Fu and Gray (2004). What is optimal? Is it optimal to learn 100 different commands and procedures that each do one thing very well (fast, precisely, . . . )? Is it optimal to learn one command and procedure that can be tweaked into doing 100 different things? (slowly, approximately, . . . ) Breaking a habit – the case of hunt & peck versus touchtyping Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 12 / 30 CITATION Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user: Stable suboptimal performance in interactive tasks. Cognitive Science, 28(6), 901–935
  13. 13. Outline 1 Mere Expertise is Not Good Enough 2 Plateaus and Asymptotes 3 Outside the Lab: Good → Better → Best! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 13 / 30
  14. 14. PLATEAUS VERSUS ASYMPTOTES: POLE VAULTING Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 14 / 30 A history of technological innovations as the composition of the pole changed from ash (wood), to bamboo (wood), to fiberglass/carbon. Each technology, enabled pole vaulters to break new records Followed by invention of new methods that resulted in new rounds of record breaking as those methods were adopted and adapted by athletes. Asymptote → New technology → New methods → Asymptote
  15. 15. PLATEAUS VERSUS ASYMPTOTES: HIGH JUMPING Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 15 / 30 A history of innovations in methods. Method – Fosbury Flop could have been invented earlier. Plateau → New Method → New Plateau The Scissors and Straddle technique for high jumping. The Fosbury Flop technique for high jumping.
  16. 16. PLATEAUS AND ASYMPTOTES How do plateaus and asymptotes come about?? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 16 / 30
  17. 17. ASYMPTOTE DUE TO ARTIFACT DESIGN Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 17 / 30 Crossman’s (1959) study of cigar rollers in Cuba. Plot shows a continued increase in performance over a two year period (estimated as 3 million cigars) and then a flattening of the curve. Newell and Rosenbloom (1981, p. 7) attribute this flattening to a “known lower bound for the performance time” in this task; namely, the “cycle time of the machine.”
  18. 18. ASYMPTOTE DUE TO SYSTEM DESIGN Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 18 / 30 A field trial of two workstations for Telephone Operators (Gray, John, & Atwood, 1993). Expected call times to decrease across the 4-mon trial. But after 2-mon worktimes stabilized with times per call slower than for the old workstation Slow enough to increase annual operating costs by $6.2 million (in 2014 dollars). Diagnosis: Designers believed call time driven by the # of keys-per-call. Predicted savings of 4.1 s in mean item per call for annual savings of $24m. BUT cognitive modeling showed that old workstation enabled Operators to interleave keypresses, chats with customer, and wait time for external databases. Conclusion: Based on the models, Operators were becoming more expert at the new workstation but asymptotes due to systems design prevented these gains in expertise from yielding performance increments.
  19. 19. ASYMPTOTE DUE TO MEASUREMENT METHOD Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 19 / 30 Example from Space Fortress in which a measurement method was introduced about 1994 that has affected many of the conclusions reached by researchers ever since. To simplify the story . . . changed scoring so that it included 4 component scores and one overall score. These measures can be shown to be (1) not independent of each other and (2) two of these measures asymptote even as skilled performance increases.
  20. 20. ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN The Digit Span Task – An important part of the Wechsler Adult Intelligence Scale (WAIS) IQ test (and others). Digits (0-9) are read at the rate of 1 per sec. Followed immediately by ordered recall. If all digits were recalled correctly, the length of the next run of digits was increased by 1. If all are not correct, the next run is decreased by 1. The population norm is 7 ± 2. Is this an asymptote due to limitations built into the human brain? or Is this a plateau due to massive stable suboptimal performance on the part of the entire human population? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 20 / 30
  21. 21. ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN Well? . . . This is an IQ test item!! Therefore it MUST be measuring an individual difference variable that differs between humans but is stable, or asymptotic, for any given individual. Right . . . ? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 21 / 30
  22. 22. IS HUMAN DIGIT SPAN A POPULATION ASYMPTOTE OR PLATEAU? – PLATEAU!! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 22 / 30 The difference between a plateau and asymptote is made clear by the existence of extreme experts with a known history of transcending the plateau. DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD SFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSF 10 20 30 40 50 60 70 80 0 10 20 30 40 50 Practice (in 5 day blocks) DigitSpan Perhaps the human asymptote is 80 ± 2 instead. . . ????
  23. 23. Strategy Plateaus and the Paradox of the Active User. Issue: The difference between a plateau and an asymptote may be hard to determine. Asymptotes may reflect a problem that can be fixed by design (whether artifact design or system design); however, plateaus due to strategy-induced suboptimality may arise when the strategies deployed do not enable utility maximization in the task environment. Overcoming such strategy-induced suboptimality is usually very difficult. Example of transfer from visually-guided to touch typing. Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 23 / 30
  24. 24. Outline 1 Mere Expertise is Not Good Enough 2 Plateaus and Asymptotes 3 Outside the Lab: Good → Better → Best! Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 24 / 30
  25. 25. Real World Tasks? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 25 / 30
  26. 26. Real World Tasks? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 26 / 30
  27. 27. Real World Tasks? Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 26 / 30
  28. 28. Outside the Lab – Resolving the Paradox of the Active User Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 27 / 30 If “most people and professionals reach a stable performance asymptote within a limited time period” (Ericsson, 2004), then practice does not make perfect. Unless, as a society, we can be content with “stable suboptimal performance plateaus” then something more is needed. We suggest that this “something more” is research into the acquisition of expertise in mundane (i.e., everyday) task environments.
  29. 29. Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 28 / 30 Thank You!!
  30. 30. REFERENCES I Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology of the telegraphic language. Psychological Review, 4(1), 27–53. Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: the acquisition of a hierarchy of habits. Psychological Review, 6(4), 345–375. Crossman, E. R. F. W. (1959). A theory of the acquisition of speed-skill. Ergonomics, 2(2), 153–166. Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10, S), S70–S81. Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 29 / 30
  31. 31. REFERENCES II Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user: Stable suboptimal performance in interactive tasks. Cognitive Science, 28(6), 901–935. Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from a computer game trainer to flight. Human Factors, 36(3), 387–405. Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world performance. Human-Computer Interaction, 8(3), 237–309. Newell, A. & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and the law of practice. In J. R. Anderson (Ed.), Cognitive skills and their acquisition (pp. 1–55). Hillsdale, NJ: Lawrence Erlbaum Associates. Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of Learning. NYC: Teachers College, Columbia University. Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 30 / 30

×