Citation Information: Anglim, J., & Wynton, S. K. (2015). Hierarchical Bayesian Models of Subtask Learning. Journal of Experimental Psychology. Learning, Memory, and Cognition. Online First. http://dx.doi.org/10.1037/xlm0000103 Abstract: In this talk I present some recent work looking at the question of how to understanding learning complex computer-based tasks in terms of component learning processes. The research tests and examines what Lee and Anderson (2001) labelled the "decomposition hypothesis" : i.e., that learning complex tasks can be understand as the result of learning many simpler subtasks. To test these ideas, we get participants to practice computer-based tasks where all mouse clicks and key presses are logged. We then extract a range of measures of strategy use, subtask performance, and overall task performance. We then use Bayesian hierarchical methods to test models of how strategy use and performance changes with practice at the individual-level. Overall, these model provide a more nuanced representation of how complex tasks can be decomposed in terms of simpler learning mechanisms. The research also presents a case study of how Bayesian methods can be used to yield novel insights to well-established psychological questions. Bio: Dr Jeromy Anglim is a lecturer at Deakin University in Melbourne. He completed his PhD at University of Melbourne on mathematical models of learning, and his Post Doc in the Melbourne Business School on applications of Bayesian hierarchical models to psychology. His research interests are at the interface of statistics and industrial / organisational psychology with particular interest in skill acquisition, performance, individual differences, Bayesian data analysis, psychometrics, and selection and recruitment. He has a particular interest in refining and promoting methods for open and reproducible research in psychology. For further information go to http://jeromyanglim.blogspot.com

1. Bayesian Hierarchical Models of Individual
Differences in Skill Acquisition
Dr Jeromy Anglim
Deakin University
22nd May 2015

2. Functional form of the learning curve
• Researchers have long been interested in functional
form of the learning curve
– Power law of practice (Newell and Rosenbloom, 1981;
Snoddy 1926)
– Evidence for exponential function at individual level
(Heathcote, Brown, & Mewhort, 2001)
Early example: 1024
choice-reaction time
task
Data from Seibel
1963; shown in
Delaney et al 1998
Task Results

3. Relating subtask to overall task learning
• Issue of how to integrate basic findings from
cognitive psychology with learning on more
complex tasks
• Lee and Anderson (2001) proposed reducibility
hypothesis suggesting that learning a complex
task could be understood as the culmination of
learning many component subtasks
• They also proposed that subtask learning will be
consistent across subtasks and follow the power
law of practice

4. Lee & Anderson (2001)
Overall Task Performance
KA Air-Traffic Controller Task
Task Analysis
Subtask Performance
Source: Lee, F. J., & Anderson, J. R. (2001). Does learning a complex
task have to be complex?: A study in learning decomposition.
Cognitive Psychology, 42(3), 267-316.

5. Gaps / Issues
Gaps
• Reliance on group-level analysis
• Need to refine definitions and tests of subtask
learning consistency
• Lack of incorporation of trial level strategy use
data
Approach
• Need for task that facilitates measurement of
strategy use and subtask performance
• A Bayesian hierarchical approach offers benefits
over piece-wise individual-level analysis.

6. Wynton-Anglim Booking (WAB) Task
1. Information
Gathering (I)
2. Filtering (F)
3. Timetabling (T)

7. Bayesian Hierarchical Models
• Increased interest in application of Bayesian
Methods in psychology
• Benefits of Bayesian Approach
– Clear and direct inference
– Flexible model specification
– Range of sophisticated model comparison tools
(e.g., DIC, Posterior predictive checks)
– Well-suited to modelling repeated measures
psychological data (i.e., observations nested
within people)

8. Models of Overall Performance

9. Models of Subtask Performance

10. Aims
1. Assess support for power and exponential
functions on overall and subtask
performance
2. Assess degree of consistency in subtask
learning
3. Estimate effect of strategy use on subtask
performance
4. Assess degree to which strategy use could
explain inconsistency

11. Method
• Participants
– 25 adults (68% female)
• Procedure
– Read WAB Task instructions
– Complete as many trials as possible in 50 minutes
• Processing
– Extract strategy use, subtask performance and overall
task performance
– Trial performance was aggregated into average block
performance (15 blocks with approximately equal
numbers of trials)

12. Data analytic approach
• Bayesian hierarchical models were estimated
using MCMC methods using JAGS with
supporting analyses performed in R
• Model comparison
– Graphs overlaying model fits and data
– Deviance Information Criterion (DIC)
– Posterior predictive checks

13. 1. Overall performance
Does a power or exponential model
provide a better model of the effect of
practice on overall task performance?

14. Overall performance (group-level)

15. Overall task completion time by block
(individual-level)

16. Overall performance: Parameter estimates and
model comparison (DIC)
Interpretation
• Power has larger deviance but
smaller penalty and smaller DIC
• Differences are small
DIC = Mean Deviance + Penalty
Rules of thumb for DIC difference:
10+: rule out model with larger DIC
5-10: model with smaller DIC is better

17. 2. Subtask performance
Does a power or exponential model provide
a better model of the effect of practice on
subtask performance and what is the effect
of constraining subtask learning curve
parameters?

18. Subtask performance (group-level)

19. Subtask performance (individual-level)

20. Subtask performance: Parameter estimates
Subtask Abbreviations:
I = Information Gathering
F = Filtering
T = Timetabling
Parameters
1: Amount of learning
2: Rate of learning
3: Asymptotic performance

21. Subtask performance: Model comparison (DIC)
• Power has lower DIC (3862 vs 3885); but larger mean deviance
• Constraints substantially damage fit

22. Subtask performance: Model comparison
(posterior predictive checks)
Interpretation:
• When data is
simulated from a
model and statistics
are calculated on
simulated data, good
models generate
statistics similar to
actual data
• Bolding reflects
discrepancies

23. 3. Strategy Use on Subtask Performance
What is the effect of strategy use on
subtask performance?

24. Strategy use (group-level)

25. Strategy use on performance: Parameter
estimates
Note:
• Parameter estimates (i.e., exp (lambda)) for
strategy covariates on subtask performance
• exp(lambda): expected multiple to task
completion time resulting from strategy use
• exp(lambda) greater than 1: strategy use
increases task completion time
• exp(lambda) less than 1: strategy use
decreases task completion time

26. 4. Strategy Use and Subtask Learning
Consistency
To what extent does strategy use
explain subtask learning
inconsistency?

27. Strategy use explaining subtask inconsistency
(group-level)

28. Strategy use explaining subtask inconsistency
(individual-level)

29. Subtask performance with strategies: Model
comparison (DIC)
• Strategies improve fit (e.g., 3885 – 3506 =
379)
• Damage to DIC fit of constraints is less with
strategies (e.g., 3794 – 3506 = 288) than
without strategies (e.g., 4497 – 3885 = 612)

30. Subtask performance: Model Comparison
(Posterior predictive checks)

31. Concluding Thoughts

32. Concluding thoughts
• Differences between power and exponential are
fairly subtle
• Task learning may be decomposed into subtask
learning but functional form of subtask learning can
vary
• Strategy use both expresses learning and learning to
trade-off time on subtasks is a strategy itself
• More generally, the study provides a case study of
Bayesian hierarchical methods

33. Future Work
• Further Bayesian skill acquisition research
– Formal models of strategy acquisition
– Models of discontinuities in the learning curve
– Integrating traits (ability and personality) into
dynamic models of performance
• Extending Bayesian Hierarchical methods to a
range of domains
– personality faking, longitudinal life satisfaction
data, diary employee well-being data

34. Notes
• Code and data
– https://github.com/jeromyanglim/anglim-wynton-2014-subtasks
• Publication
– Based on work with Sarah Wynton
– Anglim, J., & Wynton, S. K. (2015). Hierarchical Bayesian
Models of Subtask Learning. Journal of Experimental
Psychology. Learning, Memory, and Cognition. Online First.
http://dx.doi.org/10.1037/xlm0000103
• My Contact details
– jeromy.anglim@deakin.edu.au
– http://jeromyanglim.blogspot.com

35. Thank you
Questions?