The indicators project - ASCILITE version

  • 1,373 views
Uploaded on

A reworked version of presentation for ASCILITE'09 paper (http://tr.im/FD7a).

A reworked version of presentation for ASCILITE'09 paper (http://tr.im/FD7a).

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,373
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
13
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • To the rescue come academic analytics and related work. Folk doing studies of LMS usage through a variety of means. All aiming to give us better data on which to make decisions. All finding certain patterns, correlations and/or relationships
  • But this work is not with out its limitations. There is room for more work, or at least we thought so. Explain each in turn
  • There are significant limitations on what this information can tell on it’s own. Needs to be applied intelligently in a contextual and contingent way. Needs combination with other methods.
  • Let’s talk about the CQU context
  • Quickly talk about the different types of students. Explain the reason for mentioning this is that this is different from purely on-campus. Mixed delivery mode might reveal mixed results?
  • Make the point that most of CQU’s data for decision making shares certain characeteristics with other institutions
  • We embarked on the indicators project, with a specific aim.
  • We’re still taking baby steps and looking for interesting patterns to see if our idea is worthy of moving forward.
  • So now we look at some of the interesting patterns, lets see if you agree. Level of feature adoption
  • Explain what we’re using to measure participation
  • We’re still taking baby steps and looking for interesting patterns to see if our idea is worthy of moving forward.
  • Some quick reflections
  • We’re still taking baby steps, still learning, still wobbling. Need some help.
  • It’s a big task, complex combination of factors, educational, technical, statistical, political, social….
  • There are significant limitations on what this information can tell on it’s own. Needs to be applied intelligently in a contextual and contingent way. Needs combination with other methods.
  • But there’s enough of a hint here to indicate that there are rewards.

Transcript

  • 1. Twitter: #eair
    The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors
    Colin BeerDavid JonesKen Clark
    http://tr.im/FvRQ
  • 2. Surprising patterns in CQU LMS data
    Different from established patterns
    Due to limitations in current work?
    Apparent benefit in LMS usage analysis that is
    Cross-LMS
    Cross-institutional
    Longitudinal
    Want to play?
    http://www.flickr.com/photos/anders-vindegg/3408838186/
  • 3. Overview
    Context
    Level of feature adoption
    Link between participation & grades
    Reflection and Future work
    http://www.flickr.com/photos/atbaker/1577665185/
  • 4. Overview
    Context
    Level of feature adoption
    Link between participation & grades
    Reflection and Future work
    http://www.flickr.com/photos/atbaker/1577665185/
  • 5. The data that we, educators gatherand utilize is all but garbage.
    (Wiley, 2009)
    http://www.flickr.com/photos/dnorman/251646154/
  • 6. (Dawson, 2009; Dawson et al, 2009)
    (Bakharia et al, 2009)
    (Dawson et al, 2008)
    (Black et al, 2008)
    (Dawson, 2006)
    (Heathcoate et al, 2005)
    (Malikowski, 2008; Malikowski, et al, 2007;Malikowski et al, 2006)
    (Romero et al, 2007)
    (Griffiths, 2007)
    (Campbell et al, 2007)
    (Hornik et al, 2008)
    http://www.flickr.com/photos/mikebaird/2087879492/
  • 7. http://www.flickr.com/photos/mwichary/2188958154/
  • 8. Limited views
    LMS reporting tools
    Only what is in the LMS
    http://www.flickr.com/photos/mwichary/2188958154/
  • 9. Limited views
    Some surveys
    LMS reporting tools
    Human recollection
    Only what is in the LMS
    http://www.flickr.com/photos/mwichary/2188958154/
  • 10. Limited views
    Some surveys
    LMS reporting tools
    Human recollection
    Only what is in the LMS
    Error prone
    Manual checks
    Time-consuming
    http://www.flickr.com/photos/mwichary/2188958154/
  • 11. Limited views
    Some surveys
    LMS reporting tools
    Human recollection
    Only what is in the LMS
    Error prone
    Manual checks
    Time-consuming
    Limited longitudinal data
    http://www.flickr.com/photos/mwichary/2188958154/
  • 12. Limited views
    Some surveys
    LMS reporting tools
    Human recollection
    Only what is in the LMS
    Error prone
    Manual checks
    Time-consuming
    Limited longitudinal data
    Limited cross-LMS comparisons
    http://www.flickr.com/photos/mwichary/2188958154/
  • 13. Limited views
    Some surveys
    LMS reporting tools
    Human recollection
    Only what is in the LMS
    Error prone
    Manual checks
    Time-consuming
    Limited longitudinal data
    Limited cross-LMS comparisons
    Limited cross-institutional comparisons
    http://www.flickr.com/photos/mwichary/2188958154/
  • 14. “LMS” situation
    Students
  • 15. Pre-2010 “LMS”
    Webfuse 1997-2009
    WebCT 1999-2004
    Blackboard 2004-2009
  • 16. Post-2010 “LMS”
    Webfuse 1997-2009
    Moodle….
    WebCT 1999-2003/4
    Blackboard 2004-2009
  • 17. Students - 2009
    4436
    CQ on-campus
  • 18. Students - 2009
    8444
    4436
    CQ on-campus
    Distance education
  • 19. Students - 2009
    8444
    7962
    4436
    CQ on-campus
    Distance education
    AIC on-campus
  • 20. The data that we, educators gatherand utilize is all but garbage.
    (Wiley, 2009)
    http://www.flickr.com/photos/dnorman/251646154/
  • 21. Enabling comparisons of LMS usageacross institutions, platforms and time
    http://indicatorsproject.wordpress.com
  • 22. http://www.flickr.com/photos/david_jones/36379215/
  • 23. Overview
    Context
    Level of feature adoption
    Link between participation & grades
    Reflection and Future work
    http://www.flickr.com/photos/atbaker/1577665185/
  • 24. There are more similarities than differences among LMS
    (Black et al, 2007)
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 25. Level 1: Most used
    Level 2: Moderately used
    Level 3: Rarely used
    More than 50% of courses
    More than 20% of courses
    (much) less than 20% of courses
    CreatingClassInteractions
    EvaluatingStudents
    TransmittingContent
    ComputerBasedInstruction
    Evaluatingcourse and instructors
    Move to features in unexplored categories, until each category is considered for different learning needs
    (Malikowski et al, 2007)
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 35. CQU
    Bb
    Wf
    What we did
    LMS logs
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 36. LMS Independent Data
    CQU
    Bb
    Wf
    LMS logs
    What we did
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 37. % of courses adopting “Malikowski features”
    LMS Independent Data
    (Malikowski et al, 2007)
    CQU
    Bb
    Wf
    LMS logs
    What we did
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 38. # course sites per “LMS”
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 39. Malikowskitop & bottomrange
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 40. Blackboard
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 41. Webfuse
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 42. http://www.flickr.com/photos/chokingsun/3495110670/
  • 43. http://www.flickr.com/photos/chokingsun/3495110670/
  • 44. http://www.flickr.com/photos/chokingsun/3495110670/
  • 45. http://www.flickr.com/photos/wadem/2730257498/
  • 46. The particular trajectory of emergence is not wholly determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two
    (Jones, 1999)
    http://www.flickr.com/photos/wadem/2730257498/
  • 47. Why?
    Future Work
    How
    Will Moodle be different?
    What about other institutions?
    Are Malikowski ranges out of date?
    When is a feature adopted?
    …….
    http://www.flickr.com/photos/nedrichards/3234490934/
  • 48. Overview
    Context
    Level of feature adoption
    Link between participation & grades
    Reflection and Future work
    http://www.flickr.com/photos/atbaker/1577665185/
  • 49. The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
    (Dawson et al, 2008)
    Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
    (Dawson, 2009; Morris et al, 2005)
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 50. What about
    CQU context
    Different student types?
    Different levels of staff interaction?
    (Fresen, 2007)
    Different staff academic background?
    http://www.flickr.com/photos/jamuraa/813966437/
  • 51. Feature usage
    LMS Independent Data
    (Malikowski et al, 2007)
    CQU
    Bb
    Wf
    LMS logs
    What we did
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 52. Feature usage
    LMS Independent Data
    (Malikowski et al, 2007)
    Student records
    CQU
    Bb
    Wf
    LMS logs
    What we did
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 53. Feature usage
    Participation, grades,external factors
    LMS Independent Data
    (Malikowski et al, 2007)
    Student records
    CQU
    Bb
    Wf
    LMS logs
    What we did
    http://www.flickr.com/photos/chokingsun/3495110670/
  • 54. Participation
    Student/courses
    N = 510,158
    Hits
    Discussion forum
    Entire course site
    # replies
    # posts
    Just course forum
    http://www.flickr.com/photos/adactio/1259245482/
  • 55. http://www.flickr.com/photos/david_jones/36379215/
  • 56. Mixed design ANOVA
    All reported effects are significant
    Significant interaction effectbetween grades and participation
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 57. Distance education students
    Hits on course site
    Hits on course forum
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 58. CQ on-campus students
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 59. AIC students
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 60. FLEX
    CQ
    AIC
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 61. Distance education students
    # of replies
    # of posts
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 62. CQ on-campus students
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 63. AIC students
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 64. The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
    (Dawson et al, 2008)
    Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
    (Dawson, 2009; Morris et al, 2005)
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 65. http://www.flickr.com/photos/wadem/2730257498/
  • 66. What about
    CQU context
    Different student types?
    Different levels of staff interaction?
    (Fresen, 2007)
    Different staff academic background?
    http://www.flickr.com/photos/jamuraa/813966437/
  • 67. Course groups based on staff hits
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 68. High staff participation courses
    Hits
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 69. High staff participation courses
    Replies & posts
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 70. Super low staff participation courses
    Hits
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 71. Super low staff participation courses
    Replies & posts
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 72. Posts comparisons
    High
    Superlow
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 73. The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
    (Dawson et al, 2008)
    Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
    (Dawson, 2009; Morris et al, 2005)
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 74. http://www.flickr.com/photos/wadem/2730257498/
  • 75. Examination of super-low
    Future Work
    - Course size
    Forum content
    Networks of interaction
    (Bakharia et al, 2009; Dawson, 2009)
    http://www.flickr.com/photos/nedrichards/3234490934/
  • 76.
  • 77. What about
    CQU context
    Different student types?
    Different levels of staff interaction?
    (Fresen, 2007)
    Different staff academic background?
    http://www.flickr.com/photos/jamuraa/813966437/
  • 78. Course offerings with
    n=167
    Staff with grad certs in L&T
    n=4
    Significant input from instructional designer
    n=196
    Staff who have received teaching awards
    N=362
    Staff teaching education courses
    n=3288
    None of the above
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 79. None of the above
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 80. Grad cert in L&T
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 81. Teaching awards
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 82. Education discipline
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 83. Instructional design
    http://www.flickr.com/photos/orinrobertjohn/3321067724/
  • 84. Overview
    Context
    Level of feature adoption
    Link between participation & grades
    Refleection and Future work
    http://www.flickr.com/photos/atbaker/1577665185/
  • 85. Some interesting patterns
    Useful idea
    Needs more work
    http://www.flickr.com/photos/mikebaird/2985066755/
  • 86. Research
    T&L Practice
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    LMS Independent Data
    Institution A
    Institution B
    Institution C
    LMS 1
    LMS 2
    LMS
    LMS
    http://www.flickr.com/photos/atbaker/1577665185/
  • 87. Research
    T&L Practice
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    LMS Independent Data
    Enabling
    Institution A
    Institution B
    Institution C
    LMS 1
    LMS 2
    LMS
    LMS
    http://www.flickr.com/photos/atbaker/1577665185/
  • 88. Research
    T&L Practice
    What
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    LMS Independent Data
    Enabling
    Institution A
    Institution B
    Institution C
    LMS 1
    LMS 2
    LMS
    LMS
    http://www.flickr.com/photos/atbaker/1577665185/
  • 89. Avg course hits
    By age
    DE under grad students
    http://www.flickr.com/photos/wolfgangstaudt/2279651479/
  • 90. Why
    Research
    T&L Practice
    What
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    LMS Independent Data
    Enabling
    Institution A
    Institution B
    Institution C
    LMS 1
    LMS 2
    LMS
    LMS
    http://www.flickr.com/photos/atbaker/1577665185/
  • 91. Why
    Research
    T&L Practice
    How
    What
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    Useful Data/Information
    LMS Independent Data
    Enabling
    Institution A
    Institution B
    Institution C
    LMS 1
    LMS 2
    LMS
    LMS
    http://www.flickr.com/photos/atbaker/1577665185/
  • 92. http://www.flickr.com/photos/david_jones/36379215/
  • 93. http://www.flickr.com/photos/dmatos/540688952/
  • 94. http://www.flickr.com/photos/mwichary/2188958154/
  • 95. http://www.flickr.com/photos/sylvar/1138341328/
  • 96. http://www.flickr.com/photos/fncll/145149313/
  • 97. http://indicatorsproject.wordpress.com
    http://tr.im/FvRQ
    http://www.flickr.com/photos/jamuraa/813966437/
  • 98. Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.
    Black, E. W., K. Dawson, et al. (2008). "Data for free: Using LMS activity logs to measure community in online courses." Internet and Higher Education11(11): 65-70.
    Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends51(2): 35-39.
    Campbell, J., P. DeBlois, et al. (2007). "Academic analytics: A new tool for a new era." EDCAUSE Review 42(4): 40-42.
    Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.
    Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.
    Dawson, S., L. Macfadyen, et al. (2009). Learning or performance: Predicting drivers of student motivation. Same places, different spaces. Proceedings ascilite Auckland 2009, Auckland.
    Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230
    Dawson, S. (2006). "Online forum discussion interactions as an indicator of student community." Australian Journal of Educational Technology22(4): 495-510.
    Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning6(3): 351-362.
    Griffiths, M. E. (2007). Patterns of user acitivity in the Blackboard course management system across all courses in the 2004-2005 academic year at Brigham Young University. Department of Instructional Psychology and Technology., Brigham Young. Master of Science: 117.
    Heathcoate, L. and S. Dawson (2005). "Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS." E-Learn 2005: World conference on E-Learning in corporate, government, healthcare and higher education.
    Hornik, S., C. S. Saunders, et al. (2008). "The Impact of Paradigm Development and Course Level on Performance in Technology-Mediated Learning Environments." Informing Science. The international journal of an emerging transdiscipline11(11).
    Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.
  • 99. Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research36(2): 149-173.
    Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research36(2): 149-173.
    Malikowski, S., M. Thompson, et al. (2006). "External factors associated with adopting a CMS in resident college courses." Internet and Higher Education9(3): 163-174.
    Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education8(3): 221-231.
    Romero, C. b., S. n. Ventura, et al. (2007). "Data mining in course management systems: Moodle case study and tutorial." ScienceDirect Computers and Education (51 (2008)): 368-384.
    Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology13(3): 201-218.
    Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.