Your SlideShare is downloading. ×
0
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
The indicators project - ASCILITE version
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The indicators project - ASCILITE version

1,477

Published on

A reworked version of presentation for ASCILITE'09 paper (http://tr.im/FD7a).

A reworked version of presentation for ASCILITE'09 paper (http://tr.im/FD7a).

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,477
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • To the rescue come academic analytics and related work. Folk doing studies of LMS usage through a variety of means. All aiming to give us better data on which to make decisions. All finding certain patterns, correlations and/or relationships
  • But this work is not with out its limitations. There is room for more work, or at least we thought so. Explain each in turn
  • There are significant limitations on what this information can tell on it’s own. Needs to be applied intelligently in a contextual and contingent way. Needs combination with other methods.
  • Let’s talk about the CQU context
  • Quickly talk about the different types of students. Explain the reason for mentioning this is that this is different from purely on-campus. Mixed delivery mode might reveal mixed results?
  • Make the point that most of CQU’s data for decision making shares certain characeteristics with other institutions
  • We embarked on the indicators project, with a specific aim.
  • We’re still taking baby steps and looking for interesting patterns to see if our idea is worthy of moving forward.
  • So now we look at some of the interesting patterns, lets see if you agree. Level of feature adoption
  • Explain what we’re using to measure participation
  • We’re still taking baby steps and looking for interesting patterns to see if our idea is worthy of moving forward.
  • Some quick reflections
  • We’re still taking baby steps, still learning, still wobbling. Need some help.
  • It’s a big task, complex combination of factors, educational, technical, statistical, political, social….
  • There are significant limitations on what this information can tell on it’s own. Needs to be applied intelligently in a contextual and contingent way. Needs combination with other methods.
  • But there’s enough of a hint here to indicate that there are rewards.
  • Transcript

    • 1. Twitter: #eair
      The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors
      Colin BeerDavid JonesKen Clark
      http://tr.im/FvRQ
    • 2. Surprising patterns in CQU LMS data
      Different from established patterns
      Due to limitations in current work?
      Apparent benefit in LMS usage analysis that is
      Cross-LMS
      Cross-institutional
      Longitudinal
      Want to play?
      http://www.flickr.com/photos/anders-vindegg/3408838186/
    • 3. Overview
      Context
      Level of feature adoption
      Link between participation & grades
      Reflection and Future work
      http://www.flickr.com/photos/atbaker/1577665185/
    • 4. Overview
      Context
      Level of feature adoption
      Link between participation & grades
      Reflection and Future work
      http://www.flickr.com/photos/atbaker/1577665185/
    • 5. The data that we, educators gatherand utilize is all but garbage.
      (Wiley, 2009)
      http://www.flickr.com/photos/dnorman/251646154/
    • 6. (Dawson, 2009; Dawson et al, 2009)
      (Bakharia et al, 2009)
      (Dawson et al, 2008)
      (Black et al, 2008)
      (Dawson, 2006)
      (Heathcoate et al, 2005)
      (Malikowski, 2008; Malikowski, et al, 2007;Malikowski et al, 2006)
      (Romero et al, 2007)
      (Griffiths, 2007)
      (Campbell et al, 2007)
      (Hornik et al, 2008)
      http://www.flickr.com/photos/mikebaird/2087879492/
    • 7. http://www.flickr.com/photos/mwichary/2188958154/
    • 8. Limited views
      LMS reporting tools
      Only what is in the LMS
      http://www.flickr.com/photos/mwichary/2188958154/
    • 9. Limited views
      Some surveys
      LMS reporting tools
      Human recollection
      Only what is in the LMS
      http://www.flickr.com/photos/mwichary/2188958154/
    • 10. Limited views
      Some surveys
      LMS reporting tools
      Human recollection
      Only what is in the LMS
      Error prone
      Manual checks
      Time-consuming
      http://www.flickr.com/photos/mwichary/2188958154/
    • 11. Limited views
      Some surveys
      LMS reporting tools
      Human recollection
      Only what is in the LMS
      Error prone
      Manual checks
      Time-consuming
      Limited longitudinal data
      http://www.flickr.com/photos/mwichary/2188958154/
    • 12. Limited views
      Some surveys
      LMS reporting tools
      Human recollection
      Only what is in the LMS
      Error prone
      Manual checks
      Time-consuming
      Limited longitudinal data
      Limited cross-LMS comparisons
      http://www.flickr.com/photos/mwichary/2188958154/
    • 13. Limited views
      Some surveys
      LMS reporting tools
      Human recollection
      Only what is in the LMS
      Error prone
      Manual checks
      Time-consuming
      Limited longitudinal data
      Limited cross-LMS comparisons
      Limited cross-institutional comparisons
      http://www.flickr.com/photos/mwichary/2188958154/
    • 14. “LMS” situation
      Students
    • 15. Pre-2010 “LMS”
      Webfuse 1997-2009
      WebCT 1999-2004
      Blackboard 2004-2009
    • 16. Post-2010 “LMS”
      Webfuse 1997-2009
      Moodle….
      WebCT 1999-2003/4
      Blackboard 2004-2009
    • 17. Students - 2009
      4436
      CQ on-campus
    • 18. Students - 2009
      8444
      4436
      CQ on-campus
      Distance education
    • 19. Students - 2009
      8444
      7962
      4436
      CQ on-campus
      Distance education
      AIC on-campus
    • 20. The data that we, educators gatherand utilize is all but garbage.
      (Wiley, 2009)
      http://www.flickr.com/photos/dnorman/251646154/
    • 21. Enabling comparisons of LMS usageacross institutions, platforms and time
      http://indicatorsproject.wordpress.com
    • 22. http://www.flickr.com/photos/david_jones/36379215/
    • 23. Overview
      Context
      Level of feature adoption
      Link between participation & grades
      Reflection and Future work
      http://www.flickr.com/photos/atbaker/1577665185/
    • 24. There are more similarities than differences among LMS
      (Black et al, 2007)
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 25. Level 1: Most used
      Level 2: Moderately used
      Level 3: Rarely used
      More than 50% of courses
      More than 20% of courses
      (much) less than 20% of courses
      CreatingClassInteractions
      EvaluatingStudents
      TransmittingContent
      ComputerBasedInstruction
      Evaluatingcourse and instructors
      Move to features in unexplored categories, until each category is considered for different learning needs
      (Malikowski et al, 2007)
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 35. CQU
      Bb
      Wf
      What we did
      LMS logs
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 36. LMS Independent Data
      CQU
      Bb
      Wf
      LMS logs
      What we did
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 37. % of courses adopting “Malikowski features”
      LMS Independent Data
      (Malikowski et al, 2007)
      CQU
      Bb
      Wf
      LMS logs
      What we did
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 38. # course sites per “LMS”
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 39. Malikowskitop & bottomrange
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 40. Blackboard
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 41. Webfuse
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 42. http://www.flickr.com/photos/chokingsun/3495110670/
    • 43. http://www.flickr.com/photos/chokingsun/3495110670/
    • 44. http://www.flickr.com/photos/chokingsun/3495110670/
    • 45. http://www.flickr.com/photos/wadem/2730257498/
    • 46. The particular trajectory of emergence is not wholly determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two
      (Jones, 1999)
      http://www.flickr.com/photos/wadem/2730257498/
    • 47. Why?
      Future Work
      How
      Will Moodle be different?
      What about other institutions?
      Are Malikowski ranges out of date?
      When is a feature adopted?
      …….
      http://www.flickr.com/photos/nedrichards/3234490934/
    • 48. Overview
      Context
      Level of feature adoption
      Link between participation & grades
      Reflection and Future work
      http://www.flickr.com/photos/atbaker/1577665185/
    • 49. The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
      (Dawson et al, 2008)
      Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
      (Dawson, 2009; Morris et al, 2005)
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 50. What about
      CQU context
      Different student types?
      Different levels of staff interaction?
      (Fresen, 2007)
      Different staff academic background?
      http://www.flickr.com/photos/jamuraa/813966437/
    • 51. Feature usage
      LMS Independent Data
      (Malikowski et al, 2007)
      CQU
      Bb
      Wf
      LMS logs
      What we did
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 52. Feature usage
      LMS Independent Data
      (Malikowski et al, 2007)
      Student records
      CQU
      Bb
      Wf
      LMS logs
      What we did
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 53. Feature usage
      Participation, grades,external factors
      LMS Independent Data
      (Malikowski et al, 2007)
      Student records
      CQU
      Bb
      Wf
      LMS logs
      What we did
      http://www.flickr.com/photos/chokingsun/3495110670/
    • 54. Participation
      Student/courses
      N = 510,158
      Hits
      Discussion forum
      Entire course site
      # replies
      # posts
      Just course forum
      http://www.flickr.com/photos/adactio/1259245482/
    • 55. http://www.flickr.com/photos/david_jones/36379215/
    • 56. Mixed design ANOVA
      All reported effects are significant
      Significant interaction effectbetween grades and participation
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 57. Distance education students
      Hits on course site
      Hits on course forum
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 58. CQ on-campus students
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 59. AIC students
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 60. FLEX
      CQ
      AIC
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 61. Distance education students
      # of replies
      # of posts
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 62. CQ on-campus students
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 63. AIC students
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 64. The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
      (Dawson et al, 2008)
      Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
      (Dawson, 2009; Morris et al, 2005)
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 65. http://www.flickr.com/photos/wadem/2730257498/
    • 66. What about
      CQU context
      Different student types?
      Different levels of staff interaction?
      (Fresen, 2007)
      Different staff academic background?
      http://www.flickr.com/photos/jamuraa/813966437/
    • 67. Course groups based on staff hits
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 68. High staff participation courses
      Hits
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 69. High staff participation courses
      Replies & posts
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 70. Super low staff participation courses
      Hits
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 71. Super low staff participation courses
      Replies & posts
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 72. Posts comparisons
      High
      Superlow
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 73. The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
      (Dawson et al, 2008)
      Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
      (Dawson, 2009; Morris et al, 2005)
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 74. http://www.flickr.com/photos/wadem/2730257498/
    • 75. Examination of super-low
      Future Work
      - Course size
      Forum content
      Networks of interaction
      (Bakharia et al, 2009; Dawson, 2009)
      http://www.flickr.com/photos/nedrichards/3234490934/
    • 76.
    • 77. What about
      CQU context
      Different student types?
      Different levels of staff interaction?
      (Fresen, 2007)
      Different staff academic background?
      http://www.flickr.com/photos/jamuraa/813966437/
    • 78. Course offerings with
      n=167
      Staff with grad certs in L&T
      n=4
      Significant input from instructional designer
      n=196
      Staff who have received teaching awards
      N=362
      Staff teaching education courses
      n=3288
      None of the above
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 79. None of the above
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 80. Grad cert in L&T
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 81. Teaching awards
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 82. Education discipline
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 83. Instructional design
      http://www.flickr.com/photos/orinrobertjohn/3321067724/
    • 84. Overview
      Context
      Level of feature adoption
      Link between participation & grades
      Refleection and Future work
      http://www.flickr.com/photos/atbaker/1577665185/
    • 85. Some interesting patterns
      Useful idea
      Needs more work
      http://www.flickr.com/photos/mikebaird/2985066755/
    • 86. Research
      T&L Practice
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      LMS Independent Data
      Institution A
      Institution B
      Institution C
      LMS 1
      LMS 2
      LMS
      LMS
      http://www.flickr.com/photos/atbaker/1577665185/
    • 87. Research
      T&L Practice
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      LMS Independent Data
      Enabling
      Institution A
      Institution B
      Institution C
      LMS 1
      LMS 2
      LMS
      LMS
      http://www.flickr.com/photos/atbaker/1577665185/
    • 88. Research
      T&L Practice
      What
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      LMS Independent Data
      Enabling
      Institution A
      Institution B
      Institution C
      LMS 1
      LMS 2
      LMS
      LMS
      http://www.flickr.com/photos/atbaker/1577665185/
    • 89. Avg course hits
      By age
      DE under grad students
      http://www.flickr.com/photos/wolfgangstaudt/2279651479/
    • 90. Why
      Research
      T&L Practice
      What
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      LMS Independent Data
      Enabling
      Institution A
      Institution B
      Institution C
      LMS 1
      LMS 2
      LMS
      LMS
      http://www.flickr.com/photos/atbaker/1577665185/
    • 91. Why
      Research
      T&L Practice
      How
      What
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      Useful Data/Information
      LMS Independent Data
      Enabling
      Institution A
      Institution B
      Institution C
      LMS 1
      LMS 2
      LMS
      LMS
      http://www.flickr.com/photos/atbaker/1577665185/
    • 92. http://www.flickr.com/photos/david_jones/36379215/
    • 93. http://www.flickr.com/photos/dmatos/540688952/
    • 94. http://www.flickr.com/photos/mwichary/2188958154/
    • 95. http://www.flickr.com/photos/sylvar/1138341328/
    • 96. http://www.flickr.com/photos/fncll/145149313/
    • 97. http://indicatorsproject.wordpress.com
      http://tr.im/FvRQ
      http://www.flickr.com/photos/jamuraa/813966437/
    • 98. Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.
      Black, E. W., K. Dawson, et al. (2008). "Data for free: Using LMS activity logs to measure community in online courses." Internet and Higher Education11(11): 65-70.
      Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends51(2): 35-39.
      Campbell, J., P. DeBlois, et al. (2007). "Academic analytics: A new tool for a new era." EDCAUSE Review 42(4): 40-42.
      Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.
      Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.
      Dawson, S., L. Macfadyen, et al. (2009). Learning or performance: Predicting drivers of student motivation. Same places, different spaces. Proceedings ascilite Auckland 2009, Auckland.
      Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230
      Dawson, S. (2006). "Online forum discussion interactions as an indicator of student community." Australian Journal of Educational Technology22(4): 495-510.
      Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning6(3): 351-362.
      Griffiths, M. E. (2007). Patterns of user acitivity in the Blackboard course management system across all courses in the 2004-2005 academic year at Brigham Young University. Department of Instructional Psychology and Technology., Brigham Young. Master of Science: 117.
      Heathcoate, L. and S. Dawson (2005). "Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS." E-Learn 2005: World conference on E-Learning in corporate, government, healthcare and higher education.
      Hornik, S., C. S. Saunders, et al. (2008). "The Impact of Paradigm Development and Course Level on Performance in Technology-Mediated Learning Environments." Informing Science. The international journal of an emerging transdiscipline11(11).
      Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.
    • 99. Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research36(2): 149-173.
      Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research36(2): 149-173.
      Malikowski, S., M. Thompson, et al. (2006). "External factors associated with adopting a CMS in resident college courses." Internet and Higher Education9(3): 163-174.
      Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education8(3): 221-231.
      Romero, C. b., S. n. Ventura, et al. (2007). "Data mining in course management systems: Moodle case study and tutorial." ScienceDirect Computers and Education (51 (2008)): 368-384.
      Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology13(3): 201-218.
      Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.

    ×