The indicators project - ASCILITE version

Twitter: #eair
The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors
Colin BeerDavid JonesKen Clark
http://tr.im/FvRQ

Surprising patterns in CQU LMS data
Different from established patterns
Due to limitations in current work?
Apparent benefit in LMS usage analysis that is
Cross-LMS
Cross-institutional
Longitudinal
Want to play?
http://www.flickr.com/photos/anders-vindegg/3408838186/

Overview
Context
Level of feature adoption
Link between participation & grades
Reflection and Future work
http://www.flickr.com/photos/atbaker/1577665185/

The data that we, educators gatherand utilize is all but garbage.
(Wiley, 2009)
http://www.flickr.com/photos/dnorman/251646154/

(Dawson, 2009; Dawson et al, 2009)
(Bakharia et al, 2009)
(Dawson et al, 2008)
(Black et al, 2008)
(Dawson, 2006)
(Heathcoate et al, 2005)
(Malikowski, 2008; Malikowski, et al, 2007;Malikowski et al, 2006)
(Romero et al, 2007)
(Griffiths, 2007)
(Campbell et al, 2007)
(Hornik et al, 2008)
http://www.flickr.com/photos/mikebaird/2087879492/

http://www.flickr.com/photos/mwichary/2188958154/

Limited views
LMS reporting tools
Only what is in the LMS

Limited views
Some surveys
LMS reporting tools
Human recollection

Limited views
Some surveys
LMS reporting tools
Human recollection
Error prone
Manual checks
Time-consuming

Limited views
Some surveys
LMS reporting tools
Human recollection
Error prone
Manual checks
Time-consuming
Limited longitudinal data

Limited views
Some surveys
LMS reporting tools
Human recollection
Error prone
Manual checks
Time-consuming
Limited cross-LMS comparisons

Limited views
Some surveys
LMS reporting tools
Human recollection
Error prone
Manual checks
Time-consuming
Limited cross-LMS comparisons
Limited cross-institutional comparisons

“LMS” situation
Students

Pre-2010 “LMS”
Webfuse 1997-2009
WebCT 1999-2004
Blackboard 2004-2009

Post-2010 “LMS”
Webfuse 1997-2009
Moodle….
WebCT 1999-2003/4
Blackboard 2004-2009

Students - 2009
4436
CQ on-campus

Students - 2009
8444
4436
CQ on-campus
Distance education

Students - 2009
8444
7962
4436
CQ on-campus
Distance education
AIC on-campus

The data that we, educators gatherand utilize is all but garbage.
(Wiley, 2009)
http://www.flickr.com/photos/dnorman/251646154/

Enabling comparisons of LMS usageacross institutions, platforms and time
http://indicatorsproject.wordpress.com

http://www.flickr.com/photos/david_jones/36379215/

Overview
Context

There are more similarities than differences among LMS
(Black et al, 2007)
http://www.flickr.com/photos/chokingsun/3495110670/

Level 1: Most used
Level 2: Moderately used
Level 3: Rarely used
More than 50% of courses
More than 20% of courses
(much) less than 20% of courses
CreatingClassInteractions
EvaluatingStudents
TransmittingContent
Chat
Discussion forum
Email
Quizzes
Drop box
Content/files
Announcements
Gradebook
ComputerBasedInstruction
Evaluatingcourse and instructors
Some quizzes
Adaptive release
Surveys
Move to features in unexplored categories, until each category is considered for different learning needs
(Malikowski et al, 2007)

CQU
Bb
Wf
What we did
LMS logs

LMS Independent Data
CQU
Bb
Wf
LMS logs
What we did

% of courses adopting “Malikowski features”
CQU
Bb
Wf
LMS logs
What we did

# course sites per “LMS”

Malikowskitop & bottomrange

Blackboard

Webfuse

http://www.flickr.com/photos/wadem/2730257498/

The particular trajectory of emergence is not wholly determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two
(Jones, 1999)

Why?
Future Work
How
Will Moodle be different?
What about other institutions?
Are Malikowski ranges out of date?
When is a feature adopted?
…….
http://www.flickr.com/photos/nedrichards/3234490934/

Overview
Context

The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course
Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance.
(Dawson, 2009; Morris et al, 2005)
http://www.flickr.com/photos/wolfgangstaudt/2279651479/

What about
CQU context
Different student types?
Different levels of staff interaction?
(Fresen, 2007)
Different staff academic background?
http://www.flickr.com/photos/jamuraa/813966437/

Feature usage
CQU
Bb
Wf
LMS logs
What we did

Feature usage
Student records
CQU
Bb
Wf
LMS logs
What we did

Feature usage
Participation, grades,external factors
Student records
CQU
Bb
Wf
LMS logs
What we did

Participation
Student/courses
N = 510,158
Hits
Discussion forum
Entire course site
# replies
# posts
Just course forum
http://www.flickr.com/photos/adactio/1259245482/

Mixed design ANOVA
All reported effects are significant
Significant interaction effectbetween grades and participation

Distance education students
Hits on course site
Hits on course forum

CQ on-campus students

AIC students

FLEX
CQ
AIC

Distance education students
# of replies
# of posts

CQ on-campus students

AIC students

What about
CQU context
(Fresen, 2007)

Course groups based on staff hits
http://www.flickr.com/photos/orinrobertjohn/3321067724/

High staff participation courses
Hits

High staff participation courses
Replies & posts

Super low staff participation courses
Hits

Super low staff participation courses
Replies & posts

Posts comparisons
High
Superlow

Examination of super-low
Future Work
- Course size
Forum content
Networks of interaction
(Bakharia et al, 2009; Dawson, 2009)
http://www.flickr.com/photos/nedrichards/3234490934/

What about
CQU context
(Fresen, 2007)

Course offerings with
n=167
Staff with grad certs in L&T
n=4
Significant input from instructional designer
n=196
Staff who have received teaching awards
N=362
Staff teaching education courses
n=3288
None of the above

None of the above

Grad cert in L&T

Teaching awards

Education discipline

Instructional design

Overview
Context
Refleection and Future work

Some interesting patterns
Useful idea
Needs more work
http://www.flickr.com/photos/mikebaird/2985066755/

Research
T&L Practice
Useful Data/Information
Institution A
Institution B
Institution C
LMS 1
LMS 2
LMS
LMS

Research
T&L Practice
Enabling
Institution A
Institution B
Institution C
LMS 1
LMS 2
LMS
LMS

Research
T&L Practice
What
Enabling
Institution A
Institution B
Institution C
LMS 1
LMS 2
LMS
LMS

Avg course hits
By age
DE under grad students

Why
Research
T&L Practice
What
Enabling
Institution A
Institution B
Institution C
LMS 1
LMS 2
LMS
LMS

Why
Research
T&L Practice
How
What
Enabling
Institution A
Institution B
Institution C
LMS 1
LMS 2
LMS
LMS

http://www.flickr.com/photos/dmatos/540688952/

http://www.flickr.com/photos/sylvar/1138341328/

http://www.flickr.com/photos/fncll/145149313/

http://indicatorsproject.wordpress.com
http://tr.im/FvRQ

Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.
Black, E. W., K. Dawson, et al. (2008). "Data for free: Using LMS activity logs to measure community in online courses." Internet and Higher Education11(11): 65-70.
Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends51(2): 35-39.
Campbell, J., P. DeBlois, et al. (2007). "Academic analytics: A new tool for a new era." EDCAUSE Review 42(4): 40-42.
Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.
Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.
Dawson, S., L. Macfadyen, et al. (2009). Learning or performance: Predicting drivers of student motivation. Same places, different spaces. Proceedings ascilite Auckland 2009, Auckland.
Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230
Dawson, S. (2006). "Online forum discussion interactions as an indicator of student community." Australian Journal of Educational Technology22(4): 495-510.
Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning6(3): 351-362.
Griffiths, M. E. (2007). Patterns of user acitivity in the Blackboard course management system across all courses in the 2004-2005 academic year at Brigham Young University. Department of Instructional Psychology and Technology., Brigham Young. Master of Science: 117.
Heathcoate, L. and S. Dawson (2005). "Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS." E-Learn 2005: World conference on E-Learning in corporate, government, healthcare and higher education.
Hornik, S., C. S. Saunders, et al. (2008). "The Impact of Paradigm Development and Course Level on Performance in Technology-Mediated Learning Environments." Informing Science. The international journal of an emerging transdiscipline11(11).
Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.

Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research36(2): 149-173.
Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research36(2): 149-173.
Malikowski, S., M. Thompson, et al. (2006). "External factors associated with adopting a CMS in resident college courses." Internet and Higher Education9(3): 163-174.
Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education8(3): 221-231.
Romero, C. b., S. n. Ventura, et al. (2007). "Data mining in course management systems: Moodle case study and tutorial." ScienceDirect Computers and Education (51 (2008)): 368-384.
Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology13(3): 201-218.
Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.

The indicators project - ASCILITE version

by David Jones on Dec 02, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 100

Statistics

The indicators project - ASCILITE version — Presentation Transcript