This document describes an approach to evaluating discussion board use in Blackboard at a UK university with over 10,000 students. Both quantitative and qualitative methods are used. Quantitatively, data on user interactions such as messages and participants are analyzed. Qualitatively, the Community of Inquiry model is applied through content analysis to assess cognitive, social and teaching presence. Results show wide variation in discussion board usage across schools. An engagement index is also calculated to compare student and tutor participation levels. Overall, the evaluation aims to understand how discussion boards are operationalized and to inform future technology implementations.

1. An Approach to Evaluating
Discussion Board Use in Blackboard
Neil McKeown

2. Outline
• Context
• Quantitative
– User interactions
• Qualitative
– Community of Inquiry model + content analysis
(Garrison, Anderson and Archer. 2000)
– Application of the model
• Results
– Operational uses
– Tutor/Student engagement index
• Context of evaluation within UK HEIs

3. ―Perhaps the overriding change that this entails
is shifting the field away from asking ‗state-of-
the-art‘ questions about technology and towards
asking questions that can be described as being
concerned with the ‗state-of-the-actual‘.‖
(Selwyn N. (2011) ―In praise of pessimism—the need for negativity in
educational technology‖, British Journal of Educational Technology, Vol. 42, Iss.
5. p.715.)

4. Context: The Institution
• Centre for Educational Development SENIOR
MANAGEMENT
• 10,000FTE
• 7 schools (faculty)
– Social & International Studies
– Life Sciences
– Computing, Informatics and Media
– Engineering, Design & Technology
– Health Studies
– Lifelong Education & Development
– Social & International Studies
– Management
• Implementation of technologies done in an ad hoc
fashion

5. Context: the approach
• To develop an approach to evaluation ex post
facto
– Evaluation of a technology already embedded
– Evaluation of the previous year‘s DB use
• Holistic (institutional)
• No surveying
• Quantitative and Qualitative elements
– Scale, manner, quality
– Links between the two?

6. Quantitative Data: Course DBs
ID STA STU TOC TotFora withDESC TotMess ofWhchSTU ofWhchSTA STUPrtipant's STAPrtipants AvgMslgth frSTU fr STA TotReds DistRders Grps
1170 8 35 YES 17 17 156 78 78 21 1 1580 1514 1647 3131 34 0
1173 7 31 YES 2 2 2 2 0 1 0 873 873 0 31 23 0
1179 23 159 NO 1 1 3 3 0 3 0 230 230 0 50 20 2041
1201 15 60 YES 1 0 6 2 4 2 1 311 128 402 115 22 0
1218 9 81 YES 3 3 9 3 6 3 1 792 193 1092 399 45 0
1277 2 13 YES 5 5 5 0 5 0 1 377 0 377 2 4 0
1285 4 34 YES 16 16 22 16 6 10 1 3635 4632 448 591 35 0
1290 5 79 YES 2 2 1 1 0 1 0 591 591 0 16 15 0
1295 9 32 YES 1 1 35 35 0 29 0 24 24 0 880 32 0
1305 19 183 YES 4 4 2 2 0 2 0 18 18 0 53 31 0
1314 15 206 YES 2 2 9 3 6 3 1 1592 282 2247 856 116 0
1319 9 36 YES 6 0 36 17 19 9 1 237 286 192 1366 34 0
1332 3 17 YES 1 0 1 1 0 1 0 572 572 0 11 8 0
1334 4 17 YES 2 1 5 1 4 1 1 1441 5649 389 59 10 0
1338 20 61 YES 8 1 89 37 52 6 1 1483 2178 789 1508 32 0
1347 16 219 YES 1 0 16 10 6 9 1 266 222 339 765 63 0
1348 19 196 YES 1 1 9 9 0 7 0 161 161 0 163 35 0
1359 19 72 YES 2 2 1 1 0 1 0 419 419 0 39 37 0
1386 7 66 YES 1 1 5 5 0 4 0 343 343 3 2 0
1434 18 61 YES 1 1 4 0 4 0 1 42 0 42 87 21 0
1439 6 64 YES 3 0 3 2 1 2 1 410 396 440 92 21 0

7. Qualitative approach
Social Cognitive
Presence Presence
EDUCATIONAL
EXPERIENCE
Search string Titles in Google Scholar
Teaching “Social Presence” 22,200
“Cognitive Presence” 2,760
Presence “Teaching Presence” 3,080
Community of Inquiry model: Elements of an Educational Experience
(Garrison, Anderson and Archer, 2000. p.88)

8. Identification of ‘Community of Inquiry’
using content analysis
• Cognitive Presence
Phase In brief
IV. Metacognition New knowledge is applied/reflected upon.
III. Collaboration New knowledge is constructed.
II. Conversation Two or more are communicating i.e. The content of posts is
being read and responded to.
I. Information Someone has posted something!
Exchange
(Gunawardena, C.N., Lowe, C.A. and Anderson, T., 1997, “Analysis of global online debate and the development of
an interaction analysis model for examining social construction of knowledge in computer conferencing”, Journal
of Educational Computing Research, Vol. 17, No. 4, Pp. 397-431.)

9. Identification of ‘Community of Inquiry’
using content analysis
• Social Presence:
Autobiographical details
Acknowledging other contributions
Offering help and support
...plus more...
• Teaching Presence:
Diagnosing misconceptions
Summarising learning outcomes
Structuring DB
...plus more...

10. Qualitative data: content analysis
ID Purpose Presence
Cognitive Social Teaching
Instructional Management
Open Communication
Emotional Expression
Direct Instruction
Group Cohesion
Facilitation
Phase
106 Other I - information Xch 1 2 1 1 3 1
110 Task (non DB) I - information Xch 1 1 0 1 0 3
131 Administrative I - information Xch 1 2 1 1 1 1
317 Task (non DB) I - information Xch 1 1 1 1 0 2
380 Task (non DB) I - information Xch 0 1 0 1 0 0
385 Task (non DB) I - information Xch 0 0 0 1 0 0
396 Social I - information Xch 0 0 0 1 0 0
398 Other II - conversation 3 3 3 1 1 0
399 Subject (informal) III - collaboration 3 3 3 2 3 1
400 Other II - conversation 3 3 2 1 0 0

11. Inter-rater reliability
Social Presence Teaching Presence
Direct Instruction
Group Cohesion
Communication
Management
Instructional
Facilitation
Expression
Emotional
Course ID
Cognitive
Purpose
Open
Agree
Disagree
Disagree+ 110
317
399
703
720
989
1218
1285
1295
1677
1757
1809
5 6 8 6 6 6 1 4
41.7% 50% 66.7% 50% 50% 50% 8.3% 33.3% 43.75%

12. Why was the content analysis
inconsistent?
• Too large
– A numerical approach to qualitative data?
– Content analysis another quantitative approach?
• No negotiation between raters

13. Discussion
• 50%+ Available Course DBs with no fora:
ID STA STU TOC TotFora withDESC TotMess ofWhchSTU ofWhchSTA STUPrtipant's STAPrtipants AvgMslgth frSTU fr STA TotReds DistRders Grps
1170 8 35 YES 17 17 156 78 78 21 1 1580 1514 1647 3131 34 0
1173 7 31 YES 2 2 2 2 0 1 0 873 873 0 31 23 0
1179 23 159 NO 1 1 3 3 0 3 0 230 230 0 50 20 2041
• DB descriptions?
• Lurking

14. Discussion
Overall Course DB activity 2010-1:
All available with forum with extant messages
courses Numbers % Numbers %
School A 183 12 6.6 6 3.3
School B 178 2 1.1 2 1.1
School C 316 51 16.1 42 13.3
School D 187 9 4.8 6 3.2
School E 110 2 1.8 2 1.8
School F 439 53 12.1 37 8.4
School G 260 22 8.5 14 5.4
Activity in those DBs:
School All Course Total messages of which created by student of which created by tutor
DBs with
fora Mean (SD) Median Mean (SD) Median Mean (SD) Median
School A 6 38.3 (54.1) 15.5 25.2 (38.3) 10.5 13.2 (16.4) 5.5
School B 2 13.5 (10.5) n/a 10 (9) n/a 3.5 (1.5) n/a
School C 42 257.6 (542.4) 30 243 (520.7) 24.5 14.6 (28.8) 2
School D 6 105 (221) 4 79.5 (168.9) 2 25.5 (52.1) 2.5
School E 2 1.5 (0.5) n/a 0.5 (0.5) n/a 1 (0) n/a
School F 37 17.7 (29.9) 5 9.8 (15.8) 2 7.9 (15.4) 3
School G 14 70.3 (68.2) 48 66.1 (64.7) 40.5 4.1 (5.8) 1

15. Discussion
• School C had 6 DBs with over 900 messages
• One DB had a total of 2442 messages
• Goldilocks approach to the qualitative
investigation? Group and investigate according
to level of use:
• 0-3 messages
– Not used
• Many hundreds of messages
– May well be unmanageable for some users
• ‗Just right‘
– A ‗successful‘ level of engagement

16. Student/Tutor ‘engagement index’
• Per school....
Student Tutor Student to tutor ratio
School A 2.9 13.2 4.6
School B 2.9 3.5 1.2
School C 12.1 15.3 1.3
School D 5.7 10.9 1.9
School E 1.0 2.0 2
School F 1.7 10.6 6.2
School G 3.3 5.7 1.7

17. Community of Inquiry
Social Cognitive
Presence Presence
Teaching
Presence
―The main finding over the last decade with regard
to teaching presence is the growing evidence as to
the importance of this element.‖
Garrison, D.R., Anderson, T. and Archer, W. , 2010 ―The first decade of the community of
inquiry framework: A retrospective‖ Internet and Higher Education, Vol. 13. Pp. 7

18. Answers?
Email: n.m.mckeown@bradford.ac.uk
Twitter: neilmack55