This document discusses classroom research with technology. It provides an overview of topics related to conducting educational research using technology, including literature reviews, theoretical frameworks, research design, instructional design, data collection, and analysis. Examples are provided of specific classroom research studies investigating the use of wikis, peer tutoring, and analyzing online discussion forum and learning management system activity log data. The document is intended as a guide for educators conducting research on integrating technology into classroom instruction.

1. CLASSROOM RESEARCH
WITH TECHNOLOGY
Vasa Buraphadeja, PhD
Educational Technology, University of Florida
Support comes from
MSM&E Business and Economics Research Center

2. http://www.northern.dpsnc.net/faculty/northern-academic-coach/resources-for-classroom-management/Unruly%20Classroom.jpg
2

3. Classroom Research is about...
Trust,
practice,
and mutual response
3

4. Classroom Research is about…
Cooking from your garden
4

5. Problem
Literature review
Theory & Framework
Research Design
Instructional Design
Data Collection
Analysis
Write-up
5

6. Media and Learning
Learning with media
(Kozma)
Media never influence learning
(Richard E. Clark)
Restructuring the Debate
(David Jonassen)
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
6

7. Mode of Learning
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
Face to Face
Blended
Learning
Online | e-
learning
7

8. Should we compare f-2-f with
online learning?
www.NoSignificantDifference.org
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
8

9. How should we ask (measure)
our students?
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
Google insights on “survey fatigue”
9

10. Want Data? Ask Students. Again and
Again (Lipka, 2011)
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
Challenges
Accountability movement
Straight-line respondents
Will answers make a difference?
How to entice students?
Solution
Share results & actions
Gift card, iPad?
Get them to involve
Not more than one out at once
10

11. < Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
Community of Inquiry
(Anderson, Rourke, Garrison, & Archer, 2001)
Teaching
Presence
Educational
Experience
Setting
Climate
Selecting
Content
Supporting
Discourse
Social
Presence
Cognitive
Presence
Open
Trust
Purposeful
Collaborative
responsibility
11

12. < Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
Community of Inquiry
(Anderson, Rourke, Garrison, & Archer, 2001)
Teaching
Presence
Educational
Experience
Setting
Climate
Selecting
Content
Supporting
Discourse
Social
Presence
Cognitive
Presence
Critical reflection
Systematic inquiry
Resolution
Metacognition
Assessment
12

13. Research Design
random
assignment?
randomized
or true
experiment
quasi-
experiment
non-
experiment
control group
or multiple
measures?
http://www.socialresearchmethods.net/kb/destypes.php
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
13

14. Research Design
Posttest only
randomized
experiment
Pretest-posttest
nonequivalent
groups quasi-
experiment
Posttest only
non-experiment
R X O
R O
N O X O
N O O
X O
http://www.socialresearchmethods.net/kb/destypes.php
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
14

15. Instructional Strategies as
Treatments
Posttest only
randomized
experiment
Pretest-posttest
nonequivalent
groups quasi-
experiment
Posttest only
non-experiment
R X O
R O
N O X O
N O O
X O
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
15

16. Summarizing and note
taking
Homework and practice
–peer feedback
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
16

17. Identifying similarities
and differences
–classify
–create metaphor/analogy
–Venn diagram
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
17

18. Cooperative learning
–paired work
–group discussion with
role assignment
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
18

19. Setting objectives and
providing feedback
–project based
–learning contract
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
19

20. CASE STUDIES
20

21. Midterm Evaluation
Swift & succinct: ask what
students like/dislike or
what they learn
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
21

22. Midterm Evaluation
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
22

23. Wikis in Plain English
mediaWikis _ Common Craft.flv
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
23

24. Research with Wikis
Using Wikis to Collaboratively Prepare for
Qualifying Examinations (DiPietro et al., 2010)
Clear instruction
Timeline
Expected outcome
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
24

25. Research with Wikis
Data Collection in wikis
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
25

26. Peer Tutoring
Enhancing the sense of community and learning
experience using self-paced instruction and
peer tutoring in a computer-laboratory course
(Buraphadeja & Kumnuanta, 2011)
freshmen / face-to-face /
Computer lab / non-credit
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
= 14 weeks
3(1) + 3(1) + 4(1) + 1
26

27. Peer Tutoring
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
= 14 weeks
3(1) + 3(1) + 4(1) + 1
27

28. Peer Tutoring
Self-Paced Learning + Interaction
= Learning Community
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
Existing
New Design
Teaching
Strategy
Step-by-step
instruction
Self-paced instruction
Location of
instructor
In front of the
class
In front of the class
Responsibility
to learning
Teacher
Student
Implication in
classroom
management
Conservative &
dominating
Liberal &
self-regulating
http://www.flickr.com/photos/barretthall/
28

29. Peer Tutoring
32 sections + 56 students = 1,7XX students
220 (Control) + 57 (experimental)
Classroom Community Scale
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
http://www.graphpad.com/quickcalcs/ttest1.cfm
29

30. Peer Tutoring
Descriptive statistics
Group x Gender ANOVA
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
Group Gender N
Connectedness Learning Community
M SD M SD M SD
Control F 148 22.11 4.16 21.91 4.19 44.01 6.35
M 72 21.68 5.23 22.29 5.45 43.97 8.53
Total 220 21.97 4.53 22.03 4.63 44.00 7.12
Experimental F 32 23.91 4.34 24.31 4.58 48.22 7.95
M 25 23.76 4.31 22.44 5.75 46.20 7.90
Total 57 23.84 4.29 23.49 5.16 47.33 7.92
df F p
Connectedness
Within groups = 273
Group 1 7.858 0.005 **
Gender 1 0.410 0.523
Learning
Within groups = 273
Group 1 4.297 0.039 *
Gender 1 0.041 0.840
Community
Within groups = 273
Group 1 9.432 0.002 **
Gender 1 0.276 0.600
Note: The mean difference was significant: * <.05; ** <.01 30

31. Forum Discussion:
Social Network Analysis
An assessment of knowledge construction in an online
discussion forum: the relationship between content
analysis and social network analysis (Buraphadeja, 2010)
SNAPP
http://research.uow.edu.au/learningnetworks/seeing/snapp/index.html
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
31

32. Activity Logs Data
Data for free: Using LMS activity logs to measure
community in online courses (Black et al, 2008)
“Non-invasive”
Classroom Community Scale[iv]
(Rovai, 2002)
Linear regression
LMS activity logs[dv]
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
32

33. Activity Logs Data
Data for free: Using LMS activity logs to measure
community in online courses (Black et al, 2008)
log files were determined to be a predictor
of sense of community (Adj R2=.086, F(2,
64)=4.090, p=.021)
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
33

34. Activity Logs BIS3315
Programming and Algorithms
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
34
Online
Activity
Duration Mark
Allocations
Online
Videos
Throughout the semester 2
Quizzes Throughout the semester 4
Forum
Discussions
1 forum before midterm, 1 forum
after midterm
5
Assignments 3 assignments before midterm 7
Workshops 3 workshops after midterm 12
Total 30

35. Activity Logs BIS3315
Programming and Algorithms
Percentage of activities
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
35

36. Activity Logs BIS3315
Programming and Algorithms
Comparison of overall and programming course activity logs
* Size of bubble denotes average logs per user in a given week
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
36
5.60 13.40
18.31
26.66
24.16
22.86
20.34
18.66
17.46
20.98
23.22
19.53
18.04
12.61
15.49
18.91
15.54
18.00
19.07
14.72
16.46
16.53
14.31
75.33 22.17 73.33 15.29 37.93 26.37 49.81 67.33 32.74 22.23 17.96 8.12 22.63 52.92 28.35 38.96 36.74 30.88 16.36 42.21 19.38
-20000
0
20000
40000
60000
80000
100000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

37. Activity Logs BIS3315
Programming and Algorithms
Comparison of overall and programming course activity logs
* Size of bubble denotes average logs per user in a given week
< Problem | Lit review | Theory & Framework | Research Design | Instructional Design | Data Collection | Analysis >
S C
T
37
5.60
13.40
18.31
26.66 24.16
22.86 20.34 18.66 17.46 20.98 23.22 19.53 18.04
12.61
15.49 18.91
15.54 18.00 19.07
14.72 16.46 16.53
14.31
75.33
22.17
73.33
15.29
37.93
26.37
49.81
67.33
32.74
22.23 17.96
8.12
22.63
52.92
28.35
38.96 36.74 30.88
16.36
42.21
19.38
1
10
100
1000
10000
100000
1000000
10000000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

38. Q&A
38

39. References
Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing Teacher Presence
in a Computer Conferencing Context. Journal of Asynchronous Learning Networks, 5(2),
1–17.
Black, E. W., Dawson, K., & Priem, J. (2008). Data for free: Using LMS activity logs to
measure community in online courses. The Internet and Higher Education, 11(2), 65–70.
Buraphadeja, V, & Kumnuanta, J. (2011). Enhancing the sense of community and learning
experience using self-paced instruction and peer tutoring in a computer-laboratory
course. Australasian Journal of Educational Technology, 27(Special issue, 8), 1338–
1355.
Buraphadeja, V. (2010). An assessment of knowledge construction in an online discussion
forum: the relationship between content analysis and social network analysis.
University of Florida, Florida. Retrieved from http://0-
www.eric.ed.gov.opac.acc.msmc.edu/ERICWebPortal/detail?accno=ED522091
Clark, R. E. (1994). Media will never influence learning. Educational Technology, Research
and Development, 42(2), 21–29.
DiPietro, J. C., Drexler, W., Kennedy, K., Buraphadeja, V., Liu, F., & Dawson, K. (2010).
Using wikis to collaboratively prepare for qualifying examinations: an example of
implementation in an advanced graduate program. TechTrends, 54(1), 25–32.
Jonassen, D., Campbell, J., & Davidson, M. (1994). Learning with media: restructuring the
debate. Educational Technology, Research and Development, 42(2), 31–39.
Kozma, R. B. (1991). Learning with media. Review of educational research, 61(2), 179–211.
Lipka, S. (2011). Want data? Ask students. Again and again. The chronicle of higher
education. Retrieved from http://chronicle.com/article/Want-Data-Ask-Students-
Again/128537/
Rovai, A. P. (2002). Development of an instrument to measure classroom community. The
Internet and Higher Education, 5(3), 197–211.
39