Presentation at the IU Online Conference on November 11, 2016

1. A Simple Strategy for Maximizing
Student Engagement (without
Overwhelming Instructors or Students)
Presented on November 11, 2016 at the
Indiana University Online Conference, Indianapolis, IN
Daniel T. Hickey
Professor of Learning Sciences
Center for Research on Learning and Technology
Indiana University, Bloomington

3. Research Context
3
2009 in Sakai
@IU
2012 in
CourseBuilder
BOOC
2013 in Canvas
@ IU & IUHS

4. Use public contexts give
meaning to knowledge tools
Reward productive
disciplinary engagement
Grade artifacts through local
reflections
Let individuals assess their
understanding privately
Measure aggregated
achievement discreetly 4
James
Greeno

5. Tools Needed to Implement PLA
• Use public pages as artifacts (“wikifolios”)
– I use discussion headers in Canvas
– IUHS teachers use Google docs (“gPortfolios”)
• Use threaded comments on pages for local
interaction
– Discussions in Canvas
– Comments on Google docs
• Online assessments
– Use open ended items for private self-assessments
– Use selected-response items for discreet module exams

6. Preparing for PLA
• Organize ideas and resources into chunks
– Around 4-6 elements seems to work
• Distinguish between disciplinary knowledge and
practices
– Knowledge is what experts know, independent of context
(i.e., textbook knowledge)
– Practices are what experts do in contexts that afford
expertise
• Define personalized engagement context or practice
– Organizing goal, question, project, etc.
– Should embody experience, interests, and aspirations
• Define networking groups (optional)

7. Engle & Conant’s (2002) Design Principles
for Productive Disciplinary Engagement
Problematize Content
- From learner’s perspective
- Open up issues that experts see as closed
Give Students Authority
- Avoid “known answer” questions
- Position students as stakeholders
Establish Disciplinary Accountability
- Defend positions
- Be responsive and respectful
Provide Relevant Resources
- Include time and information
- Provide easy contextualized access
7

8. • Ed Psych 540: Learning &
Cognition in Education
• Challenging to Teach
– Diverse experience & ambitions
among students
– Ambitious but busy instructor!
– Started adjuncts in 2011
• Typical Accountability
– Use textbook item bank
– Include a pretest
• Online in Sakai in 2009
– Canvas in 2013

9. 9
The P540 Team
Gina Howard
Lauren Smith
Andi Rehak
Xinyi Shen
Joshua Quick
Firat Soylu

12. • Educational Assessment Big Open Online Course
– P507 Assessment in Schools embedded for credit
• Offered to 500 students for free in 2013 with a grant
from Google using CourseBuilder
– 460 registered, 160 started, 60 finished, 9 in P507
• Offered a smaller streamlined version starting 2014
– Added videos and quizzed and automated key features
– Still open but not advertised widely

13. Supplemental Examples
Indiana University High School
• Fully online school
– One of the largest in the US
• Collaborating with teachers
since 2013
• Biology class use online
textbook and resources
• History and English classes
only use OERs
• Moved to self-paced in 2016
Rebecca Itow
(PhD Student)
Courtney Gaylord
(English & Comp)
Jody Duncan
(Biology)
Christine Hitchcock
(Social Studies)

14. 1. Use public contexts to give
meaning to knowledge tools
• Reframe course resources and concepts as
knowledge tools.
• “Problematize” knowledge from each
learners’ perspective.
• Learner define a personally meaningful
context.
– Disciplinary question, problem etc.
– Nascent disciplinary practice.
– Embodies their experience, interest, and
aspirations.
• Learners redefine context as they progress in
the course.
• Use public contexts for the work.
– Avoid “known answer” questions in public.
Rogers
Hall

16. My Learning Goal: Radiography students learning how to work as a team. After module 2, I
decided to narrow my learning goal. It was too broad.
Updated Learning Goal: Radiology students learning how to use critical thinking skills during
a trauma situation.
My Learning Context: I chose ‘Radiography students learning how to work as a team’
because I have a class of students who struggles with working together and acting
professionally. The students are separated in various hospitals in the area for several days in
the week and have class two days on campus. The behavior is mostly during class on
campus, but occasionally in the hospital too.
Our team of faculty has had guest speakers from various professions come and speak to the
students regarding student conduct, professionalism, etc. We have also had to place them
in a seating arrangement in class. I feel like this has helped marginally, but I want to also
prevent this behavior from happening when the new class of Radiography students begins
in the Summer II/Fall.
I have to teach an ‘Introduction to Clinical Radiography’ course in Summer 2 and would like
to incorporate some team strategies in the classroom. I’m hoping I can get some ideas from
this course or at least investigate why the current ‘troubled’ class is acting this way.

18. Theory of Learning: I believe that we all learn by life
experiences. The older we get, the more we learn from our
experiences. We observe others and learn from their and our own
mistakes. I think we retain information better when we are
motivated and determined to succeed.
In regards to my learning goal, I think the students are not motivated
to get their degree. I am not sure if it has to do with prior life
experiences in which they have not had to work hard for ‘something’
or if they are not interested altogether. I think to work together as a
team, every individual has to be motivated to get the job done. This
is my dilemma for students; how do I get them motivated in the
classroom?
Preferred Networking Group: First choice: Science; Second choice:
Comprehension; Third choice: Literacy

20. 21

21. Most Relevant Cognitive Themes: I think all of the 8 themes described in
the book are relevant to my learning goal, but number 3 is true for student
radiographers, ‘extended practice is needed to develop cognitive skills’ (p.
6). The students practice positioning different body parts to obtain a
desired image for the exam. It is important that they practice this with
each other and on patients as much as possible to become more familiar
and comfortable. Working together frequently will also help build these
skills needed for the job and team development.
Theme number 1, learning is a constructive process, is the least relevant
for my learning goal (p. 5). Learning to work with other healthcare workers
requires tremendous knowledge and skill in order to sufficiently perform
the required duties. According to the text, learning is the construction of
meaning by the learner (p. 6). This is true when developing teamwork skills
by extended practice, however, the knowledge must be there first to
perform specific duties sufficiently.
It was extremely difficult for me to choose a theme that was most/least
relevant. From my experience as a Radiographer, they all are important
aspects when students are learning how to work as a team.

24. Relevance of the Implications (when referring to working as a team in the hospital)
#1 2. Automaticity facilitates learning by reducing resource limitations. I chose this as the most
relevant for my goal because the more the student works with a team, the better they
become. Confidence will increase and the skills used within the team will become more automatic.
#2 3. Perception and attention are guided by prior knowledge. I chose this second because having prior
experience is always a plus for working in a team. If students have had prior experiences as working as a
team, the more successful they will be working within the team because they know what to
expect. This gives them more energy for other skills to learn.
#3 4. Perception and attention are flexible processes. The same as my choices for #1 and #2. I feel like
this is a repeat in that the more the student works in a team, the more it becomes automatic. They will
become more productive.
#4 6. All students should be encouraged to “manage their resources.” The more self-regulated students
are within a team, the more successful the team. They will be able to strategize more efficiently in a
team to complete the task.
#5 5. Resources and data limitations constrain learning. It may take some students longer to grasp the
skills needed to work as a team. This could slow the team down.
#6 7. Information processing is easier when to-be-learned information is distributed in working
memory. In a mock team scenario, visual and auditory rehearsals will allow students to retain the
information gathered longer.
#7 1. Information processing is constrained by a “bottleneck” in sensory and short-term memory. I
chose this as least important (although it is relevant) because in the healthcare environment, you
have to be on your toes and pay attention to everything that is going on around within the team.

25. Takeaway Relevant Specifics in Chapter 2
• Information about sensory memory helped me realize that I need to slow down
on the amount of material given to the students. On sensory registry, there is
only so much information they can handle at one time (p.20). Finding more
meaningful activities with course work could help them retain material more
efficiently.
• Presenting new information visually and auditorially will help the student
perceive the material (p.20). Having the students act out a scenario together
will help them retain information learned within the specific scenario- especially
if something wrong or right happened.
• Attention! Clear instructions for working as a team will equal better quality and
performance of the task (data-limited tasks p. 22-23). Student should know
their role within the group to enhance their performance and eliminate
confusion.
• Automaticity (as mentioned earlier) is important for teams. Extended practice of
skills will make those skills automatic and need little attention (p. 23).
• ‘Working memory works over time” (p. 28). Students who are older (may) have
more experience and know more particulars about working within a team.

26. Results:
Overall Public Engagement in P540
• Wikifolios averaged 1569 words (144-3291)
• Groupwikis averaged 3477 words (2705-4630)
0
1000
2000
3000
4000
5000
1 2 3 4 5 6&7 8 9 10 11 12 13 14 15
Average Words per Wikifolio & Groupwiki
Wikifolios
Groupwikis

27. 28

35. 36
Lesson 1: Introduction to Narrative Writing
CCSS W.10.3 Write narratives to develop real or imagined
experiences or events using effective technique, well-chosen
details, and well-structured event sequences.
Briefly: Tell a story that readers enjoy reading including a beginning,
middle, and end. Points along the way of your story should keep the
story flowing.
Getting started…Narrative is just another way to say "story-telling," and people do
it all the time. Talking with friends, writing letters, or just telling your folks what
happened while you were out today are all narrative. Other examples of narrative
would include history, the daily news, TV dramas, movies, and even children's fairy
tales. Humans are great storytellers and love a great story. We communicate our
culture to our children through stories and morality plays. Without narrative, life
would be pretty boring. The important part about a good narrative is making sure
the story is an interesting one, and putting all the parts together so that the story
flows from point to point toward the climax of the story. A good narrative leaves
the reader wanting more.

36. 37
Part One—Evaluate and Rank Resources:
Go to the following sites and read/watch the information/examples on
narrative writing. Rank their relevance to you on you wikifolio. Find and
share other relevant resources if you wish.
1. Summarize each resource: In two or three sentences, explain what
this resource tells you.
2. Evaluate each resource: Consider for whom it is intended and what
its goals seem to be. How effective is it at accomplishing those goals
for that audience?
3. Rank each resource: How useful will this resource be for you as you
work on the next assignment? Please explain briefly why you ranked
each one as you did.

37. EVELYN’S EVALUATING NARRATIVE RESOURCES
Purdue OWL Website: This site goes into a decent amount of detail
telling the reader about how a narrative essay is written. It goes on saying
that this type of writhing are often anecdotal, experiential, and personal.
It allows the one who is writing them to express themselves in creative
and most of the time quite ways.
Evaluation: This resource is good in describing to the reader key points in
how a narrative essay is written. Telling the reader such things as, If
written as a story, the essay should include all the parts of a story, when
would a narrative essay not be written as a story, the essay should have a
purpose, the essay should be written from a clear point of view, use clear
and concise language throughout the essay, the use of the first person
pronoun ‘I’ is welcomed, and finally as always, be organized. Over all this
resource tells the reader in a very detailed way how to correctly write a
narrative essay.
Ranking: I give this a 1 because it goes into some decent detail about the
learning how to write a narrative essay, going into further detail about
telling all parts of a narrative story to the reader.

38. 39
LADONNA’S EVALUATING NARRATIVE RESOURCES
Purdue OWL: Essay Writing- Rank 1
Summary: This resource is an informational guide on the things you should
include in your narrative essay. It gives you several tips on what people will
be looking for in your essay and uses understandable language to convey it's
information to you. This resource is easy to read and understand. It's a little
short, but its full of information.
Evaluation: I've used OWL as a resource before, so this resource was easy
for me to read and understand. I feel that this resource would be useful for
anyone who chooses to read it; adults could read it and have no problem
with understanding it and so could younger children. I like this resource
because it uses plain and simple language to show you what it's talking
about and it appears to be written by a person, not a computer or machine,
if that makes any sense.
Rank: I chose rank 1 for this resource because I found it to be the most
informational and useful. As I said, I've used the Purdue OWL before and
have been able to understand it easily, I found that to be true with this page
of the website too. As with the other resources, I found this one to be short,
but this one had more information in it than the other two.

39. 40
Part Two: Choosing a Topic for a Personal Narrative
Call this section of your Wiki “Topics.”
First: Think of five particularly important events in your life. For
example, a big move for your family, an injury or setback, a challenge,
a victory or other key event.
Second: Briefly describe your events. What happened? Who was
involved? When did this happen? Where? How did it shape you into
the person you are today or how did it change your views on a
subject?
Third: Support your topic. It's not good enough for a narrative to just
list the event. You also need to explain why it was so important. After
each event you think of, explain what made it such an important one
in your life. List your reasons in a logical order.
Fourth: Choose the topic that you think you would like to write
about. Explain why you have chosen it.

40. 41
The topic I would most like to write
about is the head injury I sustained.
I wish to write about this because it is
one of the most major event in my life
that I can remember. Therefore it is
easy to remember and easy to give all
the necessary details, so that it makes
sense and people area able to follow
along and completely understand
what I am writing.

41. 42
I think the topic I'd most like to write
about is the injury I've sustained to
my wrist. There are a lot of aspects
about it that I think are really
interesting and there were so many
different opinions from each doctor
that I think it would make an
interesting story. There are a lot of
other things I like about my other
topics, but I think that this topic is
probably the most interesting and
diverse one that I have.

42. Reward productive
disciplinary engagement
53
Randi
Engle
• DE involves declarative knowledge and
cultural practices
• PDE connects knowledge to practices,
makes connections, finds resources, etc.
• Support PDE so it can be rewarded
– Comment directly & locally on wikifolios
– Work in the open and encourage “lurking”
– No grades or mandatory posts
• This really requires DBR to get right
– Easier for me because of my research

48. 1. Wikifolios averaged
8.9 comments & 95
words/comment
2. Groupwikis forums
averaged 61 posts
3. Groupwiki averaged
40 comments & 92
words/comment
4. 2062 comments
included 182 badges
0
5
10
15
1 2 3 4 5 6&7 8 9 10 11 12 13 14 15
Average Number Comments per Wikifolio in P540
Unthreaded
Threaded
0
20
40
60
80
100
Forum Posts by Group and
Comments by Groupwiki
Forum
Posts
Groupwiki
Comments

49. Agreement & Disagreement
in Badges vs. Comments
0
10
20
30
40
50
60
70
80
90
100
Agreement Disagreement Perspective
Change
Reinforcement
PercentageFrequeincy
Content (potentially co-ccuring)
Comments
Badges

50. 61

51. 62

52. 63

53. 64
Part Three: Collaborate with classmates.
Go to the discussion forum for this activity and paste
your dialogue exchange into the forum.
Then respond to at least two of your classmates. In your
comments, consider how well the writer has followed the
rules and conventions of writing dialogue. If the writer
needs to make changes in the formatting, spacing, or
punctuation in order to follow the rules of dialogue, let
them know how to make those changes. If the speech
doesn’t sound fluid and natural, let them know how to
make it sound more fluid and natural. Dialogue is difficult
to write, so work together to make all of your dialogue
stronger!

54. 65

55. 66
Courntey: But it's important to keep in mind that in order to be
"narrative-worthy" you need to focus on an experience that really
taught you something and changed you in some way. You say that
you think about the head injury a lot, but you haven't explained
what made it important in your life. Your move to SD certainly
sounds as though it might produce a narrative that is more reflective
and interesting than any of these injuries
Ladonna: But I am wondering, I know you chose the event of the
plates crashing on your head because you say it was the most
memorable for you, but I was wondering, what are the other factors
that made that event important in your life? For instance, I chose my
wrist injury because it has changed me as a person. Did your injury
change you, do you think? Or did you just chose it simply because it
was something you remember vividly?

56. Grade artifacts through
“local” reflections
• Wikifolios generate lots of text and
comments
– Grading artifacts and comments undermines
participation
• Grade reflections on PDE.
– Contextual engagement (suitability of personal
context for practicing using concepts)
– Collaborative engagement (what students
learned from others)
– Consequential engagement (consequences of
new knowledge for disciplinary practice )
• Summative assessment of engagement
– Proleptically shapes prior engagement
– Formative assessment of knowledge
71
Melissa
Gresalfi

57. Reflect. After you've commented on your classmates' work and received a
sufficient number of comments from classmates, post three reflections at the
bottom of your chapter 2 wikifolio page. Insert a bold header called Reflection and
write at least one paragraph on each aspect of your engagement.
Critical engagement: How suitable was your instructional goal and educational
context for learning the ideas this week? Did your classmates have goals where
these implications were more directly relevant? For example, some instructional
goals make bigger demands on short term memory than others. You are not being
asked to criticize your understanding or your work; rather you are critiquing the
suitability of your instructional goal and experience for learning to use these ideas.
Collaborative engagement. Review the comments from your classmates and
reflect on any insights that emerged in the discussions or anything that was
particularly useful or interesting. Single out the classmates that have been
particularly helpful in your thinking, both in their comments and from reading their
wikifolios.
Consequential engagement. What are the consequences of what you learned this
week for your instructional goal and in general? How would you teach or instruct
differently based on what you learned? What are the real world consequences of
these ideas? Think generally but be specific.

58. Contextual engagement:
I found this chapter to be suitable for my goal, but I
realized that maybe I need to take a step back and
not make it so broad. Working as a team covers a
wide area in my field and can be used in multiple
settings like the classroom, hospital and as a group
(graduating class). When I was trying to figure out
the order of relevant implications, I had to state
what area of teamwork (hospital) would show the
relevance. Other students’ goals were more
directly relevant and not as broad. That made me
realize I need a more specific goal.

59. Collaborative engagement:
Yue mentioned the importance of differentiating and learning
different roles in the team so that they can perform better as a
team. This made me think of the importance of when each team
member is to anticipate what is coming next. It also reminded me of
how important the students need to understand who is involved in
the room and what their duties involve.
Alexander mentioned that there may be something else beside the
cognitive load that would need work (other than teamwork skills). I
think of motivation again and the students’ inability to want to work
in a team. But, I think this is more of a cognitive load issue in that
there is so much information to learn in a short amount of
time. The students are expected to work together and they are not
performing well until they have practiced more and more. He also
suggested simulations in a ‘real-world’ experience to help practice.

60. Consequential engagement
After studying the implications for this chapter, it made me realize that radiology
students have an incredible amount of information that must be stored and
practiced in order for their skills to become automatic and useful in a team
situation. Most of the information is given to them in a short period of time, so
they need to be able to distinguish what material is the most important and move
on. I realize that I need to make more specific learning goal! The following is an
addendum to my learning goal:
Radiology students learning how to use critical thinking skills during a trauma
situation.
Based on what I learned, I need to make sure the students are getting the basic
concepts of how to do a basic exam (on an ideal ‘walkie-talkie’ patient). Once
these skills are achieved, then they can start thinking outside of the box for a
more realistic trauma patient. I could do mock examinations with the students in
an x-ray room. Setting up scenarios for them to work together and work out the
different ways of imaging a patient with limited movement (on the patient’s
part). They could then take these skills to the hospital and work on patient’s with
similar situations.

61. Let individuals assess
understanding privately
• Offer ungraded open-ended
items and performance tasks
• Cover all of the big ideas in the
assignment
• Use to re-engage rather than
remediate.
– Items themselves have little formative
value beyond the correct answer
77
Pamela
Moss

62. 78
2. Self-Assess Your Knowledge. When you
have completed your wikifolio and reflections and
you feel you are ready to move on, complete the
ungraded Chapter 2 Self Assessment Quiz. If you
can't write an acceptable answer for most of the
items from memory, you will not do well on this
part of the module exam. You should go back re-
read the text, re-engage with your classmates and
review their implications and specifics. Note that
these items will not be included in the module
exam.

63. Measure achievement
discreetly
• Standards-oriented distal tests
– Necessary for measuring gains, comparing instruction,
documenting course improvement
– Useful for motivating prior engagement
• Protect test security
– Provide only item-level feedback
• Refine tests (but don’t teach to them)
– Use LMS item analyses
– About 85% average with 1-2 perfect scores
• Limit point value to 30-40% of the course
– Will need to refine over time
• Time tests and use high quality items
– Include many “best answer” items
Me!

65. P540 Exam Scores (percent correct)
0
10
20
30
40
50
60
70
80
90
100
MC Pretest
(25 items, 50
minutes, 0
points)
MC Midterm
(40 minutes,
20 items, 10
points)
Essay Midterm
(50 minutes, 5
items 5 points)
MC Final
(60 minutes,
30 items 9
points)
Essay Final
(120 Minutes,
6 items, 6
points)
Mean Exam Scores (Percent Correct)

66. Course Evaluation Results
0 1 2 3 4 5
My instructor is WELL-PREPARED for class
meetings.
My instructor treats students with
RESPECT
Overall, I would rate this INSTRUCTOR as
outstanding.
Overall, I would rate the quality of this
COURSE as outstanding.
14 Other Sections This Course
Campus-Wide Items
(11/15, 5-point Likert, SD to SA)

67. Course Evaluation Results
(11/15, 5-point Likert, SD to SA)
0 1 2 3 4 5
This distance education format helped me feel IN
CONTROL of my own learning
The online technology used in this course was
RELIABLE
The ONLINE FORMAT was well suited for this course.
The level of INTERACTION in this distance course
was sufficient.
I would take ANOTHER ONLINE COURSE from the
School of Education
The instructor used the available TECHNOLOGY well.
School-Wide Online Learning Items
14 Other Sections This Course

68. Issues with Semi-Formal (Cohorted)
Course Format
 Assumes regular instructor effort
 Is not what online learners expect
 Not flexible
 Can’t get behind
 Led many students to drop out of the BOOC
 Led many IUHS students to stay with
distance ed courses

69. Moving to Self-Paced Courses
 Not simple to get interaction
 Need to find like-minded peers
 Need to find peers working on same unit
 Need a large number of learners
 Currently creating a self-paced BOOC
 New participant list shows current learners
 New wikifolio feature allowing archiving
 Learners can look at archived work
 Currently creating self-paced IUHS courses
 Modules with exams and Badgr

70. Moving to Open Learning Resources
 Need to automate many features
 Automated registration and homepage
 Real-time participatory feedback
 Exam scoring and badges issuer
 Relying extensively on open ed resources
 Use ranking feature with OERs
 Pending NSF proposals for IUHS and IU

72. Thank you very much!
More info at www.RemediatingAssessment.blogspot.com
Email me at dthickey@Indiana.edu
Hickey, D. T., & Uttamchandani, S. U. (in press). Beyond hype, hyperbole, myths, and paradoxes:
Scaling up participatory learning in a Big Open Online Course. In Losh, E. (Ed.). The
MOOC Moment: Experiments in Scale and Access in Higher Education. Chicago, IL: The
University of Chicago Press.
Hickey, D. T., & Willis, J. E., & Quick, J. D. (2015). A framework for interactivity in competency-
based online courses. EDUCAUSE Review, July. Online.
Hickey, D. T. (2015). A situative response to the conundrum of formative assessment. Assessment
in Education: Principles, Policies, and Practices.
http://dx.doi.org/10.1080/0969594X.2015.1015404
Hickey, D. T., & Rehak, A. (2013). Wikifolios and participatory assessment for engagement,
understanding, and achievement in online courses. Journal of Educational Media and
Hypermedia, 22 (4), 229-263.
Hickey, D. T., & Zuiker, S. J. (2012). Multi-level assessment for discourse, understanding, and
achievement in innovative learning contexts. The Journal of the Learning Sciences, 22, (4)
1-65
Hickey, D. T., Kelly, T. A, & Shen, X. (2014). Small to big before massive: Scaling up
participatory learning and assessment. Proceedings of the Fourth International Conference
on Learning Analytics and Knowledge, Indianapolis, IN (pp. 93-97)