Open Education Conference 2015 Presentation on preliminary results from a study that aimed to explore the appropriateness of assessing student performance through student annotation use in digital learning environments
Under the direction of Gardner Campbell, the Vice Provost of Learning Innovation and Student Success, my home university, VCU is trying to take learning through public, digital scholarship to a very large scale.
Our goal is to build a virtual city-campus, not just for distance learners, but for everyone. Students, faculty, and staff have their personal blog spaces that can be networked at a local level for formal and informal learning. Because these are public spaces, they integrate smoothly with the global network.
And a specific kind of course experience seems to be emerging from this experiment – Course materials, activities, and communications are found on a public course website Students complete at least some of their assignments through the act of blogging on their personal websites – these are aggregated by an RSS feed for the course website Certain level of public discourse which counts towards a participation grade, be that commenting on their colleagues’ blogs, Twitter, Discussion Forums,
Our reasons for promoting these sorts of experiences are multifold, but we are hoping to promote certain learning dispositions, one of which is connectivity.
One of the main learning dispositions is the concept of connectivity – the ability to recognize, understand, and act strategically on connections made across content, people, space, and time.
Meaningful assessments are those that Document what you are hoping to achieve Advance the learning of the students by providing actionable feedback Meet goals of 21st century assessment – scalability, feasibility, peer/self assessment
Social learning analytics may offer a solution. Ferguson and Buckingham Shum (2011) propose a shift in emphasis from the highly technical, individualized, and algorithmically motivated work of learning analytics to what they define as social learning analytics, a subset of learning analytics meant to capture, organize, and demonstrate the inherently social, open, and connective aspects of networked participatory learning. The authors identify five research methodologies that show promise in advancing digital age learning: content, disposition, context, network, and discourse analytics.
My study focuses on the annotations of blogging and tweeting. These are the symbols and phrases that are distinct from but included within the communication, meant to demonstrate communicative intent. Specifically, I focused on hyperlinks, embedded images, mentions, and hashtags
My argument for
Three types of information are consistently available from clicking on hyperlinks in student works: the types and sources of the materials, and what purpose it serves. To save time I won’t review the types and sources of materials that CC students used – however they were appropriate for student level and the course content.
Students hyperlinked to supporting documents. Oftentimes this took the form of traditional citing or referencing, or in the undergraduate courses, it was an “Additional Resources” list at the end of the post. However, sometimes, CC students used hyperlinks to define, describe, or provide examples of something in the narrative in ways that wouldn’t be possible on paper. In the two examples listed, students hyperlinked to provide explanations of what they mean by Mad Lib or nonverbal, rather than explaining it in the primary narrative. If they had stopped to explain it, it would have cluttered up the primary narrative.
Students also used hyperlinks to link to previous work or experience. They used them to piece together their personal narrative.
Students also hyperlinked to the course website, to provide context for what they were writing. This is interesting to me because it speaks to an awareness of audience beyond people who are in the class.
But you can also make a dashboard, like this.
1. Students must complete the work, as demonstrated by the number of posts completed. 2. The number hyperlink-embeds per post provides information about quantity. Content type (image- vs. text-based materials) addresses the most basic characteristic of the connections being made. The presence of broken or misdirected links and aesthetic images offer measures of quality control for hyperlinks and embedded images, respectively.
While the CC cohort included a high number of exemplary students, there was still some stratification in performance. At the time of sampling, students should have completed eight or nine posts; the variation occurred because the sampling frame did not precisely match the assignment completion schedule. However, students with only seven posts were falling behind. Students are not necessarily ordered with each category.
Exemplary student work included hyperlinked or embedded materials frequently, averaging seven to nine hyperlinks per post. In general, these students used fewer aesthetic images and had few, if any, broken links.
Average student work included just as many hyperlinks and embedded images as exemplary students, but did not have the same level of quality.
Two of the three students in the final grouping had fallen behind on completing assignments. Furthermore, they had neither the frequency nor quality of hyperlinking or embedding, suggesting that intervention or, at least, further review, is required.
This figure shows a three-post sample work from three exemplars: CC-S4 (exemplary), CC-S1 (average), and CC-S10 (needs improvement).
CC-S4 tends to incorporate hyperlinks that have a variety of purposes in each post. For example, exemplary work might be a post that uses hyperlinks to support a brief description of course or personal context, followed by hyperlinks or embedded images to support their argument (description, citations, and illustration), followed by a hyperlinked list of references (references).
An average post may have less links, some of which may be broken, concentrated in a list of references.
Posts that need significant improvements may only include several hyperlinked references and embedded clip art that punctuate section headings but do little to further the narrative
Annotation-Centric Assessment of Blogging in Higher Education
Exploring the What, Why, & How of
Social Learning Analytics
Annotation-Centric Assessment of
Blogging in Higher Education
Laura Gogia - @Googleguacamole
Academic Learning Transformation Lab
Virginia Commonwealth University, Richmond VA
Slides are available on SlideShare.net
Hello. My Name Is...
Do you teach online? Blog and tweet with student-
participants? In public? To what end?
How do you know if you are achieving your goals?
VCU is exploring the intersection of
connected learning & open education
Photo Credit: http://graduate.admissions.vcu.edu/why/
Personal Blog Sites
Emerging-Evolving Course Experience:
• Public – Open Course Website
• Public and Aggregated Student Blogging
• Public Discourse
By introducing students to this sort of loosely
structured, flexible, open digital learning experience,
we hope to promote certain learning dispositions,
Creating, recognizing, understanding, and acting
on connections made across content, disciplines,
living spheres, people, space and time.
Interdisciplinary Learning – Self-Reflection – Transferability –
Social Learning – Intersection of formal and informal – Holistic
But how do we assess connectivity?
We are thinking hard about how to build
meaningful assessments in our course designs.
(1) Documenting connectivity.
(2) Advancing the learning.
(3) Meeting 21st century goals for assessment.
Social Learning Analytics: A subset of learning
analytics meant to capture, organize, and
demonstrate the inherently social, open, and
connective aspects of networked participatory
--Ferguson & Buckingham Shum (2011)
Are there ways to take advantage of the
uniquely digital aspects of blogging and
tweeting to assess student blogging and
tweeting in these environments?
Symbols & phrases that are distinct from but included within
the communication, meant to demonstrate communicative
Hyperlinks – Embedded Images – Mentions – Hashtags
Initial argument for annotation-centric
• While specific to digital, they point to the art of
• Annotation is an act that lends itself to strategic
reflection (“Why did you annotate that?”)
• Documentable, extractable, quantifiable.
Strands of Inquiry
1. How are student-participants using annotations in
course-related blogging and tweeting?
1. How can this information be used (organized and
visualized) to inform meaningful assessment?
4 ONLINE COURSES
Undergraduate & Graduate
Multidisciplinary & Gen Ed
Instructors & Assistants (n = 10)
Students (n = 60)
Open Participants (n = 12)
Other Participants (n = 200)
1618 Posts (500 Sampled)
Hyperlinks (n = 800)
Embedded Images (n = 400)
Hyperlinks (n = 430)
Mentions (n = 3000)
Hashtags (n = 130)
Social Network Analysis
Two Things Happened.
1. Classification systems for describing how
1. Dashboards for documenting and assessing
CC was a graduate level elective which aimed to introduce its
students to community engaged research.
On average, students blogged several times a week in three
formats: (1) digital makes; (2) reflective posts; (3) research
They also engaged in weekly, structured Twitter chats.
Enrollment: 10 students. 12 open participants.
• Types and sources of
• Purpose or impact of the
annotation in the context
of the post
But why did CC students hyperlink?
Types and sources of materials that CC students
used were appropriate for student level and the
1. To link to supporting documents.
• Traditional citing and referencing.
• Defining, describing, and providing examples in
ways not supported by formal writing styles.
• “This could be filled in, Mad Libs style.”
• “...verbal or nonverbal communication...”
2. To link to previous work or
• “As I discussed in a previous post...”
• “As I reflect on my proposed research
3. To provide course context.
• “I am writing this because I am taking a course...”
The pedagogical value of
embedded images and videos
varied, but in a way that can be
defined and organized into a
BASIC LEVEL (IMAGES):
• Serves no obvious purpose other than
contributing to an aesthetic.
INTERMEDIATE LEVEL (IMAGES):
• Provides additional information
• Makes an otherwise unstated them explicit.
• Inspires deeper questions.
ADVANCED LEVEL (IMAGES & VIDEOS):
• Further the narrative (e.g. a table, chart, or
infographic that the student refers to or explains in
• Demonstrate a personal connection to the subject
(e.g. a photograph, graphic, or video the student
made themselves and explains in the narrative)
How does can typologies like
these inform student
Rubrics – Peer Assessment – Self Assessment