Did It Work? Evaluating the Impact of the Learning Resource Metadata Initiative (LRMI) for OER
•Download as PPTX, PDF•
1 like•536 views
Presented at SXSWEdu March 6, 2014 Austin, TX, USA Sure, aligning commercial content to LRMI is becoming a standard for major publishers and those with deep pockets, but what about non-profits, museums, or research organizations? The Smithsonian will present their process for identifying resources, selecting teacher metadata creators, and implementing the code on their websites and into the Learning Registry. Attendees will learn how to develop effective methodologies to evaluate the impact on discoverability of LRMI-tagged content. Darren Milligan Senior Digital Strategist Smithsonian Institution, Center for Learning and Digital Access @darrenmilligan Melissa Wadman Manager of Program Evaluation Smithsonian Institution, Center for Learning and Digital Access Sue Cowden Senior Director, Content Partnerships inBloom #lrmiforoer - See more at: http://schedule.sxswedu.com/events/event_EDUP23498#sthash.TmtkMUrU.dpuf
Recommended
Recommended
More Related Content
Similar to Did It Work? Evaluating the Impact of the Learning Resource Metadata Initiative (LRMI) for OER
Similar to Did It Work? Evaluating the Impact of the Learning Resource Metadata Initiative (LRMI) for OER (20)
More from Darren Milligan
More from Darren Milligan (9)
Recently uploaded
Recently uploaded (20)
Did It Work? Evaluating the Impact of the Learning Resource Metadata Initiative (LRMI) for OER
- 1. Did It Work? Evaluating the Impact of the Learning Resource Metadata Initiative for OER #LRMIforOER Darren Milligan Senior Digital Strategist Smithsonian Institution, Center for Learning and Digital Access @darrenmilligan Melissa Wadman Manager of Program Evaluation Smithsonian Institution, Center for Learning and Digital Access Sue Cowden Senior Director, Content Partnerships inBloom
- 3. Audience Participation Time Image adapted from the Department of Education, http://www.flickr.com/photos/departmentofed/9602545478/, used under a CC BY 2.0 license.
- 4. Audience Participation Time (Raise Your Hands) Who has visited one or more Smithsonian museums in DC or New York?
- 5. Audience Participation Time (Raise Your Hands) Who has visited one or more Smithsonian museums in DC or New York? Who has visited a Smithsonian website, used a digital resource, or interacted with a Smithsonian scientist, curator, educator, or other staff online?
- 7. 1995 23.6 million physical visits to our museums 72,942 web visits
- 8. 1995 23.6 million physical visits to our museums 72,942 web visits 2013 30 million physical visits to our museums 140 million digital visits
- 9. 1995 23.6 million physical visits to our museums 72,942 web visits 2013 30 million physical visits to our museums 140 million digital visits Physical: 30,000,000-23,600,000 / 23,600,000 X 100 = 27.12% increase Digital: 140,000,000-72,942 / 72,942 X 100 = 191,833.32% increase
- 10. 1995 23.6 million physical visits to our museums 72,942 web visits 2013 30 million physical visits to our museums 140 million digital visits Physical: 30,000,000-23,600,000 / 23,600,000 X 100 = 27.12% increase Digital: 140,000,000-72,942 / 72,942 X 100 = 191,833.32% increase
- 11. 2010 30.2 million physical visits to our museums 97.2 million digital visits
- 12. 2010 30.2 million physical visits to our museums 97.2 million digital visits 2013 30 million physical visits to our museums 140 million digital visits
- 13. 2010 30.2 million physical visits to our museums 97.2 million digital visits 2013 30 million physical visits to our museums 140 million digital visits Physical: 0.66% decrease Digital: 44.06% increase
- 14. What Can the Smithsonian Do to Help Young People Be Successful in Their Lives?
- 15. What Can Our Digital Resources Do to Help Young People Be Successful in Their Lives?
- 16. “We have ambitious plans to use new technologies to reach new audiences. … We have much to offer students and teachers in art, science, history, education, and culture. We want to give learners of all ages access to America’s treasures and our creative experts who bring them to life.” – G. Wayne Clough
- 22. LRMI Survey Report August 2013 Update 43.7%: search online several times a week 30.5%: search online daily 64.8%: report “irrelevant results” 86.8%: more satisfied if they could filter
- 24. LRMI Survey Report August 2013 Update Survey responses from educators indicating the relative importance of descriptive metadata for learning resources (LRMI Survey Report August 2013 Update, Ease and Discoverability: Educators and Publishers on the Search for Educational Content).
- 26. based on The Content Developers Guide to the Learning Resource Metadata Initiative and Learning Registry, 2013
- 28. Train and Share Training Workshops Training Video Virtual Tagathons LRMI Guide
- 32. Who are the best partners for developing strategy, implementation, and evaluation of an LRMI tagging project? How do you evaluate/demonstrate the impact of LRMI implementation? How does a large OER publisher implement LRMI?
- 33. Questions? #LRMIforOER Darren Milligan Senior Digital Strategist Smithsonian Institution, Center for Learning and Digital Access milligand@si.edu / @darrenmilligan Melissa Wadman Manager of Program Evaluation Smithsonian Institution, Center for Learning and Digital Access wadmanm@si.edu Sue Cowden Senior Director, Content Partnerships inBloom sue.cowden@inbloom.org
Editor's Notes
- To begin, let’s take a look first at this place: -the Smithsonian is the world’s largest museum and research complex. Founded in 1846 with the mission of: the increase and diffusion of knowledgecollection of 19 museums and galleries, 9 major international research center, the National Zoo, a nationwide network of 184 affiliate museums, and a collection of programmatic offices focusing on targeted initiatives.- since 1846 the Smithsonian has grown in size, adding new museums and centers, but also in visitation, both physical and digital.
- So, let’s do some easy, don’t worry, easy audience participation:
- Raise your hands: Who has visited one or more Smithsonian museums in DC or New York?And let’s do, since 1995. You will see why in a minute.
- Again, since 1995….Raise your hands:Who has visited a Smithsonian website, used a digital resource, or interacted with a Smithsonian scientist, curator, educator, or other staff online?
- The reason for the 1995 restriction was that the Smithsonian went online in 1995, so let’s start there, and see how things have changed.Let’s take a look at how these numbers compare
- 1995 (19 years ago)23.6 million physical visits to our museums72,942 web visits(about 3% of the physical visitors)
- So if we just ahead 19 years to 2013, to last year, we see a pretty dramatic increase in visitation, both physical and digital.201330 million physical visits to our museums140 million digital visits
- An if we do some algebra, solve for x, we see thatPhysical visitation increased by about 27%But Digital visitation to the Smithsonian increased by 191,833%Source:(Estimated number of people who visited Smithsonian public websites during the report period as counted by SI's WebTrends reporting software. Website tracking software counts a "unique" visitor once even if they visit a site multiple times, but for technical reasons the result is considered only an approximation.Data Source: Office of the Chief Information Officer (OCIO))
- So let’s let those numbers sink in for a minute. Almost 200,000% increase in digital visition!I suppose you can argue that is to be expected as we have seen a global explosion in the use of digital tools and the expectation of people of all ages for access to rich digital content. In fact, most museums now operate in three different spaces: the physical museum floor, online on the fixed web in places like websites and social media platforms, and most recently in mobile apps and mobile sites.- so, let’s take a look more recently, say in the past 3 years, we find:
- 201030.2 million physical visits to our museums97.2 million digital visits
- And the 2013 numbers again30 million physical visits to our museums140 million digital visits
- you’ll have to trust me on the math on this one, but what we find here is- Physical: -0.66% decrease; basically a leveling out of visitationDigital: 44.06% increaseA tiny decrease in the physical. Physical visitation ahs leveled out.This staggering and sustained growth in our digital visitation reflects not only the increasing work of those here at the Institution, but the access desires of the people we serve. The Smithsonian will always be, for the foreseeable future, more of a digital resource than a physical destination.Source:(Estimated number of people who visited Smithsonian public websites during the report period as counted by SI's WebTrends reporting software. Website tracking software counts a "unique" visitor once even if they visit a site multiple times, but for technical reasons the result is considered only an approximation.Data Source: Office of the Chief Information Officer (OCIO))
- Given this enormous change in how the world sees and interacts with our Institution, we can now begin to ask ourselves questions like this:What Can the Smithsonian Do to Help Young People Be Successful in Their Lives?
- Or more specifically:What Can Our Digital Resources Do to Help Young People Be Successful in Their Lives?
- Its not just practitioners like myself asking these kinds of questions, but the entire Institution is reflecting. This is a quote from Wayne Clough, former president of Georgia Tech and current Secretary of the Smithsonian.“We have ambitious plans to use new technologies to reach new audiences. … We have much to offer students and teachers in art, science, history, education, and culture. We want to give learners of all ages access to America’s treasures and our creative experts who bring them to life” –G. Wayne Clough
- This is part of what educational outreach at Smithsonian looks like:discussion of units education websites/pages, more than 32 different places to startlots of great work being done to connect resources with educators, experts with learners, etc., but being done still in a way that works really well if you understand our buildings, our geography, but not so great if you don’t.
- One of the ways that the Institution combats this problem is through the creation of the Smithsonian Center for Learning and Digital Access. We are a center of excellence within the Institution focused on establishing the Smithsonian as a learning laboratory for everyone. So, what that means is that, in this new reality of massive digital access and desire, It’s our mission to provide models and methods that enable learners to access everything the Smithsonian has to offer and empower them to explore their own interests and engage with others. Today, we are going to be focusing on a project that aspires to increase access, specifically, to digital learning content available from the Smithsonian.
- One way we do this is through SmithsonianEducation.org, the central education portal for the entire Institution
- Main feature is an indexed collection of learning resources that are aligned to all state, national, and Common Core standards of learning. The site’s database has simple metadata describing 2,000 resources such as lesson plans, video and audio clips, and interactive instructional games.
- The metadata accessible here on this site, is currently also being syndicated through partnerships with educational resource content aggregators: non-profit, for-profit, state departments of education as well as published into the Federal Learning Registry.Current collaborating sites:Brokers-CA Department of EdPromethean PlanetLearning.comNettrekkerNYC Department of EdPearson FoundationLearning Registry*
- We are a very data-driven center at the Smithsonian, so we always start with an attempt to understand better our audiences needs. A recent survey of educators found:- Nearly half (43.7%) of educators said they search online for instructional resources at least several times a week, and nearly a third (30.5%) search daily. - Two thirds of educators (64.8%) said they get too many “irrelevant results.” - Nearly 9 in 10 educators (86.6%) said they would be more satisfied with Internet searches if they could filter results by standard instructional criteria such as grade level, subject area, media type, etc.But, we also recognize that most of our users, the teachers and learners we hope to reach with our resources, will most likely never come to our home page and search. They may not discover our games through one of our content partners. Source:Winter Group. (2013) LRMI Survey Report August 2013 Update, Ease and Discoverability: Educators and Publishers on the Search for Educational Content. Association of Educational Publishers. August 2013. Consulted January 30, 2014. http://www.lrmi.net/wp-content/uploads/2013/08/LRMI-Survey-Report-August-2013-Update.pdf
- Most likely, if they find our content at all, they do it the way that we all find content.
- We realize that if the Smithsonian hopes to contribute to the potential of Personalized learning, places like our museums and research centers, who already produce open educational content, need to improve the discoverability and retrieval of their digital resources. We need to listen to the needs of those we are trying to serveWe must develop complete learning-appropriate descriptions of what we have and share this descriptive language with users in many settings. Source:Winter Group. (2013) LRMI Survey Report August 2013 Update, Ease and Discoverability: Educators and Publishers on the Search for Educational Content. Association of Educational Publishers. August 2013. Consulted January 30, 2014. http://www.lrmi.net/wp-content/uploads/2013/08/LRMI-Survey-Report-August-2013-Update.pdf
- How do content producers like the Smithsonian ensure that educational users can quickly locate and analyze potentially crucial digital resources? The solution to this problem was to develop a standard way of tagging online content. This standard, now described at schema.org, was developed by Google, Bing, and Yahoo! to provide a consistent “markup schema” for general web content. The project encouraged specialized communities (like those producing learning resources) to extend the schema to meet their own needs. The Learning Resource Metadata Initiative (LRMI) emerged from a partnership between the Association of Educational Publishers and the Creative, as a response to this exact problem.LRMI was established to first understand the needs of the learning community and then to develop the specific metadata specification that they would need to accurately describe their content. In May of 2013, the LRMI specification was accepted and published into schema.org. Now, once LRMI metadata is embedded within a resource page, search engines can index this descriptive language and can “recognize” learning resources. Sue Buesing (bee-sing) and Michael Jay did a very thorough overview of this at their session yesterdayAn extension to the LRMI specification was adopted by schema.org as recently as January 18, 2014. These four additional properties (shown in Table 1 below) describe the accessibility features and hazards that a given resource contains. Sources:Association of Educational Publishers. (2013) LRMI is now part of Schema.org. Consulted January 30, 2014. http://www.lrmi.net/lrmi-is-now-part-of-schema-orgCapiel, G. (2014) Schema.org Accepts our Proposal. Consulted January 31, 2014. http://www.a11ymetadata.org/schema-org-accepts-our-proposal/Redd, B. (2013) Introduction, within The Content Developer’s Guide to theLearning Resource Metadata Initiative and Learning Registry. Association of Educational Publishers. March 2013. Consulted January 30, 2014. http://www.lrmi.net/wp-content/uploads/2013/03/lrmi_lr_guide.pdf
- So, we want to talk specifically today about the process we will follow to generate LRMI metadata for more than 2,000 educational resources from the Smithsonian, but perhaps more importantly, a process we are developing to measure quantitatively and qualitatively the effect of this effort on discoverability and usage of our content.Finally we will share with you how we plan to keep this effort sustained and some of the tools available to do this quickly.- Now I would like to introduce my colleague, Melissa Wadman, Manager of Program Evaluation for the Smithsonian Center for Learning and Digital AccessSource:Association of Educational Publishers, inBloom, and Educational Systemics. (2013) The Content Developers Guide to the Learning Resource Metadata Initiative and Learning Registry. March 2013. Consulted July 18, 2013. http://www.lrmi.net/wp-content/uploads/2013/03/lrmi_lr_guide.pdf
- Quantitative Evaluation
- Quantitative – Google AnalyticsThe Link Between Metadata and Sales - Nielsen UK ISBN Agency. 2012.http://www.isbn.nielsenbook.co.uk/uploads/3971_Nielsen_Metadata_white_paper_A4(3).pdf, accessed July 30, 2013.
- Qualitative Evaluation