Downloaded 33 times
![Thank You! James King NIH Library, Information Architect [email_address]](https://image.slidesharecdn.com/roiandbeyond-king-110606213330-phpapp01/85/ROI-and-Beyond-King-23-320.jpg)
Uploaded bySLA Government Information Division
ROI and Beyond - King
The document discusses various metrics that can be used to evaluate research performance and impact. It describes metrics for productivity, influence, efficiency, relative impact compared to norms, specialization within or across disciplines, emerging areas, and indirect impact. Examples of calculations and uses for different metrics are provided.
More Related Content
Similar to ROI and Beyond - King
ROI and Beyond - King
- 1. The library’s (potential)role in creating research metrics for their organization June 15, 2011 Special Libraries Association Conference Philadelphia, PA James King, NIH Library Office of Research Services
- 2.
- 3.
- 4.
- 5. What does thismean? Snap out of it! We live in a time when: Change is constant Flexibility matters Attitude is everything We need to be: Positive Interesting Inventive Persistent
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13. Trends in scholarlyresearch Competition for government research funding increasing Available funding decreasing Competition for top research faculty is on the rise Accountability: Research spending Demonstrating return on investment (ROI) Proving the institution’s quality of research to: Prospective students Prospective faculty members/research staff Investors/donors Result: Institutions seek objective data on research performance, for data-based decision making Thanks to Thomson Reuters for this and the next couple slides
- 14. Different stakeholders inresearch evaluation Library Government agencies/funding organizations /accreditors/state government External Entities Individuals Faculty, staff Institutional Depts/Divisions Deans; Department Heads Institutional assessment, academic affairs Institutional Management Management, including committees, provost, vice provosts, rectors, chancellors
- 15. What is agood citation count? Journal and category performance ratios compare this researcher’s citation counts to the norms or “expected” rates in his discipline. Boonen is performing above average on both the journal and category levels (1.14 ,1.48). Mean percentile reflects the percentile performance of the researcher’s work, on average. e.g. in top 33.91 percent Relative Impact/ Bench-marking Journal actual/expected 1.14 Category actual/expected 1.48 Mean percentile 33.91 % papers in top 10% of their field 19% % papers in top 1% of their field 5% Special-ization Disciplinarity index 0.97 Productivity # papers 149 Total influence # citations 1,136 Self-citing papers 57 H-index 22 Efficiency Avg. citation rate 7.62 Percent of papers cited 43
- 16.
- 17. People Opportunities AwardedGrants Publications ANALYTICS
- 18. Overview of SciValSolutions Performance & Evaluation Demonstration of Capabilities Research Enhancement Institution/Country Researcher/Group Institution Researcher “ Showcase ” Q4 2011 SciVerse Scopus Aggregation Primary User Customer Challenges The Solution Addresses Office of Research Dean Dept. Head Researchers Students Researchers Researchers Administrators Researchers What are my interdisciplinary strengths? Who are the high performers in my strategic priorities? How can I act upon my competitive advantage? How are my researchers performing relative to their peers? I want to evaluate researchers with the criteria that are important to me. I want to demonstrate my institution’s research expertise in the areas that are of strategic importance to us. I want to find researchers within my institution with whom I can collaborate. I want to understand ‘who knows what’ in my institution. I want to find funding opportunities I need to determine the right funding strategy I want to i ncrease success rates What are the most highly-cited articles and authors within a research area? What is the h-index of an author? Overview of all of an author’s or institution’s publications
- 19. Who can useVIVO? … and many more! http://www.vivoweb.org
- 20.
- 21. Electronic Scientific PortfolioAssistant Multi-Dimensional Science Management Tool eSPA supports the NIH mission by providing a comprehensive solution for program planning and evaluation staff and NIH leadership to search, build, and collaboratively review and analyze portfolios of NIH research projects.
- 22. Information Wants toBe Free “ On the one hand information wants to be expensive, because it's so valuable. The right information in the right place just changes your life. On the other hand, information wants to be free, because the cost of getting it out is getting lower and lower all the time. So you have these two fighting against each other.” Stewart Brand First Hackers' Conference in 1984
- 23. Thank You! JamesKing NIH Library, Information Architect [email_address]
- 24. Types of citationmetrics and what they measure Metric Calculation Evaluator questions Productivity -# papers -share of papers in field -# papers -# papers in field/ papers in field What is the research output of X? (a country, institution, researcher , etc.) Total influence # citations # citations What is the overall impact of a body of work? H-index Number of papers ( N ) with at least N citations each. What is the impact and productivity of a body of work? Efficiency Average citation rate Total citations/Total papers What is the rate at which a body of work is cited? Percent of papers cited # papers with at least one citation/ Total # papers in population How many papers get cited? Never get cited?
- 25. Types of citationmetrics and what they measure Metric Calculation Evaluator questions Relative Impact/ Benchmarking Journal performance ratio (actual/expected) Sum of citations/sum of journal expected citation rates Expected citation rate calculated for a journal, for each year and document type combination (e.g. JAMA , 2001, review) Has this body of work performed better than average vis-à-vis the journals represented? Category performance ratio (actual/expected) Sum of citations/sum of category expected citation rates Expected citation rate calculated for a journal category, for each year and document type combination (e.g. Physics, 1995, article) Has this body of work performed better than average vis-à-vis the specific disciplines represented?
- 26. Types of citationmetrics and what they measure Metric Calculation Evaluator questions Relative Impact/ Benchmarking Aggregate Performance Indicator Impact of an institution or country relative to an expected citation rate for the entity. The indicator is normalized for field differences in citation rates as well as size differences among entities and time periods. Where ci is the total citations for institution (or country) i, Pift is the number of publications from institution i in field f and year t, cft/pft is the average citation rate of papers in field f and year t. Sums are over all years and all fields. What is the overall impact of my institution’s or country’s research amongst the fields we publish in?
- 27. Types of citationmetrics and what they measure Metric Calculation Evaluator questions Relative Impact/ Benchmarking % papers in top x% of their field e.g. 10% of Dr. Lopez’ papers are in the top 1% of their fields What proportion of a body of work achieves a specific level of performance? Percentile in category and mean percentile Percentile placement of article within a journal category (e.g. oncology, 2002) How has this body of work performed compared to the disciplines represented?
- 28. Types of citationmetrics and what they measure Metric Calculation Evaluator questions Specialization Disciplinarity index where s is the share of papers in category i and n is the number of categories How multidisciplinary is a body of work? Interdisciplinarity index where p is the share of papers in category i and n is the number of categories How dispersed is a body of work across disciplines? Emerging areas of research Research Fronts Clusters of highly cited papers identified via co-citation analysis What are the emerging areas of research in chemistry? Indirect impact Second generation citations Citations received by a paper’s citing papers Did the papers citing a body of work go on to have impact?
Editor's Notes
- #2 Thank you for allowing me a few minutes to talk about the amazing work that happens at the National Institutes of Health Library 06/08/11 NIH Library
- #3 Thinking back to my childhood, I can rattle off a lot of things I’ve seen for the first time that are now commonplace like personal computers, cable TV, e-mail, the Web, cell phones, laptops, and smart phones. What is my daughter Sarah (left) and her BFF Karah (right) going to have easy access to when she grows up? Will she use a computer, a smart phone, or an embedded device as her primary access to the Web? Will she have easy access to an ‘enhanced reality’ like you can currently see with Google Goggles? Will she “lug” a backpack to school with a Nook-like device rather than a large stack of books? Will that school include virtual sessions with the best educators on the planet, combined with immersive experiences to not only learn about science and history, but to live it? If technology is changing the potential life of my 2 year old daughter, why isn’t it changing our mindset in the libraries and organizations in which we work? 06/08/11 NIH Library
- #4 John Cotton Dana, founder of SLA, got it right well over a hundred years ago. We as information organizations need to continue to modify ourselves to the needs of the organizations that we serve. Reference Davenport and Prusak article “Blow Up the Corporate Library” Granted, libraries have been able to adapt to the changes over the past century, incorporating technical reports, microfiche, and even CD-ROM into our collections and processes. Unfortunately, the Web and digital technology is forcing not just adaptations to existing processes, but a rethinking and overhaul of the entire mindset of libraries. The technology is reshaping our profession and redefining the world in which we live. Now, more than ever, we need to be value-driven and relevant. 06/08/11 NIH Library
- #5 Jack Welch said that when the change on the outside is greater than the change on the inside, the end is near. If we are not rapidly adapting to the needs of our client base, in this tight economic market, we will be viewed as expendable. Information recognizes no boundaries! A world bombarded with information must have people who can navigate knowledge with confidence. I see several growth areas or opportunities before us: Embedded librarians or informationists Content integration, embedded services, and distilled content – high risk areas Better understanding of our user community through metrics and persona development Handheld technologies Evaluation tools/scientific measures I will spend my few remaining minutes talking about the last point, scientific metrics 06/08/11 NIH Library
- #6 Information recognizes no boundaries! A world bombarded with information must have people who can navigate knowledge with confidence. This is the Chinese word for ‘CRISIS’ The first symbol represents ‘Danger’ – libraries are facing a lot of them: Budget cuts/downsizing Old school librarians focused on preservation of the status quo Managers that are focused on bottom line IT reaching beyond the ‘pipes’ – taxonomies, KM, etc. The second symbol is ‘Opportunity’ – we have several of them ahead of us, depending upon our perspective and response to the changing environment. Today, I’m going to touch on three growth areas or opportunities before us: Embedded librarians or informationists Handheld technologies Evaluation tools/scientific measures 06/08/11 NIH Library
- #7 The first opportunity I’ll discuss is the concept of embedded librarianship. The NIH Library established the informationist (or librarian in context) service in 2001 and now supports over 40 groups in 16 institutes/centers across NIH. It was started to improve outreach and service to clinical research groups in response to the shift to digital resources and services which reduced face-to-face interaction in the library, an important component of service evaluation. Informationists are specially trained, often with dual degrees, and spend up to 75% of their time with their target group. Informationists bring the wide range of NIH Library services into focus for their target group, often providing customized versions of library services. One of the custom services that we provide through the Informationists program is the creation of custom database sets for specific work projects. These could either take the form of a Endnote file usable by the researcher or a ASP/SQL based web page displaying the results of the literature search. This service, though rudimentary, provided an open door for the NIH NIAID institute to seek out our library to help them get control over their pandemic collection. The 1918 Influenza Pandemic was a devastating event in the world’s history, happening so fast and impacting so many people that detailed record-keeping was difficult to maintain so maps like this are broad interpretations. The NIH NIAID studies this and related pandemics to help them better understand the epidemiology (study of disease impact), etiology (study of disease origin), diagnosis and treatment of the 1918 pandemic influenza and influenza-related diseases. They have amassed a collection of thousands of documents dating back to the 12 th century and have had trouble managing and even finding portions of the collection when needed. After speaking with them, it was apparent that this project needed more than a good bibliographic database and basic cataloging skills, it also needed strong collaboration tools to support the creation of a virtual community. We determined that a Virtual Research Environment was the best solution for this project and that Drupal was the most advanced platform to build it upon. I’m now currently working on three major projects in parallel, including this pandemic project, a national site focused on Down syndrome research, and one to help research directors to identify gaps in oral health intervention research. As we complete these, the lessons learned will be easily ported to other opportunities that arise as a new service from the library. Since any new code generated is also contributed back to the open source community, other organizations will also be able to offer the same service, furthering science in many ways. 06/08/11 NIH Library
- #8 In addition to creating bibliographic databases, the need for full-text repositories is also an important component. The NIH Library, focused on the internal researchers at NIH, has partnered with the National Library of Medicine (focused on public health) to jointly develop a public-facing digital repository of historic material in our collections. We are using the open source Fedora Commons software and have been collecting material that had been previously digitized and started digitizing material specifically for this effort. The NIH Library’s contribution was to digitize a number of the NIH Annual Reports held in our collection. Using the Internet Archive’s FEDLINK contract called FedScan, we were able to scan nearly a thousand volumes containing over 400,000 pages for a mere $40k. To me, this shows what a non-issue digitization is today so if you have unique collections, you should explore digitizing them! 06/08/11 NIH Library
- #9 The second opportunity is mobile technology. This is my new smart phone, an Android My Touch. I previously had a Blackberry Bold and I must admit that this new device has dramatically changed my habits in a short period of time. Rather than simply viewing the handheld as an e-mail reader (which the Blackberry is great at), the Android is becoming my mobile computer and gateway to the cloud. Because of this device, I’ve now switched my master to do list for all parts of my life (work, home, church, SLA, etc.) to Toodledo – a web and mobile application to manage tasks. I’ve also switched personal notes, new photos and documents I’m working on to a secure file sharing service called DropBox. These cloud services are enhancing my mobile experience and encouraging me to use it more. Just downloaded Google Docs app which can take a picture of a document and make it editable! The NIH Library has made a strategic goal to build a mobile competency in the staff over the next couple years. This is being accomplished by providing mobile devices to the entire staff (some with phone service and others with just data plans), holding brown bag sessions to help each other learn new features, and hosting a Mobile Technology Expo earlier this month. We’re also using this initiative to make our own library Web site mobile-friendly and to build a collection of links to mobile versions of the journals and databases we provide. We feel that we need to first have a basic understanding of the technology and try to do our own job with a mobile device so that we can see the barriers that we’re imposing on using mobile with the library’s services. Why the big focus on mobile? 06/08/11 NIH Library
- #10 As you can see from this chart, In 2010, over 80% of all shipping devices were mobile so the stationary desktop computer is quickly becoming a minority to its mobile cousins the netbook, notebook, table, and smartphone. 06/08/11 NIH Library
- #11 Though this chart only shows from 2007-2009, you can see a dramatic growth in mobile web usage in that time. With the explosive growth of smartphone and tablets since then, I expect that the percentage will go higher quickly. In fact, many are predicting that mobile will be the primary access by 2020. When I was at the Drupal conference earlier this year, the key developer told an audience of 3,000 that if they were building the massive open source content management system today rather than 10 years ago, it would have been a mobile-focused effort – a testament to where connectivity is heading in the near future. 06/08/11 NIH Library
- #12 When working on a speech for an Elsevier Metrics program last fall, I ran across the Becker Medical Library provides a great model for libraries to use to assess the impact of research. Though focused on biomedical research, it can easily be applied to any research setting. They are working on an update to this model this summer and are in the audience. They’ve developed a tree-based visualization of the model which I encourage you to check out or talk to Kristi Holmes and/or Cathy Sarli. I encourage you to check out the site in more depth but to summarize, they mention several ways to measure research impact: By the numbers – counting how many publications were made and tracking the various outputs Referenced - determining if the research was referenced or reused, including counting how many references were made to those publications Adoption - was this used in a patent, a medical protocol, or in some other way moved from research to practice Difference? – did the research result in some difference in economics, efficiency, effectiveness, or quality of life where it was applied Official – did the research impact laws, policies, or regulations in their sphere of influence 06/08/11 NIH Library
- #14 Thanks to Thomson Reuters This slide shows what the people we serve are concerned about. We need to understand their needs and specifically jump in to meet them. 06/08/11 NIH Library
- #15 Another way to look at the needs is by stakeholder.. Funders: Need to fund new types of research cross cutting science, translational medicine, collaborative endeavors; old qualitative approaches are no longer viable Universities’ need metrics to support strategic direction decisions and understand the impact of their research programs that get beyond the simplicity and skewed nature of national rankings Dept. and Individuals: Being found, getting attribution, and finding colleagues in a global research environment is imperative for collaborating with the local and global research community Library is in the objective position to extend its role in new ways as the institution implements ongoing assessment and impact exercises 06/08/11 NIH Library
- #16 Whether it be in PubMed, in an internal database or using commercial database tools such as Scopus or Web of Science, a focused list of publications of a research organization or department allows you to build an impressive suite of measures utilizing tools that many organizations already have. These examples from Thomson Reuters show what can be measured simply by looking at publishing patterns. An explanation of each measure is listed in the back of this presentation. 06/08/11 NIH Library
- #17 The Naval Research Laboratory has been working over the past several years on one approach to creating useful metrics. By identifying and capturing the metadata of all journal articles, conference proceedings, book chapters, US patents, and technical reports written by NRL researchers and engineers, they can automatically create a number of reports easily. For example, a bragging list of the top 25 highest cited NRL papers of all time, most popular journals published in, and with some analysis by a third party, the Patents that have cited NRL work. What made this project appealing was that much of the initial work was captured using API access to the Thomson-Reuters Web of Science database and the Elsevier Science Scopus database, keyed to the author affiliation fields. A number of other Federal agencies such as NASA have also pursued the creation of internal databases of all agency-produced materials and I firmly believe that there is a clear role for libraries to play a role in this. Of course, I should also mention that NLM is also doing something similar by requiring all publications resulting from NIH grant funding should be deposited into PubMed. I recall Dr. Zerhouni, then Director of NIH, pushed for this specifically so that he could have a tool to measure research productivity. Whether it be in PubMed, in an internal database or using commercial database tools such as Scopus or Web of Science, a focused list of publications of a research organization or department allows you to build a slate of measures. 06/08/11 NIH Library
- #18 Metrics and value are now big business. Thomson Reuters and Elseiver have both made large acquisitions and developed a life-cycle approach to grant management, creating products to meet each area of need.. Before starting, I want to state one simple but often unremarked fact that I think all in this room will agree with: research is important. Indeed, along with art, and music, and writing, it is one of the high capabilities that define in the best sense what it means to be human, to be curious, to strive to understand. People are the beginning and end of the entire process of scientific advancement. Researchers and their talents and creative ideas are the primary foundation of all scientific inquiry, the sine qua non of pushing the boundaries of knowledge outward. Over time, as funding and compliance requirements for research have gotten more difficult to navigate, other people, from research administrators to VP’s for research, compliance committees and research development professionals, deans, department chairs, mentors and department administrators have also become essential for this task, even while remaining secondary to the researchers themselves. Finding the right people for the right opportunities is a complex and highly demanding task. For every successful grant application, there is the application that might have been, that could have been written and have succeeded in securing funding, if only there was time to put the right team together. Only by doing this matching very well and very consistently can one move successfully to the next critical step. Grant Awards are the lifeblood and enabler for both the investigator and the institution. Money and protected time are fundamental for expanding the horizons of science and driving socioeconomic advances, allowing for the production of publications, patents, and conference proceedings, the entire research information ecosystem that drives and disseminates new ideas. The grant and publication records of researchers, in turn, drive career advancement for individuals and institutional reputation and prestige. All of these phases of the grants lifecycle can be illuminated and improved by the use of relevant analytics. And that brings us back again to people, those who drive everything and those whose mission it is to help them achieve their goals. It is to assist these people that Elsevier has committed itself to creating the SciVal suite of products. Thank you. 06/08/11 NIH Library
- #19 Elsevier’s goal is to provide services to every phase of the grant life cycle. Thomson Reuters has a similar approach and both have made significant acquiaisions to expand their reach into this arena. Whether you agree with the commercialization or not, it clearly shows that there is a need and high-impact stakes in this space. Libraries can and should play a role. 06/08/11 NIH Library
- #20 The effort was started by Cornell and expanded to additional partners and made open source through a NIH grant. It is being run at a dozen organizations and growing each day. This effort goes beyond the social web of Web 2.0 and into the Web originally envisioned by Tim Berners Lee, a semantic web of relationships. With the addition of bibliometrics, I believe that VIVO can be the answer to our need for a platform to provide scientific measures to our clients., Talking points: VIVO is useful to many different users and audiences. Users will come from within and outside of the participating institutions. VIVO can result in increased grant and donor funding. For instance, the Cornell Alumni Affairs and Development Office was contacted by a major company interested in making research funds available to scientists working on issues related to urology; a search in VIVO-Cornell quickly returned several faculty in diverse disciplines working in this area, translating to real dollars. The technology is versatile and flexible enough to easily accommodate changes based on new and innovative demands that users make as they utilize VIVO. 06/08/11 NIH Library
- #21 Some organizations like the National Institutes of Health has also been fortunate enough to have the resources to work with A&I providers to create robust, custom views of their organizations’ data. The first I want to look at is Research Portfolio Online Reporting Tools (or REPORT) which is a tool designed to support the extramural research community, providing per-year data on grants, and disease portfolios powered by the Collexis search tool. It allows users to search a repository of both intramural and extramural NIH-funded research projects from the past 25 years and access publications (since 1985) and patents resulting from NIH funding. Search results can include the research project number, project title, contact principal investigator, performing organization, fiscal year of funding, NIH administering and funding Institutes and Centers (IC), and the fiscal year total costs provided by each funding IC. 06/08/11 NIH Library
- #22 The second NIH-hosted service is called the Electronic Scientific Portfolio Assistant or eSPA and is a tool that helps the intramural community to evaluate the outcomes (including outputs and impact) of NIH funding. Currently focused on helping to review and analyze portfolios of research projects for program planning and evaluation. By combining research funding with publications, custom portfolios of research can be created to help program managers and administrators to track and evaluate their research. You can see many of the internal and external data sources used in this effort. Since the NIH Library is also focused on supporting intramural research efforts, they have started to work with eSPA to expand their support of research publications-based evaluations similar to what I’ve been describing today. I’ve found both REPORTER and ESPA to lacking in bibliometrics and accurately capturing all of the publications of our researchers. 06/08/11 NIH Library
- #23 As libraries continue to move into the digital era, I think it is important to look at the context of what has become the battle cry for much of this past decade’s advances. Everyone has heard that “information wants to be free” and has spurred the open access movement and other related movements to open up access to content. We’ve also heard that the distribution costs in the digital world are zero which puts tremendous pressure on the traditional view of library as a place, especially library as a warehouse. But if we remember the first part of the quote that “information wants to be expensive, because it is so valuable. The RIGHT information in the RIGHT place just changes your life.” Correct me if I’m wrong, but isn’t a librarian motto to get the right information to the right person at the right time? Well placed information services and resources that specifically meet the needs of our community will continue to make the difference between success and failure, life and death. The question is, will WE as information professionals be there – whether it be through building a customer-focused Web environment that integrates library-provided content into the workflow of a customer group, or at the other end of a handheld while a researcher is checking the credentials of the next speaker at a conference, or creating an authoritative report of the impact of a group’s research project that will sway the funders to fund (or not fund) the project next year? To be relevant in this era, we need to be willing to take risks. 06/08/11 NIH Library
- #24 Thanks! 06/08/11 NIH Library