Successfully reported this slideshow.
Your SlideShare is downloading. ×

Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
Nih dempwolf 20160408-v4
Nih dempwolf 20160408-v4
Loading in …3
×

Check these out next

1 of 34 Ad

Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations

Download to read offline

More than an umbrella term, open science is moving towards broadening and integrating the open access movement to scholarly literature on other fronts, such as open scientific data, open scientific tools, open notebook science, open education, and citizen science.

This “movement of movements” transforms the scenario and the dynamics of science collaboration, communication and dissemination, expanding its ability to respond to contemporary new and complex issues, while posing new challenges. On the one hand, new possibilities arise for the generation of social, economic and environmental benefits, as well as innovation, associated to increased reach, speed and quality of production and circulation of scientific knowledge, its results and possible uses. On the other hand, new institutional and technological requirements are imposed on the adoption of open research policies, strategies, and practices (regulations, capacities, infrastructures, and tools), and the costs derived therefrom. A new economics of open science is being developed, together with new business models, with repercussions on the present and future of scientific journals and their relationship with other scientific publication and publicization systems emerging from this framework, as well as with the monitoring, evaluation and research financing apparatuses.

At the same time, it is about facing the challenge of bridging the gap between science (and its various forms of data availability) and policy. Today there is an abyss in this interface that should be narrowing so that, increasingly, political decisions, particularly those that affect social and environmental issues more directly are based on quality and plural science. To strengthen this relationship, efforts are needed to reconcile languages and times that allow virtuous dialogue between these two fields.

In the end, it is also important to recognize the different implications of this changing scenario regarding more and less developed countries, placing new opportunities and barriers for their science, technology and innovation systems and their respective repositioning in the global scenario.

Syllabus
Open science, science communication and the challenges of sustainable development; open publications and innovation; the new economy (politics) of open science and its infrastructures of scholarly communication: costs and benefits (academic, social, and economic); political and institutional requirements; business models emerging from open scientific publications; opportunities and challenges for developing countries.

More than an umbrella term, open science is moving towards broadening and integrating the open access movement to scholarly literature on other fronts, such as open scientific data, open scientific tools, open notebook science, open education, and citizen science.

This “movement of movements” transforms the scenario and the dynamics of science collaboration, communication and dissemination, expanding its ability to respond to contemporary new and complex issues, while posing new challenges. On the one hand, new possibilities arise for the generation of social, economic and environmental benefits, as well as innovation, associated to increased reach, speed and quality of production and circulation of scientific knowledge, its results and possible uses. On the other hand, new institutional and technological requirements are imposed on the adoption of open research policies, strategies, and practices (regulations, capacities, infrastructures, and tools), and the costs derived therefrom. A new economics of open science is being developed, together with new business models, with repercussions on the present and future of scientific journals and their relationship with other scientific publication and publicization systems emerging from this framework, as well as with the monitoring, evaluation and research financing apparatuses.

At the same time, it is about facing the challenge of bridging the gap between science (and its various forms of data availability) and policy. Today there is an abyss in this interface that should be narrowing so that, increasingly, political decisions, particularly those that affect social and environmental issues more directly are based on quality and plural science. To strengthen this relationship, efforts are needed to reconcile languages and times that allow virtuous dialogue between these two fields.

In the end, it is also important to recognize the different implications of this changing scenario regarding more and less developed countries, placing new opportunities and barriers for their science, technology and innovation systems and their respective repositioning in the global scenario.

Syllabus
Open science, science communication and the challenges of sustainable development; open publications and innovation; the new economy (politics) of open science and its infrastructures of scholarly communication: costs and benefits (academic, social, and economic); political and institutional requirements; business models emerging from open scientific publications; opportunities and challenges for developing countries.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations (20)

Advertisement

More from SciELO - Scientific Electronic Library Online (20)

Recently uploaded (20)

Advertisement

Stephanie Faulkner - Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations

  1. 1. | www.plumanalytics.com1 Moving Beyond Citation Indexes – Measuring Societal Impact through Clinical, Policy and Patent Citations 26 September 2018 Stephanie Faulkner | Product Manager S.Faulkner@Elsevier.com | +1.732.216.5104
  2. 2. | www.plumanalytics.com2 Who is Plum Analytics? • Founded to Tell the Story of Research • Developed comprehensive metrics across 5 categories (Citations, Usage, Captures, Mentions, Social Media) • Timeline: − 2012: Company founded − 2014: Merged with EBSCO − 2017: Joined Elsevier • 250 worldwide customers, including: − University of Southern California − University of Pittsburgh − Universidad del Rosario − University of Western Australia − University of Helsinki, Finland − Chalmers University, Sweden − Nanyang Technical Institute, Singapore − UNICEF, Italy − CSIRO, Australia − University of Sydney, Australia
  3. 3. | www.plumanalytics.com3 Metrics Drive Decisions
  4. 4. | www.plumanalytics.com4
  5. 5. | www.plumanalytics.com5
  6. 6. | www.plumanalytics.com6
  7. 7. | www.plumanalytics.com7 Comprehensive Metrics Give You an Edge
  8. 8. | www.plumanalytics.com8 What are your institution’s / country’s strengths vs. your peers What research areas are getting funding? How can we demonstrate ROI (Return On Investment) of research money? Who are rising stars amongst early career researchers? Use metrics to tell the story of research Research Metrics Can Be Used to Identify…
  9. 9. | www.plumanalytics.com9 • Funders and governments increasingly require real world or societal proof – consider the 2014 Research Excellence Framework exercise conducted in the UK. • 6,975 impact case studies were submitted categorized by a variety of impact types including: − Political − Health − Technological − Economic − Legal − Cultural − Societal − Environmental • Almost 25% of all the cases submitted were labeled Societal Impact http://impact.ref.ac.uk/CaseStudies/ https://www.ugc.edu.hk/doc/eng/ugc/rae/2020/framework.pdf Societal Impact – Why does it matter? 9
  10. 10. | www.plumanalytics.com10 • Measuring societal impact is not always easy as it can take a long time for this type of impact to be reflected and measurable. • Sources vary and the accessibility to them also varies. • There is no standard measure across research areas. Societal Impact - Challenges 10
  11. 11. | www.plumanalytics.com11 Measuring Societal Impact – PlumX Metrics
  12. 12. | www.plumanalytics.com12 • We represent societal/economic impact in our Citations area in three ways: − Clinical Citations − Policy Citations − Patent Citations • Displayed separate from traditional citation indexes, but included in our overall citation count Our approach 12
  13. 13. | www.plumanalytics.com13 Policy and Clinical Citations • Demonstrates: − Public engagement with research − Impact on government policy and cultural life − Societal impact in general • Can drive policy makers and legislators to a particular course of action. • They can also be actual statements of policy by public policy making institutions.
  14. 14. | www.plumanalytics.com14 Policy Citations Policy from different types of organizations: • Non-profits • Governmental agencies • NGOs • Think tanks Policy documents can be: • Guidelines • White Papers • Monographs • Pamphlets • Articles • Books • and more
  15. 15. | www.plumanalytics.com15 Societal Impact – Policy Citations
  16. 16. | www.plumanalytics.com16 Policy Citations in PlumX
  17. 17. | www.plumanalytics.com17 Policy Citations in PlumX
  18. 18. | www.plumanalytics.com18 “…clinical medicine draws heavily on basic science, but not vice versa. The result is that basic medicine is cited three to five times more than clinical medicine, and this is reflected in journal impact factors.” Clinical Research & Citations
  19. 19. | www.plumanalytics.com19 Clinical Citations Come From… Clinical Alerting Services Clinical Guidelines Systematic Reviews Clinical Trials
  20. 20. | www.plumanalytics.com20 Clinical Citation Sources: Practice Guidelines • Based on PubMed / Medline − “Guideline” is a classification by PubMed − Any time an article is cited by a “Guideline” this will be counted as a clinical citation − PlumX will calculate this as a type of “clinical citation count” for a given article Note we will also have a separate “citation index” count for citations across all of PubMed. • Based on NICE, UK (Clinical Guidelines) • And many more Guidelines to follow…
  21. 21. | www.plumanalytics.com21 Clinical Citations: Practice Guidelines
  22. 22. | www.plumanalytics.com22 Societal Impact – Clinical Citations
  23. 23. | www.plumanalytics.com23 Clinical Citation Sources: DynaMed Plus Topics • Clinical reference tool • Designed for point-of-care • Content: Extensive network of clinical experts − Review 500 journals and 120 guideline organizations − select the best and most appropriate evidence − confirm the clinical applicability of content − peer-review topics
  24. 24. | www.plumanalytics.com24 Societal Impact – Clinical Citations
  25. 25. | www.plumanalytics.com25 DynaMed Plus Topics
  26. 26. | www.plumanalytics.com26 Patent Citations
  27. 27. | www.plumanalytics.com27 Patent Citations • A patent citation is any document cited by a patent applicant, a third-party, or a patent office examiner because it's relevant to a patent application. • Gain an understanding of how much research is being used in the creation of products by seeing the total number of outputs that received patents. • Provide information about the economic impact of research.
  28. 28. | www.plumanalytics.com28 Patent Citations • Citation of scholarly output in patents indicates a connection between academia and industry. • Patents citation contains valuable data and if analyzed well, may sometimes reveal concealed mysteries of the information flow between countries, laboratories, companies, and universities. Sharma, P., & Tripathi, R.C. (2017). Patent citation: A technique for measuring the knowledge flow of information and innovation. World Patent Information, 51, 31-42. doi: 10.1016/j.wpi.2017.11.002
  29. 29. | www.plumanalytics.com29 Societal Impact – HERL at University of Pittsburgh
  30. 30. | www.plumanalytics.com30 Societal Impact – Patent Citations
  31. 31. | www.plumanalytics.com31 Societal Impact – Patent Citations
  32. 32. | www.plumanalytics.com32 Summary • It’s exciting to see the real-life impact that research can take – whether impacting law, medical care or the invention of new technologies. • Clinical research is an area underserved by traditional citation metrics, yet is important to the day- to-day welfare of people around the world. Being able to measure the impact of clinical research can encourage more clinicians to innovate and find cures for diseases. • PlumX will continue to add new clinical, policy and patent sources and innovate around additional measures to help further demonstrate societal impact.
  33. 33. | www.plumanalytics.com33 Questions?
  34. 34. Thank You s.faulkner@elsevier.com +1.732.216.5104 Stephanie Faulkner Find out more information at: http://www.plumanalytics.com

Editor's Notes

  • If you don’t know where to eat you will choose metrics such as restaurant reviews and ratings to choose a restaurant.
  • Businesses like Starbucks use metrics like foot traffic, auto traffic, costs, etc. to determine the best place to locate a new store.
  • Uber has used metrics to optimize revenue by implementing surge pricing when they detect a big demand.
  • Thus, comprehensive metrics give you an edge.
  • Research metrics can be used to identify
    An institution, journal, countries strengths
    What research areas are getting funding
    Return on investment (particularly for funders)
    What new researchers are getting recognition

    Metrics can help you tell the story about research
  • More and more regional assessment exercises – like the REF in the UK, the ERA in Australia and the RAE 2020 in Hong Kong, where they are specifically looking to assess the reach and significance of impact on the economy, society and/or culture -- need to have these types of societal impact metrics for these types of assessments.

    Funders are asking for metrics that demonstrate real world proof of the how a piece of research has made an impact.

    Researchers and Institutions want to know what impact their work has made.

    The purpose of the REF in the UK, for example, is to provide assessment outcomes for each submission from higher education institutions. These outcomes will be used for or by:

    The 4 higher education funding bodies to help determine the allocation of research grants to which institutions they fund
    Accountability for public investment in research and it’s benefits
    Benchmarking information and establishing reputational yardsticks, for use within the Higher Education sector and for public information.


    In addition, there are 3 other benefits identified:

    to provide a rich evidence base to inform strategic decisions about national research priorities
    to create a strong performance incentive for higher education institutions and individual researchers
    to inform decisions on resource allocation by individual higher education institutions and other bodies

    Just looking at the last REF in the UK from 2014, they received almost 7,000 impact case studies which the classified into a variety of categories. Of the almost 7,000, 25% of the cases were labeled as Societal impact.

    As the REF prepares for the 2021 report, they state that they are specifically assessing the ‘reach and significance’ of impacts on the economy, society, culture, public policy or services, health, the environment or quality of life – Societal Impact.
  • Measuring this type of impact can be a challenge – patent applications can take 18 months or longer (in the US) to be approved and have the reference data publicly available. Many governments do not have policy documents (and the associated references) in a format for automated harvesting, so manual curation is needed.

    And how do you equally measure the impact of an astrophysicist compared to an economist? It’s like comparing apples to oranges.

    Source: “'What is Real World Impact?’ Looking at Impact Across a Spectrum of Influence, Attention and Affecting Change ” presented at the Society for Scholarly Publishing 2018 annual conference. By David Crotty of the Oxford University Press.

  • PlumX currently represents societal and economic impact in our citations metric category, but separate from traditional citation indexes. We currently categories our societal and economic impact metrics as:

    clinical citations (which we have pioneered and I’ll speak more about in detail),
    policy citations and
    patent citations
  • Let’s first talk about Policy Citations. We know that finding references to research in policy documents is important because there are new demands on researchers and those who support them to demonstrate public engagement with their research, its impact on government policy and cultural life, and societal impact in general. These policy documents can be designed to drive policy makers and government officials to a particular course of action. They can also be actual statements of policy by public policy making institutions like a government.
  • However, the term “policy documents” can be a vague way of describing documents that result from the research of non-profits, governmental organizations and think-tanks. They can take the form of white papers, monographs, pamphlets, articles, books, book chapters or reports. They can contain words in the title like “Policy”, “Guidelines”, “Recommendations” or “Guidance” or they may just be research objects that are used in the creation of policy.
  • For example, the World Health Organization’s primary role is to direct and coordinate international health within the United Nations system. They work to combat diseases influenza and HIV, cancer and heart disease.

    They help mothers and children survive and thrive so they can look forward to a healthy old age. They ensure the safety of the air people breathe, the food they eat, the water they drink – and the medicines and vaccines they need.

    The World Health Report in 2013 focused on "Research for universal health coverage" argues that universal health coverage – with full access to high-quality services for prevention, treatment and financial risk protection – cannot be achieved without the evidence provided by scientific research.

    This article on the SciELO platform was cited in that WHO 2013 report on universal health coverage.
  • In another example, not from the SciELO platform, you can see that this article has received 5 different policy citations. Let’s take a look at the policy details…
  • If you look at the title and source of the policy documents, you can see that this one article had an impact on food and nutrition policy in different ways, including economic impact with both the World Health Organization and the World Bank referred to this article in their policy works.
  • With regards to clinical research, we know that it does not get cited as much as basic research---recognized as far back as 1997. Why is this a problem? Researchers have difficulty getting funding so early career researchers opt out of translational medicine. Hospitals and research institutions cannot showcase their talents.

    It hurts us all when our researchers don’t want to turn basic science into ways to treat illness and disease. Or, when our doctors pioneer new treatments but have no motivation to publish them. We need to measure clinical impact. And PlumX wants to help researchers and institutions understand what is “impactful” in the clinical realm.

    So, we created a special category for references in clinical policy because of the history of clinical and translational science receiving less citations — and with that - less prestige and funding. 

  • Clinical citations come from a variety of sources including PubMed Guidelines, Dynamed Plus and NICE (National Institute for Health Care) Clinical Guidelines in the UK. We will be adding Elsevier’s clinical key soon as well as looking for additional clinical sources to add.
  • Our current sources include PubMed guidelines, NICE guidelines from the UK and Dynamed Plus Topics (an EBSCO product). We’re adding ClinicalKey which is an Elsevier product as well.
  • PubMed practice guidelines are available as an article type.
  • Here’s another example of an article that has 4 clinical citations – 3 from PubMed and one from NICE in the UK. We link back to the guideline so you can see where the research is referenced.
  • If you are not familiar with DynaMed Plus, here a brief description. The content is developed using a strict, evidence-based editorial process, written by a team of specialized physicians and researchers with content being updated several times a day to include the latest evidence-based research.
  • When the Zika virus outbreak occurred in 2015 and hit North America, doctors were unprepared to diagnose and treat it. In 2015 there were 62 reported cases, but in 2016, there were 5,168 cases according to the Center for Disease Control which is part of the US National Institutes of Health. That’s a huge increase in a very short period of time. So doctors in hospitals and private practices needed to turn to sources like PubMed Guidelines or DynaMed Plus topics to be able to quickly diagnose and treat this rapidly spreading virus. This is an example where this article is cited in a DynaMed Plus topic.
  • Here’s what we link to from the PlumX Dashboard.
  • Another measure of societal impact is the Patent citation. Patent citations can demonstrate economic impact, as well as helping provide an understanding of how much research is being used in the creation of products.

  • Patent citations can illustrate a connection between the academic and corporate worlds and according to one article on the topic, can also show collaboration between not only researchers, but countries.
  • The Human Engineering Research Laboratories at the University of Pittsburgh’s mission is to “continuously improve the mobility and function of people with disabilities through advanced engineering in clinical research and medical rehabilitation” They want to create a world where all people with disabilities have unencumbered mobility and function so they can fully participate and contribute to society. While they publish a good amount of research in books, journals, conference proceedings, etc. they also research and develop all sorts of things that from wheelchairs that run on compressed air (versus a battery) to splints to treat hand injuries.

    In 2012 they received a patent for a joystick that allows individuals with substantial upper extremity motor impairments to have
    A wheelchair joystick with customized fit and function using mechanical and software customizations.
  • That joystick was patented. In PlumX, you can see that the joystick patent is being referenced by 10 other patents. All of this research contributing to the quality of life for those who are wheelchair-bound.
  • Clicking through to the US Patent and Trade Office database, you can see the other patents that reference it.
  • We look forward to your feedback and questions

×