Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

1. | www.plumanalytics.com1 Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books 24 September 2018 Stephanie Faulkner | Product Manager S.Faulkner@Elsevier.com | +1.732.216.5104
2. | www.plumanalytics.com2 Supporting and developing metrics for the full lifecycle of research We strive for broader use of open metrics that can be used by everyone responsibly. Research Metrics at Elsevier 2 Our Approach
3. | www.plumanalytics.com3 Defining altmetrics • Altmetrics = Alternative ways of measuring the use and impact of scholarship • Altmetrics combine traditional impact measures (citation counts) with non-traditional measures • Altmetrics = ALL METRICS “Altmetrics are measures of scholarly impact mined from activity in online tools and environments” – Jason Priem, Co-Founder, ImpactStory Using Altmetrics to Illustrate the Impact of Open Access on Graduate Student Research (Barnett, Collister, Chan 2014)
4. | www.plumanalytics.com4 USAGE (clicks, downloads, views, library holdings, video plays) CAPTURES (bookmarks, code forks, favorites, readers, watchers) MENTIONS (blog/news posts, comments, reviews, Wikipedia links) SOCIAL MEDIA (tweets, likes, shares, comments) CITATIONS (citation indexes, patent citations, clinical citations) Metrics Categories
5. | www.plumanalytics.com5 USAGE Library holdings, Clicks, Downloads, Views, Video plays • Is anyone reading our work? • Which libraries have our book in their collections? • Usage is the #1 stat researchers want to know after Citation counts • PlumX is the only product that includes Usage
6. | www.plumanalytics.com6 CAPTURES Readers,Exports/Saves, Favorites, Followers • Captures indicate that someone wants to come back to the work • How many time has a book citation been exported • The number of times a citation/abstract has been saved, emailed or printed • Early indicator of future citations
7. | www.plumanalytics.com7 MENTIONS Blog posts, News posts, Book reviews, Wikipedia references • This category measures people truly engaging with your research • Automatically uncover the conversations about your research • Discover feedback, opinions, etc.
8. | www.plumanalytics.com8 SOCIAL MEDIA Shares, likes, & comments, Tweets • Social media measures how well a publisher or researcher is promoting their work • Track the buzz and attention around your research output • This is especially important for early career researchers to measure and understand
9. | www.plumanalytics.com9 CITATIONS Citation Indexes, Patent Citations, Clinical Citations, Policy Citations • Citations still a standard of long-term impact • Including citations enables side-by-side analysis with other metrics categories
10. | www.plumanalytics.com10 Visualizing Impact: Plum Print • Includes the 5 categories of metrics • Circles dynamically change size based on metrics in each category
11. | www.plumanalytics.com11 • Articles • Blog posts • Book chapters • Books • Cases • Clinical Trials • Conference Papers • Data Sets • Figures • Grants • Interviews • Letters • Media • Patents • Posters • Presentations • Reports • Source Code • Theses / Dissertations • Videos • Web Pages ALL Types of Research Artifacts
12. | www.plumanalytics.com12 ACI Amazon Airiti bepress bit.ly CABI CrossRef Dryad dSpace DynaMed Plus EBSCO ePrints Facebook figshare Github Goodreads Google+ Mendeley NICE (UK) OJS Journals PLOS PubMed PubMed Central Reddit RePEc SciElo Scopus SlideShare SourceForge SSRN Stack Exchange Twitter USPTO Vimeo Wikipedia Worldcat YouTube http://plumanalytics.com/learn/about-metrics/ D MP Scholarly Data Exhaust – Data Source Examples
13. | www.plumanalytics.com13 Identifying Research Tracking anything is a challenge. To track a piece of research you need a way to identify it. The Digital Object Identifier (DOI) has been widely adopted to identify articles. But what about non-article output? Or articles that do not have a DOI? Plum Analytics tracks as many identifiers as possible, including ISBNs for books, SlideShare IDs for presentations, YouTube IDs for videos and so much more.
14. | www.plumanalytics.com14 START END >20 URLS & 3 IDENTIFIERS URL http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0054922 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0054922 PMCID PMC3566209 PMID 23405137 Repository URL http://d-scholarship.pitt.edu/17881/ URL http://dx.doi.org/10.1371/journal.pone.0055344 URL http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3602431/ URL http://www.ncbi.nlm.nih.gov/pubmed/23526927 DOI 10.1371/journal.pone.0055344
15. | www.plumanalytics.com15 Books Matter How many libraries hold this book? Books are often the seminal work of researchers in Humanities and Social Sciences. Yet, citations do not do them justice – they never have and never will. This puts researchers from these disciplines at a disadvantage. Luckily, there are other ways to measure the impact of books. How many Wikipedia articles reference this book? What are the reviews on Amazon about this book? How many people added this book to their bookshelves? How many abstract views, downloads, and clicks did this eBook get? How many books & book chapters? 15,121,312
16. | www.plumanalytics.com16 Measuring the impact of books is challenging • In a 2016 study, several bibliometricians analyzed over 70,000 books, collecting and comparing traditional and non-traditional (PlumX) metrics. • Metrics varied across types of books and content • They concluded there isn’t one measure that captures the impact of books across all types -- you need to look a multiple or different measures including: • Social Media Mentions • Downloads • Readers • Viewers • As well as Citations • They also concluded that it is important for publishers to utilize social media to raise awareness to their books (including text books). Halevi, G., Nicolas, B., & Bar-Ilan, J. (2016). The Complexity of Measuring the Impact of Books. Publishing Research Quarterly, 32(3), 187-200. doi:10.1007/s12109-016-9464-5
17. | www.plumanalytics.com17 Most frequently mentioned in social media
18. | www.plumanalytics.com18 Most frequently viewed & downloaded books Most frequently mentioned in social media
19. | www.plumanalytics.com19 Most frequently mentioned in reviews
20. | www.plumanalytics.com20 Book reviews
21. | www.plumanalytics.com21 PlumX links directly to Goodreads • See reviews and ratings • See individuals who have read or want to read the book
22. | www.plumanalytics.com22 Uncover the conversations the author was not even aware of. References
23. | www.plumanalytics.com23 Usage
24. | www.plumanalytics.com24 Finding insights through the PlumX Dashboard Organized by Publisher Export raw metrics Filter and analyze
25. | www.plumanalytics.com25 Quickly find which book has the most metrics in a specific category Each metric column can be sorted (publication year and title, too!)
26. | www.plumanalytics.com26 PlumX artifact page
27. | www.plumanalytics.com27 Analytics – Metrics by publication year
28. | www.plumanalytics.com28 Analytics – Metrics by publication year
29. | www.plumanalytics.com29 Overall Impact Working from the middle out, find the largest section in the third ring to see the book with the most combined impact.
30. | www.plumanalytics.com31 Summary • Tracking over 67 research artifact types such as articles, presentations, patents, books, book chapters, datasets, videos, musical scores, thesis and dissertations and more, from over a 50 different sources. • Comprehensive metric data grouped into five key categories, vetted by our partners and librarians including – Usage, Captures, Mentions, Social Media and Citations. We are the only provider including article level usage statistics. Sources of book specific metrics are extensive and coverage growing. • Measured versions? Artifact (journal, book, book chapter, datasets, etc.) level of measurement includes not only the Publisher (Gold), but, also Green Open Access, Preprint, Aggregated versions and Abstracting & Index versions. • We currently have metrics for over 70+ million artifacts in our PlumX index and these numbers continue to grow—-covering a wide breadth of the world’s research output.
31. | www.plumanalytics.com32 Questions?
32. Thank You s.faulkner@elsevier.com +1.732.216.5104 Stephanie Faulkner Find out more information at: http://www.plumanalytics.com

Editor's Notes

Our Metrics Manifesto

Need to use different metrics and common sense
Decisions should be based on both quantitative and qualitative input
Should always use at least two metrics (more than one way to ‘excellence’)

The methodologies should be open, transparent, valid and replicable
Modern research metrics give a more comprehensive and holistic view of impact. These new metrics are more timely than citation metrics and can keep pace with new formats faster than the entrenched, legacy practices. They are now available to provide fresh insights into how research is interacted with, shared, commented on, promoted, and more. So we define altmetrics as ALL METRICS. And PlumX is the only metrics provider that includes usage.
We developed 5 categories of metrics of how to represent the data exhaust that we are collecting. Usage, Captures, mentions, Social Media and Citations. I will give you a brief overview of each category.
Usage is very important – knowing how many times a particular article or artifact is downloaded, viewed or played (if it’s a video). One of the usage metrics we have for books is how many libraries hold a book.
Captures goes beyond viewing the artifact. Someone is saving it or bookmarking it for later or adding it to Mendeley of CiteULike. This indicates that the person wants to come back to this piece of research. Several studies have shown that captures are really good indicator of future citations. People save and store work that they will come back to when they are ready to write their paper or create their research output.
Mentions show engagement – someone is commenting, blogging or reviewing research or even referencing it in a Wikipedia article. It’s where we say the stories are hidden.
Many people think of altmetrics synonymously with social media. We think social media is a great way of measuring how work is promoted – by the researcher or the publisher.
Citations are a part of the overall impact assessment and are important to consider alongside the other metric types we capture. In addition to traditional citation indexes, PlumX metrics also include clinical, patent and policy citations which are a nice measure of impact in society.
The Plum Print changes dynamically based on each artifact’s metrics. It is like a fingerprint – unique to the specific research output. You can see the color-coding and easily identify which of the 5 categories of metrics has the most or least impact. We’re working towards getting the Plum Print on the Scielo Livros platform.
We track 67 different types of output in our database of over 70 million artifacts and 9.5 billion interactions.
Currently we provide metrics from over 50 different types of sources, and these are just a few examples of what we are measuring. Some metric sources update in near-real time, others daily or weekly.
How do we track the different types of output being generated? We can use an array of unique identifiers from ISBN, to ORCID ID, Slideshare Slideshow ID, of course a DOI. Even a Repository handle URI can be used.
Our technology starts with that single identifier and goes and finds additional identifiers related to the research. It could be an ISBN, publisher URL or DOI. In this example, we started with a single repository URL. From that one URL, we found multiple URLs and other identifiers. In the end we are tracking metrics on 3 identifiers and over 20 URLs. We take all of those identifiers and look for them in our over 50 metrics sources.
We know that measuring the impact of books is challenging as books and book chapters are underserved by traditional citation analysis. PlumX includes inventive new measures for books, including how many libraries hold a book, ebook downloads, reviews from Amazon and GoodReads, Wikipedia references, and more.
In 2016 a study was conducted by several bibliometricians who analyzed metrics on over 70,000 books.

They compared traditional and non-traditional metrics (using PlumX metrics) and grouped books into subject areas. The authors acknowledge other research on books that states in general, books are not cited as often as articles and in in many disciplines books become obsolete much faster than articles.

Abrizah, A., & Thelwall, M. (2014). Can the impact of non-Western academic books be measured? An investigation of Google Books and Google Scholar for Malaysia. Journal of the Association for Information Science and Technology, 65(12), 2498-2508. doi:10.1002/asi.23145
In the study, it was interesting to see which subject areas performed better in specific metric categories that other subject areas. This image illustrates which book subject areas tend to receive the most social media attention. Notice the subject areas with much less social media impact, like Education, Engineering, Ecology and Religion.
The analysis also revealed which book subject areas had the most views and downloads (usage). Notice how the subject areas with lesser social media impact, have better usage. I would imagine with a stronger social media presence, these same areas would have even higher usage.
Finally, a look at subject areas with the most book reviews. Medicine and Business, which had high social media impact, are not even listed here.
Let’s take a look at some book examples on the SciELO platform, starting with this publication with a good number of reviews. From any PlumX artifact page (an artifact is what we call any piece of research output), we not only display the number of reviews,
we link to the reviews and ratings on the platform (Goodreads or Amazon). This allows you to see exactly what people are writing.
In this example, you can see this book has 9 Wikipedia references. PlumX currently covers 4 Wikipedia editions – English, Spanish, Portuguese and Swedish. Each language has unique content – the references are not just translations. We link to the full Wikipedia articles where you can see the reference to the artifact. More languages are coming soon. It’s interesting to see the different types of Wikipedia articles that references this single book.
When I look at usage for this particular publisher, this book has high usage compared to other books on the SciELO platform. You can see that for all of our metric categories, we link back to as many sources as possible. Some of our sources (like EBSCO for example), only provide us with a count. This book has high abstract views on the EBSCOHost platform.
We have loaded all of the SciELO book identifiers into a PlumX Dashboard which you will have access to. A PlumX Dashboard allows you view metrics in a variety of ways and uncover insights about the books. The SciELO books dashboard is organized by publisher. Click on the publisher name to see the list of books and associated metrics. Throughout the dashboard you always have the option to export the metrics and perform your own analyses or use our built in analytic reports. A PlumX Dashboard gives you lots of ways to drill into or find information with several ways to filter – by specific metric type, by artifact type or by publication year range. Let’s take a closer look…
Each column can quickly be sorted to see which books have the highest impact . Clicking on the “plus” symbol below the metric category displays the metrics in more detail. After you sort the column, click on the title of any book to view all of the metrics.
View the source for each metric by clicking on the “down arrow” to expand. We display news and blog mentions, original tweets, Wikipedia references and other details if we have them.
When you click on the Analytics button from the PlumX Dashboard, you will find several prepared reports. The first report - artifacts by publication year -- shows you the last 10 years of metrics in each of our 5 categories so you can identify trends over time. The blue line represents the number of books in the year. The colored bar graphs represent each of our metric categories.
Usage is in a separate graph from citations, captures, social media and mentions because usage tends to be a much larger number compared to the other metric categories.

Mouse over a particular year to see the detailed metrics category counts. Look at the high social media metrics in 2009. Click on any bar graph to go to the filter view and sort by social media to see why this category is so high in this year.
You can see that same book from the previous example published in 2009 by Editora FIOCRUZ had a huge amount of social media compared to all the other books by this publisher in the same year.

Expand the social media column, or click on the title to see the social media metric breakdown.
Another report in the analytics area is the Sunburst. The sunburst provides a nice visualization to find the book with the most overall impact by a particular publisher. Work from the inside ring out to the third ring and click on the largest section to see the book details. You can click on the title in the pop-up window to go to the artifact page.
And don’t forget that you can always export all of the detailed metrics as a spreadsheet by clicking the Export Data button on the dashboard. This will allow you to perform your own analyses, selecting the metrics that you choose to highlight.
To summarize: we track 67 types of research output from over 50 sources, grouping them into five main metric categories. We track not only published versions of research, but preprint, A&I and open access and currently have metrics for over 70 million artifacts.
We look forward to your feedback and questions

Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

You are reading a preview.

Check these out next

Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

More Related Content

Slideshows for you (20)

Similar to Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books (20)

More from SciELO - Scientific Electronic Library Online (20)

Recently uploaded (20)

Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

Editor's Notes

Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

You are reading a preview.

Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

Recommended

More Related Content

Slideshows for you (20)

Similar to Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books (20)

More from SciELO - Scientific Electronic Library Online (20)

Recently uploaded (20)

Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

Editor's Notes