This document discusses challenges in measuring adoption and impact of electronic lab notebooks (ELNs) for research data management. It provides background on ELN implementation at Cornell and Wisconsin universities and describes prior efforts to survey ELN users about data management practices. Specifically, it examines difficulties in defining and assessing concepts like data management and adoption, and getting user perspectives on the value of ELNs for record keeping, metadata capture, and archiving data over time. Input is sought on how to improve questions that evaluate the degree to which ELNs help with various data management needs and goals.

1. 0
200
400
600
800
1000
Jul-12
Sep-12
Nov-12
Jan-13
Mar-13
May-13
Jul-13
Sep-13
Nov-13
Jan-14
Mar-14
May-14
Jul-14
Sep-14
Nov-14
Jan-15
Mar-15
May-15
Jul-15
Sep-15
Nov-15
Jan-16
Mar-16
Accounts
Created
Monthly
Unique
Logins
ADOPTION
Frequency of ELN usage varies by discipline, lab and even individual, so traditional metrics for measuring
adoption and usage of the platform is challenging. Also, how do we accurately gauge how we compare with
our peers?
at Cornell (Total Faculty + Staff = 9,757; Total Student Population = 21,904*)
at Wisconsin (Total Faculty + Staff = 21, 608; Total Student Population = 43, 389**)
Measuring adoption of Electronic Lab Notebooks and their impact on research data management practices‘
Wendy Kozlowski, Data Curation Specialist, Cornell University (wak57@cornell.edu)
Jan Cheetham, Research Cyberinfrastructure Liaison, University of Wisconsin-Madison (jan.cheetham@wisc.edu)
INTRODUCTION
Electronic Lab Notebooks (ELNs) are a relatively new addition to laboratory workflows and the data
management toolkit for research and teaching in higher education. Driven by researcher needs for a digital
platform for recording data and documenting the research process, a number of universities have launched
campus-wide ELN services in recent years. At the institutional level, ELNs can provide a system for protecting
the security of data and intellectual property and may play a role in implementation of campus data
management/stewardship policies. However, little is known how successful campus investments in an ELN
platform have been, including the extent of adoption by researchers and how well enterprise ELNs serve the
research data management needs of users.
We would like to engage poster viewers in discussions about ways to measure both the adoption of ELNs and
their impact on the research data management practices of researchers. Prior efforts at gauging the usefulness
of ELNs for data management indicated that researchers may have different perspectives on what data
management involves and what acceptable practices are.
RESEARCH DATA MANAGEMENT IN AN ELN
The ELN is designed to collect, capture, and/or ingest many different types of digital data (and metadata) that
researchers produce.
Data management concept: One place for keeping research data
Traditional lab notebooks have been used to keep a complete record of discovery, including taping in
printouts of data. How well do ELNs serve as a complete record?
We have developed a survey of ELN adoption and impact on data management that will be used in a future study across institutions to address these questions. Components of the survey are available
online, and we’re looking for your input! Please visit: http://bit.ly/24jMsCZ.
CHALLENGES OF MEASURING DATA MANAGEMENT
Does the term “data management” resonate with researchers in this context?
• Do they see ELN entries and data files as separate things?
• What does “management” mean to them in this context?
Was the pilot period too short to gauge improvements in data management?
Are more detailed questions needed?
• What proportion of their data files can be kept in the ELN? What are the barriers for others?
• How useful is adding links to files that can’t be included in the ELN?
• What gaps do they perceive in the ELN records they are keeping?
PRIOR EFFORTS TO MEASURE DATA MANGEMENT
Both Cornell and Wisconsin did user surveys during the pilot phase of software roll-out. Below are some of the
questions asked that tried to determine if users saw ELNs as something that helped improved data
management practices.
Data management concept: Capturing metadata
ELNs have features for tagging, fields for entering image annotations, and entries for recording
experimental details. How well are these features being used to capture a complete record of the research?
Data management concept: Archiving and export
ELNs have several export format options, including PDF, packages with HTML/XML. How are researchers
using these to backup, archive, or deposit their ELN records?
User’s perception of value:
 Helps me manage my research data.
 Allows me to keep all my research records and
data in one place.





User’s perception of value:
 Lets me annotate/tag data and results in useful
ways.
 Lets me capture “metadata” such as information
about materials instrument settings, methods, etc.





User’s perception of value:
 Lets me keep track of research data over
several years.
 Provides peace of mind because I know I have
multiple copies of my work.





CHALLENGES OF MEASURING ADOPTION
Enterprise-level tools are managed - and so metrics are also often considered - at an institutional level.
• Should we also be considering different metrics at a discipline, lab or even user level? What might that
look like?
• How often do users have to log in to be considered active?
• Could total storage used be another metric of use?
• Should we track what features are users most attracted to, or find that they still wish the ELN had?
What other adoption issues could be considered?
• Does your institution have specific adoption goals that have to be met? What approaches are you using
to reach those goals?
• What are the most common barriers to adoption, and how can we help users work through those?
• Is it important to know if use is associated with a class?
at Cornell: Strongly
Disagree
Generally
Disagree
Generally
Agree
Strongly
Agree
Do Not
Know/Did
not use
Not
Applicable
Using LabArchives has (or
will) facilitate research,
data management and/or
record keeping in the lab.
0% 0% 17% 83% 0% 0%
at Wisconsin: Agree
Not sure or
neutral
Disagree
Using the ELN has allowed me to:
Improve Data Management 34% 22% 44%
Keep All Data in One Place 47% 22% 31%
Keep Data Over Several Years 63% 25% 13%
0
50
100
150
200
Jul-12
Sep-12
Nov-12
Jan-13
Mar-13
May-13
Jul-13
Sep-13
Nov-13
Jan-14
Mar-14
May-14
Jul-14
Sep-14
Nov-14
Jan-15
Mar-15
May-15
Jul-15
Sep-15
Nov-15
Jan-16
Mar-16
Accounts
Created
Monthly
Unique
Logins
COMMUNICATIONG DATA MANAGEMENT VALUE
“Data management’ can mean lots of different things to different people. To evaluate how important the
various ways ELNs help researchers with their data management needs, there are LOTS of questions that
could be asked.
Below, we highlight three concepts and pose some wording that get at user perception of the value of each.
We want your thoughts on ways to inquire about why and how researchers use ELNs for data management.
Data management concept: ____________________________
Give us ideas – what do you hear from your researchers about why they like to use ELNs? What words are
best to use to have that idea resonate with a wide variety of users?
* Data from Fall 2015: https://www.cornell.edu/about/facts.cfm. ** Data from 2015-2016: http://apir.wisc.edu/datadigest/Data%20Digest_Web_16.pdf