Kyrillidou "Research Methods and Tools: A NISO Training Series"

LIS Research Methods and
Tools: A 2022 Assessment
Training Series
Martha Kyrillidou, MLS, MEd, PhD
NISO Fall 2022

Structure
● Introductions
● An introduction to inquiry
● The structuring of inquiry
● Modes of observation
● Analysis of data

Timeline
Tuesday October 11, 2022: Introduction to Research Methods and Tools for Library and
Information Science
Tuesday October 25, 2022: Quantitative and Qualitative approaches
Tuesday November 15, 2022: Reflections on Library Assessment Methods and Tools
Tuesday November 29, 2022: What do we want to achieve in our Library?
Tuesday December 6, 2022: Identification of research questions and challenges
Tuesday, December 13, 2022: Most valuable tools and methods - why do we see these as being the most
valuable?

Human Inquiry, Knowledge, Science, and Libraries
Listen and Learn

Ask questions
•Reflect on the ways you ask questions
•What are some methods you can deploy to answer your questions?
•What are the assumptions behind these methods?
•What types of bias are introduced by these assumptions and how
can you control for such bias?
•Use different methods and explore their assumptions and biases?

Looking for Reality
● Knowledge from Agreement Reality
○ How do we know what is real?
● A scientific assertion must have both theoretical
(logical) and empirical support
○ Depending on your philosophical worldview you
may be more open to postpositivism,
constructivism, transformative, or pragmatic
worldviews

Research Design
by Creswell & Creswell
Chapter 1. The Selection of a Research Approach

My reflection - your homework
● In high school (Thessaloniki, Greece), I was good both in math and in languages though my
teacher role models inspired me to pursue humanistic studies (English language and
linguistics major at Aristotle University)
● Exposed to libraries and wanting to improve them had a lot of questions as to how they can
be improved; MLS with thesis after taking a lot of methods courses in EdPsych to complete
a survey research project as a master’s thesis project (Kent State University)
● Spending time and observations in understanding research library environments and
interactions led to appreciation of qualitative methods (in depth interviews developed
LibQUAL+); quantitative dissertation on developing LibQUAL+ Lite (UIUC)
● Keep following changes in Greek libraries to this day and emphasizing the design thinking
and transformative perspective as a library consultant (QualityMetrics, LLC since 2016)
● Appreciation for pragmatism and long term planning and perspectives for libraries through
space planning projects; doing user research with design thinking tools and with architects.

Research and Assessment
Cycle Toolkit
https://www.arl.org/research-and-assessment-cycle-toolkit/

Assessment Cycle
Questions
Aspects of your goals
Will determine methods
Focus / Goals
Modiﬁcations
Transformation
Change
Revisiting Goals
Interpretation
Sharing results
Act or not act
Making Decisions
Communicate
Visualize
Explain and Understand
Results
Systematic collection
Appropriate Methods
Review and Analysis
Interpretation
Data

Key Drivers for Assessment
A need to better understand our users (LC)
Justifying my budget and any increases
UC San Diego Library: Developing a collaborative public service model
Helping others within the library have a better picture of how their work is landing
Standardizing instruction program and proving its use/worth to students (University of Nebraska at Kearney)
Preemptive work to help justify future spending and additional staff. (BLM Library)
Determining user behavior and satisfaction, & meeting strategic planning goals (University of Oklahoma)
Effective outreach to promote services across disparate organizations (NIST)

Understanding Users
● What is the organization’s goal/purpose/mission?
● What is the organization trying to achieve?
● What does the organization actually achieve? (match
or gap)
● What is important enough to the organization for them
to allocate resources? (follow the
action/money/time/effort)

Research and Assessment Cycle Toolkit
● Are we doing the right things?
● Are we doing things right?
● How do we know they are the right
things?

Leveraging Strategic Plans
American University - university plan
Central Washington University Libraries - university - strategic plan introduction – libraries
Drew University - Strategic Plan of the Drew University Library Partnering for Impact
East Carolina University - Academic Library Services 2017-2022 Strategic Plan
Government Accountability Office - no library plan. Agency = https://www.gao.gov/products/gao-22-1sp
Jimma University - university - library
Library of Congress
National Institute of Standards and Technology (NIST) Research Library
Northern Virginia Community College - Virginia’s Community Colleges – Library Mission
Texas State University Libraries
Towson University - university – libraries
U.S. Government - Bureau of Land Management https://www.blm.gov/about/our-mission (not quite what you are looking for)
UC San Diego Library
University of Idaho Library
University of Nebraska at Kearney (Karni) (no library plan available)
University of Nebraska-Lincoln Libraries - https://libraries.unl.edu/libraries-strategic-plan
University of Oklahoma Libraries - https://www.ou.edu/leadon
Vocational Training Council
West Chester University Libraries - https://library.wcupa.edu/strategicplan

Key Drivers for Assessment
A need to better understand our users (LC)
Justifying my budget and any increases
UC San Diego Library: Developing a collaborative public service model
Helping others within the library have a better picture of how their work is landing
Standardizing instruction program and proving its use/worth to students (University of Nebraska at Kearney)
Preemptive work to help justify future spending and additional staff. (BLM Library)
Determining user behavior and satisfaction, & meeting strategic planning goals (University of Oklahoma)
Effective outreach to promote services across disparate organizations (NIST)
???
JOURNEY
MAPPING

Library Assessment Conference - highlights
Keynote by Denise Stephens (U of Oklahoma) on “Elusive Measures: The
Imperative to Identify and Utilize DEI Indicators that Shape Stakeholder and
Organizational Outcomes”

Know your state and the structure of libraries in your state
LSTA Evaluation on the IMLS website

Assessment Accelerators
CARL Library Impact Framework by Justine Wheeler (University of Calgary), Mark Robertson (Brock
University), and Tania Gottschalk (Thompson Rivers University Library)
A Roadmap to Practical Strategic Planning by Maurini Strub (University of Rochester) and Starr
Hoffman (UNLV Libraries)
Present & Future Proficiency: Updating the ACRL Assessment Proficiencies to Reflect Current and
Coming Realities by Rebecca Croxton (University of North Carolina at Charlotte), Megan Oakleaf (Syracuse
University), and Jung Mi Scoulas (University of Illinois Chicago)

Choosing Your Own Scholarly Communication Assessment Adventure: Applying and Reviewing a
Draft Engagement Matrix to Evaluate Program Growth and Development
Emily Chan (San Jose State University), Suzanna Yaukey (Towson University), Nicole Lawson
(California State University, Sacramento), and Daina Dickman (Network of the National Library of
Medicine Region 5)
Demystifying Qualitative Coding by Alisa B. Rod, Marcela Y. Isuster, and Tara Mawhinney (McGill
University)
Online Participatory Design: Activities and Approaches for User Engagement in the Remote
Environment by Jackie Belanger, Maggie Faber, and Reed Garber-Pearson (University of
Washington)

A Discussion on the Evolution of Assessment Work in Academic Libraries: Is this a fluke, or is
this our future? By Kat Bell (George Mason University, Emily Guhde (Georgetown University),
Steve Borrelli (Penn State University Libraries), and Maurini Strub (University of Rochester)
Data Ethics and Learning Analytics: Putting Privacy into Practice by Lisa Hinchliffe (University of
Illinois at Urbana-Champaign)
Introducing the Values-Sensitive Library Assessment Toolkit: A Practical Approach for Ethical
Assessment by Scott Young (Montana State University)

Designing Libraries IX highlights

Brightspot strategy blog (pre conference workshop)
Philadelphia art museum master plan
University of Virginia special collections
University of Illinois Urbana Champaign library redevelopment plan

Latest books on space planning …

University of Alaska, Fairbanks
Martha Kyrillidou led the team that reconceptualized part of the Rasmuson Library at the University of Alaska, Fairbanks as
a Student Success Center; during the pandemic we developed a programmatic approach to the Student Success Center
after numerous conversations and focus groups with potential stakeholders. Identifying “light” as an attractive element for
the student success center, we focused on ensuring a light-filled floor houses important academic services and other
student support services with a shared vested interest in its success. In addition to the campus research and the space
assessment, we designed a staffing model for the area. The project included space analysis and focus groups and
interviews with 20-30 academic support units in order to prioritize their presence in the student success center. Currently,
the space has been cleared from collections and is ready to bid for construction.
Response to the RFP
Final report delivered

Noel Wien Public Library, Fairbanks, Alaska
Noel Wien Public Library is the main public library in Fairbanks serving about 100,000 people in the furthest northern city a
commercial airline flies within the United States. The consultants were asked to create an innovative conceptual plan for
renovating the existing space to enable the library to foster creativity and learning and to meet the library and information
needs of the Fairbanks Borough residents well into the future. The renovation plan did not include any expansion but the
reconceptualization of the existing space. It is currently slotted for implementation by Noel Wien Burrough in Fairbanks. The
goal is to transform the library facility into a dynamic center of community life.
Response to the RFP
Final report delivered

For those interested in
doing a design
critique, you may find
this template of good
use

Here’s a resource
explaining Journey
Mapping

Journey Map Template
You can use this template to collect data on your own project

Logic Models
W.K. Kellogg Foundation Logic Model Development Guide

Logic Model Template
https://drive.google.com/drive/u/0/folders/1u30cm_fL3ZoTgsANfRzikDk
pkcmoYPum

Alternative Futures
Hear about the full process for Alternative Futures along with an example.
The Future’s Executive Director, Marina Gorbis, has prepared an introduction
video available on YouTube
After the Pandemic: The Deeper Disease Scenarios

Strategic Foresight
https://libguides.ctstatelibrary.org/
dld/help/strategicplan

Strategic Scenarios
https://www.arl.org/resources/the-arl
-2030-scenarios-a-users-guide-for-r
esearch-libraries/

IMLS grant - Needs of Paraprofessionals in Academic
Libraries
https://docs.google.com/presentation/d/1Zah9UETUGgGUErDGSg1nnu5ODNjWC6vD4g
JsMTnBvBw/edit?usp=sharing

IMLS grants
https://imls.gov/grants/awarded-grants?search_api_views_fulltext=&sort_by=field_
program&field_states=All&field_city=&field_institution=&field_program_categories
_text=&fulltext_search=&page=49&order=field_fiscal_year_text&sort=desc

Timeline
Tuesday October 11, 2022: Introduction to Research Methods and Tools for Library and
Information Science
Tuesday October 25, 2022: Quantitative and Qualitative approaches
Tuesday November 15, 2022: Reflections on Library Assessment Methods and Tools
Tuesday November 29, 2022: What do we want to achieve in our Library?
Tuesday December 6, 2022: Identification of research questions and challenges
Tuesday, December 13, 2022: Most valuable tools and methods - why do we see these as being the
most valuable?

Thinking Strategically About Library Futures
● What is the central work of the library and how can we do more, differently, and at less cost?
● What important set of services does the library provide that others can’t?
● What new roles are needed?
● What advantages does the library possess?
● What will be the most needed by our community of users in the next decade? How is user behavior changing?
● What should our libraries aspire to be ten years from now? What are the implications of technology driven
change?
● What are the essential factors responsible for the success of the library?

Scenario Thinking
Scenario Planning
Tool for helping
managers deal with
situations of significant
uncertainty.
Scenario Project
Series of exercises
where teams drill into
the uncertainties faced
today and in the future
Result
A set of narratives describing three or four “alternate
futures.”

Scenario Thinking
Creation of alternate futures encourages
managements teams to
“Think the unthinkable”
Anticipate surprises
Rehearse new
possibilities

Scenario Thinking
Scenario set
A variety of possible futures and uses
containing the elements and conditions the
organization will face in the future
Must consider full
set of scenarios

Strategic Direction: Transforming Research
Libraries: Outcomes & Strategies
■ Initiate visioning and scanning
activities focused on emerging roles for
research libraries in the processes of
research, scholarship, and graduate
education. Encourage and facilitate
member engagement in adopting new
roles in advancing research and
scholarship.

Acknowledgements
■ Susan Stickley, Stratus Inc
■ Karla Strieb, ARL
■ Sue Baughman, ARL
■ Charles Lowry, ARL Executive Director
■ ARL Member leaders involved in this effort
■ http://www.arl.org/rtl/plan/scenarios/index.shtml

Prepare for a DIFFERENT Future

ARL’s Scenario Development Project
Phase One - Data Gathering
Define Strategic
Focus
and strategic
conversation
Explore and uncover
core strategic
questions
How do we transform our organization(s) to create differential
value for future users (individuals, institutions, and
beyond), given the external dynamics redefining the research
environment over the next 20 years?
Expand strategic
questions

Refining the process:
Addressing concerns about maintaining and or
building relevancy that can be sustained
and valued by users.

Framing of Critical Uncertainties
Individual Users and
Researchers
The vertical axis asks will research
process and product empower users and
researchers or not? Who is most valued
in the research process?
Individual
Researchers
Research Enterprise
The horizontal axis asks will
research be highly distributed
and organic or will it be
integrated across
organizations?
Research Enterprise

The ARL 2030 scenarios allow members to
opportunity to suspend disbelief and stretch beyond
conventional wisdom about our future.

The Four ARL Scenarios
● Scenario 1: Research Entrepreneurs
● Scenario 2: Reuse and Recycle
● Scenario 3: Disciplines in Charge
● Scenario 4: Global Followers

ARL’s Scenario Project
Individual
Researchers
Research Enterprise
Aggregated Diffused
Unconstrained
Constrained
Scenario 4:
Global Followers
Scenario 3:
Disciplines in Charge
Scenario 2:
Reuse and Recycle
Scenario 1:
Research Entrepreneurs

Scenario 1:
Research Entrepreneurs
■ Hannah Chen embodies the successful entrepreneurial
researcher. She shapes her own career and research,
flexibly pursuing opportunities as she chooses. Part of
her edge comes from her personal willingness to follow
new paths to her own development as a scholar, while
she also demonstrates the benefits of canny
partnerships with other researchers.

Scenario 2: Reuse and Recycle
■ Hannah Chen is one of the lucky few who has persisted
in crafting a career as a researcher. Her ability to
“follow the money” to clients who can support her rather
mundane research projects, combined with her skill in
fostering personal relationships with other researchers,
has allowed her to craft a niche to support her research
activities, although not to fully develop her research
interests.

Scenario 3:
Disciplines in Charge
■ Hannah Chen has found success by aligning herself with
an emergent disciplinary research organization. She has
leveraged her research skills, not in service of the
discipline that trained her, but to play a leadership role in
developing the research technologies that underpin
another domain’s research program.

Scenario 4: Global followers
■ Hannah Chen’s career has unfolded along somewhat
traditional lines, while also tracking the shifting directions
of the North American research enterprise. Throughout
her development as a researcher, she has successfully
aligned her research interests with the shifting grand
research challenges identified by large-scale funding and
leveraged emerging sources of research data. As a
consequence, by 2030 her institutional allegiances have
migrated beyond the borders of the US and Canada,
although advances in technology and shifting cultural
norms allow her to continue to live in North America.

Elsevier’s Research Futures
https://www.elsevier.com/connect/elsevier
-research-futures-report

How can you use the
scenarios?
■ Engage staff in strategic discussions
■ Creating one strategic agenda from multiple
futures
■ Using scenarios to create vs test strategy
■ Scenario planning, an interactive
engagement process
■ Scenario planning and Strategic planning
■ Early indicators and ongoing monitoring

Fiat lux, fiat latebra: a celebration of
historical library functions by Krummel
https://www.ideals.illinois.edu/handle/2142/3861

THANK YOU
martha@qualitymetricsllc.com

QUALITYMETRICS
martha@qualitymetricsllc.com
www.qualitymetricsllc.com

Our mission is to foster
organizational
commitment to
evidence-based decision
making, library
assessment, research
and development.
Our focus areas include:
● vision
● mission
● planning
● infrastructure
● assessment
● evaluation
● project management

Featured Projects
Montana State University Library Strategic Plan
US Virgin Islands Public Library System: Evaluation
Noel Wien Public Library, Fairbanks, Alaska: Renovation Planning
USMAI (University System of Maryland & Affiliated Institutions) Assessment
Wash and Learn: Libraries Without Borders, IMLS Leadership Grant
Penn State University Organizational Structure Review
UCLA Library Cost Study for Indirect Cost Rate Negotiation

Kyrillidou "Research Methods and Tools: A NISO Training Series"

Kyrillidou "Research Methods and Tools: A NISO Training Series"

Recommended

Recommended

More Related Content

Similar to Kyrillidou "Research Methods and Tools: A NISO Training Series"

Similar to Kyrillidou "Research Methods and Tools: A NISO Training Series" (20)

More from National Information Standards Organization (NISO)

More from National Information Standards Organization (NISO) (20)

Recently uploaded

Recently uploaded (20)

Kyrillidou "Research Methods and Tools: A NISO Training Series"