Feliciati contribution to "User studies for Digital Libraries development" tutorial in JCDL2012 conference, Washington June 10th 2012

1. TESTING OF AN ARCHIVAL DL PROTOTYPE
ASKING TO USERS:
THE CASE OF “UNA CITTÀ PER GLI ARCHIVI”
Pierluigi Feliciati
University of Macerata (Italy)
JCDL – Washington (USA) 2012.06.10

2. INTRODUCING THIS
DISTANT PRESENTATION (1)
 I am an ICT senior researcher in Macerata
University (Italy), but my background covers
archaeology, historical archives management and
distance services development. I’m working
especially on Web services quality
 Rarely archives on line (archival DLs) are
approached from the point of view of final
Feliciati - JCDL 2012
users, and input and output activities and
environments are often mixed in an unique layer
 Users are probably not satisfied (but who knows it
2
for sure?)

3. INTRODUCING THIS
DISTANT PRESENTATION (2)
 User studies in the prototypal stage of a DL life-
cycle are rare but crucial, to finalise the quality of
services.
 The case study I will present is probably one of
hugest projects in this sense in terms of methods
and number of people involved
 This video presentation will ask you 15-20
Feliciati - JCDL 2012
minutes of kind attention, then Milena and Nicci will
be happy to ask to all of your questions instead of
me (unabled to move to Washington)
3

4. MAIN TOPICS
1. Archives on line and Users: a museum of broken
relationships
2. Testing a prototype: Why? What? How? Results
implementation policy
3. The Città per gli Archivi (Italy) project
4. Asking to users 1: internal staff
5. Asking to users 2: target users / focus groups
Feliciati - JCDL 2012
6. Asking to users 3: expert evaluation
7. First conclusions and lessons learnt 4

5. ARCHIVES ON LINE AND USERS
 Without the expert mediation
of archivists what is based on
the quality of access/use?
Feliciati - JCDL 2012
 A lesson to be learnt: Input is
not output!
 Archival standards create
encoded contents, not
automathically good displays
 Available very few (American)
studies on Archives on line
quality (from the final users
satisfaction point of view)
5

6. ARCHIVES ON LINE AND USERS
Some typical issues:
 Terminology: use of jargon
Feliciati - JCDL 2012
 Archival hierarchical
structure: browsing is guided
just by titles
 Contents: too encoded
 Searching: no intuitive tools
with no subjects
 Coverage: what could I find
here?
6

7. TESTING A PROTOTYPE:
WHY AND WHAT?
 User analysis could be very useful in the early
stages of a project (prototype)
 Evaluation is called formative, when conducted
while the DL is still under development
 Part of the data gathered during this stage will focus
on the content and the functionalities of the DL
and how these correspond to the users’ needs
Feliciati - JCDL 2012
 User needs could be checked asking directly to
users, who can provide feedback that helps in the
recognition of flaws in the interaction design.
7

8. TESTING A PROTOTYPE: WHY AND WHAT?
 Morever, expert opinions can provide assistance
when the DL prototype has been refined, giving
their insights on the quality features of the system
Feliciati - JCDL 2012
8

9. TESTING A PROTOTYPE: HOW?
 To directly involve users in the prototype step, we
could launch a survey (e.g. using on-line
questionnaires), if we have already a designed
community to address
 On the other hand, we have to organize as more
focus groups as possible trying to address user
panels who could be interested to the future DL we
are working on
Feliciati - JCDL 2012
 Finally, the expert study could involve both experts
on content and functionality, to gather advanced
qualitative data on performance, usability and
9
expected success of our project

10. TESTING A PROTOTYPE: DEFINING A RESULTS
IMPLEMENTATION POLICY
 During formative evaluations, the researcher may
return to previous stages of development and adopt
their findings, thus altering the status of the DL.
Feliciati - JCDL 2012
10

11. THE CITTÀ PER GLI ARCHIVI PROJECT
Una città per gli archivi (A City for Archives CPA) is a
project supported by two banks foundations, with the
collaboration of other institutions (archival and city
administration)
Its mission is to improve the
preservation, valorization and full access for the
most important Archives related to the
contemporary history of the city of Bologna (Italy)
Feliciati - JCDL 2012
The project focused not only on institutional archival
records but as well on audiovisual and photographic
archives
11
This huge cultural project started in 2007

12. THE CITTÀ PER GLI ARCHIVI PROJECT
 More then 200 Archives have been preserved and
described, made of 300.000 records, paper
documents, photographic and graphic
materials, sound and audiovisual recordings
 70 archivists were involved
 200.000 digital objects created
 A Web portal (an archival DL…) was developed to
enable users to access the described archives and
the digital reproductions
Feliciati - JCDL 2012
 Export of metadata towards Italian National Archival
System, Italian Cultural Portal and Europeana was
scheduled 12

13. THE CITTÀ PER GLI ARCHIVI PROJECT
A wide program of
restoration and sorting
out of Archives at risk of
loss,
redaction of guidelines for
descriptions and
authority control
Feliciati - JCDL 2012
Thus, during the
prototypal step of the
project, a user and
functionality study
13
was launched

14. CPA portal prototype
a snapshot
Feliciati - JCDL 2012
14
1
4

15. CPA USER STUDY
ASKING TO USERS 1: INTERNAL STAFF
 For the quality archival descriptions, the
competence of archivists is crucial
 Rarely those input content providers are directly
involved in the development of output environments
 The CPA users and functionalites study planned 2
special focus groups, inviting together archival
staff and DL developers (beginning and end of the
study)
 Those meetings were moderated according to a
Feliciati - JCDL 2012
special protocol, mixing typical focus groups
method with that of a moderated brainstorming
(archivists vs engineers)
15

16. CPA USER STUDY
ASKING TO USERS 2: FOCUS GROUPS
5 focus groups were organized to collect
precious data about potential target users
needs and satisfaction against CPA portal
 1 with high school students (16 pople, aged 16-
19)
 2 with university students (total 20 people, aged
23-29)
Feliciati - JCDL 2012
 1 with general public (16 people, aged 39-63)
 1 with archivists and cultural heritage (LIS and
museums) professionals (16 people, aged 29-
16
54)

17. CPA USER STUDY
ASKING TO USERS 2: FOCUS GROUPS
 All
the focus groups were based on the
same detailed protocol, asking 100 minutes
and including 6 sections:
1. Demographic and ICT skills questionnaire
2. CPA prototype presentation
3. First impressions and discussion
4. 5 tasks, whose results were recorded asking
Feliciati - JCDL 2012
3-4 questions for each task
5. Advanced impressions and discussion
6. Final discussion 17

18. CPA USER STUDY
ASKING TO USERS 3: EXPERT EVALUATION
 Finallyan expert study activity was
scheduled for next September, when the
prototype will be almost ready
 The 7 experts involved are: 3 archivists (2
university professors and a City Archive
director), 3 DL and metadata experts, 1 ICT
university professor
Feliciati - JCDL 2012
 The protocol adopted is focused on
performance, functionality, usability
issues, terminology, quality of content 18

19. FIRST CONCLUSIONS
(AND LESSONS LEARNT)
 Archives on line are special DLs: they open
peculiar issues to face users expectations and
guarantee satisfaction
 Users study are crucial, to finalise the DLs and
ensure a (really) good quality
 The prototypal stage of the project, when
possible, offers a good occasion for evaluation and
assessment activitis based on users involvment
 The Italian CPA project shows that such activities
Feliciati - JCDL 2012
are not that expansive, they need an expert (and
neutral) coordination, a good organizational staff
and a network of people and institutions to be
involved 19

20. THANK YOU FOR YOUR ATTENTION,
WISH YOU ALL A GREAT WORK TOGETHER!
pierluigi.feliciati@unimc.it
This presentation is licensed under:
CC-BY-SA