![1. INTRODUCTION ,[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
PM8_Vinagre_Rethinking quality dimensions and attributes for the digital age: A multi-item scale analysis and assessment
- 1. RETHINKING QUALITY DIMENSIONS AND ATTRIBUTES FOR THE DIGITAL AGE: A MULTI-ITEM SCALE ANALYSIS AND ASSESSMENT Maria Helena Vinagre ISCTE Leonor Gaspar Pinto Universidad de Alcalá de Henares Paula Ochôa Universidade Nova de Lisboa PORTUGAL IFLA Satellite Preconference 8th Northumbria International Conference on Performance Measurement in Libraries and Information Services (PM8) Florence, 17-20 August 2009
- 10. Tolerance Zone and Service Quality Measures Zeithaml, Berry & Parasuraman (1993), developed a model that integrates two levels of expectations: the service that represents the desired level (what should be provided to the consumer) and the appropriate service that represents the minimum level of service that the consumer is willing to accept. Between these two levels, there is a zone of tolerance. Evaluation criteria and methods
- 11. Tolerance Zone and Service Quality Measures Evaluation criteria and methods Desired Service What should be provided to an user of digital library Tolerance Zone Appropriated Service The minimum level of service that user of digital library is willing to accept Measure of Service Adequacy (MSA) = Perceived Service Measure of Service Superiority (MSS) Perceived Service =
- 12. Tolerance Zone and Service Quality Measures Based on the conceptual e-gap model suggested by Zeithaml, Parasuraman e Malhota (2002), Measure of Service Adequacy (MAS) and Measure of Service Superiority (MSS), we developed a model - Digital Library Service Quality Model - to evaluate the quality of service delivered by the Portuguese Digital Library. Evaluation criteria and methods
- 15. 4. THE EXPLORATORY STUDY FOR PSYCHOMETRIC VALIDATION OF THE MEASURES Initial Questionnaire Draft Library team Focus - group Internal Consistency Analysis Revised Questionnaire • Internal consistency analysis ( Cronbach ’ s alpha) • Exploratory Factor Analysis • Internal consistency analysis of the factor structure found • Confirmatory factor analysis Other scales Literature Review Questionnaire Pre - testing Data Analysis / Data Validation Cronbach ’ s alph Validated Questionnaire / Further Comparative Analysis The process of constructing and validating the measures
- 17. 4. THE EXPLORATORY STUDY FOR PSYCHOMETRIC VALIDATION OF THE MEASURES Item code Item Component Efficiency Competitive advantage Information adequacy P2Q145g Easiness of search information ,73 ,23 ,13 P2Q150g Easiness of use compared with other sites (e.g. Google, yahoo) ,66 ,25 ,08 P2Q140g Finding information without help ,66 ,28 ,15 P2Q139g Easiness to access electronic resources from any location (e.g. at home or at work) ,64 ,03 ,19 P2Q149g User Support Service speed of response ,60 ,30 ,04 P2Q144g Easiness to access journals backfiles ,55 -,02 ,34 P2Q152g Greater scientific credibility of search results compared with other sites (e.g. Google, yahoo) ,06 ,78 ,10 P2Q153g Easiness of use of electronic services compared with going to the library ,17 ,77 ,17 P2Q151g Easiness of search compared with searching in printed materials ,32 ,71 ,04 P2Q148g Speed download of articles ,25 ,57 ,25 P2Q142g Relevance of contents to my study and research area ,16 ,25 ,84 P2Q143g The digital library has all contents adequate to my information needs ,25 ,16 ,83 % of explained variance (with rotation) 56,96 Factor internal consistency all items (Cronbach ’ s alpha) ,85
- 18. 4. THE EXPLORATORY STUDY FOR PSYCHOMETRIC VALIDATION OF THE MEASURES Goodness-of-fit indices: χ2 (51, N = 1708) = 209.87, p< .001, IFI = .99, CFI = .99, TLI = .99 RMSEA = .043. Second order Service Quality Model ( Confirmatory Factor Analysis ) ,79 ,39 P2Q150_global ,63 ,32 P2Q149_global ,56 ,53 P2Q145_global ,73 ,24 P2Q144_global ,49 ,50 P2Q140_global ,71 ,28 P2Q139_global ,53 ,60 ,34 P2Q148_global ,51 P2Q151_global ,37 P2Q152_global ,53 P2Q153_global ,71 ,61 ,73 ,49 ,60 P2Q143_global ,62 P2Q142_global ,78 ,79 ,58 ,89 ,78 ,70 Information adequacy Competitive advantage Efficiency Service Quality
- 20. 5. THE COMPLETE DIGITAL LIBRARY SERVICE QUALITY MODEL - Results Zone of Tolerance by dimension and question Competitive advantage It is clear that users acknowledge the digital library’s competitive advantage over other information resources Information adequacy the perceived and minimum tolerated levels were quite similar Efficiency users show the lowest service level evaluation and, simultaneously, the lowest level of demand in terms of the minimum tolerated service. Users are more willing to tolerate lower Efficiency levels than lower Competitive advantage or Adequacy of information levels Efficiency Competitive advantage Information adequacy 0.00 1.00 2.00 3.00 4.00 5.00 0 1 2 3 4 5 6 7 8 9 10 11 12 minimum desired perceived o
- 21. 5. THE COMPLETE DIGITAL LIBRARY SERVICE QUALITY MODEL: GAP Analysis Results The Knowledge gap 0.16 0.15 0.05 0.05 - 0.10 0.27 -0.05 -0.11 -0.11 -0.03 -0.01 -0.15 Adequated Desired Adequated Desired B1 - U B2 - U Efficiency Competitive advantage Information adequacy
- 22. 5. THE COMPLETE DIGITAL LIBRARY SERVICE QUALITY MODEL: GAP Analysis Results Perception Gap 0.33 0.16 0.21 -0.01 -0.06 0.12 Library managers-Users LIS profissionals-DL users Efficiency Competitive advantage Information adequacy
- 23. 5. THE COMPLETE DIGITAL LIBRARY SERVICE QUALITY MODEL: GAP Analysis Results Service Quality Gap 0.1 - 0.9 - 0.1 0.7 - 0.0 0.7 - 0.1 0.4 - 0.2 0.5 - 0.1 0.3 - 0.1 - 0.8 - 0.1 - 0.9 - 0.0 0.6 - MSA MSS MSA MSS MSA MSS DL users Libray managers LIS profissionals Efficiency Competitive advantage Information adequacy
- 26. THANK YOU FOR YOUR ATENTION [email_address] [email_address] [email_address]
Editor's Notes
- The growing importance of the networked environment and the consequent change in users’ information behaviour challenged the library academia and professionals’ skills to find new and more adequate strategies and methods to assess the performance of traditional, hybrid or digital information services This presentation aims to detail the strategy and methodology used by a research team to evaluate the performance of a Portuguese Digital Library consortium.
- For more than four decades, performance evaluation of library and information services has been captivating the attention of professionals and researchers of the Information Science area. The models used in the evaluation of these services had been categorized by Hernon and Altman (1996) in the following way: Extension (amount): models that implement measures to find out “how much” inputs (financial resources, staff, documents, etc.) or outputs (activities, services’ use, etc). Efficiency : models centred in the establishment of ratios between inputs and outputs, most frequently per capita ratios. Cost : models focused on the average cost per input/output; when combined with extension and efficiency measures, they generate cost-effectiveness indicators; Quality : models developed to evaluate results (outputs) in terms of quality (reliability, relevance, etc. of a given service and user satisfaction with that service); Effectiveness : models that evaluate how an information service is attaining its goals and its alignment with the parent organization, as well as the organisation’s capacity to fulfil users’ needs.
- The advent of a new networked environment in the beginning of the 90s of the 20th century brought increased complexity to this library and information services’ diverse evaluation context: traditional, hybrid and digital libraries coexist in the same geographical space and, sometimes, within the same organisation. Among many existing definitions of Digital Library (DL), we can pick one provided by Leiner (1998): “ The Digital Library is: The collection of services And the collection of information objects That support users in dealing with information objects And the organization and presentation of those objects Available directly or indirectly Via electronic/digital means.” According to T. Saracevic (2004), the main problem in evaluating digital libraries derives from the difficulty in establishing evaluation borderlines.
- Consequently, the literature on digital libraries evaluation is characterized by a multitude of approaches and levels of analysis, although in general these libraries “...are not evaluated on more than one level” (Saracevic, 2000, p.364). The complexity of this sort of social, institutional and technical system suggests the adoption of a holistic and cumulative approach to evaluation (Nicholson, 2004). This kind of approach to evaluation was considered adequate for evaluating the Portuguese Digital Library consortium.
- From 2006 to 2007, the research team in charge of this digital library evaluation initiative developed and implemented an Integrated Evaluation Programme . The conceptual pillar of this Programme was the Digital Library Integrated Evaluation Model (Pinto, Ochôa and Vinagre, 2009) formed by five key-components: Diagnosis ; Strategical groups and performance information needs , Perspectives on performance evaluation , Evaluation criteria and methods and Evaluation points of view . The development of the fourth key-component led to the identification of six criteria for performance evaluation: Effectiveness, Extension , Efficiency , Cost, Quality and Impact. The application of the quality criterion to the evaluation of the Portuguese digital library was based on the development of a tailor-made Service Quality Model , which is the main focus of this paper.
- DIGITAL LIBRARY INTEGRATED EVALUATION MODEL 5 key-components : Diagnosis Strategical groups and performance information needs Perspectives on performance evaluation Evaluation criteria and methods Evaluation points of view . 6 criteria for performance evaluation: Effectiveness, Extension , Efficiency , Cost, Quality and Impact. The application of the quality criterion to the evaluation of the Portuguese DL was based on the development of a tailor-made Service Quality Model
- THE DIGITAL LIBRARY SERVICE QUALITY MODEL The use of a service involves a multiplicity of tangible and intangible aspects which make particularly critical the adoption of instruments that allow its measurement and evaluation. Because of its specificity, these services evaluations required the adoption of convenient instruments and scales adequate to its characteristics and users’ relevant evaluating aspects Despite the great profusion of studies on service quality only one limited academic articles dealt with e-service quality or web services quality evaluation. Because of its specificity, these services evaluation required the adoption of convenient instruments and scales adequate to its characteristics and users’ relevant evaluating aspects (Parasuraman, Zeithaml and Malhotra, 2005)
- In the evaluation of e-Service Quality some authors have noted several dimensions relevant to this sort of service. In their empirical study, Zeithaml, Parasuraman and Malhotra, (2002) developed and validated E-S-QUAL for measuring service quality delivered by web sites. In two different stages of empirical data collection, they identified two different scales: the basic E-S-QUAL with four relevant dimensions: (1) Efficiency, (2) Fulfilment, (3) System availability, and (4) Privacy; these dimensions were reduced to three dimensions in the E-RecS-QUAL (Zeithaml, Parasuraman and Malhota, 2002): (1) Responsiveness, (2) Compensation and (3) Contact. Loiacono, Watson and Goodhue (2002) developed and validated the also called WebQual, a web site quality measure with 12 core dimensions Yoo and Donthu (2001) developed SITEQUAL, a scale with four dimensions: (1) Ease of use, (2) Aesthetic design, (3) Processing speed and (4) Security. Mulvenna, Anand and Buchner (2000) found other different dimensions: (1) Access, (2) Responsiveness and (3) Customization
- As remarks Parasuraman, Zeithaml and Berry (1994) there is a general agreement on the role of expectations as points of reference in the evaluation of consumers. There is, however, a wide debate on how to best incorporate the expectations in the assessment of quality of service (e.g. Babakus and Boller, 1992; Brown, Churchill and Peter, 1993, Carman, 1990) Given the discussion on the different levels of expectations (e.g. Teas, 1993, 1994), Zeithaml, Berry & Parasuraman (1993), developed a model that integrates two levels of expectations: the service that represents the desired level (what should be provided to the consumer) and the appropriate service that represents the minimum level of service that the consumer is willing to accept. Between these two levels, there is a zone of tolerance.
- The zone of tolerance thus represents the range of performance of a service that can be considered satisfactory. In this formulation, the desired service is obtained from a Measure of Service Superiority (MSS) and the appropriate service from a Measure of Service Adequacy (MSA).
- Based on the conceptual e-gap model suggested by Zeithaml, Parasuraman e Malhota (2002), along with the Measure of Service Adequacy (MSA) and the Measure of Service Superiority (MSS), we developed a model - Digital Library Service Quality Model - to evaluate the quality of service delivered by the Portuguese Digital Library
- The model identifies some linkages between the identified strategical groups in this digital library evaluation process. These linkages have some critical points that can be identified as gaps : Knowledge gap – it refers to libraries teams and libraries managers’ deficient knowledge on users´ needs and expectations. Perception gap is the difference between users’ perceptions and libraries teams and libraries managers’ perceptions on users. Communication gap reflects misunderstandings between different operational teams and institutions. Service quality Gap - Digital libraries services intend to fulfil the expectations of its users. The difference between the perceptions of the delivered service and the expectations generates service quality (fulfilment gap).
- This model has three basic assumptions: A digital library is a multidimensional construct It is possible to evaluate two levels of expectations: the desired level (what should be provided to the digital library’s users) and the appropriate level (that represents the minimum service expected and tolerated). Therefore, the use of the Measure of Service Adequacy (MSA) and the Measure of Service Superiority (MSS) was a valid way to evaluate service quality in a digital library and its diagnostic power allows a better assessment of service shortcomings. There are four potential basic deficiencies or organizational gaps that can occur in digital library service processes and could contribute to a poor e-service quality delivery: Knowledge gap, Perception gap, Communication gap and Fulfilment gap.
- Based on the literature review and previous research carried out by members of the team, we developed a questionnaire for users that would enable the operationalization e-service quality dimensions / attributes identified as relevant to digital libraries. The first draft of the questionnaire was pretested by a group of experts and also in several focus groups sessions. After that, the questionnaire was submitted to a pilot study to assess the internal consistency of the measures in a sample of convenience consisting of 77 potential users of digital libraries. Once these two phases aimed at the development of a reliable instrument were completed, we initiated the empirical study. After making a public invitation to participate in study through the digital library website and emailing a personal invitation to all digital library potential users. Responses to the questionnaire were directly stored in a central database. The Figure illustrates the process followed to develop, purify and validate these measures. Psychometric measure validation followed the recommended procedures: Internal consistency of the initial set of measures (Cronbach’s alpha). Identification of factorial structure with an exploratory factor analysis, Internal sub-scales consistency (Cronbach’s alpha) and Confirmatory Factor Analysis (CFA)
- The sample was composed of 1.708 valid answers, with the following characteristics: 51.7% female; 4.8% had less than 20 years old; 36.4% were between 20 and 30 years old, 27.8% between 31 and 40 years old, 22.8% between 41 and 50 years old, 8.1% between 51 and 65 years old and 0.1% had over 65 years old. In what concerns their situation / occupation, 6.9% were researchers, 56.7% teachers, 16% students of postgraduate studies, masters or doctoral degrees, 2.7% doctors or other health professionals, 12.5% senior technicians and 5.2% had any other occupation.
- In order to determine service quality dimensions, we carried out an Exploratory Factor Analysis (EFA). According to the procedures suggested by Cook, Heath and Thompson (2000), the factorial analysis focused on a perceptions scale. Using Kaiser’s criterion (“Eigenvalue”>1), we extracted three components which explained 57.0% of total variance; two items were eliminated for not complying with the established criteria. Table shows the saturation matrix after Varimax rotation. Those three components that were extracted reflect the three quality dimensions relevant to digital libraries: Efficiency Competitive advantage Information adequacy In what concerns the initial theoretical preposition, there is an “integration” of items regarding Information control within the Efficiency dimension and a reaffirmation of the other dimensions - Competitive advantage and Information adequacy .
- The factorial structure found is the quality of service construct. The hypothesis that the variance of these common factors (or dimensions) is due to a higher factor (or construct) was tested using the second order confirmatory factor analysis of. The second-order Service Quality Model with the three first-order factors (Efficiency, Competitive advantage and Information adequacy) is represented on Figure As you can see in this slide the structural model has a good fit
- Considering the strategical groups that were previously identified, the research team developed four web-survey versions to evaluate the different gaps: dlQUALutil (users); dlQUALprof-ID (LIS Professionals); dlQUALresp (library managers of consortium members); dlQUALdir (top managers). After disseminating the project, a web-survey was deployed via a link on the digital library consortium website. The development of the complete Digital Library Service Quality Model was based on a sample of 1708 one thousand seven hundred and eight users, 259 two hundred and fifty-nine LIS Professionals and 50 fifty library managers of consortium-members.
- The Graphic represents the zone of tolerance for each of the twelve questions regarding service quality. For each question, the pink square represents the average of the desired service, the blue diamond the minimum appropriate level and red triangle the perceived level of service. The results demonstrate that apart from the Competitive advantage dimension, the perceptions are lower than the minimum service level required, which means they are below the zone of tolerance. It is clear that users acknowledge the digital library’s competitive advantage over other information resources, recognizing that its search results have greater scientific credibility than other sites; its electronic services are easier to use then going to the library and searching the digital library website is easier than searching in printed materials
- GAP Analysis The digital library service quality model includes the strategic inter-group gaps evaluation embodied in the Knowledge gap , Perception gap , Service quality gap and Communication gap . The Knowledge gap This gap refers to the deficit of knowledge that library managers of consortium members (B1) and LIS professionals (B2) have towards users’ (U) needs and expectations. The Knowledge gap is monitored through the analysis of the discrepancies between these three groups expectations (see Graphic 2). Concerning Efficiency items, it appears that both library managers and LIS professionals have higher expectations that digital library users in terms of the minimum adequate service level. In turn, LIS professionals are the group that shows lower levels of minimum service requirement for the adequate and desired service in all digital library service quality dimensions. Regarding the Information adequacy dimension, users are those with higher expectations towards minimum appropriated and required services.
- Perception Gap The Perception gap represents the discrepancy between the perceptions of library managers and LIS professionals (B1 and B2 towards U). This gap, like the previous one, is monitored by calculating the differences between the perceptions of these three groups - B1-U and B2-U) The consortium library managers, assess the digital library performance in a more favourable way than users in what concerns quality dimensions like Efficiency and Competitive Advantage
- Service Quality Gap The data analysis of this gap (see Graphic 4) shows that digital library users have lower Measures of Service Adequacy (MSA) than the other groups in the Efficiency and Information adequacy dimensions. That is, users perceive the digital library services as lower than their expectations of minimum service adequacy in terms of Efficiency and Information adequacy and as superior in what concerns to Competitive advantage . Regarding the desired service gap, it appears that digital-library users show a greater discrepancy (between their service perceptions and their expectations of desired service) at the Efficiency level .
- The exploratory and confirmatory factor analysis identified three dimensions. In fact, the Information control dimension considered in the theorical model was integrated into the Efficiency dimension. The dlQUAL scale is a valid and useful instrument to evaluate the quality of services offered by digital libraries to stakeholders and it can provide valuable performance information to digital libraries’ decision-makers. The different versions of the dlQUAL also permit to evaluate other stakeholders’ service quality perceptions.
- Moreover, the full Digital Library Service Quality Model is a useful instrument to check the critical points or gaps that are related to organizational deficiencies. I f these gaps are regularly monitored, it is possible to implement adequate measures to correct these critical points and improve service quality.