Internet Channel Evalution


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Internet Channel Evalution

  1. 1. ARTICLE IN PRESS International Journal of Information Management 24 (2004) 473–488 A framework for internet channel evaluation Stephen F. KingÃ, Jung-Shiuan Liou Leeds University Business School, University of Leeds, LEEDS LS2 9JT, UK Abstract The Internet is becoming an increasingly important and pervasive channel to market for many organisations. Despite its importance, and the continued pressure to justify IT expenditure, few organisations undertake comprehensive channel evaluation. Market leading evaluation firms provide technical and operational metrics for their clients such as the number of hits per page and site response times, but more sophisticated concepts such as user value and long-term business benefits remain underexplored in practice. In contrast, there is a growing academic literature on channel evaluation. Many frameworks and metrics have been proposed recently. This paper brings theory and practice together by synthesizing existing frameworks proposed by academics with those used by the market leaders in Internet channel evaluation. The resulting framework has two levels—a business- and a user-level. The framework is validated by Internet consultants, channel managers and channel users in three different sectors: retail, financial services and higher education. The framework is refined following the validation in response to the need for a simpler, more usable set of metrics. The outcome is a framework split into three ‘‘sets’’. Set A constitutes the foundation stone of an Internet channel evaluation programme and consists of a core set of objective user-level metrics. Set B contains a further set of more sophisticated user-level metrics. Set C addresses business-level metrics, which enable the long-term contribution of the Internet channel to be evaluated. r 2004 Elsevier Ltd. All rights reserved. Keywords: Internet channel evaluation; Electronic business; Metrics; Performance measurement framework; Information system assessment ÃCorresponding author. Tel.: +44-113-3434462; fax: +44-113-3434465. E-mail address: (S.F. King). 0268-4012/$ - see front matter r 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.ijinfomgt.2004.08.006
  2. 2. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 474 1. Introduction The Internet is becoming an increasingly important and pervasive channel to market for many organisations. Despite its importance, and the continued pressure to justify IT expenditure, few organisations undertake comprehensive channel evaluation. Market leading evaluation firms provide technical and operational metrics for their clients such as the number of hits per page and site response times, but more sophisticated concepts such as user value and long-term business benefits remain underexplored in practice. In contrast, there is a growing academic literature on channel evaluation. Many frameworks and metrics have been proposed recently (Straub, Hoffman, Weber, & Steinfield, 2002a). This paper brings theory and practice together by synthesising existing frameworks proposed by academics with those used by the market leaders in Internet channel evaluation to produce a new, comprehensive framework. The framework is validated and refined in the light of feedback from Internet consultants, channel managers and users. 2. Research methodology This study addresses the research question: How should Internet channel performance be evaluated in practice? The aim is to develop a practical tool for use by Internet consultants and channel managers. The research is therefore inductive, qualitative and exploratory (Robson, 2002; Saunders, Lewis, & Thornhill, 2003). The project followed three stages: conceptualisation, design and validation (Aladwani, 1996)—see Fig. 1. The first stage, conceptualisation, comprised of two steps: firstly a review of the existing literature on Internet channel evaluation; secondly a review of the market leaders in this field in order to supplement academic theory with current practice. In the second stage, design, the frameworks and metrics from the two sources are combined into a single framework. In stage three the framework is validated by way of interviews with consultants, channel managers and channel users. The consultants, from a local software development company, were interested in improving their channel evaluation capabilities. The company was in the process of developing two new Validation of the Literature Framework: Initial Version Review 1. Interviews with of Internet Final Consultants Channel Framework Case 2. Interviews with Evaluation Studies of Channel Managers Framework Market 3. Interviews with Leaders Channel Users Conceptualisation Design Validation Fig. 1. Research stages.
  3. 3. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 475 Table 1 Interviewees and the systems considered Sector Channel/system Status Consultants Channel Channel interviewed managers users Financial services Pensions system Prototype 3 1 1 Higher education Comprehensive First year in use 3 3 0 university system (marketing & teaching) Higher education University marketing Four years in use 0 3 3 system Higher education University teaching Over 5 years in use 0 4 3 system Retail Amazon Over 5 years in use 0 0 4 Total number providing feedback on framework (some interviewees 3 5 8 commented on more than one system) Internet-based systems—a pensions system and a comprehensive university system. In order to obtain a broad range of views across a variety of systems, channel managers and users of the two systems were interviewed plus managers and users of two other readily accessible university systems and Amazon (in order to test the framework against the market leading site). Table 1 describes the systems and the interviewees. Each interview comprised of two parts, firstly questions on how the channel was currently being evaluated (consultants and managers) or how it performed (users only) and secondly views on the framework content and efficacy. In total, 16 people provided feedback on the framework. Feedback from the interviews was used to refine the initial framework and to produce a final framework that is grounded in both theory and practice. 3. Internet channel evaluation theory Many kinds of Internet channel exist: informational channels, transactional channels, Intranets, B2C, B2B and more (Watson & Straub, 2002). With the Internet maturing and so many applications coming online, organisations face a challenge of how to evaluate the performance of their Internet channels. Both business and user concerns should be addressed since the Internet channel aims at serving user needs and achieving business goals at the same time (Hoffman, Novak, & Chatterjee, 1995; Schonberg, Cofino, Hoch, Podlaseck, & Spraragen, 2000; Agarwl & Venkatesh, 2002). After the dotcom crash the Internet channel evaluation market has become more operation-oriented, focusing on creating realistic financial frameworks for evaluating Internet investments (Christensen & Tedlow, 2000; Garbani, 2002; Bowen, 2003). Organisations pursue multiple objectives through their websites and through the new business processes using the Internet. From a strategic perspective, Wheeler (2002) viewed increasing customer value and building sustainable capabilities as the major goals. Furthermore, firms want to see whether their Internet investments can be used to increase revenues, reduce costs, acquire and retain customers, or generate leads (Liu & Arnett, 2000; Gomez, 2003; WebCriteria, 2003).
  4. 4. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 476 E-commerce measurement should consider strategic value and the extent that the Internet is integrated into business functions and daily operations (Turban, Lee, King, & Chung, 2000; Deluca & Richter, 2002; Straub, Hoffman, Weber, & Steinfield, 2002b). Finally, a customer- centric approach is proposed. Organisations need to identify what types of users they want to attract and what their value proposition is (Schonberg et al., 2000; Roland & Kannan, 2003). On the Internet, user experience is affected by tasks, services offered, navigation, website design, and emotional gains (Schonberg et al., 2000). Compared with traditional buyers, online consumers are generally more powerful, demanding and utilitarian in their shopping behaviour (Koufaris, 2002). Users compare the value given by the site with the costs of searching, ordering, and receiving products and services (Keeney, 1999). Therefore, the ultimate factor that determines Internet commerce success is providing a satisfactory user experience, which in turn improves customer loyalty (Roland & Kannan, 2003). The above arguments indicate that the Internet channel should be evaluated from three perspectives: strategic, operational and user-centric. 4. Internet channel evaluation practice Ferengul (2003) estimated that during the period 2003–2005 new infrastructures, application architectures, and budgetary restrictions will increase the need for Internet channel capacity planning and good systems management. In addition to employing the Internet as a channel to explore strategic opportunities, organisations are also very concerned that operational benefits are delivered too. Commercial evaluation services are expected to offer root cause analysis of availability and performance problems, which should be combined with actionable recommenda- tions in a timely manner (Garbani, 2002). More than 60 vendors offer Internet performance monitoring solutions. Keynote ( was the market leader in 2002 with about 40% of the market, followed by Mercury Interactive ( and Gomez ( (see also Semilof, 2001; Ferengul, 2003 and King, 2003). The market has developed primarily in the US but significant future growth is forecast in Europe (Garbani, 2002). Taking the three market leaders as indicative of the range of services offered, all three provide extensive technical and operational metrics (e.g. site availability, traffic volume, paths taken by users, transaction success rates and wireless performance). Channel management is supported by a range of facilities including automatic reports, fast alarms, diagnostics, benchmarking and integration with internal management information systems. Keynote surveys users for reasons why transactions were abandoned and Gomez provides information on industry trends and opportunities and advice on channel strategy. 5. An initial internet channel evaluation framework Some key themes emerge from the preceding review. The Internet channel should be evaluated from both a business (strategic and operational) and a user perspective. But few quantitative measures are proposed in the academic literature. Many of the measures remain ill defined— particularly at the business level. In contrast, commercial measurement firms take a pragmatic, bottom-up approach to measurement. They start with what can be measured, typically response
  5. 5. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 477 time, page hits, transaction success rate etc., and provide real-time information to their clients on their own performance. Some also provide benchmarking services. The commercial firms add depth to the academic frameworks by showing how certain constructs can be measured in practice. But they do not indicate any new constructs. Therefore the breadth of the academic frameworks would appear to be adequate at this stage. Drawing on this analysis, a comprehensive initial framework is proposed (Tables 2 and 3). The framework is based on a synthesis of existing academic frameworks, where overlapping and similar measures have been eliminated. Table 2 presents the business-level constructs. Table 3 presents the user-level constructs. Metrics sourced from the commercial frameworks are shown in italics. 6. Framework validation The initial framework was validated in interviews with Internet consultants, channel managers and channel users (see Table 1). The interviews were semi-structured and comprised of two parts: questions on current evaluation practice and questions on the appropriateness of the framework. 6.1. Internet consultants (discussing both the business and user-level frameworks—Tables 2 and 3) The software company does not undertake formal or quantitative channel evaluations. The consultants will give their opinions on the quality of a website, but it is a subjective judgement. They see three possible areas for channel evaluation: as a pre-sales service—‘‘scoring’’ the client’s current site; as a post-sales service—proving that the new site delivers what was promised; and as a separate product line—entering into the territory of Keynote, Mercury, Gomez and others. ‘‘It will be useful to allow us to get the target, lots of business. We can look at all the websites and give them scores from 1 to 100 and we just target ones that are underdeveloped. We definitely want to develop empirical methods to evaluate websitesy We need facts or empirical evidence that we can present to clients. In most cases, we tell clients the problems but they don’t agree. We cannot win the argument.’’ (senior consultant). The consultants suggested developing different versions of the framework of different levels of complexity. The standard version should be objective and simple, and then moving toward subjective and customised measures. Finally, they argued that the market situation in Europe is different from that in the US. Except for several big companies or big dotcoms, most firms did not consider measuring their websites therefore the benefits of channel evaluation would have to be promoted strongly to get firms to change. 6.2. Channel managers (discussing the business-level framework only—Table 2) Managers of four systems were interviewed: the pensions system and the comprehensive university system (both developed by the software company) and the university marketing system and the university teaching system (both developed by the university)—see Table 1. Firstly the pensions system. The system represents a new business model in the financial services field, therefore the channel manager is concerned more about customer needs than strategic issues at this stage. The performance of the pensions site is measured by the number of registrations
  6. 6. 478 S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 Table 2 Initial business-level framework Measures (the extent to Construct Description Literature support How measured/metrics which Internet is used to) (commercially supported metrics in italics) Benchmarking. Strategic position Using the Internet to Phan (2003), Amit and Zott Create a strategic position. ARTICLE IN PRESS Competitor analysis. achieve strategic goals. (2001), Turban et al. (2000) Maintain a strategic position. Support an innovative business model. Benchmarking Strategic Using the Internet to react Phan (2003), Hoffman et al. Time to market; speed of responsiveness rapidly to changes in the (1995). change. environment. Integration Using the Internet to Phan (2003), Deluca and Support business strategy; integrate business processes Richter (2002), Turban et al. combine business functions; both internally and (2000) reconfigure and realign externally. resources and capabilities. Referral mix. Complementarities Using the Internet to Chaffey (2002), Amit and Provide online and offline compliment other business Zott (2001), Porter (2001), complementarities. Benchmarking. channels. Evans and Wurster (1999) Support vertical complementarities (after- sale service). Benchmarking; range of Support horizontal complementarities (cross products/services cross-sold. selling). Bundle products/services. Operational efficiency Using the Internet to cut Phan (2003), Chaffey Improve daily operations; RoI; cost reduction; costs, improve productivity, (2002), Deluca and Richter integrate supply chain productivity improvement; etc. (2002), Amit and Zott activities; disintermediate time saving; frequency of (2001), Turban et al. (2000), supply chain. errors. Hoffman et al. (1995). Optimize human resources. Staff headcount.
  7. 7. Integrate Internet channel Improve management with Enterprise Management control. Systems (EMS). Communication Using the Internet to Phan (2003), Amit and Zott Facilitate communications Costs of paper and phone S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 improve communication (2001), Evans and Wurster among employees, partners calls; time saved. internally and externally. (1999), Hoffman et al. and customers. (1995) User/customer profile Increased revenues Using the Internet to Phan (2003), Chaffey Access new business analysis; increase in market increase revenues either (2002), Deluca and Richter opportunities/markets. directly, through online (2002), Wheeler (2002), share. sales, or indirectly through Amit and Zott (2001), Create profitable business Online revenue contribution; sales referrals to offline sales Schonberg et al. (2000), models. conversion rate; sales per channels e.g. call centres or Evans and Wurster (1999), branches. Hoffman et al. (1995) customer. Create sponsorship ARTICLE IN PRESS opportunities. Support customer decision- making in the transaction process. 479
  8. 8. 480 Table 3 Initial user-level framework Construct Description Literature support Measures How measured/metrics (commercially S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 supported metrics in italics) Load test; Stress test; Initial page load Availability Ease of access to the Phan (2003), Aladwani (2002), Aladwani Ease of access; access time; Validation of different platforms Internet channel for and Palvia (2002), Chaffey (2002), reliability and software; availability via search the user. Deluca and Richter (2002), Garbani (2002), Palmer (2002), Liu and Arnett engines; availability via domain name. (2000), Chaffey (2002) Website availability. Percentage site downtime; site recovery time. Response time; Page load time; Traffic Speed Channel response Phan (2003), Aladwani and Palvia Waiting time. volume; Data variation; Download time at time as experienced (2002), Chaffey (2002), Devaraj et al. network-level (DNS lookup time, route by user. (2002), Garbani (2002), Kim et al. ARTICLE IN PRESS time or redirect time); Download time at (2002), Palmer (2002), Amit and Zott web-level (GIFs, Javascript, flash (2001), Liu and Arnett (2000) content), ISP performance comparison. Content/ Quality of site Agarwl and Venkatesh (2002), Aladwani Understandability; reliability; information quality content as perceived (2002), Aladwani and Palvia (2002), relevance; adequacy; up-to- by the user. Chen and Hitt (2002), Deluca and date; usefulness. Richter (2002), Devaraj et al., 2002, Kim Depth and breadth. Benchmarking. et al. (2002), Palmer (2002), Torkzadeh Product/service Benchmarking. and Dhillon (2002), Amit and Zott differentiation. (2001), Liu and Arnett (2000) Assurance. Content check. Path trace; Clickstream analysis; Link Navigation/Ease of Ease of use as Agarwl and Venkatesh (2002), Aladwani Web page layout; sequence; completeness; Third-party software use perceived by the (2002), Aladwani and Palvia (2002), structure. validation; Streaming (multimedia) user. Chaffey (2002), Chen and Hitt (2002), measurement. Deluca and Richter (2002), Devaraj et al. (2002), Kim et al. (2002), Koufaris Design consistency & style. (2002), Palmer (2002), Torkzadeh and Users feel confident/in control; Search facilities; comparison facilities; Dhillon (2002), Amit and Zott (2001), simplicity. reversibility of actions; feedback on Liu and Arnett (2000), Nielsen (2000) status. Transaction process time. Transaction Efficiency and Aladwani (2002), Devaraj et al. (2002), Shopping effort; ease of effectiveness of Kim et al. (2002), Koufaris (2002), ordering. transactions Torkzadeh and Dhillon (2002), Amit Delivery. Accuracy of delivery process; choice of performed via the and Zott (2001), Liu and Arnett (2000), delivery methods. channel. Hoffman et al. (1995) Payment. Multiple payment mechanisms. Vendor trust. Conversion/attrition rate; frequency of User satisfaction Overall user Phan (2003), Roland and Kannan User satisfaction. complaints; online satisfaction survey. satisfaction with (2003), Agarwl and Venkatesh (2002), Conversion/attrition rate. using the channel. Chaffey (2002), Deluca and Richter Perceived value gap.
  9. 9. (2002), Palmer (2002), Wheeler (2002), Amit and Zott (2001), Evans and Wurster (1999), Hoffman et al (1995) System Help provided by Aladwani and Palvia (2002), Chaffey Customer service and support. FAQ facility; online help mechanism. responsiveness the system to the (2002), Devaraj et al. (2002), Kim et al. S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 user in order to (2002), Palmer (2002), Liu and Arnett Interactive consultancy. E-mail response rate; alternative call complete the task. (2000) centre channel available. Interactivity Site responds in Agarwl and Venkatesh (2002), Aladwani Customization/ Personalised web page facility. customised manner and Palvia (2002), Chen and Hitt (2002), personalisation. to user behaviour. Deluca and Richter (2002), Devaraj et al. Users learn from using Feedback mechanism provided. (2002), Koufaris (2002), Palmer (2002), the site. Liu and Arnett (2000), Hoffman et al. (1995) Emotional The extent to which Agarwl and Venkatesh (2002); Aladwani Attractiveness; playfulness; involvement the user becomes and Palvia (2002), Koufaris (2002), enjoyment; excitement; emotionally Torkzadeh and Dhillon (2002), Liu and challenge. ARTICLE IN PRESS involved through Arnett (2000) Feeling of participation. Online community available. interacting with the site. Number of members/subscribers; Customer loyalty The degree of Roland and Kannan (2003), Agarwl and Lock-in; likelihood of return; performance of loyalty programmes; loyalty exhibited by Venkatesh (2002), Chaffey (2002), customer retention. channel users e.g. Koufaris (2002), Palmer (2002), Amit proportion of repeat visitors; frequency repeat business. and Zott (2001), Winer (2001), of site use. Schonberg et al. (2000), Armstrong and Hagel (1996) 481
  10. 10. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 482 recorded and by user activities: ‘‘We are able to get reports every week or every day. How users have done, how they quotey We can look at the activities to see whether our site is being used well.’’ (pensions channel manager). As for the framework, all of the constructs were seen as important though customers are regarded as the present priority. However, as a start up and a pure player, she commented that some of the measures were not relevant, such as integration and offline complementarities. Secondly, the three university systems were discussed with each of the three departmental channel managers. All three managers confirmed the value of the initial framework. With reference to the business-level framework, the managers see two aspects to integration: across the functions (e.g. degree programme management and the careers service) and across all degree programmes. In terms of the functions, the university wishes to integrate programmes, the alumni network, and the careers service, which are currently operated separately, into a single web-based platform. In terms of the programmes, it will pilot by full-time MBA, but later will integrate all degree programmes. The university would like to enhance its strategic position by improving the service for students, and to keep competitive in the education marketplace. An Internet channel is fundamental to achieving this aim: ‘‘Because the students put pressure on usy they keep telling [us] why don’t you do this? Why don’t you do that? It is becoming clear to me now that we must have [an Internet channel] in order to compete in the MBA market. We have to do this, so not doing it is not an option.’’ (departmental channel manager 1). Increasing revenues tends to be a long-term effect, not an immediate outcome. Rather than the additional incomes through the use of the Internet channel, the university is more concerned with its reputation, which might be its most important asset in the education market. The university employs the Internet as a complementary tool for improving teaching quality and better serving student needs, but it should be noted that the Internet cannot replace face-to-face contact. In terms of interactivity, the university wishes to build an environment that provides personalised service and up-to-date news via the Internet: ‘‘For this website, we want an interactive site for studentsy For all MBA students, they will get something called my myMBA. They will get their diary there which has items of news, such as whether a class is cancelled, or say, you’ve got three emails from the office, or there are two jobs and you have to take a look.’’ (departmental channel manager 1). The Internet channel is utilised to better organise all activities and administrative processes to enhance operational efficiency, which is usually measured in terms of costs and time savings: ‘‘How do we best organise events through this [the Internet], which will save time and money and also improve efficiency and effectivenessy From a careers perspective, as a service for students, I need to have an Internet face for information, for managing process, for allowing people to use other useful resources, to manage their data, career searches and self-development.’’ (departmental channel manager 2). Thirdly, the channel manager for the university’s existing intranet-based teaching system was interviewed. Because the system is an integral part of the teaching process he believed that it was very difficult to calculate return on investment (RoI) separately. Furthermore, the university does not use performance measurement software and the channel manager saw little value in formal site evaluation: ‘‘All of these [metrics] are done by very subjective measures, we rely on user complaints and saying that they are doing a certain function that is taking a long time, then we have to see what is going wrong.’’ (central channel manager). To summarise, the consultants and the channel managers felt that the framework covered all aspects of channel measurement. But the interviews revealed some hitherto hidden issues. Firstly,
  11. 11. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 483 managers have different priorities based on the maturity of their channels. For example, the pensions site is newly established, so more focus is given to user satisfaction and customer loyalty to consolidate the business rather than thinking of strategic issues. As for the university departmental managers, they have had a web presence for years and the basic functions have been well developed. Now they are adopting a more strategic perspective and emphasising the opportunities for process change and new forms of student engagement. Secondly, not all constructs are applicable in all situations. For example, the pensions site is a pure Internet channel, so measuring offline complementarities is not relevant. Similarly, the return on investment for the university teaching sites depends mainly upon improving teaching quality and acquiring a better reputation. This is not easy to calculate. Thirdly, most of constructs at the business level are measured subjectively, such as the extent of integration or the effectiveness of communication. Though developing objective metrics is desirable, it is hard to assess channel performance based only on figures. Most managers are clear about what they wish to achieve through the Internet, but they do not assess performance by setting measurable goals. 6.3. Channel users (discussing the user-level framework only—Table 3) Users of the pensions site, the two established university sites, and Amazon were interviewed (see Table 1). Each interviewee was asked to reflect on their experiences of using the site in question before commenting on the content of the evaluation framework. Firstly, the four Amazon users. Before examining the framework, the interviewees stated that first impressions, including site appearance, speed, and ease of use determine whether they are willing to use a site: ‘‘Anything that takes a very long time to download, such as flash and animation, takes me away from the websitey The worst site I can think is the audio website, the cinema chain, because they have huge flashy When you get in, the navigation is terrible, sometimes it works; sometimes it doesn’t.’’ (Amazon user 1). Efficiency and data security are also fundamental concerns: ‘‘For a transactional website, I prefer the site has as few steps as possible before I place an order. And I don’t have to key in all my details every single time. If it is a website that I have used before, like Amazon, I am happy to give my credit card details.’’ (Amazon user 1). After assessing Amazon, all interviewees agreed that the user-level framework covers most of their concerns. The main reason that people prefer using Amazon is convenience, which can be quantified in terms of less shopping effort and shopping time, and competitive prices. Availability, speed, content and ease of use were seen as the four main drivers here. Emotional involvement was not seen as an important factor for Amazon because people just wanted to transact: ‘‘Emotional involvement is not so important for me. When I go shopping, I feel I don’t want to be emotionally involved, that could make me buy more than I needy I want to stay cool.’’ (Amazon user 2). ‘‘It doesn’t engage me at an emotional levely If I need that, I go to a bookshop.’’ (Amazon user 1). Interestingly, cultural differences were found when evaluating Amazon. The Japanese interviewee felt uncomfortable about paying online and disclosing personal details suggesting the need to pay particular attention to the transaction and interactivity constructs in certain countries. Secondly, one user of the prototype pensions site was interviewed. The site provides information and quotations for pensions products. Emotional involvement strongly affected the user’s willingness to follow the instructions on the site: ‘‘It’s not very exciting, I want to feel
  12. 12. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 484 something like ‘hey, I have to take a look’y Make it more personal, user-friendly.’’ (pensions user). And navigation could be made simpler. Considering the target audience of older users, she felt that the site should use bigger fonts and clear pictures (content quality issues): ‘‘For me to be able to sort out the answers from the different quotes, I don’t like going to lots of pages. y When you go to loan sites, you type the amount you want, and then it [the site] displays it [the quote]yI think I am expecting that.’’ (pensions user). The user also expected richer content, such as more literature and links to other financial websites. Again the interviewee confirmed the usefulness of the user-level framework. Thirdly, three users of the two existing university websites, the marketing site and the teaching site, were interviewed. The former is the ‘‘external’’ face of the university on the web and contains general information about the university, its courses and its research activities. The latter contains teaching-related materials for current students and is password protected. The interviewees indicated that content and ease of use are imperative for university websites. One questioned the reason for using two different websites: ‘‘The content of the [marketing] university website is very important because international students are looking for information. For example, for prospective students, they want to apply to [the university], so they need detailed information. For the current students, [it is] also important for them to find module information’’ (university user 1). However, in comparison to Amazon, users of the university sites are more patient. They will make more effort to get what they need and are willing to wait longer. Again, all interviewees agreed that the user- level framework provides effective criteria for measuring the websites. To summarise, like the consultants and channel managers, the users felt that the evaluation framework was very comprehensive. But again, prioritisation of constructs emerged as a key issue. System factors (availability and speed) and interface design factors (content and navigation) were seen as fundamental to the user experience. Transactional performance is also important. However, responsiveness and interactivity seem to have less impact and depend on individual needs. As for emotional involvement, it gets very different reactions. It was viewed as unimportant for Amazon and the university websites, but was emphasised for the pensions site. The reasons may be found in the nature of different sites and in the users’ needs. People use Amazon and the university sites for purchasing and for information seeking respectively. The interviewees felt little need for emotional engagement with these sites. However, only current students were interviewed and there are arguably differences between the emotional needs of current and prospective students. Current students have to use the teaching site for their studies, but prospective students, particularly from overseas, rely on the Internet channel as their main source of information regarding a life-changing move. They are therefore likely to feel a greater sense of emotional involvement with the channel in its role as the (virtual) face of the university. Similarly, a pension is a major issue for someone approaching retirement. They expect to be reassured that they are choosing the right product and that it will support them adequately over many years. This analysis suggests two sets of user-level metrics. The core set consists of basic constructs, including availability, speed, content/information quality, navigation/ease of use and transaction. These are the basic dimensions of channel performance. The second set contains responsiveness, interactivity, and emotional involvement. These are higher-level dimen- sions that differentiate a website from its competitors. However, compared with the core constructs, these dimensions are more abstract and their importance depends on the purpose of the site.
  13. 13. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 485 7. Discussion This study is an exploratory study drawing on the academic literature and current industry practice, together with the views of Internet consultants, channel managers and channel users to develop and validate a framework for Internet channel evaluation. The research question motivating this study is: How should Internet channel performance be evaluated in practice? From the interviews we find that the initial version of the framework effectively covers both business- level and user-level concerns. However, it can be further improved by prioritising according to the needs of managers and users. For each website certain aspects are more important than others. Hence, identifying these critical elements is imperative. The evaluation framework should help managers prioritise their investments to obtain optimal return. For example, for the university teaching site, it may not be appropriate to invest in enhancing emotional engagement because students will seek face-to-face interaction to satisfy their needs. On the other hand, for a university marketing site for overseas students, encouraging emotional participation may be crucial for successful student recruitment. Taking personalisation as another example, it may not be appropriate in countries such as Japan where users may prefer not to disclose personal details. Furthermore, organisations at different stages of Internet development have different priorities. For new sites user satisfaction is the main concern. In contrast, organisations with established channels are likely to consider strategic issues too. And, since the idea of Internet channel evaluation is still in its infancy, it is better to start with well-known concepts and solid figures. More subtle items and subjective judgments can be introduced later. For these reasons, the initial framework has been split into three sets of measures (see Fig. 2), which range from simple, objective, user-level measures (Set A) to more complex, subjective, strategic-level measures (Set C). Set A (user level) Set B (user level) Set C (business level) • • • Availability System Strategic position • • Responsiveness Speed Strategic responsiveness • • • Content/Information Interactivity Integration • • quality Emotional Complimentarities • • Navigation/Ease of use involvement Operational efficiency • • • Transaction Customer loyalty Communication • User satisfaction • Increased revenues impacts Internet User Customer Increased Channel revenues satisfaction loyalty RoI influences Fig. 2. The final framework.
  14. 14. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 486 Set A contains basic user-level measures of channel performance essential to all organisations at all stages of Internet maturity. Set B may be viewed as optional. Here, more subtle concepts such as system responsiveness, interactivity and emotional involvement are introduced. The impact of these factors depends on the nature of the sites as discussed earlier. Set C covers strategic issues. Except for operational efficiency, few measures here can be related to metrics because these constructs rely mainly on subjective judgment, benchmarks and competitive intelligence, which requires a deep understanding of market drivers and market dynamics. Set C should be used where the Internet channel is well-established. Channel evaluation should be an ongoing process because a channel that is performing better than the competition now may not be doing so in a few months time and users who are satisfied now may not be in the future. As well as presenting the three sets of measures, Fig. 2 also suggests a causal model for Internet channel RoI. Here, the dependent variables user satisfaction, customer loyalty and increased revenue are extracted from the sets and shown again at the bottom of the model. Devaraj, Fan, and Kohli (2002) and Roland and Kannan (2003) showed that e-commerce channel satisfaction influences channel preference and loyalty. Greater loyalty can, in turn, lead to more profitable customers and thus to increased revenues. Increased revenues contribute to a better RoI, assuming costs are controlled. 8. Conclusion To effectively evaluate Internet channel performance both business and user concerns must be addressed. An initial framework was developed by integrating academic and practitioner frameworks and measures. The framework was validated against the views and experiences of Internet consultants, channel managers and channel users. All three groups agreed with the range of constructs proposed. However, the interviews revealed a need to prioritise the constructs to take account of the unique circumstances of each organisation and its users. For example, whilst user-level measures are important for all organisations, they are particularly important for newly established channels. Without satisfied users, more strategic concerns such as strategic position are irrelevant—for the channel will not survive for long. Through prioritising different constructs, the framework was divided into three sets, which ranged from simple, objective, user-level measures to complex, subjective, strategic measures. Given the immature environment of Internet channel evaluation, with Europe not as advanced as the US, it is better to start with well-known concepts and solid figures, and gradually introduce more subtle items and subjective judgments as confidence in evaluation grows. References Agarwl, R., & Venkatesh, V. (2002). Assessing a firm’s web presence: A heuristic evaluation procedure for the measurement of usability. Information Systems Research, 13(2), 168–186. Aladwani, A. M. (1996). Factors that influence the performance of IS project teams: a theoretical model and empirical validation. Carbondale, USA: Southern Illinois University. Aladwani, A. M. (2002). The development of two tools for measuring the easiness and usefulness of transactional web sites. European Journal of Information Systems (11), 223–234.
  15. 15. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 487 Aladwani, A.M., Palvia, P.C. (2002). Developing and validating an instrument for measuring user-perceived web quality. Information & Management (39), 467–476. Amit, R., & Zott, C. (2001). Value creation in e-business. Strategic Management Journal (22), 493–520. Armstrong, A., & Hagel, J. (1996). The real value of online communities. Harvard Business Review, 74(3), 134–141. Bowen, D. (2003). Chasing customers through the internet maze, Financial Times, 3 September 2003. Chaffey, D. (2002). E-business and E-commerce Management. Essex: FT/Prentice-Hall. Chen, P.-Y., & Hitt, L. M. (2002). Measuring switching costs and the determinants of customer retention in net-enabled businesses: A study of the online brokerage industry. Information Systems Research, 13(3), 255–274. Christensen, C. M., & Tedlow, R. S. (2000). Patterns of disruption in retailing. Harvard Business Review, 78(1), 42–45. Deluca, L. M., & Richter, R. (2002). Putting technology to the test. Association Management, June 2002, [online] Did=000000126828181&Fmt=4&Deli=1&Mtd=1&Idx=10&Sid=1&RQT=309 [26 June 2003]. Devaraj, S., Fan, M., & Kohli, R. (2002). Antecedents of B2C channel satisfaction and preference: Validating e-commerce metrics. Information Systems Research, 13(3), 316–333. Evans, P., & Wurster, T. S. (1999). Getting real about virtual commerce. Harvard Business Review, 77(6), 85–94. Ferengul, C. (2003). Web Application Management Tools. META Group, 10 April 2003, [online] http:// [4 August 2003]. Garbani, J.-P. (2002). Market review update: Web monitoring services. Giga Information Group, a subsidiary of Forrester Research, 23 September 2002, [online] 2002-100702.pdf [9 June 2003]. Gomez. (2003). [online] [28 May 2003]. Hoffman, D. L., Novak, T. P., & Chatterjee, P. (1995). Commercial scenarios for the web: Opportunities and challenges. Journal of Computer-mediated Communication, [online] hoffman.html [18 June 2003]. Keeney, R. L. (1999). The value of internet commerce to the customer. Management Science, 45(4), 533–542. Kim, J., Lee, J., Han, K., & Lee, M. (2002). Businesses as buildings: Metrics for the architectural quality of internet businesses. Information Systems Research, 13(3), 239–254. King, R. O. (2003). Keynote: A leader in web performance management. Web Hosting Monthly, 1(5), 15–16. Koufaris, M. (2002). Applying the technology acceptance model and flow theory to online consumer behavior. Information Systems Research, 13(2), 205–223. Liu, C., & Arnett, K. P. (2000). Exploring the factors associated with web site success in the context of electronic commerce. Information & Management, 38, 23–33. Nielsen, J. (2000). Designing Web Usability. Indianapolis, IN: New Riders Publishing. Palmer, J. W. (2002). Web site usability, design, and performance metrics. Information Systems Research, 13(2), 151–167. Phan, D. D. (2003). E-Business development for competitive advantage: a case study. Information & Management (40), 581–590. Porter, M. E. (2001). Strategy and the internet. Harvard Business Review, March, 43–78. Robson, C. (2002). Real World Research (2nd ed). Oxford: Blackwell. Roland, T. R., & Kannan, P. K. (2003). E-service: A new paradigm for business in the electronic environment. Communications of the ACM., 46(6), 37–42. Saunders, M., Lewis, P., & Thornhill, A. (2003). Research Methods for Business Students (3rd ed). Englewood Cliffs, NJ: Prentice-Hall. Schonberg, E., Cofino, T., Hoch, R., Podlaseck, M., & Spraragen, S. L. (2000). Measuring success. Communications of the ACM., 43(8), 53–57. Semilof, M. (2001). Mercury moves to root cause tool?, InternetWeek, 11 December 2001, [online] http://, [26 June 2003]. Straub, D. W., Hoffman, D. L., Weber, B. W., & Steinfield, C. (2002a). Measuring e-commerce in net-enabled organisations: An introduction to the special issue. Information Systems Research, 13(2), 115–124. Straub, D. W., Hoffman, D. L., Weber, B. W., & Steinfield, C. (2002b). Towards new metrics for net-enhanced organisations. Information Systems Research, 13(3), 227–238.
  16. 16. ARTICLE IN PRESS S.F. King, J.-S. Liou / International Journal of Information Management 24 (2004) 473–488 488 Torkzadeh, G., & Dhillon, G. (2002). Measuring factors that influence the success of internet commerce. Information Systems Research, 13(2), 187–204. Turban, E., Lee, J., King, D., & Chung, H. M. (2000). Electronic Commerce: A Managerial Perspective. Englewood Cliffs, NJ: Prentice-Hill. Watson, R. T., & Straub, D. W. (2002). New horizons for ICT research in net-enhanced organizations. Working Paper. Athens, GA: University of Georgia. WebCriteria (2003). The Executive Guide to Improving Website ROI, [online] [26 June 2003]. Wheeler, B. C. (2002). NEBIC: A dynamic capabilities theory for assessing net-enablement. Information Systems Research, 13(2), 125–146. Winer, R. S. (2001). A framework for customer relationship management. California Management Review, 43(4), 89–106. Stephen King is Senior Lecturer in Information Management at Leeds University Business School. He teaches on Leeds’ MBA and executive short course programmes. His research interests are in electronic business, electronic government and IS evaluation. He is currently managing a three year project exploring the role of social networks in the development and dissemination of good practice in local electronic government. He has published in a number of journals including: European Journal of Information Systems, Information Systems Journal, Journal of Information Technology and Information and Software Technology. Before becoming an academic he worked as an IS consultant for the Rover Group and for Unisys. Jung-Shiuan Liou is Product Manager in the Storage Business Department of BENQ Corporation, Taiwan.