Email/Calendar Microsoft Exchange 2003 End User Pilot Survey ...
University Information Systems
Central Administration IT (CAIT) Email & Calendaring Microsoft End User
Pilot Survey Summary
February 2, 2005
Author: Susan DeLellis, UIS
CAIT released a Request for Proposal (RFP) to vendors in early June, 2004 to evaluate and
procure a new Email and Calendaring System for CAIT and its customers. Two vendor finalists
were chosen out of eight to take to the next level of evaluation which included on-site back end
testing as well as an in-depth end-user production pilot. These two finalists included IBM, with its
Lotus Notes/Domino solution and Microsoft’s Outlook/Exchange 2003.
The CAIT Email/Calendar project team solicited approximately 70 end users to participate in the
Email/Calendar pilot for each vendor solution. The end user community represented staff and
students from a number of CAIT departments and schools, including UIS, OHR, President’s
Office, FAS, GSD, HSPH, Divinity School and the Law School. End user participants were asked
to utilize the pilot email/calendar system as their production email system for a period of at least a
week. An informal training and information session was provided to the pilot members prior to the
go-live pilot date, with full documentation and installation instructions. In addition, a production
help-desk service was provided by the UIS helpdesk to provide trouble ticketing work flow and
additional help to pilot users.
At the end of the formal pilot period, each end user was encouraged to fill out a twenty question
on-line user survey to express the level of satisfaction with the Email/Calendar solution provided
during the test. This document summarizes the results of the on-line survey for the Microsoft
Exchange 2003 Pilot. A separate evaluation is also provided for the IBM Lotus Notes/Domino.
For more information about this survey or the Email & Calendaring project, please contact Susan
DeLellis at firstname.lastname@example.org .
Table Of Contents:
Summary Statistics of the Survey………………. 2 - 10
Copy of the Survey (html Doc). …………………. 11
University Information Systems
EMAIL/Calendar Microsoft Exchange 2003 End User Pilot Survey
Pilot Customer Base Surveyed: 70
Number of Respondents: 22
Response Rate: 31%
Staff Members 81.8% Central Admin: 54.5% HSPH: 0.0%
Students 18.2% FAS: 31.8% HDS: 0.0%
Faculty 0.0% Law School: 9.1% GSD: 4.5%
• Primary Email Client in Use Today
o 45.5% Eudora
o 18.2% Microsoft Outlook
o 4.5% Mac OS X Client
o 20.0% Microsoft Outlook Express
o 0.0% Web Client
o 31.8% Other – please specify- (5-Mozilla/Thunderbird, 1-forwards to gmail,1
• Primary desktop/laptop operating System Environment in Use Today
o 86.4% Windows 2000 or XP
o 9.1% Mac OS X
o 0.0% Linux
o 0.0% Windows 98 or Older
o 4.5% Other – please specify (Solaris)
• Primary Calendar system/client in Use Today
o 72.7% Meeting Maker Native Client
o 4.5% Microsoft Exchange Outlook Client
o 13.6% Do not currently use a Calendar System/Client
o 9.1% Other (Outlook with PDA)
The following responses relate to the recent testing of the Microsoft Exchange 2003 Solution:
• Type of Non-Web Client used in the pilot testing
o 86.4% Microsoft Outlook 2003 for Windows
o 0.0% Microsoft Outlook 2003 for Mac Os X
o 13.6% Other (Thunderbird and SnapperMail (email), Microsoft Entourage and
Apple iCal (calendaring UIs)3- used web client only)
The following relate to the Microsoft Exchange 2003 solution features and functionality:
• Level of Satisfaction with the functionality and use of the Microsoft Exchange 2003
System using the Native Client
Very Satisfied Somewhat Dissatisfied Very
Satisfied Satisfied Dissatisfied
Customization of Client
10.0% 55.0% 30.0% 5.0% 0.0%
40.9% 54.5% 4.5% 0.0% 0.0%
Message Formatting 36.4% 50.0% 13.6% 0.0% 0.0%
Contact Lists 23.8% 61.9% 14.3% 0.0% 0.0%
Search Functions 15.0% 50.0% 20.0% 15.0% 0.0%
Message Filters 30.0% 40.0% 20.0% 10.0% 0.0%
Folder Management 38.1% 47.6% 9.5% 4.8% 0.0%
Task Management 15.0% 70.0% 10.0% 5.0% 0.0%
Distribution Lists 5.3% 68.4% 26.3% 0.0% 0.0%
Meetings and Apt.
10.0% 75.0% 15.0% 0.0% 0.0%
Calendar Proxy Mgt. & 5.3% 63.2% 21.1% 5.3%
Calendar Views 15.0% 75.0% 5.0% 5.0% 0.0%
Calendar Suggested 10.5% 68.4% 15.8% 5.3%
Encryption & Security 11.1% 61.1% 16.7% 5.6% 5.6%
On-line Help 35.3% 29.4% 23.5% 11.8% 0.0%
Spell Check 35.3% 41.2% 17.6% 5.9% 0.0%
• Features/Functionality of the Microsoft Exchange solution found to be most Useful
or liked the best (free text field)
Overall, UI is very intuitive. This was the best feature.
Receiving meeting invitations as email messages in Inbox; offline support; fast and responsive
in pilot; keyboard commands and shortcuts made sense
The calendar and task features were extremely easy to use, and I actually found myself using
calendar and mail in one
I have used Outlook with Exchange in the past. I like the calendaring, task management and
synch ability with pocketpc devices
-Outlook 2003 strong UI -Good Exchange performance -Nice message flagging -Excellent
parity with OWA on Internet Explorer -Search folders - very nice feature
In the above questions, you need to add a selection for Did Not Try. There are several that I
can't respond to because I didn't use them--calendaring, for instance. I have no use for
calendaring, and therefore I have no feedback.
The feature integration and intuitive design of the client made it very easy to use.
I like the way Outlook is fully integrated into the Windows environment, including browser, o/s
and MS office.
I really liked the Web Client. It was very easy to use and when using IE, I liked the functionality
of drag and drop. At times I forgot I was using the web client all together.
Calendar announcements in email
Outlook was easy to use on an intuitive level. The help was informative and usually solved
whatever issue I had without having to involve the Helpdesk or another third party.
In-windows dragging and dropping, fast response time and synchronized calendar - with
Outlook alone, the calendar is only on one computer.
Overall a good interface and easy to use; I think our customers will find it much easier than
I liked how the Inbox was visually organized. Also liked how a new message resulted in a "pop
up" in the bottom corner of the screen, which then faded away.
Easy and intuitive to use. Task to-dos have more functionality than we have today. I like the in-
line spell check
I found it to be much more intuitive to use that Domino. I had no trouble figuring out how to
organize and customize.
I know I'm in the minority, but I like what MS has done with the UI in Office 2003 in general
despite being generally critical of Microsoft. Perhaps owing to student familiarity with other MS
products, my guess is undergrads would have a much easier time with Outlook than with Lotus.
Outlook's pref settings are also more coherent, and the web client is impressive.
The preview pane is nice, and I do like the ability to switch back and forth between mail,
contacts, calendar etc. from the main screen. The calendar is very intuitive, and I can definitely
see the advantages of an office having access to a system like this.
The MacOS interfaces for calendaring through Exchange are much more usable and there are
more options available. The IMAP email capabilities of Exchange aren't distinguishable from
other IMAP servers, including the existing Harvard one. I used my regular email client to
access the Exchange email server, so there was no difference in UI experience.
• Features/Functionality of the Microsoft Exchange 2003 solution found to be most
difficult to use or liked the least (free text field )
Entourage for the Mac does not have parity with Outlook for the PC. Non-industry standard alt-
F4 to close windows. No integrated chat native to Exchange.
Scheduling options are rather complicated; menus in full Outlook client were more complex
than clients may be used to; more difficult (Web client only, perhaps?) to view full headers;
special characters like ampersand, etc. cause messages not to be viewed
Well, I use Gmail, and the sort of sorting that gmail does I feel is optimal. But for a standard
conventional email client, this one is fair.
it was kinda hard to customize everything to my satisfaction
It would not let me send emails as a different account name but I suspect that was just because
the pilot addresses where on a different server in comparison to the other email accounts I
Because the most important use for me is the ability to work on somebody else's email, I
couldn't test the features that I really needed to test. It is critical that I be blind copied whenever
xxxx sends mail from one of us computers, and this option was not available. Also, the ability to
toggle with ease between xxxx’s name and my own when sending email seemed complicated.
This is a problem.
Delegation/Proxy wasn't as nice as MM. There is no listing of users who have granted right to
you when trying to open a calendar.
As an existing Outlook user I really didn't have trouble adapting to an Exchange back end.
Nothing at all.
Setup of multiple accounts
I was only able to access one other person's calendar. I'm not sure if that was an issue with
user preferences or the way that the pilot was set up. It would also have been nice if instant
messaging was fully functional. As it was I could not access anyone else to instant message.
The instant messaging function is not that intuitive to use.
Web Access has limited functionality, and search folders or filters don't work for IMAP servers.
Took up WAY too much screen real estate despite tweaking. I like to have email open in one
window and do *work* in another window sometimes and this was difficult. Continue to have
I'd like to be able to customize my dashboard more.
I could not figure out how to make it automatically remember the order in which I like to display
my incoming mail. Sometimes it would revert back to the default settings. I would have liked it
to automatically display the contents of the next email after I've deleted the email message
It's a resource-intensive package even in light of its functional breadth. It's occasionally slow. It
defaults to HTML messages. Its attachments are often not receivable for non-Outlook users
(although I didn't personally hit this problem, winmail.dat). It seems like bad news if you want to
check multiple email accounts. It should auto-delete deleted messages on exit, to conserve
I used this very little, so I can't comment on it too much, other than to say that I don't like not
being able to restrict my email viewing to text-only (or at least I couldn't figure out how to do it.)
That's a security issue for me.
The Exchange IMAP service seems a bit slower than others that I use, including the current
Harvard IMAP service.
• Overall level of Satisfaction with the Microsoft Exchange 2003 service features and
o 27.3% Very Satisfied
o 59.1% Satisfied
o 4.5% Somewhat Satisfied
o 9.1% Dissatisfied
o 0.0% Very Dissatisfied
The following responses relate to the Microsoft Exchange 2003 Experience and Ease of Use:
• Length of time taken to “learn” the Microsoft Exchange 2003 system & features
o 59.1% Less than one hour
o 13.6% About one hour
o 22.7% About two hours
o 4.5% Other – (already familiar)
• Level of Intuitiveness or Ease of Use of the Microsoft Exchange 2003 system
o 47.6% Very Intuitive
o 19.0% Intuitive
o 33.3% Somewhat Intuitive
o 0.0% Not Intuitive at all
• Experience with Microsoft Exchange 2003 system & client compared to current
email/calendar system & client in use at Harvard today
o 0.0% Much easier to use
o 40.9% Easier to use
o 40.9% About the same
o 18.2% Not as easy to use
• Overall Satisfaction with the Microsoft Exchange 2003 solution Experience & Ease
o 36.4% Very Satisfied
o 45.5% Satisfied
o 13.6% Somewhat Satisfied
o 4.5% Dissatisfied
o 0.0% Very Dissatisfied
The following responses relate to the Microsoft Exchange 2003 Web Client called OWA:
• Satisfaction Level with the features & functionality of the Microsoft Exchange 2003
o 28.6% Very Satisfied
o 47.6% Satisfied
o 9.5% Somewhat Satisfied
o 0.0% Dissatisfied
o 0.0% Very Dissatisfied
o 14.3% Did not use the OWA Web Client during Pilot testing
• Features/Functionality of the Microsoft Exchange 2003 Web Client (OWA) found
most useful or liked the best (free text question)
Was very intuitive.
Very fast and responsive; very good interface within IE. Overall a much better Web client than
The calendar and task features were extremely easy to use, and I actually found myself using
interface was very nice
Very similar to the actual Outlook client which is nice
The experience was very similar to the native client which made it very easy to learn.
I like the fact that OWA is almost exactly the same as Outlook.
It was almost exactly like the full client itself. Very easy to use and navigate.
To be able to easily access my calendar remotely from my Mac at home was the best feature.
I like that it mostly resembled the native client.
Look & Feel is very similar to the native client. I couldn't figure out how to view and change
proxy delegations on the web client for Calendaring. This would be a big drawback of the web
client. Spell check works nicely. On-line help is good too.
I did not use the Web Client as much as the desktop version. However, I really liked that it
seemed to look and behave very much like the desktop version.
I didn't use it much, but I was impressed with its speed and the carryover of features. It also
mirrors the interface of the software client, which will be helpful for student types.
It seemed fairly instinctive, I mostly used the basic client via Firefox, and didn't have any trouble
navigating or reading emails, other than the subject line restrictions.
• Features/Functionality of the Microsoft Outlook Web Client (OWA) found most
difficult to use or liked the least (free text question)
Wasn't fully functional on the Mac.
Initial page could sometimes take a while to load; compatibility with non-IE browsers not as
I couldn't use premium with mozilla firefox it took eons to load (seriously)
It would not let me change my password - pop up dialog would come up server not found no
matter which web browser I used.
Nothing in particular, other than in general I don't like Web clients.
Nothing at all.
Set up of new account
Because I have a Mac at home I really missed the drag and drop functionality available with the
Limited filtering was annoying, and the discrepancy between the web client's and native client's
ability to filter was confusing. I normally use Mozilla Firefox, but I found the "basic" web client
very limited. This meant I had to have two different browsers open for the duration of the beta
test (I would never give up Firefox's extensive functionality and customizable security for IE, in
Proxy delegation to calendar.
nothing comes to mind.
It doesn't support Mozilla at all, which is a huge sin. I can imagine talking to undergraduate
users over the phone: User: "My internet explorer is broken in ways X, Y, and Z." Me: "I could
fix it, but you'd be better off using Mozilla." User: "But I heard I can't check my OWA that way."
Argh. (Perhaps it works with Firefox? I haven't tried.)
The inability to log in via the Mozilla Suite. The reduced functionality of non-IE browsers (and
the requirement to run Active X in an IE browser.) The security restrictions on subject lines
were super ridiculous and very inconvenient. The security messages the browser flashes need
a lot of fine tuning to avoid upsetting or confusing clients.
• Experience with Microsoft Exchange 2003 Web Client compared to current
email/calendar system Web Client or other Email web clients used at Harvard in the
o 33.3% Much easier to use
o 28.6% Easier to use
o 19.0% About the same
o 14.3% Not as easy to use
o 4.8% Never Used Email Web Clients in the past
The following responses relate to the Importance of Aspects of Communication Services:
• Importance of each of the following items as it relates to providing Collaboration
and Communication Services
Absolutely Very Important Somewhat Not at All
Critical Important Important Important
New Value Added
Services (IM, Web 9.1% 9.1% 27.3% 22.7% 31.8%
81.8% 18.2% 0.0% 0.0% 0.0%
More Cost Effective
13.6% 50.0% 13.6% 18.2% 4.5%
18.2% 27.3% 36.4% 9.1% 9.1%
Enhanced Security 36.4% 50.0% 9.1% 4.5% 0.0%
18.2% 50.0% 9.1% 9.1% 13.6%
client for Services
End User help desk
22.7% 40.9% 13.6% 18.2% 4.5%
18.2% 31.8% 22.7% 18.2% 9.1%
End User Training 13.6% 40.9% 22.7% 13.6% 9.1%
4.5% 31.8% 18.2% 22.7% 22.7%
Seamless Access to 4.5% 45.5% 27.3% 13.6%
Services via PDAs
Robust Web Client 54.5% 36.4% 4.5% 4.5%
for Remote Access
• Other Comments shared by Pilot Users (free text)
While I liked the product a lot, I'm concerned that many enhanced features require an additional
the web client was way too slow...otherwise, it was good
Overall Outlook/Exchange was vastly superior to Domino/Notes.
I really liked The outlook experience, partially because it is like the rest of the Office suite and
very easy to support and use.
Note: We primarily support Eudora but also have a lot of users using Silkymail or Squirrelmail
web clients Did not test distribution lists. This survey needs a column for N/A or Not tested!!!
Also we tested both web clients and the native client and there isn't much of a way to indicate
this on this survey Very disappointed that support for the palm client was not part of this test.
I came in expecting to be less than impressed with Outlook, having used Exchange in a
previous workplace through Outlook 97, but the speed and usability improvements in Outlook
2003 are quite impressive. If there's a way to check Exchange mail through IMAP/POP and the
server maintenance/cost end of things isn't any worse than the existing Unix system, Outlook
actually seems well worth looking into, at least from an undergrad perspective, provided we
could give out free copies to undergrads.
I would like to see more testing with non-MS clients with the Exchange backend to see if it
would be possible to retain heterogeneity and reduce the security risk of relying on Microsoft
software. Also, the browser restrictions are really prohibitive for students. And did anyone do
testing from slow modem connections or off-campus locations? I worry about the ability of
someone to access this from abroad. Also, the text limits on feedback are a bit stifling.
Exchange calendaring interface (Entourage) is much more intuitive and friendly than the Lotus
Notes client on the Mac. More options are available for using calendar services on Mac and
with Palm devices, compared to Meeting Maker and Lotus. Email is fine, no real advantages to
Exchange IMAP over existing service or Lotus Notes email service.
• Interest in testing the IBM Outlook Connector which allows use of a Microsoft
2003 Exchange Client to access an IBM Lotus Domino back end messaging system
o 59.1% Yes
o 27.3% No
o 13.6% No, but I would be interested in hearing about this option
Microsoft Exchange 2003 End User Pilot Email & Calendar
To view a copy of the Microsoft Exchange 2003 End User Pilot Survey questionnaire, click on the
following link. Please note this is an html document. This survey was constructed using the
iCommons Polling Tool.
Not available at this time.