Healthcare enterprises are faced with a number of challenges. Among them are the needs to operate
more efficiently; to expand the scope of the enterprise; provide greater access to information from a
variety of locations and platforms (including mobile/wireless), and to greatly improve the security
and privacy of information. These challenges can, at times, seem contradictory. The response to these
challenges will necessitate many different initiatives—including security planning, creation of wide
area networks, and adoption of wireless/mobile platforms. One initiative that can address a number of
these challenges involves implementing a virtual private network (VPN) infrastructure. Virtual
Private Networks combined with greater use of the public internet can offer institutions a cost-effec-
tive means to enhance security, lower costs, and expand the accessibility of information.
Through a case study approach, this paper will seek to provide the reader with an understanding of
the underlying technology of VPNs as well as some of the issues faced in the planning, selection,
implementation, and use of a VPN. The intention is to use real experiences as a basis for illustrating
the benefits as well as some of the challenges associated with implementing a VPN. As with most
complex projects, the manner and approach utilized allowed the participants to learn valuable les-
sons, some of which will be highlighted in this paper, and thus point the way to alternative approaches
were the project to be done another time.
In order to make the applicability of this material as broad as possible, we have chosen to avoid the
focus on specific vendor names and product name. We also note that since this project was initiated at
the beginning of 2000, many of the vendor product offerings have changes since then, so excessive
focus upon specific vendor product features and function would distract the reader from gaining the
benefit of our experiences.
As we will discuss, VPNs can form a key element of overall network security. Therefore, we have
elected to omit discussions of specific configuration parameters to avoid publicizing technical details
that might aid unscrupulous individuals in attempting attacks on the security infrastructure.
Additionally, every network is different and thus specific configuration parameters will need to be
considered in light of each specific institution’s environment.
This case study is based upon a project at a major academic medical center (MAMC). The project
was initiated at the beginning of 2000. The project went through a number of different phases begin-
ning with the formulation of the plan and culminating in the rollout of the VPN to the campus com-
munity, which includes the Medical Center, in mid-2001. The rollout of the VPN is ongoing, as will
The project faced a number of significant changes in direction from original scope and intention. As
originally defined, the VPN project was tied to several other security-related initiatives, some of
which did not move along their timetable as rapidly as necessary to support the original VPN project
scope, thus necessitating a change in the VPN project scope.
Prior to the initiation of the VPN project, MAMC had a remote access environment that was prima-
rily dial-up in nature. The major means for users (approximately 10,000 users of varying levels of
activity) to gain access to the enterprise-wide network are summarized in Table 1. The majority of
users had full access to internal resources only through dial-up into a shared remote access server
(RAS), which was often bottlenecked due to capacity limitations. There was a great deal of dissatis-
faction due to the inability to gain reliable remote access to the enterprise network. Several of the
other full-access solutions were extremely expensive (Private Service and ISDN) and limited to a
small number of individuals.
Users who obtained access through one of the internet service provider (ISP) options were not able to
access some internal servers, due to access control list restrictions on key routers (one of the many
security approaches used to control unauthorized access). In a compromise of security, some key
internal systems, especially email servers had little to no access restrictions to compensate for the
poor remote access alternatives, and thus were open to full access from the internet (allowing ISP
users access, but also exposing the email servers to unauthorized users).
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 2
Table 1: Existing Remote Access Modalities
Operating the various remote access modalities was costly. Within the last year, MAMC had spent
over $100,000 replacing the core RAS system, as it moved from 28.8 kbps modems to a system based
upon 56 kbps modems. However, MAMC had no viable means to meet the growing demand for
higher speed access, as faculty, employees, and students migrated to DSL. Further compounding the
dissatisfaction, vendors of several of the services, which MAMC users relied upon, had announced
plans to discontinue offerings.
There was also concern due to the need to plan for the more stringent requirements of HIPAA (Health
Insurance Portability and Accountability Act), which at the beginning of 2000 had not yet been final-
ized. After a high-level consideration of the alternatives, and faced with mounting complaints from all
areas of the campus and Medical Center, a decision was made to acquire comprehensive remote con-
nectivity services, as a turnkey set of managed services.
REQUEST FOR PROPOSAL FOR COMPREHENSIVE REMOTE
The division responsible for the development and management of the enterprise network embarked
upon a formal Request for Proposal (RFP) process for a comprehensive remote connectivity solution
(“The Solution”). The Solution requested was for the creation, implementation, management, and
support of remote access services; a total turnkey approach. Among the major drivers of this decision
were the following:
• Rapidity of technology change in access technologies and the unwillingness of MAMC to con-
tinue to fund regular replacement of access technology.
• Cost of acquisition, implementation and maintenance of owned remote access solutions.
• Cost and personnel requirements associated with the infrastructure and user support aspects of a
self-managed remote access solution. MAMC, like many other healthcare enterprises, was
already challenged in filling many technical positions.
• Rapidly increasing demand for robust and secure remote connectivity.
• Need to assure compliance with requirements of local, state, and federal regulations, including
those, such as HIPAA, which require increased security and confidentiality of medical data.
In addition to more effectively meeting existing demands for remote access, MAMC also expected
the existing environment to become more complex resulting in an increased need for robust, high-
bandwidth, scalable, and secure remote access demand levels due to the following factors:
• Greater demand for access to clinical systems
• More use of image-rich clinical systems
• Increase in telecommuting.
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 3
• Need for off-hours work and systems support.
• Greater use of distance learning.
• Further integration of systems into the care delivery, academic environment, and teaching
At the time, the need to plan for remote handheld devices was not a major impetus, however, this has
become another driver of VPN use in many environments.
A committee of representatives from across the user community was formed to provide input on the
draft of the RFP. Once the RFP was finalized, it was issued electronically to over twenty vendors. A
Bidders conference was held to answer questions. The major sections of the RFP are listed in Table 2.
Table 2: Key RFP Sections
A subset of vendors submitted compliant responses. Based upon an objective element scoring system
(OESS), the compliant responses were reviewed and scored by the committee. The top four semi-final-
ist vendors were invited to make presentations on their proposals. Based upon clarifications obtained
from the presentations, and submitted in writing by the vendors, the field was narrowed further.
In general, the semi-finalists were major ISP or telephone companies; i.e., companies with well-
developed business lines in providing turnkey solutions, as well as some major providers of internet
Objective Element Scoring System (OESS)
To assure an unbiased basis for selection of the semi-finalist vendors, an objective element scoring
system was utilized. Each reviewer scored the RFP responses by assigning one of three grades (defi-
cient, acceptable, exceeds expectations) to each of the over 200 specific RFP response requirements.
Prior to receipt of the RFP responses, the committee had weighted each of the response requirements
(optional/nice to have, Important, Required) as well as created a distribution, totaling 100%, that
reflected the relative importance of each section to MAMC. The weighted grades for each section
were computed and the score for each section was then multiplied by the section weighting.
Separately, the pricing from each RFP response was normalized to correct for any variations in serv-
ices definition between vendors. The total multiple-year cost for each vendor was divided by the
score to identify the vendor offering the greatest value to MAMC at the lowest cost. Unlike the acqui-
sition of a commodity, the services sought had significant differences, thus purely selecting the low-
est cost vendor would have not met MAMC’s needs. The OESS provided a basis for selecting the
overall best solution, not merely the lowest cost solution.
The total cost, when normalized, were substantial. The vendors had offered a variety of different
approaches in their proposals. Most of the vendors provided a fully turnkey solution, which required
no upfront capital investment by MAMC. They also had provided, in most cases, 24-by-7 end-user
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 4
support. Overall, the semi-finalist proposals met many of the requirements for providing a solution
that required minimal active MAMC personnel involvement for the creation, implementation, man-
agement, and support of remote access services accessible from around the world. Total annual costs
exceeded $2 million at the 5000-user level.
There were a number of areas in which the vendor proposals fell short of MAMC’s requirements. One
of the key issues was support was provided for only newer model Microsoft Windows computers; this
left Macintosh and Unix systems as ineligible to access the proposed solutions. As an enterprise
where computers based on operating systems other than Microsoft Windows formed a significant
portion, solutions that did not address all major platforms were not viable. Despite concerted work by
the semi-finalist vendors, the gap between required functionality and ability to deliver was too great
for a solution to be acquired. The Macintosh issue, especially, continued to play a significant role in
the further development of the overall solution.
BACK TO THE DRAWING BOARD
After terminating the RFP process, MAMC determined that the appropriate course of action was to
internally develop the capability to acquire, implement, and manage the core VPN technology.
Concomitant with this decision, it was decided to issue an RFP for internet connectivity services. The
combination of the internal VPN infrastructure effort and the acquisition of remote internet connec-
tivity services together were designed to meet MAMC’s needs.
The internet connectivity services RFP process was conducted in a manner similar to the first RFP.
The primary respondents were internet service providers who could deliver worldwide access serv-
ices. The internet connectivity services RFP was put on a separate track from the internal VPN infra-
structure projects; thus allowing the two efforts to proceed in parallel.
SELECTING A VPN SOLUTION
Beginning in July 2000, MAMC ramped up the search for a suitable VPN solution. Having made the
decision to build a VPN infrastructure through internal efforts, MAMC surveyed the enterprise VPN
marketplace to identify the solutions from the major vendors, as well as niche vendors. Product func-
tion and feature were surveyed; analysis and comparison of specifications based upon vendor litera-
ture and direct vendor discussions were compiled; and vendor presentations were arranged.
Based upon this preliminary work, products from a number of vendors were acquired under demon-
stration agreements. Given the expected cost of entire systems, MAMC reached agreements with a
subset of the VPN system vendors to acquire loan of equipment for testing purposes.
MAMC looked at three separate approaches to deployment of VPN connectivity present in the
• Router-based solutions
• Software-only solutions that would run on Windows NT or Unix platforms
• Dedicated VPN systems.
Narrowing to one approach
Given the volume of users and need for a robust VPN solution, MAMC discarded options one and
two, based upon a combination of analysis of materials and conversations with other users of the var-
ious solution types. The desired solution would need to be capable of supporting a minimum of
5,000-to-10,000 simultaneous users. This level of traffic would overwhelm systems not designed for
this level of service. Furthermore, MAMC had determined that only by implementing IPSec using
Triple DES with 128-bit keys could an appropriate level of security be maintained. These require-
ments, plus others, essentially ruled out router-based and software only solutions.
This narrowed the field to providers of dedicated hardware-based VPN solutions. Some of the sys-
tems considered were carrier-class devices (carrier class devices had been primarily sold to internet
service providers and vendors providing secure private network services to Fortune 100 companies),
capable of supporting thousands of simultaneous users. These carrier-class devices had a good track
record of performance and the ability to provide robust support for large user populations.
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 5
Small capacity devices
Companies who recommended using multiple hardware-based VPN systems to achieve the simulta-
neous user counts anticipated represented the other extreme. In some cases, this would have required
acquisition of over 100 devices to meet the same user count as a single carrier class device. These
products were also relatively untested and did not have the same level of installed based in terms of
length of service or similar sites. A key deficiency of the micro-capacity systems was the absence of
a robust and full-featured management platform; a critical component if multiple systems would be
required to service the user population.
Another shortcoming of the smaller devices was the inability to form a single pool for remote access.
Thus, users would need to be allocated to a specific device, thus it would be possible for a single
device, among the collection, to be at capacity while open tunnel sessions on other devices existed.
The smaller devices could not meet MAMC’s needs, except in one regard: Macintosh support.
Like a basilisk, the Macintosh issue kept rearing up to kill off the most viable alternatives. The only
devices that purported to deliver, in a shipping product, Macintosh support were the small devices.
However, as outlined, these devices could not meet the scalability and capacity requirements.
Furthermore, the majority of the solutions purporting to support Macintosh as remote nodes were not
able to support full IPSec compatibility. Often they provided Macintosh support through less robust
and less secure protocols (such as Point to Point Tunneling Protocol; PPTP).
The vendors of the carrier class devices had varying levels of inclination to support the Macintosh as
a remote IPSec client. The essential issue for many of the carrier class VPN vendors is the small mar-
ket share that Macintosh holds in their traditional marketplace: large international companies. As dis-
cussed below, the Macintosh basilisk continued to rear up and cause problems as the internal
development went forward.
After testing of a number of solutions, MAMC settled on a single solution for in-depth testing. The
chosen system was one of the more widely-utilized carrier-class solutions. References from a number
of internet service providers and private network service providers, plus a handful of Fortune 100
companies all provided a good comfort level that the product was robust and would meet the traffic
requirements. The vendor was unable to provide any academic or healthcare provider references, as
these class of devices had not been typically sold into these markets. While the vendor did not have a
shipping Macintosh solution, they indicated that a viable solution was available. With these assur-
ances, MAMC began in-depth testing of the solution.
To make the testing as pertinent as possible, the vendor provided a test configuration that, assuming
everything went well, would be the system to be acquired. The configuration consisted of two sys-
tems capable of 5,000 simultaneous sessions each; providing a raw capacity of 10,000 simultaneous
sessions. The systems were configured with multiple NICs to provide over 200 Mbps of network traf-
fic throughput. Encryption acceleration processors were added to assure that there would be no bot-
tlenecks in the processing of packets.
At the time, it was not possible to combine the two VPN systems in a fully-redundant fail-over pair.
However, with two systems and the necessary software, it would be possible to assure that failure of
a single VPN system would not result in the total loss of VPN connectivity. Later software develop-
ments will permit this improvement in fault-tolerance.
Table 3 summarizes some of the key features and functions of the solution chosen for in-depth test-
ing. In total, including the hardware plus installation assistance, training, management platform, and
maintenance, the cost of the tested solution was budgeted at just under $500,000.
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 6
Table 3: Key Solution Features
Given some of its specific requirements, MAMC agreed to test beta software on both the core VPN
system as well as for the remote Windows and Macintosh clients.
Phase One Testing
MAMC define several phases for the process of in-depth testing. The first phase was internal to the
network services division and included installation of the system, creation of custom configured dis-
tributable client software (using the vendor supplied toolkit), and testing from a variety of remote sys-
tems across a variety of ISPs. Phase One testing focused on Microsoft Windows-based remote clients
only, as the primary purpose of Phase One was to test the core VPN server configuration.
The largest issue that arose in Phase One testing was related to inability to connect with the VPN
server over the largest consumer ISP. Further research, during Phase Two isolated this problem to
incompatibility with older versions of the proprietary ISP client. Using the latest version of the pro-
prietary ISP client was found to resolve this interoperability issue.
Phase Two Testing
After about four weeks of successful Phase One testing, the test was expanded to volunteers from
across the enterprise. These were mostly savvy computer users. The primary foci of Phase Two were
to identify major issues with the remote client software, design of the customized self-install package,
and begin to develop user instructions. MAMC also worked to identify issues with specific ISPs and
access to various departmental servers across the enterprise.
Each week, MAMC added five to twenty users to the test, so that at the conclusion of this test period
over 200 users were regularly utilizing the VPN. Major lessons and activities from this phase
included the following:
• Issues with Windows NT and Windows 2000 that required configuration of the remote client as a
service, not an application
• The need to cancel the initial Windows login when booting the computer to permit Windows NT
server authentication to occur
• Changes necessary in the custom self-installing client
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 7
• Exploration of the ability to provide group-level controls
• Active testing of the fail over approach
• Development and refinement of user instructions
• Introduction of a RADIUS server infrastructure, moving authentication off of the VPN server itself
• Expanded communication with user departments across the enterprise.
• Testing of multiple Macintosh clients and close work with vendor staff on beta Macintosh client
Overall, Phase Two was highly successful. Close support from vendor engineers was key to further
education of MAMC lead network engineers and resolution of issues as they arose. Support for
remote Macintosh systems continued to be the one sore spot in the process.
Despite months of presentations and directed messages, the network division took a great deal of
flack due to lack of support for IPX and AppleTalk protocols. The VPN solution had been designed
and scoped from day one to be IP-only. However, this became a major issue as more individuals from
across the enterprise became part of the test group. The feedback lead MAMC to accelerate its plans
to initiate a separate project to actively move the core network to an IP-only network.
Entering Phase Three most of the technical issues had been identified and resolved; some of which
had to be resolved through new versions of the core VPN server and remote client software. Phase
Three was concentrated on building links from the internal Help Desk software to the RADIUS
Authentication server. The goal was to allow authorized end users to request a VPN User ID through
the Help Desk web interface, and be issued their User ID (after review and confirmation by Help
Desk staff of the web-submitted ticket) along with being sent the requisite client software and instruc-
tions. Due to issues with changes to the Help Desk system, full automation was not implemented, and
thus some Help Desk staff involvement and manual processes are still required.
With the exception of a total power outage that took down all of the core network equipment and a
planned outage for the relocation of the VPN server equipment, the core VPN servers did not have an
outage during the nine months of testing. The core VPN infrastructure proved extremely robust.
Macintosh issues aside, the most significant remote client issues revolved around Windows 2000;
some of these issues were addressed by the next release of the remote client software. Others were
addressed as technical staff within user departments and the central IT division gained experience
with Windows 2000. Remaining Windows 2000 issues have been rolled into the enterprise Windows
2000 rollout project as they appear to be related to the manner in which the current Windows NT
domain structure has been configured.
STATUS AND NEXT STEPS
As of today, a large number of Windows-based users are utilizing the VPN infrastructure. MAMC is
still working with the vendor in the testing of the Macintosh client. The vendor has provided clearly
exceptional support in the internal development of a Macintosh client after multiple attempts to
implement third-party Macintosh IPSec solutions failed.
Remote Branch Offices
MAMC prioritized the implementation of remote client support. The next major effort is being
devoted to developing branch office connectivity. This will allow the elimination of hundreds of
point-to-point circuits (mainly T-1s, though some Frame Relay and ISDN). By utilizing a smaller
VPN server at remote sites, MAMC will be able to swap an expensive point-to-point circuit for high-
speed DSL internet access along with the VPN infrastructure to provide remote connectivity to owned
remote sites. Depending upon distance preliminary estimates indicate an operating cost savings of
several hundred dollars per month per remote site.
Application Vendor Support
Like many other healthcare enterprises, MAMC has an enormous number of dedicated modems con-
nected to vendor-supported application systems. Not only do these modems pose a security weak-
ness, but the cost of dedicated phone lines is significant. Through the use of group definitions on the
VPN server, MAMC will be moving vendor access to these application platforms to the VPN. By cre-
ating appropriate groups on the VPN system, MAMC will be able to restrict the vendor to accessing
only the permitted systems through the VPN. This approach has additional benefits: vendor support
personnel will have higher speed access to the application platforms through the VPN, rather than
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 8
being limited to dial-up speeds; without the limitation of a single modem and phone line, multiple
vendor support personnel will be able to work in parallel; and the logging inherent in the VPN will
permit improved tracking of vendor access to supported application platforms.
IF WE HAD TO DO IT OVER AGAIN
Obviously, any process as far reaching as the MAMC remote connectivity project has been provides
a plethora of lessons. Some of the more significant ones include the following:
• Vendor support is key! MAMC benefited greatly from a very dedicated vendor technical support
team. They exhibited true partnership in working with MAMC personnel to identify problems and
help resolve them. Enterprise VPN systems are complicated and it is difficult for any customer to
master the entire scope of a complex product without partnership with the vendor.
• Ongoing communication with key end-users representatives is crucial to maintaining focus and assur-
ing that buy-in occurs. At several junctures in the overall project, the ability of influential department
heads to push for continued funding of the VPN initiative helped keep the project on track.
• Managing end user expectations is key to controlling the testing process. MAMC had some leakage
during the testing phases with eager users not in the test group trying to sneak into the test group.
This confounded the testing process and log analysis until all of the rogue testers were identified.
• The average end user does not understand and does not want to understand the technical details. If
their modem is not properly configured or their ISP is experiencing service problems, “it must be
the VPN that is broken.” While the core VPN infrastructure was up virtually all of the time,
remote-side problems occasionally snowballed into a perception of infrastructure problems. The
best solution is to keep all Help Desk and user support personnel informed about known problems
and continually reinforce the uptime statistics for the core VPN systems.
• Assuming that just because no one has reported a problem that everything is fine may be mislead-
ing. Despite all of the testing phases and requests for status reports, some problems experienced
by users that should have been caught earlier slipped into subsequent phases. The Windows 2000
differences were not caught as early as they could have been.
• Despite best efforts, obtaining support for platforms outside of the mainstream can be a challenge.
Macintosh computers from a large percentage of systems within academic institutions but a neg-
ligible portion of the overall “corporate” systems. Recognize that persistent and early effort will
be required to gain vendor support of these marginalized platforms1.
• From the time that the internal VPN project was initiated until the rollout to the enterprise many
of the vendor offerings changed. The chosen vendor maintained a consistent product line and
MAMC expects that it would make the same selection decision today. However, much of the rest
of the market has improved its offerings and this lead to a number of the vendors, which had not
been selected, trying to reverse the decision based upon products that had not been previously
available. Educating Senior Management about the costs of continually switching horses is some-
thing that must be integrated into the project plan.
• MAMC had originally planned to have the rollout to the enterprise of the internal VPN coordi-
nated with the availability of the internet connectivity services (via the RFP). However, the inter-
net connectivity RFP process significantly lagged the internal VPN project and thus MAMC was
unable to rollout a full solution with a single point of contact; thus requiring end users to sepa-
rately obtain an ISP account. This has raised the overall cost of the remote connectivity solution,
as MAMC has not been able to leverage the volume discounts. This disconnect has also made the
rollout process more complicated for end users. However, the decision to provide available VPN
functionality, even without the ISP component, has been beneficial.
• Phase rollouts are the best way to proceed. Early in the process, MAMC determined that it would
rollout functionality incrementally. Thus Windows 95/98 platforms were support first, then NT,
with Windows 2000 still in limited support. Though this has caused a fairly high level of negative
feedback from those users whose remote desktop platform were not initially supported, it allowed
a controlled process that allowed incremental improvement of the overall user experience.
Seeking a Big bang supporting all platforms at the same time would have been a disaster.
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 9
Overall, the VPN is a significant advantage for the specific institution in the case study. It has
enhanced remote access and thereby supported increased telecommuting, as well as extended the
workday for individuals (both a positive and a negative). The VPN has also allowed the replacement
of many expensive point-to-point links, supported improved access for remote offices, while lowering
costs. While a formal cost study has not been performed, the original plan for a payback period of less
than two years appears realistic.
VPNs are a significant advantage as institutions look to better secure their off-campus links. In the
world of wireless/mobile devices, the VPN also enables secure communication over the airwaves,
thereby enhancing access and protecting patient privacy. VPNs can also prompt the institution to look
at holes in their networks and develop a plan to close them. Many industries have derived benefits
from VPN technologies. Healthcare has not yet harnessed the same level of benefits. The technology
is well developed and deployed, and thus healthcare has a stable base from which to draw for tech-
nology and expertise in VPNs
This is a reflection of reality, not a comment on the suitability of Macintosh computers for any
purpose. In the specific case of MAMC, the VPN vendor was more forthcoming and involved in
identifying a solution that was the vendor of the Macintosh systems.
2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 10