Session 94 Session 94


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Session 94 Session 94

  1. 1. Proceedings • • • Slide Presentation Handouts Paper Session 94 Virtually Secure: Using Your VPNs to Enhance Access and Security AUTHOR/PRESENTER Gerard M. Nussbaum, MS, CPA, CMA Senior Manager Kurt Salmon Associates Chicago, IL 2002 ANNUAL HIMSS CONFERENCE & EXHIBITION COPYRIGHT © 2002 BY THE HEALTHCARE INFORMATION AND MANAGEMENT SYSTEMS SOCIETY. 1
  2. 2. INTRODUCTION Healthcare enterprises are faced with a number of challenges. Among them are the needs to operate more efficiently; to expand the scope of the enterprise; provide greater access to information from a variety of locations and platforms (including mobile/wireless), and to greatly improve the security and privacy of information. These challenges can, at times, seem contradictory. The response to these challenges will necessitate many different initiatives—including security planning, creation of wide area networks, and adoption of wireless/mobile platforms. One initiative that can address a number of these challenges involves implementing a virtual private network (VPN) infrastructure. Virtual Private Networks combined with greater use of the public internet can offer institutions a cost-effec- tive means to enhance security, lower costs, and expand the accessibility of information. Through a case study approach, this paper will seek to provide the reader with an understanding of the underlying technology of VPNs as well as some of the issues faced in the planning, selection, implementation, and use of a VPN. The intention is to use real experiences as a basis for illustrating the benefits as well as some of the challenges associated with implementing a VPN. As with most complex projects, the manner and approach utilized allowed the participants to learn valuable les- sons, some of which will be highlighted in this paper, and thus point the way to alternative approaches were the project to be done another time. In order to make the applicability of this material as broad as possible, we have chosen to avoid the focus on specific vendor names and product name. We also note that since this project was initiated at the beginning of 2000, many of the vendor product offerings have changes since then, so excessive focus upon specific vendor product features and function would distract the reader from gaining the benefit of our experiences. As we will discuss, VPNs can form a key element of overall network security. Therefore, we have elected to omit discussions of specific configuration parameters to avoid publicizing technical details that might aid unscrupulous individuals in attempting attacks on the security infrastructure. Additionally, every network is different and thus specific configuration parameters will need to be considered in light of each specific institution’s environment. PROJECT OVERVIEW Project Basics This case study is based upon a project at a major academic medical center (MAMC). The project was initiated at the beginning of 2000. The project went through a number of different phases begin- ning with the formulation of the plan and culminating in the rollout of the VPN to the campus com- munity, which includes the Medical Center, in mid-2001. The rollout of the VPN is ongoing, as will be discussed. The project faced a number of significant changes in direction from original scope and intention. As originally defined, the VPN project was tied to several other security-related initiatives, some of which did not move along their timetable as rapidly as necessary to support the original VPN project scope, thus necessitating a change in the VPN project scope. Technical Environment—Before Prior to the initiation of the VPN project, MAMC had a remote access environment that was prima- rily dial-up in nature. The major means for users (approximately 10,000 users of varying levels of activity) to gain access to the enterprise-wide network are summarized in Table 1. The majority of users had full access to internal resources only through dial-up into a shared remote access server (RAS), which was often bottlenecked due to capacity limitations. There was a great deal of dissatis- faction due to the inability to gain reliable remote access to the enterprise network. Several of the other full-access solutions were extremely expensive (Private Service and ISDN) and limited to a small number of individuals. Users who obtained access through one of the internet service provider (ISP) options were not able to access some internal servers, due to access control list restrictions on key routers (one of the many security approaches used to control unauthorized access). In a compromise of security, some key internal systems, especially email servers had little to no access restrictions to compensate for the poor remote access alternatives, and thus were open to full access from the internet (allowing ISP users access, but also exposing the email servers to unauthorized users). 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 2
  3. 3. Table 1: Existing Remote Access Modalities Project impetus Operating the various remote access modalities was costly. Within the last year, MAMC had spent over $100,000 replacing the core RAS system, as it moved from 28.8 kbps modems to a system based upon 56 kbps modems. However, MAMC had no viable means to meet the growing demand for higher speed access, as faculty, employees, and students migrated to DSL. Further compounding the dissatisfaction, vendors of several of the services, which MAMC users relied upon, had announced plans to discontinue offerings. There was also concern due to the need to plan for the more stringent requirements of HIPAA (Health Insurance Portability and Accountability Act), which at the beginning of 2000 had not yet been final- ized. After a high-level consideration of the alternatives, and faced with mounting complaints from all areas of the campus and Medical Center, a decision was made to acquire comprehensive remote con- nectivity services, as a turnkey set of managed services. REQUEST FOR PROPOSAL FOR COMPREHENSIVE REMOTE CONNECTIVITY SOLUTION The division responsible for the development and management of the enterprise network embarked upon a formal Request for Proposal (RFP) process for a comprehensive remote connectivity solution (“The Solution”). The Solution requested was for the creation, implementation, management, and support of remote access services; a total turnkey approach. Among the major drivers of this decision were the following: • Rapidity of technology change in access technologies and the unwillingness of MAMC to con- tinue to fund regular replacement of access technology. • Cost of acquisition, implementation and maintenance of owned remote access solutions. • Cost and personnel requirements associated with the infrastructure and user support aspects of a self-managed remote access solution. MAMC, like many other healthcare enterprises, was already challenged in filling many technical positions. • Rapidly increasing demand for robust and secure remote connectivity. • Need to assure compliance with requirements of local, state, and federal regulations, including those, such as HIPAA, which require increased security and confidentiality of medical data. In addition to more effectively meeting existing demands for remote access, MAMC also expected the existing environment to become more complex resulting in an increased need for robust, high- bandwidth, scalable, and secure remote access demand levels due to the following factors: • Greater demand for access to clinical systems • More use of image-rich clinical systems • Increase in telecommuting. 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 3
  4. 4. • Need for off-hours work and systems support. • Greater use of distance learning. • Further integration of systems into the care delivery, academic environment, and teaching curriculum. At the time, the need to plan for remote handheld devices was not a major impetus, however, this has become another driver of VPN use in many environments. RFP Process RFP A committee of representatives from across the user community was formed to provide input on the draft of the RFP. Once the RFP was finalized, it was issued electronically to over twenty vendors. A Bidders conference was held to answer questions. The major sections of the RFP are listed in Table 2. Table 2: Key RFP Sections A subset of vendors submitted compliant responses. Based upon an objective element scoring system (OESS), the compliant responses were reviewed and scored by the committee. The top four semi-final- ist vendors were invited to make presentations on their proposals. Based upon clarifications obtained from the presentations, and submitted in writing by the vendors, the field was narrowed further. In general, the semi-finalists were major ISP or telephone companies; i.e., companies with well- developed business lines in providing turnkey solutions, as well as some major providers of internet access services. Objective Element Scoring System (OESS) To assure an unbiased basis for selection of the semi-finalist vendors, an objective element scoring system was utilized. Each reviewer scored the RFP responses by assigning one of three grades (defi- cient, acceptable, exceeds expectations) to each of the over 200 specific RFP response requirements. Prior to receipt of the RFP responses, the committee had weighted each of the response requirements (optional/nice to have, Important, Required) as well as created a distribution, totaling 100%, that reflected the relative importance of each section to MAMC. The weighted grades for each section were computed and the score for each section was then multiplied by the section weighting. Separately, the pricing from each RFP response was normalized to correct for any variations in serv- ices definition between vendors. The total multiple-year cost for each vendor was divided by the score to identify the vendor offering the greatest value to MAMC at the lowest cost. Unlike the acqui- sition of a commodity, the services sought had significant differences, thus purely selecting the low- est cost vendor would have not met MAMC’s needs. The OESS provided a basis for selecting the overall best solution, not merely the lowest cost solution. Costs The total cost, when normalized, were substantial. The vendors had offered a variety of different approaches in their proposals. Most of the vendors provided a fully turnkey solution, which required no upfront capital investment by MAMC. They also had provided, in most cases, 24-by-7 end-user 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 4
  5. 5. support. Overall, the semi-finalist proposals met many of the requirements for providing a solution that required minimal active MAMC personnel involvement for the creation, implementation, man- agement, and support of remote access services accessible from around the world. Total annual costs exceeded $2 million at the 5000-user level. Process Halted There were a number of areas in which the vendor proposals fell short of MAMC’s requirements. One of the key issues was support was provided for only newer model Microsoft Windows computers; this left Macintosh and Unix systems as ineligible to access the proposed solutions. As an enterprise where computers based on operating systems other than Microsoft Windows formed a significant portion, solutions that did not address all major platforms were not viable. Despite concerted work by the semi-finalist vendors, the gap between required functionality and ability to deliver was too great for a solution to be acquired. The Macintosh issue, especially, continued to play a significant role in the further development of the overall solution. BACK TO THE DRAWING BOARD After terminating the RFP process, MAMC determined that the appropriate course of action was to internally develop the capability to acquire, implement, and manage the core VPN technology. Concomitant with this decision, it was decided to issue an RFP for internet connectivity services. The combination of the internal VPN infrastructure effort and the acquisition of remote internet connec- tivity services together were designed to meet MAMC’s needs. The internet connectivity services RFP process was conducted in a manner similar to the first RFP. The primary respondents were internet service providers who could deliver worldwide access serv- ices. The internet connectivity services RFP was put on a separate track from the internal VPN infra- structure projects; thus allowing the two efforts to proceed in parallel. SELECTING A VPN SOLUTION Beginning in July 2000, MAMC ramped up the search for a suitable VPN solution. Having made the decision to build a VPN infrastructure through internal efforts, MAMC surveyed the enterprise VPN marketplace to identify the solutions from the major vendors, as well as niche vendors. Product func- tion and feature were surveyed; analysis and comparison of specifications based upon vendor litera- ture and direct vendor discussions were compiled; and vendor presentations were arranged. Based upon this preliminary work, products from a number of vendors were acquired under demon- stration agreements. Given the expected cost of entire systems, MAMC reached agreements with a subset of the VPN system vendors to acquire loan of equipment for testing purposes. Three Approaches MAMC looked at three separate approaches to deployment of VPN connectivity present in the marketplace: • Router-based solutions • Software-only solutions that would run on Windows NT or Unix platforms • Dedicated VPN systems. Narrowing to one approach Given the volume of users and need for a robust VPN solution, MAMC discarded options one and two, based upon a combination of analysis of materials and conversations with other users of the var- ious solution types. The desired solution would need to be capable of supporting a minimum of 5,000-to-10,000 simultaneous users. This level of traffic would overwhelm systems not designed for this level of service. Furthermore, MAMC had determined that only by implementing IPSec using Triple DES with 128-bit keys could an appropriate level of security be maintained. These require- ments, plus others, essentially ruled out router-based and software only solutions. Carrier-class devices This narrowed the field to providers of dedicated hardware-based VPN solutions. Some of the sys- tems considered were carrier-class devices (carrier class devices had been primarily sold to internet service providers and vendors providing secure private network services to Fortune 100 companies), capable of supporting thousands of simultaneous users. These carrier-class devices had a good track record of performance and the ability to provide robust support for large user populations. 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 5
  6. 6. Small capacity devices Companies who recommended using multiple hardware-based VPN systems to achieve the simulta- neous user counts anticipated represented the other extreme. In some cases, this would have required acquisition of over 100 devices to meet the same user count as a single carrier class device. These products were also relatively untested and did not have the same level of installed based in terms of length of service or similar sites. A key deficiency of the micro-capacity systems was the absence of a robust and full-featured management platform; a critical component if multiple systems would be required to service the user population. Another shortcoming of the smaller devices was the inability to form a single pool for remote access. Thus, users would need to be allocated to a specific device, thus it would be possible for a single device, among the collection, to be at capacity while open tunnel sessions on other devices existed. The smaller devices could not meet MAMC’s needs, except in one regard: Macintosh support. Macintosh Support Like a basilisk, the Macintosh issue kept rearing up to kill off the most viable alternatives. The only devices that purported to deliver, in a shipping product, Macintosh support were the small devices. However, as outlined, these devices could not meet the scalability and capacity requirements. Furthermore, the majority of the solutions purporting to support Macintosh as remote nodes were not able to support full IPSec compatibility. Often they provided Macintosh support through less robust and less secure protocols (such as Point to Point Tunneling Protocol; PPTP). The vendors of the carrier class devices had varying levels of inclination to support the Macintosh as a remote IPSec client. The essential issue for many of the carrier class VPN vendors is the small mar- ket share that Macintosh holds in their traditional marketplace: large international companies. As dis- cussed below, the Macintosh basilisk continued to rear up and cause problems as the internal development went forward. IN-DEPTH TESTING After testing of a number of solutions, MAMC settled on a single solution for in-depth testing. The chosen system was one of the more widely-utilized carrier-class solutions. References from a number of internet service providers and private network service providers, plus a handful of Fortune 100 companies all provided a good comfort level that the product was robust and would meet the traffic requirements. The vendor was unable to provide any academic or healthcare provider references, as these class of devices had not been typically sold into these markets. While the vendor did not have a shipping Macintosh solution, they indicated that a viable solution was available. With these assur- ances, MAMC began in-depth testing of the solution. Configuration To make the testing as pertinent as possible, the vendor provided a test configuration that, assuming everything went well, would be the system to be acquired. The configuration consisted of two sys- tems capable of 5,000 simultaneous sessions each; providing a raw capacity of 10,000 simultaneous sessions. The systems were configured with multiple NICs to provide over 200 Mbps of network traf- fic throughput. Encryption acceleration processors were added to assure that there would be no bot- tlenecks in the processing of packets. At the time, it was not possible to combine the two VPN systems in a fully-redundant fail-over pair. However, with two systems and the necessary software, it would be possible to assure that failure of a single VPN system would not result in the total loss of VPN connectivity. Later software develop- ments will permit this improvement in fault-tolerance. Table 3 summarizes some of the key features and functions of the solution chosen for in-depth test- ing. In total, including the hardware plus installation assistance, training, management platform, and maintenance, the cost of the tested solution was budgeted at just under $500,000. 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 6
  7. 7. Table 3: Key Solution Features Given some of its specific requirements, MAMC agreed to test beta software on both the core VPN system as well as for the remote Windows and Macintosh clients. Phase One Testing MAMC define several phases for the process of in-depth testing. The first phase was internal to the network services division and included installation of the system, creation of custom configured dis- tributable client software (using the vendor supplied toolkit), and testing from a variety of remote sys- tems across a variety of ISPs. Phase One testing focused on Microsoft Windows-based remote clients only, as the primary purpose of Phase One was to test the core VPN server configuration. The largest issue that arose in Phase One testing was related to inability to connect with the VPN server over the largest consumer ISP. Further research, during Phase Two isolated this problem to incompatibility with older versions of the proprietary ISP client. Using the latest version of the pro- prietary ISP client was found to resolve this interoperability issue. Phase Two Testing After about four weeks of successful Phase One testing, the test was expanded to volunteers from across the enterprise. These were mostly savvy computer users. The primary foci of Phase Two were to identify major issues with the remote client software, design of the customized self-install package, and begin to develop user instructions. MAMC also worked to identify issues with specific ISPs and access to various departmental servers across the enterprise. Each week, MAMC added five to twenty users to the test, so that at the conclusion of this test period over 200 users were regularly utilizing the VPN. Major lessons and activities from this phase included the following: • Issues with Windows NT and Windows 2000 that required configuration of the remote client as a service, not an application • The need to cancel the initial Windows login when booting the computer to permit Windows NT server authentication to occur • Changes necessary in the custom self-installing client 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 7
  8. 8. • Exploration of the ability to provide group-level controls • Active testing of the fail over approach • Development and refinement of user instructions • Introduction of a RADIUS server infrastructure, moving authentication off of the VPN server itself • Expanded communication with user departments across the enterprise. • Testing of multiple Macintosh clients and close work with vendor staff on beta Macintosh client Overall, Phase Two was highly successful. Close support from vendor engineers was key to further education of MAMC lead network engineers and resolution of issues as they arose. Support for remote Macintosh systems continued to be the one sore spot in the process. IP-only Despite months of presentations and directed messages, the network division took a great deal of flack due to lack of support for IPX and AppleTalk protocols. The VPN solution had been designed and scoped from day one to be IP-only. However, this became a major issue as more individuals from across the enterprise became part of the test group. The feedback lead MAMC to accelerate its plans to initiate a separate project to actively move the core network to an IP-only network. Phase Three Entering Phase Three most of the technical issues had been identified and resolved; some of which had to be resolved through new versions of the core VPN server and remote client software. Phase Three was concentrated on building links from the internal Help Desk software to the RADIUS Authentication server. The goal was to allow authorized end users to request a VPN User ID through the Help Desk web interface, and be issued their User ID (after review and confirmation by Help Desk staff of the web-submitted ticket) along with being sent the requisite client software and instruc- tions. Due to issues with changes to the Help Desk system, full automation was not implemented, and thus some Help Desk staff involvement and manual processes are still required. Testing Results With the exception of a total power outage that took down all of the core network equipment and a planned outage for the relocation of the VPN server equipment, the core VPN servers did not have an outage during the nine months of testing. The core VPN infrastructure proved extremely robust. Macintosh issues aside, the most significant remote client issues revolved around Windows 2000; some of these issues were addressed by the next release of the remote client software. Others were addressed as technical staff within user departments and the central IT division gained experience with Windows 2000. Remaining Windows 2000 issues have been rolled into the enterprise Windows 2000 rollout project as they appear to be related to the manner in which the current Windows NT domain structure has been configured. STATUS AND NEXT STEPS As of today, a large number of Windows-based users are utilizing the VPN infrastructure. MAMC is still working with the vendor in the testing of the Macintosh client. The vendor has provided clearly exceptional support in the internal development of a Macintosh client after multiple attempts to implement third-party Macintosh IPSec solutions failed. Remote Branch Offices MAMC prioritized the implementation of remote client support. The next major effort is being devoted to developing branch office connectivity. This will allow the elimination of hundreds of point-to-point circuits (mainly T-1s, though some Frame Relay and ISDN). By utilizing a smaller VPN server at remote sites, MAMC will be able to swap an expensive point-to-point circuit for high- speed DSL internet access along with the VPN infrastructure to provide remote connectivity to owned remote sites. Depending upon distance preliminary estimates indicate an operating cost savings of several hundred dollars per month per remote site. Application Vendor Support Like many other healthcare enterprises, MAMC has an enormous number of dedicated modems con- nected to vendor-supported application systems. Not only do these modems pose a security weak- ness, but the cost of dedicated phone lines is significant. Through the use of group definitions on the VPN server, MAMC will be moving vendor access to these application platforms to the VPN. By cre- ating appropriate groups on the VPN system, MAMC will be able to restrict the vendor to accessing only the permitted systems through the VPN. This approach has additional benefits: vendor support personnel will have higher speed access to the application platforms through the VPN, rather than 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 8
  9. 9. being limited to dial-up speeds; without the limitation of a single modem and phone line, multiple vendor support personnel will be able to work in parallel; and the logging inherent in the VPN will permit improved tracking of vendor access to supported application platforms. IF WE HAD TO DO IT OVER AGAIN Obviously, any process as far reaching as the MAMC remote connectivity project has been provides a plethora of lessons. Some of the more significant ones include the following: • Vendor support is key! MAMC benefited greatly from a very dedicated vendor technical support team. They exhibited true partnership in working with MAMC personnel to identify problems and help resolve them. Enterprise VPN systems are complicated and it is difficult for any customer to master the entire scope of a complex product without partnership with the vendor. • Ongoing communication with key end-users representatives is crucial to maintaining focus and assur- ing that buy-in occurs. At several junctures in the overall project, the ability of influential department heads to push for continued funding of the VPN initiative helped keep the project on track. • Managing end user expectations is key to controlling the testing process. MAMC had some leakage during the testing phases with eager users not in the test group trying to sneak into the test group. This confounded the testing process and log analysis until all of the rogue testers were identified. • The average end user does not understand and does not want to understand the technical details. If their modem is not properly configured or their ISP is experiencing service problems, “it must be the VPN that is broken.” While the core VPN infrastructure was up virtually all of the time, remote-side problems occasionally snowballed into a perception of infrastructure problems. The best solution is to keep all Help Desk and user support personnel informed about known problems and continually reinforce the uptime statistics for the core VPN systems. • Assuming that just because no one has reported a problem that everything is fine may be mislead- ing. Despite all of the testing phases and requests for status reports, some problems experienced by users that should have been caught earlier slipped into subsequent phases. The Windows 2000 differences were not caught as early as they could have been. • Despite best efforts, obtaining support for platforms outside of the mainstream can be a challenge. Macintosh computers from a large percentage of systems within academic institutions but a neg- ligible portion of the overall “corporate” systems. Recognize that persistent and early effort will be required to gain vendor support of these marginalized platforms1. • From the time that the internal VPN project was initiated until the rollout to the enterprise many of the vendor offerings changed. The chosen vendor maintained a consistent product line and MAMC expects that it would make the same selection decision today. However, much of the rest of the market has improved its offerings and this lead to a number of the vendors, which had not been selected, trying to reverse the decision based upon products that had not been previously available. Educating Senior Management about the costs of continually switching horses is some- thing that must be integrated into the project plan. • MAMC had originally planned to have the rollout to the enterprise of the internal VPN coordi- nated with the availability of the internet connectivity services (via the RFP). However, the inter- net connectivity RFP process significantly lagged the internal VPN project and thus MAMC was unable to rollout a full solution with a single point of contact; thus requiring end users to sepa- rately obtain an ISP account. This has raised the overall cost of the remote connectivity solution, as MAMC has not been able to leverage the volume discounts. This disconnect has also made the rollout process more complicated for end users. However, the decision to provide available VPN functionality, even without the ISP component, has been beneficial. • Phase rollouts are the best way to proceed. Early in the process, MAMC determined that it would rollout functionality incrementally. Thus Windows 95/98 platforms were support first, then NT, with Windows 2000 still in limited support. Though this has caused a fairly high level of negative feedback from those users whose remote desktop platform were not initially supported, it allowed a controlled process that allowed incremental improvement of the overall user experience. Seeking a Big bang supporting all platforms at the same time would have been a disaster. 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 9
  10. 10. CONCLUSION Overall, the VPN is a significant advantage for the specific institution in the case study. It has enhanced remote access and thereby supported increased telecommuting, as well as extended the workday for individuals (both a positive and a negative). The VPN has also allowed the replacement of many expensive point-to-point links, supported improved access for remote offices, while lowering costs. While a formal cost study has not been performed, the original plan for a payback period of less than two years appears realistic. VPNs are a significant advantage as institutions look to better secure their off-campus links. In the world of wireless/mobile devices, the VPN also enables secure communication over the airwaves, thereby enhancing access and protecting patient privacy. VPNs can also prompt the institution to look at holes in their networks and develop a plan to close them. Many industries have derived benefits from VPN technologies. Healthcare has not yet harnessed the same level of benefits. The technology is well developed and deployed, and thus healthcare has a stable base from which to draw for tech- nology and expertise in VPNs REFERENCES 1 This is a reflection of reality, not a comment on the suitability of Macintosh computers for any purpose. In the specific case of MAMC, the VPN vendor was more forthcoming and involved in identifying a solution that was the vendor of the Macintosh systems. 2002 HIMSS Proceedings: Educational Sessions Session 94 / Page 10