course materials

1. Assignment: Healthcare IT
Assignment: Healthcare IT ON Assignment: Healthcare ITSee
attachment.pdfw7case404.docxUnformatted Attachment Preview9-303-097 REV: AUGUST
11, 2005 F. WARREN MCFARLAN ROBERT D. AUSTIN CareGroup “The good news,” reported
John Halamka, CareGroup CIO, as he opened his November 21, 2002 presentation to the
board of directors, “is that health care did not suffer.” Over the next 20 minutes, Halamka
recounted a remarkable tale that explained why CareGroup information technology (IT)
systems had completely collapsed for three and a half days the previous week, the steps
staff members and vendors had taken to recover, and how the hospital had reverted to
paper-based systems, many of which had not been used for a decade or more. Though his
story contained challenges and travails, in the end the paper-based systems and recovery
efforts had worked well. Care to some patients had been delayed, but not a single adverse
event related to the outage had been reported. Even so, there were numerous lessons
learned and some line items to be added to the IT budget; Halamka now outlined these for
the board. CareGroup CareGroup was a team of health-care professionals dedicated to
providing the best quality care to patients in a highly personalized manner. CareGroup and
its members offered a broad spectrum of health services to residents of eastern
Massachusetts in a variety of settings, ranging from worldrenowned academic health
centers and outstanding community hospitals to physician offices and community health
centers. CareGroup hospital members included Beth Israel Deaconess Medical Center in
Boston, Mount Auburn Hospital in Cambridge, New England Baptist Hospital (NEBH) in
Boston, Deaconess-Glover Hospital in Needham, and Deaconess-Nashoba Hospital in Ayer.
With more than 13,000 employees and 2,000 medical staff, CareGroup offered community-
based primary care and a wide range of specialty services close to where individuals lived
or worked. CareGroup had been formed in a three-way merger on October 1, 1996.The Beth
Israel Hospital, the Deaconess Hospital, and the Mount Auburn Hospital came together on
that day. The Beth Israel and Deaconess Hospitals, which were physically adjacent, merged
into a single hospital; every department was merged and headed by one individual (e.g., the
two surgical units were merged and a head of surgery appointed). The Mount Auburn
Hospital, located in Cambridge, reported to CareGroup management as a separate entity, as
did four other hospitals that had formerly been a part of the Pathway Network, assembled
over the previous decade by the Deaconess Hospital. (See Exhibit 1 for the organizational
chart of CareGroup and Exhibit 2 for descriptions of the different hospitals.)
_________________________________________________________________________________________________________

2. _______ Professors F. Warren McFarlan and Robert D. Austin prepared this case. HBS cases
are developed solely as the basis for class discussion. Cases are not intended to serve as
endorsements, sources of primary data, or illustrations of effective or ineffective
management. Copyright © 2003 President and Fellows of Harvard College. To order copies
or request permission to reproduce materials, call 1-800-545-7685, write Harvard Business
School Publishing, Boston, MA 02163, or go to http://www.hbsp.harvard.edu. No part of
this publication may be reproduced, stored in a retrieval system, used in a spreadsheet, or
transmitted in any form or by any means—electronic, mechanical, photocopying, recording,
or otherwise—without the permission of Harvard Business School. This document is
authorized for use only by FLORIN PANESIU (flopanesiu@icloud.com). Copying or posting is
an infringement of copyright. Please contact customerservice@harvardbusiness.org or 800-
988-0886 for additional copies. 303-097 CareGroup This merger produced a hospital group
with $1.6 billion in revenue, the second-largest group of hospitals in eastern Massachusetts.
The merger was driven by the intense competitive environment in the mid-1990s. On
December 13, 1994, the Massachusetts medical community was stunned to discover that
the two largest and most prestigious hospitals in Boston, Massachusetts General Hospital
and the Brigham and Women’s Hospital, had agreed to merge in a holding-company
structure called Partners. Assignment: Healthcare ITThe perceived pressure that this large
organization was going to be able to put on health maintenance organizations (HMOs, which
stood between individual patients and employees and the hospitals and doctors) led many
hospitals to conclude they had subcritical mass in terms of their number of “covered lives”;
in this new world, you would need a critical mass to get survival prices from the HMOs (the
more covered lives you had, the more the HMOs needed you). The hospitals felt they needed
bulked-up negotiating power to push back against the HMOs, which themselves were under
great financial pressure from companies in the region such as FleetBank and Raytheon that
were trying to drive down their medical costs. Specifically, the factors that brought
CareGroup together were: 1. The contracting power needed by the hospitals against the
HMOs 2. The possibility of developing integrated services across the hospitals that could
improve quality of care and drive down costs 3. Assignment: Healthcare ITThe need for a
strong balance sheet in a complex price war, in which there were more than 40% excess
hospital beds in the region. Because of this excess supply, some hospitals were closing their
doors. Over the next seven years, under the leadership of two CEOs, CareGroup sought to
deal with these issues. A brief summary of progress would include the following
observations: 1. It was a time of extraordinary financial pressure for CareGroup. First the
Mount Auburn Hospital, which had been profitable for over 15 years, suddenly produced a
$10 million loss and had to reengineer its operations. Just as it recovered, Beth Israel
Deaconess began losing significant amounts of money. Not until mid-2002, under the
leadership of a new CEO, Paul Levy, did it began to recover. At about that same time, the
Baptist Hospital suddenly incurred a $20 million loss, which led to the appointment of new
management there. By early 2003, these problems were largely behind CareGroup. Each
hospital in the group was headed by a different CEO than at the time of the merger, and
financial stability issues were fading into the past. 2. Assignment: Healthcare ITOperational
coordination across the hospitals had turned out to be extremely difficult because of their

3. history of independence. Financial synergy in terms of lower debt costs, however, had been
a strong feature. Similarly, joint contracting with the HMOs had been very successful. 3. An
unexpected glittering success was the development of an integrated technology system that
linked the entire group together. The system was widely touted nationally; it was
considered not only the best in health care, but also one of the very best in any industry. In
2003 CareGroup senior management consisted of a CEO/chief legal officer, a CFO, and a CIO.
In the past several years, one of the subsidiary hospitals, Waltham Hospital, had run into
difficulty, and the CareGroup Board had begun the process of closing it down. This led to a
real estate specialist stepping in and, in return for a third of the hospital’s land, providing
Waltham with additional funding that allowed it to emerge as a stand-alone organization
with strong community . 2 This document is authorized for use only by FLORIN PANESIU
(flopanesiu@icloud.com). Copying or posting is an infringement of copyright. Assignment:
Healthcare ITPlease contact customerservice@harvardbusiness.org or 800-988-0886 for
additional copies. CareGroup 303-097 On December 31, 2002, Deaconess-Nashoba Hospital
was spun off to Essent, a for-profit organization that committed significant funds to a
massive renovation of the hospital over the next two years (which CareGroup’s capital
structure did not permit). Finally, Deaconess-Glover Hospital was reorganized as a part of
the Beth Israel Deaconess Medical Center because of its strong referral pattern there. All
three of these hospitals continued to get all their IT through CareGroup’s IT organization,
which had become a de facto outsourcing vendor for the hospitals no longer part of
CareGroup. On November 1, 1998, Halamka became CIO of CareGroup as it was
systematically working through a $41 million project to deal with the Y2K problem. At the
time, the IT organization had 380 staff members and annual expenditures in excess of $50
million. Halamka brought an extraordinary background to the task. As an undergraduate at
Stanford University majoring in computers and economics, he had founded a software
company in his dorm room at age 18. After Stanford, he enrolled simultaneously at UCSF
Medical School and Berkeley Engineering School, where he completed a combined
mechanical/electrical/engineering and medical school program. At residency time, he sold
his software company, which had by then grown to 35 people, and became an emergency
medicine specialist in Los Angeles. In his spare time, he authored several books on
computing subjects and wrote a hypertext system to coordinate all clinical information in
the county hospital in Los Angeles. In 1996, he moved to Harvard to practice emergency
medicine at the Beth Israel Deaconess Medical Center and undertook postdoctoral work at
the Massachusetts Institute of Technology in medical infomatics. He also headed a group of
50 data analysts and Web specialists developing Web applications for CareGroup. Halamka,
in 2000, assumed the additional role of CIO of the Harvard Medical School. (Mount Auburn
and Beth Israel/Deaconess were both Harvard teaching hospitals.) In talking about his
background, Halamka observed: The reason why I’ve been successful is because I know all
the technologies, I program in 12 languages, and I’ve written books on Unix system
administration. I’m a doctor, so I understand the clinical domain and the technical
requirements. But as I tell people, my own blind spot—I’ve wired a telephone closet, I’ve
built 100 servers, hundreds of desktops, but never built anything beyond a home network.
It’s just the reality. But, I have now. CareGroup IT The IT organization that Halamka took

4. over in October of 1998 was a decentralized, nonstandardized operation. Each of the
hospitals ran its own homegrown legacy systems that predated the merger. Assignment:
Healthcare ITAt the Beth Israel Deaconess Medical Center, computer operations were
complicated by having to run a mishmash of the systems from each of the merged hospitals.
In addition to internal IT staff, CareGroup employed 78 consultants. Deaconess-Nashoba
had an antiquated lab system with hand-typed results (IBM Selectric), no e-mail, and few
PCs. DeaconessGlover had a pathology system hosted at Neponset Valley with no electronic
outputs and financials hosted by a third-party vendor that would fail during each
thunderstorm. Deaconess-Waltham had a 10-year-old homegrown system with
nonintegrated lab and radiology systems. NEBH had $500,000 per year outside consultants,
a problematic lab system, a self-developed payroll system, a failed operating room system
installation, and a limited network incapable of remote access. Mount Auburn Hospital, the
most sophisticated of the nonmedical center hospitals, was running on its own homegrown
system built around the Meditech package. 3 This document is authorized for use only by
FLORIN PANESIU (flopanesiu@icloud.com). Copying or posting is an infringement of
copyright. Please contact customerservice@harvardbusiness.org or 800-988-0886 for
additional copies. 303-097 CareGroup By 2002, all of these hospitals had been brought
together on a common system with the Meditech software at the center. All had state-of-
the-art e-mail, networking, PCs, and clinical/financial information systems at costs similar
or reduced from those at the time Halamka took over. For example, NEBH’s IT budget
dropped $135,000 in fiscal year 2002, and it was expected to drop another $400,000 in
fiscal year 2003. By 2003, CareGroup believed it had the most advanced network in health
care, the most advanced e-mail system in health care, the most advanced voice/wireless
system in health care, the most advanced data center in health care, and the most advanced
Web infrastructure in health care. It served 3,000 physicians, processed 40 terabytes of data
per day, and handled 900,000 patient records dating back to 1977, all ed by a staff of 200.
All applications were Web enabled. In late 2002, the case author, using Halamka’s
password, was able to access all of his own personal health records, X-rays, and records of
office visits for the past decade on a PC five miles away from the hospital where he had
received those services (he was delighted to find that they were accurate). The IT
organization ran a complete “lights-out” data center with three backup generators and had
not suffered a data center power outage in three years. Assignment: Healthcare ITIn the
first quarter of 2003, the final IBM mainframe was scheduled to be decommissioned, and
the network would be exclusively built around clustered Unix/Linux servers. Storage was
100% EMC (with no local server-based storage). Paired central processing units (CPUs) for
development, testing, and production assured that new software performed well by the
time it was in use. Hewlett-Packard was the primary supplier of the Unix boxes, and
Compaq (now HP) was the primary supplier of Wintel boxes; Dell supplied machines for
Linux clusters. Following a McKinsey study at Harvard, IBM came in with a bid that
undercut Dell prices for PCs by 50%. As a result, over a five-year period, all desktops
(including at CareGroup, because it included Harvard teaching hospitals) were being
replaced by IBM PCs. The data center housed tape-backup systems that did incremental
backups on the 40 terabytes of daily data, transferring them onto three-gigabyte tapes.

5. Every night those tapes were taken to an Iron Mountain storage facility that had once been a
missile silo. PeopleSoft software handled HR, payroll, accounts payable, and the general
ledger. Physicians were provided free e-mail accounts, and they had begun experimenting
with wireless messaging devices such as Blackberry text pagers. As of the first quarter of
2003, all networks became IP based; Novell IPX would be replaced by year-end 2002, and
AppleTalk would be gone by the end of the following quarter. Also in 2002, there were two
events that were not considered major when they happened but that took on significance in
retrospect. First, at Halamka’s request, Cisco conducted a study of the CareGroup overall
network in the summer, delivering its final report, complete with detailed
recommendations for modernizing the network, in October; the results of the study were
being analyzed in November, but nothing in it suggested imminent peril. Assignment:
Healthcare ITSecond, CareGroup’s networking guru, who had long provided the last word
on anything to do with the overall network, left his position in CareGroup IT to pursue
another opportunity; although replacement was being sought, no immediate impacts were
apparent as a result of his departure. CareGroup had cut capital budget expenditures by
90% in the three years prior to 2003 (see Exhibit 3). The Meditech installations in the
various hospitals were done without consultants in half the usual time, at 20% of the cost.
(Exhibit 4 shows the IT operating costs for the various hospitals. Exhibit 5 shows
CareGroup’s operating expenses vis-à-vis Partners and Gartner standards.) In September
2001, the CareGroup IT organization was ranked number one in America by Information
Week, and it had been in the Information Week top 100 companies for the last three years.
By November 2002, CareGroup’s IT systems and services were widely viewed as critical to
building clinical loyalty, and they were believed to provide the best knowledge management
service in the United States. All of this was exhilarating and exciting, but . . . 4 This document
is authorized for use only by FLORIN PANESIU (flopanesiu@icloud.com). Copying or posting
is an infringement of copyright. Please contact customerservice@harvardbusiness.org or
800-988-0886 for additional copies. CareGroup 303-097 November 13, 2002 In the days
leading up to November 13, 2002, a researcher on the CareGroup network had begun
experimenting with a knowledge management application based on file sharing, a sort of
“Napster for health care.” The software was designed to locate and copy information across
the network automatically. No sooner had this researcher set up the software in its original
configuration than he received a call from his wife telling him she was in labor. He departed
hurriedly for a three-week paternity leave. Assignment: Healthcare ITThe new software was
left running in a basic mode not yet tested or tuned for the environment in which it was
operating. The new application began to explore the surrounding network, seeking out and
copying data in larger and larger volumes from other computers. By the afternoon of
Wednesday, November 13, the rogue software program was moving terabytes of data
across the network. The Network Collapse These huge data transfers quickly monopolized
the services of a centrally located network switch. No other data could get through this
switch, nor was it able to respond to queries from other network components asking if it
was still functioning. Other network components concluded, reasonably enough, that
something had happened to the monopolized switch, that it had ceased to function as a
reliable part of the network. Other network components then began to compute alternative

6. paths for data flow through the network, paths that did not traverse the troubled switch.
Fortunately, the network was physically redundant throughout (see Exhibit 6); there were
alternative paths along which data could flow. Network components had a built-in ability to
recompute data paths in the event of failures. Theoretically, any computer on the CareGroup
network could still communicate with any other computer, even though a major switch was
no longer available. Unfortunately, the evolved complexity of the overall network—the way
individual smaller networks had been added one at a time, none of them resulting in issues
when they were added— had created a hidden problem that kicked in with a vengeance
now that the network had lost the services of a major switch. Assignment: Healthcare ITAs
network components tried to calculate new paths along which data would flow, as they
decided which redundant network components would now act as primaries and which
would act as backups, they became confused. As the many smaller networks had been
“glued” together over time, the network had gradually crept “out of spec”; algorithms for
computing alternative data paths could no longer operate correctly. Redundant components
intended to operate in tandem, one as primary and the other as backup, began to operate at
cross purposes, both becoming primary. They began to duplicate each other’s functionality.
Each relayed the other’s messages; one switch would relay a single message to the other,
then that switch would relay the same message back to the first, which would then relay it
to the second again. Assignment: Healthcare ITAll messages on the network began
repeating in this way, reproducing rapidly in an endless loop until the network was totally
disabled. (See Exhibit 7 for a more detailed account of this problem and Exhibit 8 for a
graphic of network traffic levels during the outage.) None of these details of what was
happening was apparent to users, operators, or managers of CareGroup’s IT systems and
network. All anyone could see on that November afternoon was that every software
application that required network communication had stopped working, suddenly and
without warning. 5 This document is authorized for use only by FLORIN PANESIU
(flopanesiu@icloud.com). Copying or posting is an . Assignment: Healthcare IT