The Wellmont Health System migrated from an aging ATM backbone to a new Gigabit Ethernet infrastructure using CWDM technology to address bandwidth and reliability issues. Wellmont built the new infrastructure by laying dark fiber and installing Cisco networking equipment. This included running Gigabit Ethernet over leased fiber from Sprint and using CWDM to transmit additional network traffic over the fiber. The new infrastructure provides improved bandwidth for applications like medical imaging, as well as increased redundancy and reliability. It positions Wellmont's network to support future growth and applications for the next 5-6 years.

1. How do you spell relief? If you’re northeast
Tennessee’s largest health care system and
you’re distressed by an aging, temperamental, under-
powered ATM network that hiccups at the least provo-
cation, you build a Gigabit Ethernet backbone and run it
over dark fiber you’ve coaxed out of a reluctant telco.
Then you add a little Course Wavelength Division
Multiplexing (CWDM), the passive optical technology
targeted at metro networks, and you’ve built yourself an
enterprise network infrastructure that’s good for at least
five or six years. At least, that’s the prediction of Darren
Ramsey, the senior network specialist who shepherded
the migration from ATM to Gigabit Ethernet for the
Wellmont Health System, a leading health care provider
in northeast Tennessee and southwest Virginia.
When fully operational this summer, the new multi-
service infrastructure will solve a plethora of business-
critical issues for Wellmont. Not the least of these is
delivering a simplified, all IP-based network with
enough bandwidth to easily transport the health care
organization’s patient records, as well as graphics files
created by the hospital system’s Agfa electronic med-
ical imaging system.
The new infrastructure will also add high-availabil-
ity failover redundancy for disaster recovery between
Wellmont’s three main facilities; Differentiated Services
Code Point (DSCP) for end-to-end QoS; and the ability
to deploy several bandwidth-intensive applications, in-
cluding medical imaging, health information systems,
VoIP, and an IP SAN that supports both Internet SCSI
(iSCSI) and Fibre Channel over IP (FCIP) over the new
backbone. All of this will be carried over three pairs of
leased fiber optic circuits from Sprint, which initially
balked when Wellmont inquired about renting its un-
used dark fiber capacity.
When matched with the appropriate CWDM boxes
and modules from Cisco Systems, Wellmont can scale
in practice
48 NETWORK MAGAZINE I 07.04 I www.networkmagazine.com
+
by Jim Carr
Darren Ramsey shepherded Wellmont’s migration from an aging ATM
backbone to a new Gigabit Ethernet infrastructure run over CWDM.
The Wellmont Health System
is a leading health care
provider in northeast Ten-
nessee and southwest Vir-
ginia. Created in a merger of
two equals—Bristol Regional
Medical Center and Holston
Valley Medical Center—in
1996, the health care system
consists of five medical cen-
ters and hospitals, 40 clinics,
an assisted living center, a
hospice, and a wellness cen-
ter. Its services range from
community-based acute
care to specialized tertiary
services that include neona-
tal intensive care and two
trauma centers.
Because the merger oc-
curred so quickly, the ensu-
ing entity’s IT departments
had little time for long-term
planning, instead deploying
a 25Mbit/sec ATM backbone
between the two main facili-
ties. By 1999, that backbone
was reaching its limit, and
with the addition of several
large mission-critical applica-
tions, it became apparent
that the hospital needed to
rebuild its infrastructure.
To that end, the organiza-
tion staged a phased de-
ployment of Gigabit Ether-
net and plans to add CWDM
capabilities in the near fu-
ture. This should ensure that
its infrastructure is “good for
five or six years,” according
to Darren Ramsey, Wellmont
Health System’s senior net-
work specialist.
he long and short of it : : :
Healing Network IllsThe Wellmont Health System turns to Gigabit Ethernet and CWDM over dark fiber
to rebuild its antiquated ATM infrastructure.

2. its Gigabit Ethernet backbone up to 8Gbits/sec over its
existing fiber circuits, although initial plans call for
using CWDM just for 2Gbit/sec connectivity.
These changes didn’t happen quickly, Ramsey ad-
mits. In fact, he likens the task of migrating Wellmont’s
infrastructure from ATM to Gigabit Ethernet to that of
turning an ocean liner: “Everything is so big and inter-
dependent, it’s difficult to make a drastic change—it’s
hard to turn the ship quickly.”
But turn it around Wellmont did, and here’s how.
MERGED ENTITY
Wellmont, one of northern Tennessee’s largest employers
with about 4,500 health care professionals, was formed in
1996 in a merger between Bristol Regional Medical Center
(BRMC) and Holston Valley Medical Center (HVMC). Eight
years later, Wellmont’s integrated health care system con-
sists of five medical centers and hospitals, 40 clinics, an
assisted living center, a hospice, and a wellness center.
According to Craig Rogers, Wellmont’s manager of
technical services, the merger occurred so quickly that
the two organizations’ IT departments had little time for
long-range design and planning. In the interest of expe-
diency, they deployed an ATM backbone, the de-facto
backbone technology at the time, with 25Mbit/sec
point-to-point Permanent Virtual Circuits (PVCs) be-
tween the two main hospitals.
Despite the fact that BRMC was primarily a Unix
shop and HVMC an IBM mainframe house, and that
both had a mixture of Novell NetWare 3 and 4 servers,
everything worked fine for a few years, says Ramsey.
But by 1999, the ATM backbone had started showing
its age. Several factors were at work here.
First, the organization had purchased a data center
that would allow it to consolidate all of its IT resources,
including servers, into a single location. That, of course,
necessitated an architectural change in the ATM back-
bone. As a result, Wellmont moved from a point-to-point
environment to a multipoint environment with two PVCs
rather than one, adding bandwidth and complexity.
Wellmont also added several new enterprise-wide
applications to its existing health information system,
which provides patient records and financial function-
ality already in use. This included Microsoft Outlook
for e-mail, and Agfa’s Picture Archiving and Communi-
cations System (PACS), which is used at four of Well-
mont’s facilities as a film-free diagnostic imaging solu-
tion. In addition, the workstation count grew to more
than 2,000, according to Wellmont CIO Steve Hill.
Despite this growth, Wellmont’s network consisted
of a single layer-2 unrouted subnetwork. With Novell’s
IPX and IP running across this backbone, it was no
surprise that the network began to experience severe
performance degradation, including broadcast storms,
says Ramsey.
It got to the point where Wellmont’s networking staff
was troubleshooting the organization’s ATM switches on a
daily basis. The staff experienced “strange issues” that the
ATM vendor was unable to fix. It was obviously time for
some changes, according to Ramsey.
www.networkmagazine.com I 07.04 I NETWORK MAGAZINE 49
Core Topology
HVMC CORP BRMCRouters
1 Gigabit
1 Gigabit
1 Gigabit
1 Gigabit
1 Gigabit
1 Gigabit
CWDM 1550nm
CWDM 1590nm
Routers Routers
Wellmont’s new Gigabit Ethernet infrastructure consists of leased fiber optic circuits (denoted by the solid white lines) and two 1 Gigabit
CWDM “lambdas” (indicated by the blue and yellow lines). Expanding the backbone’s CWDM capacity requires adding modules to the
hospital’s Cisco Gigabit Interface Converter CWDM devices, which operate in conjunction with the hospital’s Cisco routers.

3. MOVE TO ROUTING
The first step in remedying Wellmont’s infrastructure
problems was to implement a routed network. Because
this required “touching” almost every device on the net-
work (minus workstations, which are assigned IP ad-
dresses automatically via DHCP) and because many of
these devices, including servers, were moved around,
the process took almost two full years, says Ramsey.
Once the network was routed and somewhat stable,
Wellmont began eliminating IPX and its NetWare
servers, shutting down the last one in April 2002.
By then, it was evident that Wellmont had outgrown
its 25Mbit/sec ATM backbone, says Ramsey. In studying
the options, Wellmont rejected going with a bigger ATM
pipe. “We thought ATM was probably dead as an enter-
prise networking technology and Ethernet is much sim-
pler, with a smaller learning curve,” he says.
Wellmont also looked at the Multilink Point-to-Point
Protocol (MLPPP), which aggregates up to eight T1 cir-
cuits into a 12Mbit/sec link. “But 12Mbits/sec is a far
cry from 25Mbits/sec, so we knew that wouldn’t work,”
Ramsey explains.
Gigabit Ethernet, on the other hand, had recently
been standardized, “and we knew the prices would
come down in 2000/2001,” he says. “We knew we
couldn’t trench it ourselves,” so Wellmont started look-
ing for dark fiber to lease.
“Initially, we went to Sprint, but they didn’t want to
offer the service,” says Ramsey. “They didn’t want to do
it because there was no competition.”
After six to nine months of negotiations, however, the
entry of several Competitive Local Exchange Carriers
(CLECs) into the area market changed Sprint’s attitude.
Although several CLECs had fiber for lease, “Sprint made
the deal attractive because it could bundle all voice, data,
and long distance into the contract,” says Ramsey. “Sprint
found out we might go with someone else, it made us a
good deal, and we ended up paying less a month for four
fiber optic links than our two ATM PVCs had cost.”
Wellmont subsequently leased fiber circuits between
its three main sites—BRMC, HVMC, and the corporate
data center—in a multi-point-to-point arrangement, with
two independent rings between the three sites connecting
Alcatel switches at each site. That configuration gave
Wellmont’s link redundancy with a failover connection
should one fiber line go down, but not switch redundancy.
From then on, “The network was like night and day,”
says Ramsey. “It went from everything being the net-
work’s fault to the point where the network was no longer
guilty until proven innocent when an application failed.”
ELIMINATING ATM
Even with these improvements, Wellmont still relied on
ATM for wiring closet-to-backbone connectivity, with
about 35 closets per facility. Its next step was to elimi-
nate ATM entirely and move strictly to Ethernet.
With a slew of new internetworking equipment to pur-
chase, Wellmont standardized on Cisco as its “next-gener-
ation vendor,” as Ramsey puts it. He says sticking with one
vendor simplifies implementation in enterprise networks.
In 2002 and 2003, Wellmont deployed a wide variety
of Cisco boxes to begin rolling out its Gigabit Ethernet
backbone. This included placing Catalyst 3550 10/100
Ethernet switches in the wiring closets, Catalyst 4506
Ethernet switches in Wellmont’s medium-sized facili-
ties, and Catalyst 6509 multiprotocol switches in each of
the BRMC, HVMC, and corporate data center locations.
By the start of 2004, Wellmont’s Gigabit Ethernet back-
bone was up and running smoothly, performance issues
with the health information systems had vanished, and
download times—particularly those involving medical
images—were dramatically improved. At that point, Ram-
sey says Wellmont’s management brought up several for-
ward-looking issues to the IT department. These included
deploying a disaster recovery plan in tandem with a SAN,
and running VoIP over the new infrastructure.
“With the backbone up, we felt a lot of pressure to
develop a business continuity plan,” he explains. That
lead to the deployment of 80Tbytes of mirrored storage
money matters
in practice
50 NETWORK MAGAZINE I 07.04 I www.networkmagazine.com
Sprint initially balked when Wellmont inquired about
renting its unused dark fiber capacity.
“ ”
Price of a two-year
upgrade to a Gigabit
Ethernet backbone
Monthly savings migrating
from 25Mbit/sec ATM to
leased fiber optic circuits
via Sprint
Cost of CWDM equipment
to expand the backbone
to 8Gbits/sec
>$1Million
>$1K
$50K

4. in EMC Symmetric, Clarion, and Centara optical stor-
age arrays. All are linked via IP over the Gigabit Ether-
net backbone. The SAN currently supports FCIP, but
Ramsey says Wellmont plans to use a CWDM circuit to
extend the hospital system’s Brocade Fibre Channel
switches to its three main sites. That will give it redun-
dant SAN transport with a pure Fibre Channel over
FCIP, he says.
Meanwhile, Wellmont also began moving some of its
voice traffic onto the IP network. First, it added Nortel Net-
works’ Internet Telephony Gateway (ITG) VoIP cards to its
phone system, which is based on Nortel Option PBXs.
This allowed site-to-site calls within the organization to
traverse Wellmont’s backbone network, eliminating several
Time Division Multiplexing (TDM) Primary Rate Interfaces
(PRIs) and the charges associated with many long distance
phone calls. In patient care areas, the hospital also inte-
grated SpectraLink’s 802.11b-compatible handsets into the
VoIP system.
“Once we started doing all that—we have our PACS
images, patient files, and SAN and voice traffic on the
network—we were in a position where the network
can’t fail,” says Ramsey. “We needed 99.999 percent up-
time, so we started looking at what we needed for the
data center to survive a catastrophic failure.”
That meant making yet another architectural shift in
the backbone. Wellmont worked with Sprint to redeploy
its fiber circuits from a strictly point-to-point topology to
a ring topology and added a second redundant Catalyst
6509 switch to each of the three main sites.
This left Wellmont with direct links between its
main facilities. Such a configuration ensured that a fail-
ure on any communications link or switch wouldn’t
negatively impact the core of Wellmont’s backbone, giv-
ing Ramsey his “five nines” reliability.
It also paved the way for Wellmont to deploy
CWDM. Ramsey says all he has to do is drop a Cisco
CWDM Gigabit Interface Converter (GBIC) with the ap-
propriate Optical Add-Drop Modules (OADMs) between
the routers in the health care provider’s three main sites
and the fiber optic circuit, “and the routers think we
lease eight pairs of fibers.” He plans to double the back-
bone’s capacity, saving the potential eightfold band-
width increase for future growth.
All this was done at bargain prices. By eliminating
the need for a second fiber circuit, which costs $10,000
to $15,000 per month to lease, Ramsey says he’s looking
at a three-month payback on his $50,000 investment in
CWDM equipment.
NOT INEXPENSIVE
Ramsey admits the backbone project itself wasn’t so in-
expensive, however. He says from 2002 to 2004, it cost
the organization more than $1 million: “More than we
spent in the previous four years, but it’s money we had
to spend,” he says.
“We could not afford to not do it,” he adds. “We’re
looking at half a million dollars lost per hour if our sys-
tems go down. How many downtimes would it take to
pay for $1 million? I’d rather not have any,” says Ramsey.
“Besides, it gives us the ability to do things we could-
n’t do before—VoIP, wireless, the SAN Fibre Channel over
CWDM—and we’re set for the next five or six years.”
www.networkmagazine.com I 07.04 I NETWORK MAGAZINE 51
2
1 10Gbit/sec Ethernet that’s more
mature and comparable in price per
port to 1Gbit/sec Ethernet. A 10Gbit/
sec backbone would be hard to
saturate in the foreseeable future
That wireless switches had been
mainstream when standalone
access points were deployed. Had it
been available, a different design
approach might have been taken
»wishlist:::
Is your enterprise making innovative use of a network-
ing technology or service that you’d like us to write
about? Contact Jim Carr, an Aptos, CA-based freelance
business and technology writer, at jecarr13@charter.net.
lessons learned
TIPS FOR
DEPLOYING
A GIGABIT
ETHERNET
BACKBONE
WITH CWDM
1) Always make network monitoring a priority. Wellmont lost a
redundant core switch and didn’t figure out what went wrong for
about a week. 2) Always purchase more bandwidth than you
think you need today. You’ll use it sooner than later. 3) Documen-
tation is key on the day of the switchover. 4) Standardizing on a
single network vendor in the enterprise simplifies implementa-
tion. 5) Design, test, and prove your solution in a lab environment.