Smart Networking Techniques in Implementing Broadband Hybrid ...


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Smart Networking Techniques in Implementing Broadband Hybrid ...

  1. 1. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Smart Networking Techniques in Implementing Broadband Hybrid Wide Area Networks by John Puetz, President MasterWorks Communications and David Blanks, CTO, Avalon Inc. Hybrid networks, comprising satellite, fiber-cable and wireless-local-loop technologies, are becoming more commonplace as service providers and enterprise IT managers seek out the best-fit network implementation for Internet access, wide area networking and virtual private networks. Economics and performance factors demand that hybrid systems be investigated and seriously evaluated. This paper presents two case studies of hybrid networks that have been implemented in the past year. For each case an overview of the technology trade-offs is presented, along with the network details and underlying technology advances that made the network implementation possible. In addition, the economics of the implementation are provided. Introduction Capacity, network reach, and economics are arguably the three most important considerations faced by service providers and enterprise IT managers today, as they rollout or expand their networks. The amount of data used in business, and the information required to stay competitive, is roughly doubling every six months. Access to the Internet, and a presence on the web, have become a necessity in today’s corporate environment. Moving large amounts of data into, around, and out-of the enterprise network for national and multi-national corporations is becoming increasing more difficult in a cost constrained, bandwidth-limited environment. While fiber and other broadband technologies have become more widely deployed, no single technology can meet the capacity, geographical reach, and financial considerations that face my many IT managers. Service providers, especially network and Internet access providers, are faced with increasing demands for higher capacity access closer to the end- users and a wider offering of services. For economic and performance considerations a combination of network technologies is becoming more prevalent as the “best-fit” approach. In concept, hybrid networking makes use of the “best” features and benefits of each Page 1
  2. 2. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 technology, while minimizing the “rough spots” that might occur at the seams where these technologies fit together. There-in lies the challenges facing equipment manufactures, integrators, service providers and users of hybrid networks. This paper presents two cases studies of cutting-edge hybrid network implementations. The first network involves using broadband satellite VSATs (very small aperture terminals) to greatly extend the reach of an existing international frame relay network for real-time data collection and infrastructure communications in an oil exploration and services company. Mobility and high data capacity were key requirements for this IP based virtual private network. A new satellite routing protocol (SRP) was implemented and integrated with a flexible rate, IP on-demand satellite networking capability to implement a transportable WAN environment. Additional enhancements were implemented to permit automatic remote node registration and network logon. The resulting network is international in scope, covering North America and portions of South America and Central Africa. The second network details a fiber/satellite/microwave network implemented by a start-up network access provider (NAP) in bringing services to ISPs and businesses in Australia. A wireless domestic network is implemented using microwave and satellite that bypasses the incumbent telecom carriers. This allows the service provider to implement an alternative service that brings a new level of performance, pricing and service to the Australian marketplace. Case Study 1 – Broadband WAN with a long reach Travel expenses and lost employee productivity due to travel time are significant to many companies around the world. But to a U.S. based Fortune-100 oil services company, lost productivity of critical expert resources was the key motivation for finding a better way to do business. Before they deployed their wireless broadband WAN across The Americas and South Africa, their experts went to where the geological data was—the well sites. These experts spent half of their time on a job in non-productive travel, getting to exploration sites at sea or in hard-to-reach land locations. Once on site, they’d analyze large amounts of data captured in real-time using specialized computing applications, and make drilling or Page 2
  3. 3. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 process recommendations on the spot yielding immediate results. This highly specialized service has brought in big money for the company, but the number of jobs that each expert can support limits company revenues—that is until now. Since they’ve deployed their broadband WAN infrastructure, the high-volume, real-time well-site data comes to their experts, who are regionally located in offices and who can now support several operations at multiple sites concurrently. Results—increased productivity, larger service revenues and happier employees. And when local conditions warrant, expert on-call help is just a videoconference or a telephone call away for the drill crews. And for field crews that remain on-site for offshore rig or shipboard operations, the new network gets used off-hours as well—for calls home. Thus, the company has significantly altered the model they use to deploy geological services and plan on expanding services into new markets. Demanding network requirements for a “mobile” WAN Oil field exploration operations are a very demanding telecommunications environment. The virtual network must “reach” hundreds of miles from the nearest frame relay access point, to provide a consistent, high-quality of service IP pipe, at data rates above 400 Kbps. Also needed is the ability to support flexible bandwidth capacity to accommodate video feeds and video conferencing as well as “basic” telephone, email and file transfers services. Additionally, operational sites may exist only for a few hours or a few days; so network setup and activation must be fast, less than 30 minutes, and simple—trained oil field personnel only, no satellite field engineers allowed. For all practical purposes, this is a mobile WAN environment; although except for shipboard installations, the network won’t operate while mobile. An additional wrinkle to the mobile environment is that networking equipment configurations are not always the same. Some field operations required more special purpose computing equipment, so the local network topologies vary. The network could have a large number of high-bandwidth applications on line simultaneously, so to make it economically attractive, the company couldn’t use dedicated links for each. Thus a way to automatically allocate bandwidth on demand and based on application need (some low speed, some high speed) was required. Page 3
  4. 4. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Implementation Trade-offs With a worldwide frame relay network infrastructure already in place, the “trick” was extending the network’s reach wirelessly and at reasonably fast data rates. Prior to the implementing the broadband VSAT network, lower bandwidth technologies were experimented with. These included CDPD wireless and low-speed interactive VSATs. While data rates were only 4.8 to 14.4 Kbps at best, these trials provided a useful function in proving out the concept and providing an environment for specialized software applications development. Digital line-of-site microwave links were considered, but except for a few locations, were deemed too expensive and too restrictive in “reach” capability, with only 20 miles coverage at best without using link repeaters. Smart broadband VSATs After evaluating a number of leading DAMA VSAT equipment providers, the oil services company selected ViaSat as the technology vendor of choice. Key to ViaSat’s technology edge is the ability to combine intelligent IP routing techniques with a sophisticated bandwidth on demand control system—both are integrated directly into their broadband VSAT terminal. IP routes are defined based on ‘Internet Application Profiles’ that take into account the destination IP address, the requesting application port, and satellite channel parameters like transmit and receive data rates and the type of connection desired (one- way, two-way equal rate or two-way unequal rate). Thus when a file transfer is required, the FTP/IP packets arrive at the VSAT based router and a two-way IP connection established between the two networks. In this example, the circuit might be configured for a 1 Mbps transmit rate and a 32 Kbps receive rate. Once the connections are made between the routers, data transfer begins. The IP route remains active as long as there is traffic transiting the connection and automatically terminates after a designated inactivity period has expired. Dozens of possible data rates are available from 9.6 to 2048 Kbps. Another capability required for successful field operations was the ability to dynamically up- speed (and down-speed) connection rates as applications come and go, without interrupting active applications. For example, a baseline WAN connection is established at 64 Kbps in both directions to support email and general networking functions. But when real-time data Page 4
  5. 5. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 needs to be transferred from the well-site to the processing center, higher speed asymmetric IP connection is established. This approach ensures that all of the critical real- time data gets to the processing center immediately and without congestion or packet loss. Now if a videoconference connection is also desired, the broadband VSAT can establish yet another circuit (if to a different destination) or increase the existing 64 Kbps connection to a rate high enough to accommodate both the video conferencing and the basic WAN connectivity. The Hybrid Network Implementation Critical to the success of deploying wireless networking is the ability to accommodate the reduced traffic capacity of the wireless networks (e.g., 256Kbps to 2 Mbps) as they are encountered by the higher speed wired WANs with their 10 to 100 Mbps capacity. Not only are wireless pipes smaller, but also when standard protocols are used over the increased delays of satellite, the effects of congestion are amplified, since the feedback mechanisms within TCP take longer. Thus congestion management and mitigation processes are important when implementing satellite based WANs. Router queue depth management and protocol responses play a very significant role in making certain data throughputs are high. Proprietary TCP/IP throughput enhancement algorithms and quality-of-service (QoS) features were implemented to ensure the desired throughput levels could be achieved. The following diagram provides an overview of the overall network. The wired WAN represented by the network cloud on the right while the remote LANs are located on the left, with the VSAT nodes and the satellite gateway equipment forming the VSAT WAN cloud. Multiple VSAT WANs may exist for different regions (e.g., one for North America, a second for South America). In a mobile WAN environment, the remote VSAT nodes come and go from the wide area network as they join and leave the network. For fixed site locations, the remote nodes join upon initial equipment installation. Routes can either be static or dynamic. Static routing means the routes are administrated manually (added and deleted) across the entire network. Unless a network is very small, static routing is difficult to support. Page 5
  6. 6. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Remote Net ‘A’ Gateway VSAT LAN WAN #1 Remote Net ‘B’ Servers Remote Net ‘C’ WAN - Internet Remote Net ‘X’ VSAT WAN #2 Remote Video Conferencing Net ‘Y’ With bandwidth on demand WAN systems, and especially in a mobile WAN environment where network nodes come and go, it’s vital that the wired WAN network “knows” which VSAT nodes are reachable and what routes are active. This is the job of routing protocols— software routines that run on routers to determine who their “neighbors” are and how to connect to them. Dynamic routing protocols like RIP (routing information protocol) and OSPF (open shortest path first), automatically keep the routing tables in all participating routers aware of the network topology, available connections, and route performance. Internet control message protocol (ICMP) messages are used to indicate route and connections status among routers. Because these routing protocols were designed for wired networks where full-time connections are available, these protocols require non-trivial amounts of data to be transferred among routers. (While OSPF is more efficient than RIP and provides for better Page 6
  7. 7. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 routing performance, there is still several hundreds of bytes of information required on a relatively continuous basis between routers.) With on-demand satellite WAN networking, connections are not always “on”, as once a connection becomes inactive, the circuit is deactivated and must be reinitiated if needed again. Thus to achieve efficient use of bandwidth, distinctions in routing metrics are made so that active connections are preferred over inactive ones. Furthermore it’s highly desirable for system capacity and bandwidth utilization reasons, to have the IP routing protocol operate over the network’s inherent network management and control infrastructure. To achieve this, ViaSat uses a proprietary routing protocol (SRP) to support routing management over the satellite network in a “background mode”—all routing housekeeping is performed using the network control channel bandwidth so that on-demand services are used exclusively for WAN data traffic. At the Gateway LAN, where the VSAT WAN and the wire WAN meet, RIPv2 is used to communicate routing updates to the terrestrial routers. The network supports mesh, star, inverse star (for data collection) and hybrid mesh-star architectures. To achieve maximum throughput of gateway resources, any gateway VSAT can communicate to any of remote VSATs. There are no pre-assigned channel allocations needed or bandwidth restrictions, so approach all gateway resources can be used before service blocking occurs. Full mesh connectivity ensures that remote VSATs can route IP packets directly to other remotes. And the network supports voice/fax communications as well. One common remote-to-remote connection path is for the company’s Gulf of Mexico port office to communicate directly with the exploration ships that it controls. Data connections are used for planning and logistic support activities. The broadband satellite WAN network has been deployed in North America, northern South America and Central America and in portions of Africa. The African VSAT WAN network interfaces into the global frame network using a teleport located in the United Kingdom, while the two America VSAT WAN networks interface at a single facility in Houston, Texas. Since the networks operate as a single entity, email traffic originating 50 miles offshore Africa, reaches its destination in a Houston processing facility in a matter of a few seconds. Page 7
  8. 8. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 A look at the economics Almost without exception the locations requiring high-speed data connectivity are well outside the range of existing terrestrial wireless infrastructures. Mobile satellite systems such as INMARSAT can provide the service coverage but are currently limited in speed to 64 Kbps or 128 Kbps using special HSD (high-speed data) services. Typically these services run from $6 to $9 per minute with effective data throughputs limited to 50 to 70 percent. This service was evaluated as being far to costly by the energy services company to be used for anything sort of an emergency. (They do employ INMARSAT and GlobalStar services for emergency phone communications.) Another approach to consider is implementing a private wireless infrastructure using terrestrial digital microwave links. This approach is in use for off-shore installation were a number of sites are progressively further from shore on a relatively straight line and separated from one another by less than the line-of-sight range of a microwave link (15 to 20 miles). However, for transportable land use and more typical offshore usage, microwave is not practical. As a practical matter, Ku-band and C-band fixed satellite services are economically appealing. As can be see from the figure below, a 25 node network employing a variety of technologies can range from $138K to over $600K per year for space segment. The figures were derived according to the following: N dedicated: full-time asymmetric connections are used operating at 384 Kbps transmit and 64 Kbps receive. Bandwidth is not shared among terminals. N BOD: bandwidth on demand configuration whereby bandwidth sharing occurs on a 95% availability figure, with the same transmit/receive rates as noted above. N BOD with flexible IP rates: same as BOD except that 64 Kbps circuits are used until a high-speed connection is required, then a 384 Kbps transmit circuit is initiated. When the high-speed service is no longer needed then the circuit returns to 64 Kbps. The assumed space segment rate is $6K per MHz per month, a typically rate paid in the United States for Ku-band space segment for partial transponder use over a two to three year period. Page 8
  9. 9. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Space Segment Price ($K/yr) $619 $700 $600 $500 $400 $248 $300 $138 $200 $100 $0 Dedicated BOD BOD with Flex IP As can be seen from this chart, there’s approximately a 60 percent savings using bandwidth-on-demand (BOD) compared with dedicated operation (with no perceivable impact to service quality) and a further 45 percent savings using a dynamic rate approach that increases (and decreases) link capacity as required. Case Study 2 – National Network Access Provider goes Wireless and Hybrid Integrating traditional fiber-wire capacity with satellite and terrestrial wireless technologies is enabling new business opportunities and services. Case in point is a new start-up network access provider (NAP) in Australia. From a business perspective, the NAP start-up has launched three service offerings in designated markets across Australia: Page 9
  10. 10. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Fast Internet – with the first implementation for Australian customers of the DIRECT PEERING concept pioneered in the USA by InterNap Inc – removing the delays and losses associated with traditional free, public peering models used by legacy carriers Virtual Private Networks (VPN’s) – offering Web access, email, IP based Video Conferencing and Database applications on a single service equipped with QoS definitions for each traffic type Streaming Media services – based on local servers supporting Microsoft and Real Networks streaming products Market environment overview Two Tier 1 carriers presently dominate Internet access and distribution in Australia. Adding to the lack of competition are the market dynamics of insufficient bandwidth and volume based usage tariffs imposed by the incumbent telcos. Currently high tariff rates exist with an added encumbrance of penalizing ISP’s for asymmetric bandwidth usage. Both Telstra and Cable & Wireless/Optus have dominated the supply side of the market, creating a new environment of opportunity for new service providers. Within Australia, the duopoly has buried many thousands of route miles of fiber cable, which is currently inactive and being held as dark fiber. As a result, multiple carriers have wired the major capital cities (four of which account for 70 percent of Australia’s population). Until Macrocom installed a Digital Microwave Network, there was no alternative inter-capital carrier. Once Macrocom provided competition, the duopoly dropped the price of E1 inter- capital connections by 67 percent. The lack of competition within the Australian marketplaces has led to a number of thin-route, independent satellite service providers. Several are using PanAmSat satellites to supply Internet access to ISPs, although in most cases the return paths are routed terrestrially. Intelsat provides an alternative satellite source, but until Intelsat’s future privatization occurs, landing rights are still restricted to C&W Optus and Telstra. Page 10
  11. 11. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Network considerations The network architecture was based on the following considerations: (a) Independence from the incumbent Telcos within Australia for the core and distribution elements (b) Route diversity and policy routing of traffic by Class-of-Service criteria (c) End-to-end Quality of Service management (d) Robust and resilient, fault tolerance (e) High availability (fiber networks have a poor reliability record within Australia) The original network concept was conceived around an ATM platform to implement the desired quality of service features. But with the availability of multi-protocol label switching (MPLS) from Cisco in late 1999, a core IP design was selected. The opportunity to have a simplified routing implementation based upon an “ingress/egress” point definition was a definite appeal to support the VPN services. Network implementation The service is essentially implemented as a wireless virtual domestic network using a combination of satellite and microwave with fiber distribution in metropolitan areas. The network connects into the Internet backbone in the United States in two ways. The first route uses a 45 Mbps DS-3 undersea fiber connection that terminates in Sydney. (The Western Australia cable route was used due to the non-availability of the new Southern Cross Cable; Pacrim East and West have no spare capacity.) From Sydney, terrestrial microwave connects Melbourne and Brisbane. Within each city, local fiber is used to complete the WAN connection. The second Internet backbone route is a 45 Mbps satellite link from Verestar’s teleport in Brewster, Washington that feeds all four cities with Internet service. A lower-speed, 10 Mbps Internet return connection is uplinked from Sydney. A full 36 MHz transponder is used on Intelsat 701 (180 E) with high throughput rates achieved from enhanced modulation techniques and FEC coding. For the other cities the return path to the Internet is made over the microwave connections back to Sydney. Because the high-speed satellite connection drops simultaneously into all sites, there is no need for expensive 45 Mbps intra-country Page 11
  12. 12. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 terrestrial connections. The satellite link also serves as an alternate routing path for the main fiber connection, providing a level of redundancy and load sharing. Brisbane Sydney Melbourne Undersea Fiber 45 Mbps - Satellite 10 Mbps - Satellite As mentioned previously Cisco’s Multi Protocol Label Switching (MPLS) is used and implemented with Cisco 7500 and 7200 series routers that provide a virtual “cloud” between the Australian cities and Brewster / Seattle in the USA. To optimize web traffic, intelligent caching engines were used. Cisco Model 590s were installed at Sydney and Brewster. And to support streaming video and audio, specialized InfoLibria Media Mall streaming servers where installed on both ends of the fiber. Page 12
  13. 13. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 Network Economics In providing Internet access as a network access provider (NAP), connectivity into the U.S. backbone is essential. A comparative analysis of satellite and fiber rates indicates that satellites can provide substantial savings in service provisioning, especially when considering the asymmetric aspect of Internet access into the backbone. The following figure provides indicative telecom pricing for T1 or E1 (full-duplex) services between the designated country and the U.S. International T1/E1 to US France Spain Venezuela Panama Hong Kong Australia Japan 0 20 40 60 80 100 120 Monthly Rate ($K, US) A number of satellite services providers supply connectivity to numerous ISPs internationally, with representative pricing to Europe of $18K per month for a 2 Mbps outbound and a 512 Kbps return link. These rates are very favorable when comparing to the $25K to $30K for a terrestrial service into Europe and almost $90K into Australia. Specific to this network, indicative pricing for a DS-3 (45 Mbps) full duplex fiber connection to the U.S. from Australia ranges from $400K to $600K per month. Using satellite, an asymmetric service of 45 Mbps outbound and 10 Mbps return link can be established for approximately $240K per month providing a hefty 40 to 60 percent savings. Furthermore, since multiple cities are served by the single 45 Mbps outbound, no additional connectivity charges are incurred to reach them—in short the incremental cost for distributing to additional sites is minimal. This is in sharp contrast to using fiber to reach the additional Page 13
  14. 14. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 cities. This approach would require a separate connection for each city thereby increasing the connectivity charges in a multiplying factor. Network expansion The start-up intends to expand beyond the launch network with additional capabilities that will expand performance or services. These include: (a) VPN Provisioning partnerships with US Service Providers to extend the scope and reach of VPN’s beyond the POP sites in Seattle and Brewster – also to overseas countries beyond the USA (b) Network based ASP offerings such as on the fly virus protection, encrypted secure IP sub-networks with encryption nodes at all co-location sites, and IP network based call centers and ACD systems (c) Advanced compression systems that allow 54:1 JPEG image compression and FTP session compressions at lesser rates – currently widely used in Europe (d) Widespread installation of streaming media servers at Australian ISP sites– this will lift the majority of ISP’s away from the threatened revenues of simple Internet access (e) The opening up of the Telstra controlled copper network to competitive ADSL suppliers has resulted in further “last mile” access options to current and prospective customers. Other last mile options include LMDS / MMDS radio ( point to point and multipoint), SDH fiber access ( usually at E1 rate) and HDSL over copper. (f) IP performance enhancement, using Mentat Sky-X protocol on the Intelsat transponder to ensure constant throughput at maximum rates Other applications fit satellite based data networking The two case networks just discussed are but a sampling of types of applications that the new broadband satellite technology advances are enabling. Additional applications include: N “instant” communications infrastructure for locations that are geographically remote or have limited capacity or facilities (e.g., only cell phone coverage, dial-up data services, etc.). N backup data communications and media diversity for critical business operations such as publishing, production and manufacturing industries. With VSAT terminals located directly at the business, a direct end-to-end link is automatically established that bypasses all wired facilities. N quick service deployment (within hours or days) of temporary or permanent services for expanding network coverage or for new operations. Facilities base service Page 14
  15. 15. Smart Networking Techniques in Implementing Broadband Hybrid WANs PTC2001 deployment typically run 2 to 6 months depending on location, and much, much longer in some areas of the world. N service augmentation of existing wire/fiber infrastructure to alleviate network congestion and provide a “must-get-there” data path for time critical content and information. N emergency communication services in responding to natural and man-made disasters. Either fixed site, portable or mobile networking environments can be quickly activated or deployed. N “Best value” transport for multicasting (distributing) data, video, multimedia and Internet. As more entrepreneurs and IT managers around the world become exposed to the cutting edge of broadband networking possibilities, the number of hybrid satellite-fiber-WLL networks will increase dramatically. Page 15