COMBINED WIDE AREA NETWORK FINAL REPORT.doc.doc.doc

  • 2,360 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
2,360
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
13
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. JOINT WARRIOR INTEROPERABILITY DEMONSTRATION 2004 (JWID 2004)NETWORK OPERATIONS WORKING GROUP (NOWG) CFBLNet FINAL REPORT NETWORK OPERATIONS WORKING GROUP July 30, 2004
  • 2. JWID 2004 Final Report Table of Contents1. Purpose...............................................................................................................................42. Scope..................................................................................................................................43. Summary............................................................................................................................44. JWID 2004 CFBLNet........................................................................................................6 4.1 Network Engineering..............................................................................................7 4.1.1 Telecommunications Circuits.....................................................................7 4.1.2 IP Routing Protocols ..................................................................................8 4.1.2.1 IP Routing Prioritization.................................................................9 4.1.2.2 IP Routing Summarization..............................................................10 4.1.2.3 Use of Internal BGP Routing Reflectors........................................10 4.1.2.4 Control Over Incoming/Outgoing EBGP Routes...........................10 4.1.3 Ethernet LAN Improved at the AITS-JPO .................................................10 4.1.4 REL ROK Network.....................................................................................11 4.1.5 Quality of Service.......................................................................................12 4.1.6 Universal “Read-Only” password...............................................................12 4.1.7 Network Time Protocol...............................................................................12 4.2 Provisioning............................................................................................................13 4.2.1 Lessons Learned (Provisioning).................................................................16 4.3 Network Accreditation............................................................................................16 4.3.1 Lessons Learned (Network Accreditation) ................................................17 4.4 COMSEC................................................................................................................17 4.5 Network Testing .....................................................................................................18 4.5.1 Lessons Learned (Network Testing) .......................................................195. Network Services...............................................................................................................19 5.1 E-Mail.....................................................................................................................19 5.1.1 Improvements.............................................................................................21 5.1.2 Lessons Learned ........................................................................................22 5.2 DNS.........................................................................................................................22 5.2.1 Improvements.............................................................................................23 5.2.2 Lessons Learned..........................................................................................23 5.3 IP Phones................................................................................................................23 5.3.1 Lessons Learned..........................................................................................24 5.4 Web.........................................................................................................................24 5.4.1 Overall Web Structure and Format.............................................................25 5.4.2 Newsgroups.................................................................................................26 5.4.3 Bubble Chart...............................................................................................26 2
  • 3. JWID 2004 Final Report 5.4.4 Lessons Learned Database..........................................................................28 5.3.3.1 Database Lessons learned...............................................................29 5.4.5 Master Scenario Events List (MSEL) & MSEL Web Based Tool (MWBT) Servers........................................................................................................30 5.5 Defense Collaboration Tool Suite (DCTS).............................................................31 5.5.1 DCTS on the CFBLNet...............................................................................31 5.5.2 Lessons Learned..........................................................................................33 5.6 Network Engineering Management........................................................................34 5.6.1 Network Monitoring Tools.........................................................................34 5.6.1.1 Netscout..........................................................................................35 5.6.1.2 HP Openview Network Node Manager (NNM).............................36 5.6.1.3 Open NMS......................................................................................36 5.6.1.4 Network Traffic Analysis System (NTAS).....................................38 5.6.1.5 Flow Tools/Flowscan......................................................................39 5.6.2 Lessons Learned..........................................................................................41 5.6.2.1 Future Considerations.....................................................................42 5.7 Network Statistics Collection.................................................................................42Network Engineering FiguresFigure 1. Backbone Bandwidth Utilization JWID Day 9 (24 June 2004).............................5Figure 2. JWID 2004 Site & Network Diagram....................................................................6Figure 3. CFBLNet Backbone Topology...............................................................................7Figure 4. JWID 2004 Routing Plan (6 Eyes Domain)...........................................................9Figure 5. Topology/connectivity of the ROK domain...........................................................11Figure 6. NTP topology.........................................................................................................13Figure 7. Correctly Functioning Pipe.....................................................................................18Figure 8. Faulty Circuit dropping packets.............................................................................19Network Services Figures and Tables:Table 1.Classification Markings for Mail Messages.............................................................21Table 2. Attachment Types for Messages..............................................................................26Figure 9. JWID 2004 Email Implementation.........................................................................21Figure 10. Example Network Bubble Chart..........................................................................27Figure 11. Example CIT Bubble Chart..................................................................................28Figure 12. Lessons Learned Database...................................................................................29Figure 13. CFBL MCU Topology.........................................................................................32Figure 14. NetScout...............................................................................................................35Figure 15. Open NMS............................................................................................................37Figure 16. NTAS....................................................................................................................38Figure 17. Flowscan...............................................................................................................40 3
  • 4. JWID 2004 Final Report 1. PURPOSEThis report provides a summary of the Joint Warrior Interoperability Demonstration (JWID)2004 CFBLNet engineering, operations, and network services to the extent established andsupported by the AITS-JPO.2. SCOPEThe data used to produce this report was based on observations and the operation of theCFBLNet and Network Services implemented to support JWID 2004.3. SUMMARYThe CFBLNet was successfully implemented to support JWID 2004. Designed as a Secret-Releasable overlay on the Defense Information Systems Network-Leading Edge Services (DISN-LES) and DISA ATM Services - Unclassified (DATMS-U) ATM backbone networks, theCFBLNet incorporated a sophisticated ATM backbone, capable of supporting high-speed datatransmission requirements of up to 45Mbps. In the US, the CFBLNet shared between 3 and6Mbps of bandwidth on the DISN-LES with the US National and the US Military Assistance toCivilian Authority (MACA) domains. Details of the US National and US MACA domains willbe discussed in separate documentation and are only mentioned where necessary forclarification.JWID 2004 officially began 2 June 2004 and ended 25 June 2004. Scenario execution started 14June and continued until 25 June; the JWID 2004 Day totaled six hours from 1600Z to 2200Z.Throughout the exercise, the overall CFBLNet availability was 99.9998%. Availability rates arebased on IP connectivity during the JWID Day to each site. Although sporadic network outagesoccurred during the execution period, only network outages that affected operations during theJWID Day were used for site availability reporting.In general, network traffic for JWID 2004 was nearly double that of the traffic generated duringJWID in previous years. The addition of the US National and US MACA domains significantlyincreased bandwidth utilization across the DISN-LES. During JWID 2004, traffic beingtransmitted from the JPO to other sites on all three networks peaked between 5-6 Mbpsthroughout the JWID Day. This increase was due in a large part to greater use of the DefenseCollaborative Tool Suite (DCTS) and the larger number of trials participating within theCoalition and the US domains. DCTS was designated a core service for JWID 2004. Figure 1shows the backbone (US) bandwidth utilization between the AITS-JPO and Dahlgren duringJWID Day 9 (23 June 2004). 4
  • 5. JWID 2004 Final ReportAt the beginning of scenario execution, a few engineering and operational issues impactednetwork performance. When discussing network performance, network problems are defined asany network event that affected the ability of a Coalition Interoperability Trial (CIT) tosuccessfully complete its presentation as planned.Additional problems identified during JWID 2004 will be addressed within each reported area. Figure 1 Backbone Bandwidth Utilization JWID Day 9 (23 June 2004) 5
  • 6. JWID 2004 Final Report 4. JWID CFBLNETThe CFBLNet for JWID 2004 consisted of 35 sites, connected with a sophisticated network ofATM transmission paths capable of providing between 3-10 Mbps of bandwidth for datarequirements. Figure 2 JWID 2004 Site & Network DiagramWhile the majority of sites were ATM-connected, some sites were connected via IP Ethernet orserial data links. All sites on the CFBLNet backbone were secured to the Secret-Releasable levelwith the use of KG-75 FASTLANEs for ATM-connected links, KG-175 TACLANEs forEthernet-based links, and KG-84 or KIV-7 encryption devices on serial data links. 6
  • 7. JWID 2004 Final ReportThe CFBLNet for JWID 2004 facilitated additional NATO site participation through Alliedbackside connections. Many of these sites were connected to and participated on the CFBLNetwith extensive local infrastructures. However this document will only discuss the directlyconnected backbone sites. See Figure 2 for the CFBLNet topology in support of JWID 2004. Figure 3 CFBLNet Backbone Topology 4.1.0 Network Engineering4.1.1 Telecommunications CircuitsThe JWID 2004 network utilized circuits on the DISN-LES and DATMS-U Networks, alongwith leased ISDN, DS-1, fractional DS-1, E-1, DS-3, and E-3 communications circuits. In theUnited States, these communications facilities supported US Unclassified, US Secret, and USSecret Releasable to Allies (AUCANZUKUS + NATO) services. The classified security levelswere separated from each other and from the unclassified service by Type-1 encryption devices.Outside the United States, these lines supported the US Secret Releasable to Allies 7
  • 8. JWID 2004 Final Report(AUCANZUKUS + NATO) service only. Encryption devices used were KG-75 FASTLANEs,KG-175 TACLANEs, KIV-7s, and KG-84s.The use of separate unclassified ATM switches at select US sites using their own PermanentVirtual Paths (PVPs) allowed the separation of JWID traffic from other user site program trafficusing the same physical layer access circuit. This allowed better tracking of JWID user trafficand bandwidth requirements since JWID traffic was on JWID PVPs and not mixed in with otheruser site program traffic. It also enhanced network fault isolation.To carry IP traffic over the ATM portion of the JWID 2004 network, static IP to Network ServiceAccess Point (NSAP) tables were entered into all ATM-enabled routers on the JWID 2004network. ISDN, DS-1, E-1 and fractional DS-1 circuits were directly IP-connected to serialrouter interfaces at other JWID sites. Those sites were then connected to the DISN-LES ATMnetwork to support JWID 2004.4.1.2 IP Routing ProtocolsTwo routing protocols were used during JWID 2004. Open Shortest Path First (OSPF) Version 2was used within a particular Autonomous System (AS), and Border Gateway Protocol (BGP)Version 4 was used between ASs. From a US perspective, OSPF connected US Sites and BGPconnected those sites to all of the allied participants (see Figure 3). These two IP routingprotocols were kept completely separate, or not “redistributed”, during JWID 2004. By keepingthese routing protocols separate, not only did each AS have more control of what IP routes werepropagated and accepted, but troubleshooting was made much easier because the origin of an IProute could be very quickly associated with an AS. JWID 04 Routing Plan (6 Eyes Domain) UK AS 1200- N.Z. AS 1003 SACLANT AS 1103 1201 NEW ZEALAND EBGP Portsdown West 512 Kbps IP via SACLAN Loopback0 ISDN dial-up T EBGP EBGP T1 IP to JPO 6 Mbps ATM/IP via RAFM EBGP EBGP NC3A AS 1109 5Mbps ATM/IP via Wahiawa AU. AS 1000 15 Mbps PVC EBGP EBGP 6 Mbps ATM/IP via 5Mbps ATM/IP via Wahiawa RAFM RED1_NC3A_3640 AUSTRALIA Loopback0 Loopback0 U.S. AS 3664 OSPF Area 0 EBGP Static EBGP 34 Mbps ATM/IP via RAFM 3Mb ATM/IP via Wahiawa 128 Kbps IP via Wahiawa OSPF Area IBGP - Internal BGP SSC-SD_7206 JPO_7206 EBGP - External BGP (dial-up) Loopback0 Loopback0 I.J.43.105/32 I.J.43.97 EBGP - External BGP CA AS 1001 Static Route SPVC CANADA NORTHCOM USFK_7206 HANSCOM_7206 EAGLE NEiF DAHLGREN_7206 Loopback0 Loopback0 Loopback0 Loopback0 Loopback0 I.J.52.106/32 I.J.52.116/32 I.J.52.108/32 I.J.52.114/32 I.J.52.97/32 MNTG-SD AS 1015 U.S. IBGP Route Reflector Cluster MNTG-SD 8
  • 9. JWID 2004 Final Report Figure 4 JWID 2004 Routing Plan (6 Eyes Domain)4.1.2.1 IP Route PrioritizationThe JWID 2004 network had few redundant circuits. The IP routing on the redundant circuitsthat did exist was designed by the network engineers to continue to provide connectivity in theevent of a failure on the primary communications circuit. Unless the highest bandwidth linkswere deliberately prioritized, it was left up to the IP routing protocol to determine which link wasused. JWID 2004 network engineers agreed to use EBGP (External BGP) Local Preferencing(LP) to prioritize IP links among sites (see Figure 4). EBGP LP assigned a preference to allEBGP-learned routes. The default LP assigned was 100. The higher the LP, the more desirablethe route was to the remote AS. All the routers within the remote AS then learned the LPassigned to the route. Therefore, if an AS has more than one route to a remote AS (circuitredundancy), LPs were compared, and the route with the highest LP was preferred while theother route served as a dynamic backup route in the event of a failure.4.1.2.2 IP Route SummarizationInstead of the US advertising several, separate IP subnets to its Allies, a single subnetencompassing the entire US IP address space, called a “supernet,” was advertised to the Allies.Likewise, the Allies consolidated their IP address space into a small number of supernets. Thiswas advantageous for several reasons. When a small number of IP routes were advertised, EBGPupdates required less bandwidth. Moreover, one supernetted IP address could often representseveral, if not hundreds, of subnets. This made it much easier for a network engineer totroubleshoot IP routing problems because fewer IP routes needed to be verified as being presentin the router’s IP table.4.1.2.3 Use of Internal BGP Route ReflectorsOnce a router learned an EBGP Route, the JWID network was designed so that each ASpropagated the learned EBGP route to other routers in its AS via Internal BGP (IBGP).However, to avoid propagating invalid routes, IBGP would only propagate an EBGP route to its“neighbors”. This meant that all routers within an AS would have to be BGP neighbors withevery other router within that AS. This was undesirable, because within the US there were sixsite routers on the network. Therefore the US implemented an alternative solution using IBGPdual route reflectors. Once a route reflector learned a route, it distributed or “reflected” thisinformation to all route reflector “clients”. During JWID 2004, network engineers configuredtwo route reflectors on the routers located at the AITS-JPO and the Space and Naval Warfare(SPAWAR) Systems Center – San Diego (SSC-SD). In the event one route reflector failed, thesecond route reflector took over updating the route reflector clients. This added an extra layer ofredundancy to the IP routing design. 9
  • 10. JWID 2004 Final Report 4.1.2.4 Control Over Incoming/Outgoing EBGP Routes EBGP allowed the network engineer to easily filter incoming or outgoing EBGP routes. This gave complete control to the network engineer to what IP routes were distributed to neighbors and what IP routes were learned from neighbors. 4.1.3 Ethernet LAN Improved at the AITS-JPO The Top-Provider CUseeMe server and several core network servers were located at the AITS- JPO for JWID 2004. It was surmised that a lot of data traffic would be transmitted and received at the AITS-JPO during this and future JWIDs. Therefore, the LAN on which these servers were connected needed to be optimized for maximum bandwidth efficiency. The engineering staff improved upon the 10/100Mbps Switched Ethernet LAN implemented at AITS-JPO for JWID 2002, and upgraded all segments of the LAN to 100Mbps Full Duplex. 4.1.4 REL ROK Network For JWID an additional network (8-eyes) was established to facilitate the participation of the Republic of Korea in Seoul, KR. The only two sites that had visibility to this 8-eyes domain were Seoul, KR, and the AITS-JPO. DNS and email were hosted by the AITS-JPO with email traffic passing through a DII Guard for classification security. GCCS COP traffic passed through a Radiant Mercury Guard. Traffic flow from the 8-eyes domain to the JWID domain was also possible via the DII and Radiant Mercury guards. Both the DII and Radiant Mercury Guards were located at the AITS-JPO. The ROK domain was cryptographically separated from the rest of CFBL using Type-1 TACLANE encryptors and no additional services were available. JPO-CFBL ROK-CFBL ASX 200 DISN-LES ASX 200 KG 75 KG 75TAC Lane TAC Lane 3550 3550 JPO KOREA Quad C IEC Inner ROK LAN ROK LAN Cisco 3550 10
  • 11. JWID 2004 Final Report Figure 5 Topology/connectivity of the ROK domain4.1.5 Quality of ServiceDuring JWID 2004, the possibility existed that the low bandwidth link to New Zealand wouldbecome saturated with data. To allow IP phone traffic to pass unhindered, Priority Queuing (PQ)was implemented. PQ is a Quality of Service method, in which the three precedence bits locatedin the IP header of the data packet are colored by the call manager to indicate the packet iscritical. The serial interface on the AITS-JPO router for the New Zealand link was configured torecognize this coloring and process these packets as priority packets.4.1.6 Universal “Read-Only” PasswordIn order to perform easy collaboration among network engineers, the same password policyimplemented in previous years’ demonstrations was reused during JWID 2004. All JWIDnetwork engineers used a universal “read-only” password to access JWID border routers, LANEthernet switches, and ATM switches. This read-only password allowed network engineers tocollaborate on connectivity, routing, switching, and ATM problems that occurred during the leadup to and execution of JWID 2004. AS engineers configured a unique “enable” password for therouters, Ethernet switches, and ATM switches in their AS. The CCCC-R was the centralrepository for all passwords. Enable passwords were transmitted to the CCCC-R via IP phones.All JWID 2004 network engineers agreed that the enable passwords would be used only by theCCCC-R when a configuration change would alleviate a network emergency, and only afterevery effort was made to contact the applicable network engineer. Such an emergency neverarose during JWID 2004.4.1.7 Network Time ProtocolIn order to synchronize all network devices and provide a reliable time source for servers andapplications, Network Time Protocol (NTP) was used. The NATO-NC3A was configured torecognize a stratum-1 time source located at NATO-NC3A as an authoritative time source. Thismade the NATO-NC3A router a stratum-2 time source. The UK, Australia, Canada, and the 11
  • 12. JWID 2004 Final Report AITS-JPO routers peered with the stratum-2 NATO-NC3A router. Each AS engineer was responsible for propagating time down to their respective LANs from one of these routers. In the event that the stratum-1 time source was inoperable, the UK, Australia, Canada, NATO- NC3A, and AITS-JPO routers were configured as stratum-5 authoritative time sources. In this case, each router would recognize their respective internal timing crystal as an authoritative time source and synchronize with each other using the defined NTP to minimize clock drift. Network Time Protocol (NTP) NATO For NATO: GBS receiver NATO Authoritative time ntp source loopback0 ntp master 5 ntp update-calendar NC3A AS 1109 ntp server A.B.4.5 (NATO stratum 1) ntp peer C.D.217.178 (Australian router) RED1_NC3A_7204 Loopback0 ntp peer E.F.245.1 (UK border router) A.B.5.1ntp peer G.H.125.1 (Canadian border router) ntp peer I.J.43.97 (AITS-JPO router ) AS. AS 1000 CANADA US AS 3664 UK AS 1200- AS 1001 1201 Portsdown West AUSTRALIA JPO_7204 Loopback0 Loopback0 CANADA Loopback0 E.F.245.1/32 C.D.217.245 Loopback0 I.J.52.97/32 G.H.125.1 For AUS/UK/AITS-JPO/CAN: ntp source loopback0 N.Z. AS 1003 ntp master 5 ntp update-calendar ntp peer A.B.5.1 (NATO border router) For NEW ZEALAND ntp peer C.D.217.245 (Australian router) NZ: ntp peer E.F.245.1 (UK border router) ntp source loopback0 ntp update-calendar ntp peer G .H.125.1 (Canadian border router) ntp server C.D.217.245 ntp peer I.J.43.97 (AITS-JPO router) ntp server A.B.5.1 Time Source Peer Associations Time Source Server Associations Figure 6 NTP topology 4.2 Provisioning Provisioning is the all-encompassing term that incorporates all actions required to support a users network requirements. Identification of user requirements, engineering design, design review and approval, equipment bills of materials, funding, equipment and circuit acquisition, 12
  • 13. JWID 2004 Final Reportequipment configuration and testing, installation, and system testing are all part of theprovisioning process. The key to success lies in the early and thorough identification of userrequirements.The provisioning of services for JWID 2004 was based on maximum use of existing DISN-LES,DATMS-U, and Allied connectivity.The following US sites were provided Unclassified, US Secret, and US Secret Releasable Allies(AUCANZUKUS + NATO) Coalition connectivity over the DISN-LES network: SSC-SD, San Diego, CA Unclass & Coalition NORTHCOM, Peterson AFB, CO Unclass, Secret, & Coalition ESC, Hanscom AFB, MA Unclass, Secret, & Coalition MARFOR/NSWC-DD, NSWC Dahlgren, VA Unclass, Secret, & Coalition NGA, Bethesda, MD Secret, & Coalition DTRA, Alexandria, VA Coalition JITC, NSWC Indian Head, MD Coalition DISA EAGLE, Falls Church, VA Unclass, Secret, & Coalition AITS-JPO, Arlington, VA Unclass, Secret, & CoalitionThe following US sites were provided coalition connectivity over the DATMS-U Network: NCTAMSPAC, Wahiawa NCS, Wahiawa, HI Unclass USFK/ROK, Yongsan Main Post, Seoul, S. Korea CoalitionThe following allied sites were provided coalition connectivity using a combination of the DISN-LES, DATMS-U, dedicated Point-to-Point (Pt-to-Pt) US leased circuits, allied Public ATM(PATM) networks, and dedicated Pt-to-Pt allied leased circuits: NATO NC3A, The Hague, Netherlands NATO Contingent NORWAY, Lillehammer, Norway NATO National NORWAY, Lillehammer, Norway NATO National NORWAY, Kolsass, Norway NATO National FRANCE, Taverny, Celar, France NATO National Germany, Euskirchen, Germany NATO National ITALY, Rome, Italy NATO National SPAIN, Madrid, Spain NATO AC-T, Norfolk, Virginia UNITED KINGDOM, DSTL, Portsdown West, UK UNITED KINGDOM, DGIA, Feltham, UK 13
  • 14. JWID 2004 Final Report UNITED KINGDOM, JARIC, RAF Brampton, UK CANADA, CTDC, Tunneys Pasture, Ottawa, Canada CANADA, CFEC, Shirleys Bay, Ottawa, Canada CANADA, J2IM, Ottawa, Canada CANADA, SBML, Shirleys Bay, Ottawa, Canada CANADA, DRDC, Valcartier, Quebec, Canada CANADA, 1CAD, Winnepeg, Canada AUSTRALIA, DSTO Fern Hill, Canberra, Australia AUSTRALIA, DNC4ISREW, Campbell Park Offices, Canberra, Australia AUSTRALIA, ADF Warfare Center, Newcastle, Australia AUSTRALIA, DSTO Edinburgh, Adelaide, Australia NEW ZEALAND, JISA Porirua, Wellington, New Zealand NEW ZEALAND, HQJFNZ, Trentham, New Zealand NEW ZEALAND, TAC NOC, Auckland, New ZealandUK leased E-3 (34Mbps) access circuits to the British Telecom Public ATM network from RAFMolesworth, UK, and Portsdown West, UK. A 2Mbps virtual path (VP) was established throughthose circuits to provide CFBLNet access from Portsdown West, UK to RAF Molesworth, UK.This provided connectivity to the coalition Point Of Presence (POP) at RAF Molesworth for theUK. From Portsdown West, the UK extended service to RAF Brampton and Feltham, UK.NATO leased a dedicated E-3 (34Mbps) circuit to CFBLNet connectivity from NC3A, TheHague, Netherlandsto the coalition POP at RAF Molesworth, UK. An E-1 (2.048Mbps) ISDNdial-up connection to the Public Switched Telephone Network (PSTN) remained in place at bothRAF Molesworth, UK, and at NC3A, The Hague, NL, to support NATO connectivity should theE-3 line fail. The E-1 ISDN was not used during the JWID 2004 event. NATO operated a T-1connection from ACT, Norfolk, VA, to the AITS-JPO in Arlington, VA. From NC3A, TheHague, NL, NATO provided connectivity to France, Germany, Italy, Norway, and Spain. AtLillehammer, Norway, both a NATO contingent and a Norway national contingent operatedwithin the facility.Australia leased a dedicated DS-3 (45Mbps) line connection between HMAS Harmon, Canberra,Australia and NCTAMSPAC, Wahiawa NCS, Oahu, HI. The DS-3 supported an ATM cell-bearing virtual path (VP) between Australian-owned NORTEL PassPort ATM switches at HMASHarmon, Canberra, Australia and NCTAMSPAC, Wahiawa NCS, Oahu, HI. The AustralianPassPort ATM switch connection was extended from HMAS Harmon, Canberra, Australia, to aPassPort switch at DSTO Fern Hill, Canberra, Australia. The DSTO Fern Hill PassPort switch 14
  • 15. JWID 2004 Final Reportwas then connected to a collocated Marconi (FORE) ATM switch. The Wahiawa PassPort switchwas connected to the collocated DISN-LES CFBLNet ATM switch. A 10Mbps PVP wasestablished from DSTO Fern Hill to DISN-LES CFBLNet Wahiawa over this path. The use of aDISN-LES node at Wahiawa allowed traffic from Australia to choose whether it needed to go toUSFK, or directly to the CONUS, thus preventing potential node isolation and path loading.From DSTO Fern Hill, Australia provided connectivity to other sites in Australia as well as theprimary connection to New Zealand. The connection to New Zealand was an IP router over anATM-based connection.Canada leased an OC-3c (155Mbps) access circuit at DRTC, Tunneys Pasture and a DS-3(45Mbps) access circuit at AITS-JPO, Arlington, VA, to the Bell Nexxia Public ATM (PATM)network. A 15Mbps PVP was ordered by Canada via these access circuits and the PATM tosupport Canadian access to the CFBLNet from Tunneys Pasture, Canada, to AITS-JPO,Arlington, VA. This link used a Canadian TACLANE with JWID key material at both ends ofthe link to encrypt network traffic.New Zealand established a new 512kbps dedicated connection from New Zealand to Australiafor CFBLNet. The existing 512Kbps ISDN dial-up circuit from New Zealand to AITS-JPO,Arlington, VA, was left in place as a back up to the dedicated service. The ISDN dial-up was notused during JWID 2004.4.2.1 Lessons Learned (Provisioning)It is essential to remove any option to add new locations after the Mid Planning Conference. Thechanges to locations and late decision to add two sites added to the complexity of standing upand testing JWID 2004 locations prior to the JWID 2004 execution.The use of separate Unclassified ATM switches at US user locations for JWID 2004/CFBL andother programs allowed the clean separation of user traffic for JWID from other user siteprograms. This was accomplished by standing up separate PVPs that terminated only on theJWID Unclassified ATM switches, allowing simultaneous program support for both JWID andother DISN-LES user programs. It also allowed better isolation of network faults by enablingpersonnel to focus only on the JWID PVPs. 4.3 Network AccreditationIn accordance with the DoD Information Technology Security Certification and AccreditationProcess (DITSCAP) and the guidelines of the Annex C, network accreditation is two-partprocess. First, a site is accredited for the infrastructure, which allows for connectivity to thenetwork and testing. Second, all initiatives that take place on the CFBLNet require an additionalaccreditation package from the US sites for that initiative. JWID 2004 is an example of just suchan initiative. Timely site accreditation became an issue during JWID 2004 for the US sites, 15
  • 16. JWID 2004 Final Reportespecially for the new sites participating in the demonstration. Another issue regarding the USsites was a lack of separation of the accreditation packages. The US sites had combined theinfrastructure and the initiative accreditations into a single package. As a result, several US sitesinitially presented only the infrastructure accreditation package for connectivity to the network.4.3.1 Lessons Learned (Network Accreditation)All US sites connecting to the DISN-LES must follow the rules and guidelines in theaccreditation connection requirements and the DITSCAP. US sites connecting to the CFBLNetmust follow the guidelines of Annex C of the CFBLNet charter. A number of the sites were slowto submit their completed accreditation packages, resulting in a degraded ability to meet theestablished timelines for network testing. Emphasis needs to be placed on utilizing the standardprocedures for future initiatives and demonstrations.Continuous tracking of each participating site’s accreditation status is essential to ensuringaccreditation is completed on schedule. This requires intense management and timelyintervention when a site begins to fall behind schedule. A finalized list of participating sites andtrials with the network connection requirements for each site must be produced as early aspossible in the planning process. This will allow the sites to present their initiative accreditationpackage in a timely manner.The security accreditation briefing for CWID 2005 will be conducted differently next year. At theend of the briefing a representative from each site will be asked to sign an attendance sheet andwill be given an accreditation CD, containing information that can be used as a guide to preparethe accreditation packages. JPO contact information, to include name, number and emailaddress, will also be available for further assistance. 4.4 Communications Security (COMSEC)All COMSEC equipment and keying material (keymat) was sent in a timely manner. TheNational Security Agency (NSA) has granted the AITS-JPO an alternative means of deployingkeymat to Allied CFBLNet members. <whats the alternative means?> Keying devices, such asthe Data Transfer Device (DTD), were used as the main storage device for all electronic keymat.With the current availability of the STU-III or STE secure telephones to the Allies, the electronictransfer of keymat to the National Distribution Accounts was performed in a matter of minutesversus days or weeks via the physical shipment method. The receipt of the COMSECimplementation messages sent via DMS continues to be an issue with Australia. At this point,the Australian AUTODIN interface seems to be the problem. This issue will continue toresearched and worked during the course of this year as we continue to provide Australia withCOMSEC support. Network encryption devices were loaned to NATO and New Zealand forsuccessful connectivity to the network. 16
  • 17. JWID 2004 Final Report4.5 Network TestingBandwidth utilization testing began on May 12, 2004 and was completed May 25, 2004. Adetailed test plan was created and distributed to all site engineers via email. The purpose of thebandwidth utilization tests was to ensure that a clean connection could be maintained to each siteto the maximum of the site’s purchased bandwidth.To perform testing, each site needed a machine able to run the MGEN 4.x program availablefrom the Naval Research Laboratory’s website (http://mgen.pf.itd.nrl.navy.mil/).The idea behind using MGEN for network testing was to be able to accurately detect networkproblems in an end-to-end manner to and from each site and the AITS-JPO.MGEN generates UDP traffic at any size and rate that is given to the program. Given this ability,a Perl script was created to start at the lowest possible packet size and steadily increase thebandwidth used until attempting to push 5% above the known pipe size. The expected results ofthis test were, when the traffic received at each site was graphed, a stair-step pattern would resultwith the top level of the graph appearing as a flat line since more traffic was being generatedthan could travel through the pipe. The figure below is an example of a correctly functioningpipe. 17
  • 18. JWID 2004 Final ReportFigure 7 Correctly Functioning PipeDuring the testing, two circuits were discovered that dropped packets once maximum throughputwas reached. The faulty circuits were isolated and located between Australia and the AITS-JPOand between SSC-SD and the JPO. In both instances, these packet drops were isolated to ATMrate-shaping memory buffer overflows. These problems were rectified before JWID 04execution in both instances by reallocating memory buffer space. The figure below is an exampleof one site that was dropping packets as described.Figure 8 Faulty circuit dropping packets 18
  • 19. JWID 2004 Final ReportSeveral sites provided test hardware/software that was either not capable of running the MGEN4.X program or the respective site missed their test date and provided the test hardware platformlate. In these cases, we noted which links may be affected and did our best to ensure that anadequate (over 3Mbps) amount of data could be successfully sustained through the link. Thiswas accomplished by using the much less reliable “ping” test to and from the remote site.4.5.1 Lessons Learned (Network Testing)The success of JWID 2004 illustrated the many benefits of competent and diligent networkplanning, implementation, and execution. A spirit of collaboration and understanding existedamong all JWID 2004 network engineers; nonetheless, there is room for improvement in thefollowing area:Importance of test schedule: The test schedule must be discussed and agreed to during thethree planning conferences. Site engineers should make every attempt to adhere to the testschedule or reschedule as soon as possible if something disrupts planned tests. The importanceof thoroughly testing the network and services must be stressed at each of the planningconferences. 5. NETWORK SERVICESA coalition staff representing Australia, Canada, New Zealand, the UK, the US and NATOdeveloped, planned, engineered, implemented and maintained the network services for JWID2004. The tasks required skills in UNIX and Windows platforms, network management tools,collaboration tools, web, Domain Name Service (DNS), and electronic mail (email).The network services requirements for JWID this year were extremely complex, given theestablishment of multiple coalition domains. In addition to the AUSCANNZUKUS + NATOdomain, three additional domains were implemented: the ROK domain, the NATO-Only domainand the Partners for Peace (PfP) domain. Each of these domains required a set of core servicesand restricted connectivity between domains. The NATO-Only and PfP domains wereimplemented on the national side of the NATO CFBLNet point of presense, and NATO held theresponsibility for standing up core services in these domains and, therefore, will not be discussedin this document. The ROK domain was executed on the national side of the US CFBLNetdomain with no other coalition participation; therefore, the US was responsible for implementingcore services in this domain. For the ROK domain, a DII Mail guard and a Radiant Mercuryguard were included in the infrastructure to allow filtered email with attachments and theCommon Operating Picture (COP) to flow between the AUSCANNZUKUS + NATO and ROKdomains. 19
  • 20. JWID 2004 Final Report 5.1 EmailFor JWID 2004, each nation was responsible for the implementation of an email server withintheir national domains as shown in Figure 9. This alleviated the requirement of managing allcoalition email accounts from the CCCC-Rear.JWID email accounts are role-based accounts. The list of role-based accounts was retrieveddirectly from the MSEL Web-Based Tool (MWBT) and fed into the server located at CCCC-Rear. Email accounts were also created for trial support personnel and site support staff on a per-request basis. The CCCC-R sent site engineers a list of email accounts and passwords to bedistributed to the role players and support personnel. Dahlgren successfully hosted its own emailaccounts during JWID 2004. UK NATO Portsdown West, UK Lillehammer, NO CA NO Kolsas, NO Shirleys Bay, CA US JWID NATO SP Madrid, SP Arlington, VA Hague, NL CFBL AS IT GE Rome, IT Euskrichen, GE Williamtown, AU NZ FR Porirua, NZ Bruz, FR CFBL NATO Only PfP ROK Figure 9. JWID 2004 Email ImplementationThe email service used Simple Mail Transport Protocol (SMTP) to ensure interoperabilitybetween nations and organizations, but the specific implementation was at the discretion of thenation or organization. Microsoft Exchange, Qmail, and Postfix existed and successfullyinteroperated on the CFBLNet. IMAP was used to download messages from the email server tothe clients.The US provided a DII guard that allowed CFBLNet approved labeled email messages withattachments to flow between the CFBL and Republic of Korea (RoK) domains. Since there wasonly one email guard for the RoK domain, all messages destined for *.kr were routed to the USemail server, and the US email server, in turn, relayed these messages to the guard. This allowed 20
  • 21. JWID 2004 Final Reportfor tighter control over potentially sensitive email messages and made the guard implementationsimpler. The US implemented the DII Mail Guard, version 3.0.2, in conjunction with MimeSweeper, version 4.3, for SMTP. The JWID multinational community developed and agreed tothe guard configuration and rule set during the JWID planning conferences. The guard wasconfigured to check the classification and releasability marking in the first line of each emailmessage, valid attachment types and a list of “dirty” words that was developed to minimize thepossibility of inadvertently sending inappropriate information from CFBL to RoK. The coalitionapproved releasability markings are shown in Table 1. Those messages marked with the labelAUSCANNZUKUS + NATO REL ROK were allowed to flow from the CFBL domain to theRoK domain. The guard rejected messages marked with any other label as well as unmarkedmessages. AU AU CA CA NZ NZ UK UK US US NO NO FR FR IT IT DE SECRET DE ES CONFIDENTIAL REL ES TU RESTRICTED TU DK UNCLASSIFIED DK HU HU NL NL NATO NATO ROK ROK RO RO SE SE FI FI AUSCANZUKUS Table 1: Valid classification markings for mail messages sent from the CFBL to Other JWID CFBL DomainsThe attachment types listed in Table 2 were allowed to flow between domains after beingchecked for viruses and other malicious code by the guard. Valid Attachment Types (6-to-RoK and RoK-to-6) Microsoft Office • Word - .doc, .rtf, .txt 21
  • 22. JWID 2004 Final Report • PowerPoint - .ppt, .pps • Excel - .xls, .csv • winmail.dat file generated by Outlook - .dat Graphic files - .bmp, .gif, .tiff, .jpg, .jpeg, .tif Adobe Acrobat - .pdf Table 2: Valid attachment types for messages sent between CFBL and ROK domains REL 5.1.1 Email Improvements- The use of IMAP this year prevented users from accidentally downloading all of their messages from the server. This was especially important in the case of shared accounts on multiple machines.- Forcing all email for the *.kr domain to go through the US email server reduced the complexity of the guard implementation and allowed a tighter control path. 22
  • 23. JWID 2004 Final Report - Using an email server that stores all of its messages in plain text format made it easier to backup and restore and to verify message delivery. - Using a highly verbose SMTP and IMAP server allowed for refined troubleshooting of client communication problems and increased auditing capabilities. - Administrative accounts for the various top-level services were created to end the mass of e-mails that routinely filter through the ccccr@aits-jpo.joint.us account. Accounts such as postmaster@ and hostmaster@ were used for email and DNS issues respectively and were a much more effective form of communication. 5.1.2 Lessons Learned Recommendations - Modifying the IMAP server to be case-insensitive should be done for CWID 2005. Even though case-sensitivity was announced, some users still had trouble formatting the long names properly. - Adding a web-based email client would be a nice feature to add to the email service for sites that do not wish to install an email client for whatever reason. - The email account names need to be static after MSEL lockdown is announced. Any changes made after that time need to be communicated up the chain to the email administrator. - A more fail-safe solution for email service needs to be applied. This comes at a cost for additional redundancy and infrastructure but would better guarantee reliability. On the UNIX-like systems, rsync or rdist over SSH may be a viable alternative when combined with an appropriate setting of round robin DNS entries. - An LDAP/S- or SASL/LDAP/S-based authentication mechanism would greatly improve service authentication and integration. The authentication could also be passed through LDAP/S to Kerberos. 5.2 DNSDNS was provided in a hierarchical structure with the US and the UK providing the top-levelroot name servers (A and B) and other coalition partners providing national root servers. Eachcountry was responsible for providing its own DNS server unless other arrangements were madewith another country or organization. 23
  • 24. JWID 2004 Final ReportThe JWID 2004 sites followed the naming convention of site.service.country with the exceptionof the Republic of Korea, which followed the naming convention of site.service.8e.country.Mail Exchanger (MX) records were assigned appropriately within each DNS server with theunderstanding that all email servers would use the records appropriately.5.2.1 Improvements on DNS ServicesDuring JWID 2004, for redundancy an additional DNS server was made available via CCCC-Rear that was kept synchronized with the primary DNS server, also located at the CCCC-Rear.5.2.2 Lessons Learned Recommendations - To allow for more controlled and secure access and management, DNSSEC should be implemented. This will require private key exchanges between the various sites that wish to perform updates. - There should be a way for an administrator at any site to manage their own DNS entries to alleviate some of the problems with personnel time differences. This method could be web-based or could rely on remote update capabilities such as Dynamic DNS. - Start of Authority (SOA) record timeouts should be standardized across CFBL except for internal zones. This will help keep update times reasonable. Recommend the following intervals in the SOA record: o Refresh Interval - 1 hour o Retry Interval - 15 minutes o Expire Interval - 1 day o Default TTL - 12 hours - More sites should consider hosting their own DNS. By hosting their own DNS, the sites will gain additional internal name resolution reliability and may experience fewer issues with finding hosts. This will require a person at each site to be able to implement and maintain a DNS server. 5.3 IP TelephonyThe implementation of IP telephony in JWID 2004 was the most extensive JWIDimplementation ever. In fact, an additional call manager was fielded at Dahlgren to reduce 24
  • 25. JWID 2004 Final Reportbandwidth to the CCCC-R. The implementation used Cisco Call Managers, version 3.32, andCisco series 79xx IP phones. The IP telephony system was configured to use the G.711uncompressed vocoder, approximately 90kbps of IP bandwidth per call, for calls between mostsites/countries. The G.711 uncompressed vocoder was chosen to support the Cisco Call ManagerConference feature.Conference capabilities were again offered to the engineering community and were used for thedaily engineering conference calls. The IP phones also proved invaluable for assisting the trialsin locating their peers at other sites/countries. This allowed for secure conversations and savedon commercial long distance calls.Another addition to JWID 2004 was the availability of a backup call manager. This call managerwas available if the primary call manager experienced hardware or software failure. Redundancyis necessary because IP telephony is such a widely used service during JWID.5.3.1 Lessons LearnedIP phones proved to be invaluable tools for both the role players and engineers throughout JWID2004. They should continue to be deployed extensively at each participating site. The effect ofIP telephony and bandwidth usage should be evaluated at each site to determine the number of IPphones they require.As IP phones are more widely deployed, the impact on site security should be evaluated. DuringJWID 2004 setup week common security tools like Nessus were run on the Cisco Call Managerand were able to bring the system down. More work should be done with the vendor todetermine the correct security settings and vulnerabilities associated with the call manager.The overall implementation of IP telephony would benefit from vendor-provided training andconfiguration. The current call manager software includes a number of useful features, such asvoicemail and call manager clustering. These capabilities along with the many new features thatare available with the latest release of the Cisco Call Manager software should be researched foruse in CWID 2005.IP phone directories for all sites should be finalized during setup week and made available to allusers via the JWID Portal prior to execution. This would reduce the number of calls to the sitemanager requesting IP phone numbers.The call manager’s success on the Coalition domain indicates that it would be very useful toimplement on the US National and MACA Domains for CWID 2005. 25
  • 26. JWID 2004 Final Report 5.4 WebThe JWID 2004 web site, http://www.joint.nz, was hosted by New Zealand. The hardware was aPentium III 666 MHz with 512 MB RAM and 26 GB hard drive running Windows 2003. Theweb site featured the JWID Portal running on a Microsoft Internet Information Services (IIS) 6.0web server, collaborative newsgroups served by the IIS Network News Transport Protocol(NNTP) service, and MySQL 4.0.20 database. A mirror site running an UNIX FTP service waslocated CCCC-R, Arlington, VA, USA, and supported the network infrastructure team. For thesake of redundancy, the JWID site was mirrored at the CCCC-R. This mirror copy was brieflyutilized while NZ was still preparing to bring its network connection online. The mirror serverhardware was a Sun Blade 1000 with 512 MB RAM and 73 GB of storage. Netscape EnterpriseServer 6.1 was installed along with FTP, Netscape Directory Server 5.1 for authentication,MySQL 3.23.32 for database storage, and Perl version 5.005_03 to host the portal and otherservices.5.4.1 Overall Web Structure and FormatThe JWID 2004 Portal web site was located at http://www.joint.nz. The JWID 2004 web siteformat was based on the JWID 2003 site in design and functionality as well as the JWID Publicwebsite that runs continuously on the Internet, located at http://www.jwid.js.mil. Documentsposted to the site were converted to HTML and PDF due to requests from the international JWIDcommunity to publish documents in non-Microsoft specific formats. Core site functionsincluded a search capability, hyperlinks to JWID Newsgroups, and a Perl/MySQL-based BubbleChart and Lessons Learned database. 26
  • 27. JWID 2004 Final Report Figure 10 JWID 2004 Web SiteMost information published to the JWID Portal was gathered prior to scenario execution. If dataneeded to be updated during the execution phase, it was sent to the CCCC-R Webmaster emailaccount or uploaded to the FTP site. The CCCC-R web administrator would make the necessarychanges and then email the files to the corresponding NZ web administrator, who would receiveand upload the file to the JWID 2004 web server. Each day the NZ web server would, via anautomated script, export the MySQL database into an Excel spreadsheet and upload it to adesignated area on the FTP site. The CCCC-R web administrator would then retrieve thespreadsheet, export the data to a comma delimited text file, and import it into the MySQL serverat the CCCC-R for redundancy.The JWID 2004 Portal design included top-level menus for All Users, CTF Staff, Site Director,Site Engineer, Links, and Lessons Learned. The front-page graphic design provided quick linksto the JWID Overview, Plan of the Day, Daily SITREP Newsgroup, Bubble Chart, CTFWebTool, DCTS, and the JWID 2004 Welcome Video.5.4.2 NewsgroupsNewsgroups were a valuable service through which JWID warfighters, engineers andmanagement staff could quickly exchange information. Users had the ability to uploadinformation and read all other articles posted. The following newsgroups were set up on thenewsgroup server: 27
  • 28. JWID 2004 Final Report Newsgroup Group of JWID Warfighters/Newsgroup function jwid04.sitrep SITREP jwid04.nwg NetworksWorking Group Table 3. Newsgroups5.4.3 Bubble ChartThe JWID 2004 Bubble Chart served as a dynamic management tool to rapidly ascertain thecurrent status of all JWID network services and Coalition Interoperability Trials (CITs). Thenetwork services chart displayed the status of core services across all sites. The CIT chartdisplayed the operational status of each CIT. Due to the large number of CITs, controls wereadded to display only a subset of all CITs at glance. Also, a mouse-over feature was added sothat the descriptive names of the CITs would appear in addition to the trial numbers. On bothcharts, the last time the status was updated was displayed in Zulu format. On the networkservices chart the US sites were rolled-up into a single line with the option to click on the link toview the status of each site.Each site manager had the capability to mark each Bubble Chart block with green, yellow, red, ornot applicable. A corresponding legend was added this year to define the four status categories.A JWID user could get an overview of the status of each site, and then click on the site name toview more detailed comments on the site’s status. In addition, a history feature enabled users tosee a complete set of all issues over the course of the demonstration. The history could befiltered by site, date, network service or CIT, or status. 28
  • 29. JWID 2004 Final ReportThe Bubble Chart content was classified as 6-eyes releasable. An Access Control List (ACL)was established to control updates to the charts. To update the chart, a site-specific passwordwas assigned and distributed prior to JWID execution. Once the appropriate username andpassword were entered into the password dialogue box, the CGI script would automatically takeusers to the update page for their respective site. Users could only update those sites for whichtheir username and password were assigned. In addition to updating status, the site managercould also include comments for each status box. Whenever the status was changed, or a newcomment was entered, the timestamp would update automatically. The latest timestamp wouldthen be reflected on the overview page for the network services or CIT Bubble Chart. Figure 11 Example Network Bubble Chart 29
  • 30. JWID 2004 Final Report Figure 12 Example CIT Bubble Chart5.4.4 Lessons Learned DatabaseThe Lessons Learned database was a CGI/Perl-based form where users could input theircomments, and make suggestions to improve future JWIDs. It served as an important feedbackmechanism. The form collected name, organization, email, phone number, area of comment,comments, and suggestions. Once added to the database, all users could view all comments.Sixteen comments and suggestions were received. 30
  • 31. JWID 2004 Final Report Figure 13 Lessons Learned Database5.4.4.1Lessons Learned about Lessons Learned DatabaseOBSERVATION: The lessons learned database provided little feedback to the user when alesson had been successfully entered into the database.IMPACT: Users might feel confused and enter lessons multiple times resulting in redundancyand a lack of consistency between multiple versions of the same lesson.RECOMMENDATION: Develop a short response for users when data is successfully enteredinto the database, such as "Lesson successfully entered into database. Would you like to enteranother?"OBSERVATION: The lessons learned database does not allow users to preview a lesson beforeit is entered into the database.IMPACT: Incorrect or inaccurate information may be presented in a lesson. 31
  • 32. JWID 2004 Final ReportRECOMMENDATION: Develop a preview capability to allow users an opportunity to double-check lessons before committing them to the database.OBSERVATION: Multiple users saw variations of the same problem, but the lessons learneddatabase does not allow for additional users to comment on a single lesson learned. Commentsby additional users may bring clarity to a particular issue and facilitate resolution orimprovement.IMPACT: Multiple lessons learned from multiple users are difficult to correlate and adequateresolution may not occur.RECOMMENDATION: Develop the capability to allow multiple users to comment on a singlelesson learned. This will provide greater clarity on the issue at hand.OBSERVATION: The lessons learned database is an opportunity for collaboration at a differentlevel than the hotwash. This opportunity enables personnel with technical knowledge of aproblem to collaborate in a way that hotwashes and final reports cannot capture.IMPACT: Technical collaboration on lessons learned will improve future JWIDs.RECOMMENDATION: Available weblogging software meets the above recommendations andallows for collaboration on multiple topics as well as for documenting the various lessonslearned throughout the course of the demonstration. Software such as Movable Type orWordpress enables project collaboration, moderation, and documentation of lessons learned.This software is specifically engineered for multi-user environments. The technical requirementsfor such software are relatively minimal, typically, a web server with a database and support forany number of scripting languages such as Perl and PHP. This software is also cost-effective.For example, Wordpress is free for commercial use, and Movable Types fee is minimal.5.4.5 Master Scenario Events List (MSEL) Web Based Tool (MWBT)During JWID 2004, the AITS-JPO hosted the servers for the MWBT and provided systemadministration support and back up operations. The Scenario Working Group provided contentdesign and maintenance. During the course of JWID 2004 planning it was determined that theMWBT resource needed to be available on all three classification domains. This required theAITS-JPO to stand up two additional servers, four servers in all. Prior to execution usersaccessed the unclassified MWBT server via the web and entered MSELs into the database. Theservers for the Coalition and the US National Domains where kept off line until 4 June. At thattime the Scenario Working Group locked down the unclassified MWBT, restricting changes tothe MSEL database. System administrators installed copies of this database on the servers,which were accessible to both the Coalition and US National domains. The locked-downMWBT was restarted on all three domains and made available to JWID users on 4 June. At theend of JWID 2004 the servers on the Coalition and the US National domains were again takenoff line. The databases were verified to be unclassified and a copy of all three is stored and 32
  • 33. JWID 2004 Final Reportaccessible via the unclassified server at http://jwid.les.disa.mil/mwbt andhttp://jwid.les.disa.mil/jdcat. Tape backup copies of all three databases are stored at the AITS-JPO. 5.5 Defense Collaboration Tool Suite (DCTS)JWID 2004 implemented DCTS, version 2, phase 1, which was the same version that was usedduring JWID 2003 execution. DCTS supported all three networks. The key to successful DCTScollaboration capabilities was to ensure sufficient availability of network bandwidth, properlysized servers and clients (high CPU, video card and memory capabilities), and adequate usertraining levels. Network statistics showed that the available bandwidth during JWID 2004 wassufficient to support the scope of DCTS for the JWID scenario despite the constraint of runningthree networks on the same physical access circuit.5.5.1 DCTS ON THE CFBLNetFourteen CUSeeMe servers, running version 6.0.6.49, were deployed on the CFBLNet (Figure14). The biggest change from previous years was the implementation of the domain model ofconference server configuration. In previous JWIDs a configuration file was distributed to allsites in order to link conference servers. All sites were then brought online systematically inorder to ensure they federated properly. If any additional conferences were requested, a newconfiguration file was created and distributed, which required an additional systematic reboot byall sites. In the domain model configuration, once a site’s conference server joined the domain,the administrative server updated the configuration information automatically. A configurationfile and a systematic reboot were not needed. By using the domain model the administration ofDCTS was centralized at the AITS-JPO. Requests for additional conferences during executionwere done without additional coordination from the remote sites. Also in previous JWIDs, theMCU at the AITS-JPO served as both the top provider as well as a conferencing server. Toalleviate traffic to the top provider an additional server was installed at the AITS-JPO to actsolely as a conferencing server for the DCTS users in the National Capital Region. 33
  • 34. JWID 2004 Final Report Top Provider AITSJPO2 Dahlgren 1 CANADA AAQ Dahlgren 2 UK HANSCOM New Zealand AITSJPO 1 EAGLE NORTHCOM SPAWAR NATO NC3A AUSTRALIA Figure 14 CFBLNet MCU TopologyConference server stability did not improve from JWID 2003 because the same version ofCUseeMe (6.0.6.49) was used; however, each DCTS conference configuration was the same asin previous years in terms of codecs: H.323 (video), G.711 (audio). In order to attempt toimprove performance with conferencing, bandwidth settings for each conference increased to512K from client to server (256K in 2003), and 1024K between CUseeMe server to CUseeMeserver (512K in 2003).T.120 connectivity between some CUseeMe Servers in certain conferences was unreliable attimes. There were instances when the T.120 portion of some conferences became unlinked fromthe Top Provider (AITS-JPO Server). This happened with the Hanscom, UK, and NZ servers.When this unlinking occurred, engineers performed two troubleshooting methods to fix theproblem. Engineers restarted the CUseeMe service on the “problematic” server. After therestart, conferences sometimes linked, and sometimes did not. The chance of linking was greaterwhen there was not much activity on the top provider, but some conferences did not link evenwhen there was no activity on the top provider. Also, the engineers deleted and recreated“problematic” conferences. After the conferences were recreated, they linked in to the topprovider and full functionality was restored. The UK and NZ conference server service seemedto stop for unknown reasons at the end of each execution day. At the end of each week ofexecution the servers were systematically rebooted to clear the servers memory and re-link allthe conferences with the top provider.While the T.120 issues provided the most numerous connectivity problems, the issues that wereexperienced during last year’s hotwash were not experienced during JWID 2004. During the 34
  • 35. JWID 2004 Final Reporthotwash the NetMeeting video stopped transmitting halfway through the briefing. Even thoughvideo stopped switching after Australia’s portion of the brief, audio continued to transmit clearly.Despite the problem with video, a reboot of the CuseeMe servers was not required. The audiotransmitted very clearly and conference participants were able to view the shared application5.5.2 Lessons LearnedIn order to fully understand the issues that affected DCTS during JWID 2004, it is crucial to haveFirst Virtual Communications, the developers of CUSeeMe, observe future JWID exercisesduring execution. First Virtual Communications viewed the configuration of the conferencesprior to JWID and noted no problems with the configurations. They pointed out that JWID usedmuch older versions of their conference server, and that their new version of CUSeeMe fixesmany of the problems that JWID users have experienced in the past. Their on-site guidanceduring execution would have been crucial when anomalies like the connectivity issues describedabove occurred.As experienced during JWID 2003, closing out Net Meeting incorrectly continued to causeproblems the next time the user connected to a conference. When a user clicked the “X” in theupper right corner of the Net Meeting window to end a collaborative session, T.120 capabilitieswere sometimes not available when that user attempted to enter a new conference. Users mustclick the “hang-up” icon when exiting a conference.Another issue was users not using their local conference server while in conferences. Thisproblem was present throughoutJWID, despite phone calls to the site informing the site of the problem.The failure of users to mute their mikes caused echoing and the transmission of unwantedbackground noise during several collaborative sessions. Because this has been an issue for sucha long time, U-Talk was developed to fix the hot mike problem during conferences. The finalrelease of U-Talk was developed and released just before execution so there was not adequatetime to train the users. The decision was made not to implement the web part on the CFBLNetand users continued to complain about the hot mike issues throughout the exercise. U-Talk wasfully implemented on the MACA and US National networks, and as a result users did notexperience many problems while in conferences, and conference servers were much more stable.The troubleshooting process was greatly improved with use of the UMon tool. Administrators atthe AITS-JPO could view the status of all conference servers instantaneously. Administratorscould also troubleshoot problems more quickly based on the information provided by UMon.The UMon tool was also able to continuously display network latency between the conference 35
  • 36. JWID 2004 Final Reportservers, allowing the DCTS administrators to inform sites that their DCTS performance wasbased on intermittent bottlenecks on the network. In previous years, data like that, which UMonprovides, was not available.DCTS user and site engineer training for all JWID sites prior to execution was imperative. Manyof the problems that were experienced during JWID were user-related or user-workstationrelated: users not exiting the conferences properly, users not using the push-to-talk methodology,etc. If DCTS training sessions are implemented at the sites, many of the problems that wereexperienced by the users could have been alleviated. Also, for future JWIDs it is critical to havea DCTS trainer on-site during JWID execution. If a DCTS trainer had been on-site most, if notall, problems would have been quickly fixed or totally avoided. The U-Talk web-part could havebeen easily implemented had a DCTS trainer been present at each site to instruct them best onhow to use the tool.5.6 Network Engineering ManagementComprehensive network monitoring/management is critical in early detection of networkproblems. Additionally, Network monitoring/management tools are capable of providingfeedback to sites and engineers, and demonstrations and applications as to why they wereexperiencing problems. During JWID execution, not only basic site connectivity, but alsobandwidth utilization categorized by application, host, port number, etc…; OSPF and BGProuting protocol status; ATM SVC status; ATM signaling errors; port and interface input andoutput errors; multicast neighbor status; and, multicast shortest tree path (STP) status should bemonitored. This information, along with the ability to monitor and initiate trouble tickets, shouldbe readily available to any site on the JWID network.During JWID 2004, the CCCC-R used HP OpenView Network Node Manager (NNM) andOpenNMS for general network monitoring. NetScout, Network Traffic Analysis System(NTAS), and Flow Tools and FlowScan were used for monitoring bandwidth utilization;TCPDump and TRPR were used for monitoring DCTS server-to-server traffic in both real andnon-real time. For trouble ticket generation, ARS Remedy was used. All of the tools were web-based which allowed site engineers from any site to easily view and/or update the tools with theappropriate permissions. Other tools were evaluated but deemed either non-applicable or toocomplex for proper setup during the timeframe available.5.6.1 Network Monitoring ToolsThis year, the CCCC-R made strides to evaluate many Open Source (OSI Certified License)applications to provide greater network monitoring and reporting capabilities at lower costs. Theinformation shown below notes some of the Open Source tools that were used along with themore traditional tools used during JWID 2004. 36
  • 37. JWID 2004 Final Report5.6.1.1 NetScoutThe NetScout system worked very well to monitor standard SNMP devices and to connect toprobes at the JPO and at other US sites. The probes allowed a much deeper look at the trafficstatistics passing across the network at various points. Figure 15 NetScoutPros: - A unified web-based interface made it very easy to get to all of the information from any machine. - The newspapers were helpful in providing customized reports to the people that needed them. - This was the only device available that was able to look into ATM traffic. - SNMP traps can be sent to other devices on noticed events 37
  • 38. JWID 2004 Final ReportCons: - The web page was not SSL encrypted. - Getting custom email alerts did not seem to be possible (this may be a configuration error). - Applying this technology to another information domain was cost-prohibitive for a time as short as JWID execution.Conclusion:The NetScout product is a very effective tool and provides a good look into the traffic along anylink. As such, having a probe at every ATM link would be advantageous. However, securityneeds to be taken into more consideration and the cost of applying the NetScout solution to otherinformation domains needs to be analyzed.5.6.1.2 HP OpenView Network Node Manager (NNM)The HP OpenView NNM was mainly used to provide a high-level view of network devices thatthe CCCC-R wished to monitor. A map was put together that contained all of the interfaces.Each interface, or parent node, would change color if the node were unreachable via “ping”.Email alerting capabilities are present in NNM as well as the ability to integrate into ARSRemedy. However, neither of these functions was configured for this installation of NNM.From this experience with OV, it would be advisable to have one person dedicated to themanagement and maintenance of OV as well as having that person dedicated to extending themonitoring capabilities of the system. As with NetScout, analysis needs to be performed todetermine whether or not applying an OpenView installation would be cost-effective on otherinformation domains.5.6.1.3 OpenNMSOpenNMS was used throughout the JWID 2004 exercise to monitor all major CFBLNetequipment and many of the services available on the network. The program provided uptimestatistics and graphs per interface, and emailed a designated account if a problem was noticed.This tool was used for statistical information while NTAS was inoperative. 38
  • 39. JWID 2004 Final Report Figure 16 OpenNMSPros: - No cost o This makes it a good candidate for short term networks or low cost deployments - Runs on a variety of Operating Systems - Provides a great deal of required functionality out of the box - Highly configurable and customizable - Can collect SNMP data and SNMP traps and send alerts on data observed - Performs all of the basic monitoring options of HP OpenView NNM - Wrapped via Apache so there is no problem with SSL support or authentication - Can wrapper Nessus for automatic system vulnerability scanning and reporting - Commercial Support AvailableCons: 39
  • 40. JWID 2004 Final Report - Difficult to customize o Utilizes JSP for all of its polling. This makes it more difficult for the average administrator to add additional polling capabilities - No functional “Network Map” (OpenView Style) - Multiple installations may be required to support full data collection on different sites. o This is not difficult just more resource intensive - Quite resource intensive (but no more than HPOV) - Only two real user ‘levels’: Administrator and ViewConclusion:OpenNMS provided a great deal of the required functionality for network equipment monitoring.More time is needed to tailor the program for the CFBLNet environment but the amount ofperformance that was gained for the cost was excellent.5.6.1.4 Network Traffic Analysis System (NTAS)NTAS is a GOTS product that was used during JWID 2003 and again in JWID 2004. NTAStakes NetFlow input from various network devices and allows statistics to be pulled from thatdata. NTAS can also provide link status between devices and link utilization. NTAS was onlyavailable on CFBLNet for the second week of the demonstration. Figure 17 NTASPros: 40
  • 41. JWID 2004 Final Report - No cost - Near real time - Gives a graphical interface to the statistical data collected - Can provide customized reports on a daily basis - Can pass the NetFlow data on to other systems for alternate collection/analysisCons: - Seemed difficult to keep running in a stable manner - Does not allow for user report customization o NTAS developers were required to accomplish this task - The interface was unwieldy - The NetFlow forwarding ability failed several times during execution - Does not support SSL web connections - Does not allow for alerts to be sent via email when a problem is noticed - Does not seem to contain a framework for adding notification abilities - System functionality failed once DISA required STIGS where applied using the required “Gold Disk”Conclusion:NTAS has the potential to be a useful tool in the CFBLNet network-monitoring arsenal if thecons mentioned above are addressed, particularly in the realm of automatic notification. Inaddition, DISA needs to develop a listing of “Not to apply” lock downs exceptions so that NTAScan function in a DISA environment.5.6.1.5 Flow Tools/FlowScanFlow Tools and FlowScan handle NetFlow accounting and reporting. The Flow Tools packageis a suite of tools for collecting, generating, and reporting on NetFlow data. FlowScan is apackage that parses the collected flow data and can generate reports based on flow content.Pros: - No cost - Provided a very efficient means for aggregating flow collections - Could generate very detailed reports on any given set of data - Could provide graphical summaries of flows across time - Would be fairly easy to combine with a wrapper to send e-mail or SNMP traps if a problem is noticed - Extremely fastCons: - Complex configuration - Cryptic (but powerful) reporting syntax 41
  • 42. JWID 2004 Final ReportConclusion:These two packages were an effective way to quickly get statistics regarding network flow dataand should be considered for use in the future. Given the proper reporting requirements and timeto generate the report structure, this software should be able to give any amount of detail that canbe extracted from a network flow. Figure 18 FlowScanCacti is a generic web-based interface that can be used to graph arbitrary linear time data. Thisis especially useful in polling SNMP statistics or statistics from log files and/or servers. Thisprogram was only briefly set up during JWID 2004 so data is limited. 42
  • 43. JWID 2004 Final ReportPros: - No cost - Highly configurable - Data sources easy to define and collect from any programming language - Looks to be quite scalable - Can be wrapped behind Apache for SSL support - Paired with Nagios (below) becomes a very powerful network monitoring solution - Commercial Support AvailableCons: - Configurability at the price of complexity - Automatically collected Router interfaces may not always show correctly o This is not a problem is they are input by hand - Knowledge of RRDTool is not necessary but useful - Cannot natively watch for threshold overruns o This ability can be added to the collection scripts but that is fairly unwieldy o In this case, pairing with Nagios would be the best implementationConclusion:Cacti is a good addition to the network management toolbox. With the ability to easily generategraphs for any time-linear numeric data, statistics can be collected from any data source.However, Cacti requires more hands-on development than OpenNMS.5.6.2 Lessons LearnedJWID network managers have traditionally focused on monitoring the availability of networkcircuits and utilization of those circuits using tools such as MRTG. However, in recent years thenetwork management challenge has expanded to include tracking the status of network servicesand applications. An important piece of this challenge is to characterize network traffic andaccount for how and where it flows. CIT 07.03 Peakflow, CIT 07.12 NetScout and the NetworkTraffic Analysis System (NTAS) developed by DISA’s Technology Integration support groupwere tools that the CCCC-R used in JWID to address this problem.In general all the tools could be useful to managers of operational DoD networks. Peakflowprovided excellent graphing of historical data, but unfortunately CCCC engineers’ evaluation ofthe product was limited due to a NetFlow bug in the Cisco IOS that was used in JWID. NetScoutprovided the best “real-time” data of the three products and was particularly valuable inmonitoring DCTS conferences. NetScout’s historical data provided good high-level reports, butCCCC engineers would like to see more visualization capabilities of the historical data. NTASprovide the simplest and most intuitive graphical user interface of the three tools, but CCCCengineers would like to see more capabilities for plotting historical data. 43
  • 44. JWID 2004 Final Report5.6.2.1 Future ConsiderationsAbove was a quick overview of some of the tools used during JWID 2004 by the CCCC-R forboth network monitoring and network reporting. Some other Open Source Software that shouldbe analyzed include: - Nagios o Network monitoring o Easily customized o Large user community o Graphical network maps o Automatic alerts o Commercial support available - Open Ticket Request System (OTRS) o Email based o Allows a more user-friendly interface to trouble ticket systems o Very easy to integrate via any program that can send email o Commercial support available - Network Top (Ntop) o Allows for the creation of very inexpensive LAN style network probes o Advanced user interface o Acts as a NetFlow collector/distributor o Easily generates RMON style reports o Commercial support available - BigSister o Monitors systems and services using custom scripts o Gives a ‘bubble chart’ type interface for status monitoring o Can send alerts or perform other actions on a service outage o Commercial support available5.7 Network Statistics Collected During JWID 2004During JWID execution, data regarding site I/O traffic, errors, and service uptime (if available)was monitored along with all DCTS data between AITS-JPO and any other site.Data was collected via SNMP from the US routers and switches and from the internationalborder routers. Additionally, core US services were monitored via OpenNMS and uptime wasreported for each service monitored. This is a vast improvement over last year when serviceswere not automatically monitored.DCTS traffic was gathered and parsed with TCPDump and TRPR using remote spanning fromthe DCTS switch at the AITS-JPO. TCPDump recorded every packet, and TRPR processed thisinformation to produce traffic information in average kbps over one-half second intervals. Liveplots of the DCTS data to and from each MCU were viewed during execution for real-time 44
  • 45. JWID 2004 Final Reporttroubleshooting purposes. Network data briefings for each day of execution are on file withconfiguration management at the AITS-JPO.The following graphs are examples of the traffic information that was collected during JWIDexecution: 45
  • 46. JWID 2004 Final Report 46