History of 100G and Internet2
Upcoming SlideShare
Loading in...5
×
 

History of 100G and Internet2

on

  • 1,301 views

Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking - summer ...

Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking - summer of networking, Chris Robb, Indiana University/Internet2

Statistics

Views

Total Views
1,301
Views on SlideShare
1,296
Embed Views
5

Actions

Likes
0
Downloads
33
Comments
0

3 Embeds 5

http://pinterest.com 2
http://www.pinterest.com 2
http://www.slashdocs.com 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • The expectation is that the cost of 10G will drop substantially while 40 and 100 G will remain high.This would tend to encourage the use of multiple 10G interfaces. Also it is unlikely that any one connector to a regional will need more then 1 10G flow.
  • Escaping the commodity network trap (being seen as a competitor to the commercial network providers)Software stack is 45 years old, core basis of applications hasn’t changed in yearsLots of uses that can’t be used using current network because network isn’t fast or smart enough

History of 100G and Internet2 History of 100G and Internet2 Presentation Transcript

  • Chris Robb, Indiana University/Internet2HISTORY OF 100G AND INTERNET2
  • Topics• Introduction• Technical Background• History of Internet2’s 100G Deployment• Current Progress• Lessons Learned
  • Who’s this guy?• Why, I’m Chris Robb!• Director of Engineering and Operations for Internet2• But I work at IU. It’s all horribly confusing.• Started in the trenches of UITS computer lab support• Joined campus networking division as a programmer in 1998• GlobalNOC engineer in 2001. First assignment: Hawaii!• Managed the 2006 Internet2 transition to Level3• Subcontracted to Internet2 in 2008• My day consists of video conferencing, lunch, then more video conferencing- usually in that order• I also occasionally haunt the halls of the CIB
  • What’s Internet2?• “Internet2 is an advanced networking consortium led by the research and education community”• Formed in 1996• More than just a network – Middleware – Net+ – Security – Research – Measurement• Based in Ann Arbor with offices in Washington DC, New York and (soon) California
  • Technical Background
  • National Backbone Organization• Fiber• Optical Equipment• Routers and Switches• Network Operations Center
  • Dark Fiber• Someone has to lay fiber in the ground• Commonly laid along railway lines and interstate roads• A lot of investment in the late 90s• Multiple conduits installed – Ability to blow fiber later• Multi-strand cable (144ct is common, per conduit)• Hut spacing defined at installation time• Huge investment – Nationally cost prohibitive – Regionally expensive, but doable• Sometimes not very well documented
  • Cable conduit is laid
  • Fiber is blown through empty conduit Photos courtesy of Steve Wallace
  • Access to fiber via handholes
  • Fiber is terminated in huts tohouse optical amplificationequipment
  • Who can buy fiber?• Depends on when you’re asking, who you’re asking, and who you are• Early 2000s: plentiful supply of fiber (and companies that are no longer around)• Now: not so much – Consolidation – Not a lot of investment in fiber (exception: high frequency trading)• Mostly carrier swaps and internal use• Sold as an Indefeasible Right to Use (IRU), typically on a 20 year term• Expensive!• Activate Google Earth
  • OK, we have some fiber….• What do you put on the fiber once it’s laid? – Too expensive to just put a single signal on each fiber• Client interfaces can only go so far – 10GBase-SR – up to 300m (depending on fiber type) – 10Gbase-LR – 1310nm – 10km – 10GBase-ER – 1550nm – 40km – 10GBase-ZR – 1550nm – 80km• Put a switch every 80km? – You saw those huts! Too little space. – Who wants to manage that many devices? Too complex. – There are only so many coal plants in the US. Too much power.
  • DWDM Optical Equipment• Multiplex!• Convert signals from client equipment (routers & switches) to specific wavelengths of light (ITU grid)• Combine the different wavelengths onto a common fiber on one end, and separate them on the other end• Uh oh! Light doesn’t travel far enough! – Amplify! – Periodically shoot the signal through erbium doped fiber spools that amplify the signal• The noise! The noise! Amplification also amplifies noise. • Periodically regenerate the signal • Break down the analog signal back into its digital form, then send it out again as a newly cleaned up analog signal
  • DWDM History• Early WDM (late 80s) –Two widely separated wavelengths (1310, 1550nm)• “Second generation” WDM (early 90s) –Two to eight channels in 1550 nm window –400+ GHz spacing• DWDM systems (mid 90s) –16 to 40 channels in 1550 nm window –100 to 200 GHz spacing• Next generation DWDM systems –64 to 160 channels in 1550 nm window –50 and 25 GHz spacing 15
  • ITU Wavelength Grid 1530.33 nm 1553.86 nm 0.80 nm 195.9 THz 193.0 THz 100 GHz • ITU-T grid is based on 191.7 THz + 100 GHz • It is a standard for laser in DWDM systemsFreq (THz) ITU Ch Wave (nm) 15201/252 15216 15800 15540 15454 192.90 29 1554.13 x x x x x 192.85 1554.54 192.80 28 1554.94 x x x x x 192.75 1555.34 192.70 27 1555.75 x x x x x 192.65 1556.15 192.60 26 1556.55 x x x x x
  • Modern Optical Components• Fiber• Erbium Doped Fiber Amplifier (EDFA) (“Amp”)• Optical Add/Drop Multiplexor (“OADM”)• Reconfigurable Optical Add/Drop Multiplexor (“ROADM”)• Tributary (“Trib”) card• And a few more concepts – Directionless – Colorless• To the whiteboard!!
  • DWDM Components 1 15xx 1...n 2 3 Transponder Optical Multiplexer 1 1 2 1...n 2 3 3Optical De-multiplexer Optical Add/Drop Multiplexer (OADM)
  • More DWDM ComponentsOptical Amplifier Optical Attenuator (EDFA) Variable Optical Attenuator Dispersion Compensator (DCM / DCU)
  • Optical Amplifier Optical Add/Drop
  • Transmission Effects • Attenuation: – Reduces power level with distance• Dispersion and nonlinear effects: – Erodes clarity with distance and speed • Noise and Jitter: Leading to a blurred image
  • National Routed Network• How do you get millions of people connected to this capacity?• Routers direct your data transfers around the country• Generally large boxes at the national level ($500K-$1M each) – Higher interface speeds (10Gbps+) – Not a lot of edge features (traffic shaping, policing, etc.)
  • Internet2 Network Topology• 1998-2003 Evolution – Cisco GSR routers – Qwest OC-12/OC-48 backbone – Average Connector Speed: 450-600Mbps• 2003-2007 – Juniper T640 routers – Qwest OC-192 backbone – Average Connector Speed: 1-4Gbps• 2007-2009 – Juniper T640/T1600 routers – Level(3) OC-192/10GigE backbone – Average Connector Speed: 8.3Gbps• 2009-2011 – Juniper MX960 Routers – Level3 Multiple 10GigE Backbone – Average Connectors Speed: 10+ Gbps• 2011 and beyond – Juniper T-1600 Routers – Owned Infrastructure with 100GigE Backbone
  • Potential Edge Bandwidth Growth 260 2 4 8 GbsTotal Potential Gbps from Connectors and Peers 220Gbps 200 150 Cisco GSR Juniper T640 Juniper T640 Qwest Footprint Qwest Footprint 124Gbps Level(3) Footprint 2.5G Backbone 10G Backbone 10G+ Backbone 100 50 50Gbps 1.2Mbps 4.8Gbps 3.4Gbps 14Gbps 0 Jan- 02 Jul- 02 Jan- 03 Jul- 03 Jan- 04 Jul- 04 Jan- 05 Jul- 05 Jan- 06 Jul- 06 Jan- 07 Jul- 07 Jan- 08 Jul- 08 Jan- 09 Time
  • Measured Perabytes into Internet2 per Month12 11.2 10.31086420
  • Internet2 Network Backbone Speeds (in Gbps) 100100908070605040 3030 2020 1010 0.6 2.4 0 1997 1998 2002 2008 2010 2011+
  • Internet2’s McLean, VA POP
  • Internet2’s McLean, VA POP
  • History of 100G Deployment
  • A Community Defines Its Future• In June 2009, Internet2 and the R&E community author the “Internet2 Architectural Directions” Document….• Multiples of 10GigE will be the primary transport to Regional and State Networks over the next 3-5 years. – 10G cost low compared to the cost of 40 or 100G – Multiple large sub-10G flows the norm• Internet2 Network access will be divorced from physical interface speeds and available for apportionment across the network – Flexibility for connectors an important success factor.
  • Architectural Principles• Native 100GigE at the optical layer is an important technology to adopt today – Take advantage of current opportunities to lay the groundwork for future expansion.• Collapsing Layer2 and Layer3 services onto a single delivery platform is an important step toward the hybridization of the network – Reduce overall operating expenses to the Connectors – Candidate technologies include MPLS L2 VPNs, Layer2 Ethernet VLANs and Virtual Private LAN Service (VPLS).
  • Architectural Principles• The Internet2 IP and Layer2 Networks need a migration path to 40G and 100G in the next few years – Backbone must be able to efficiently handle multiple simultaneous 7-10 G flows and individual flows >10Gbps• The Internet2 Network emphasis should be on additional services and technologies that will drive transport bandwidth requirements – The use case for the network drives the technology of the network.
  • Architectural Principles• Internet2 will coordinate with the Regional and State Network partners to determine the most optimal node quantity and locations – Offer a flexible partnership with the connectors. – Create more options for connections.• As mission-critical applications become more integral to the Regional cost-recovery model, the Internet2 Network must focus on enhanced redundancy where needed – Many recent services and uses of the network require increasingly reliable/redundant/resilient connectivity
  • Architectural Principles• The Internet2 Network will continue to be instrumented and operated in a transparent fashion that supports the end-to-end model – The more information that is available about the network the better everyone understands the need for and requirements of the network.
  • Why is 100G So Important?• Edge speeds are outpacing backbone speeds – mid 2009 potential edge bandwidth was 17x the edge bandwidth of 2001• Traffic is steadily growing• Large flows are becoming increasingly commonplace• 100Gbps will scale and become economically cost effective• Drive to innovate!
  • History of 100G and Internet2• June 2009: Internet2 staff began polling the community for set of technical principles to guide staff efforts over the next several years – Resulted in the “Internet2 Architectural Directions” document that specifically called out 100Gbps networking as a strategic direction – AOAC approval of document on October 5th, 2009• August 2009 Internet2 released an RFI to the industry that sought industry feedback on a 100Gbps optical platform
  • Funding History• In late summer of 2009, the NTIA released a Notice of Funding Availability (NOFA) for broadband funding – Broadband Technologies Opportunities Program (BTOP) – $7.2B in funding• Internet2 decided not to respond to the first round of funding so as not to compete with the regional networks that were applying for funds
  • Funding• Second round of BTOP funding in early 2010• Internet2 submits a joint proposal with NLR, IU and the Northern Tier Networking Consortium (NTNC) – Obtain a partial national footprint and equip it with an optical network – Leverage the existing NLR and Internet2 optical paths – Upgrade the NTNC network between Chicago and Seattle – Upgrade all Internet2 routers to be 100G capable – Upgrade the TR-CPS routers to newer hardware – Create a “low-latency” layer of small routers at 27 sites
  • Award and Change of Plans• Internet2/NLR/NTNC were selected as the only “national middle mile” network in the BTOP program in late spring 2010• During the due diligence phase, NLR opted to step out of the partnership – Network topology was refactored to a national footprint to cover those sections of the country that had been covered by NLR’s in-kind contribution• Project officially started in July, 2010
  • Seattle Spokane Bozeman Miles City Portland Missoula Bismarck Billings Fargo Eugene Dickinson Albany Boise Minneapolis Boston Madison Buffalo Eau Claire Chicago Detroit New York Milwaukee Salt Lake City Reno Denver Cleveland Philadelphia Sacramento Pittsburgh Indianapolis CincinattiSunnyvale Ashburn Washington DC St. Louis Kansas City Las Vegas Louisville Raleigh Nashville Charlotte Memphis Chattanooga Albuquerque Tulsa Los Angeles San Diego Phoenix Jackson Atlanta El Paso Dallas Jacksonville Optical Add/Drop Facility San Antonio Optical Regeneration Facility Baton Rouge Houston
  • Preliminary Optical RFI• February 2010: meetings were scheduled with several companies to review their offerings – Evaluation was based on a number of different factors: • Optical properties • Control plane development • Physical properties • Road map for product availability and others – From this review it was determined that there were optical vendors with viable options for building the U.S. UCAN network – Accordingly they were given the option to respond to the RFP for optical equipment in September 2010
  • RFP Issuance• Based on this research potential vendors were identified: – Cisco – Infinera – Ciena• Internet2 released the U.S. UCAN Optical Network Request for Proposal (RFP) on September 13, 2010, in collaboration with Indiana University• Vendor responses were due on September 22 – The response due date was extended three times: • on September 17 Internet2 granted an extension to September 29 • on September 23 Internet2 granted an extension to October 5th • on October 5 Internet2 granted an extension to October 9th
  • Review Team• At the September NTAC meeting, Internet2 solicited volunteers from the community to review the RFP responses• A team of community members, Internet2 staff and IU NOC staff was assembled to evaluate the responses
  • Team Outcome• All three responses were outstanding and each vendor included compelling and creative support for this community effort. – Infinera will provide an upgrade path for the Northern Tier Networking Consortium on their next generation platform – Ciena will provide the national footprint on its Activeflex (formerly OME) 6500 platform – Cisco is being engaged on a variety of fronts for parallel efforts• Recommendation presented to the AOAC on November 10th – AOAC endorsement and forward to Internet2 senior management for final approval
  • Ciena Platform• 100Gbps capable 80-channel DWDM system – 100G cards shipping today• ROADM-based solution at most or all add/drop facilities• Directionless capability in metro areas• Non-Dispersion-Shifted approach provides economical approach that reduces CAPEX• Compact, scalable footprint that adapts to the changing needs of our community
  • Current Progress
  • Upgraded Internet2 Infrastructure benefits • Network Infrastructure – Dark Fiber on a national footprint – Wave Capacity • Optical equipment to light new fiber with 100Gbps-capable equipment • Upgrade of Northern Tier Networking Consortium Infinera equipment to be 100G-capable – 100Gbps IP Backbone • 100G and 10G routers for R&E network • 10G routers for TR/CPS • Enhanced Services – Increased connectivity to Commercial Exchange Points – Regional interconnects • Research Opportunity – Stronger collaboration – Enhanced capability48 – 6/17/2012, © 2009 Internet2
  • Internet2 Optical Network Topology Seattle Spokane Bozeman Miles City Portland Missoula Bismarck Billings Fargo Eugene Dickinson Albany Boise Minneapolis Boston Madison Buffalo Eau Claire Chicago Detroit New York Milwaukee Salt Lake City Reno Denver Cleveland Philadelphia Sacramento Pittsburgh Indianapolis CincinattiSunnyvale Ashburn Washington DC St. Louis Kansas City Las Vegas Louisville Raleigh Nashville Charlotte Memphis Chattanooga Albuquerque Tulsa Los Angeles San Diego Phoenix Jackson Atlanta El Paso Dallas Jacksonville Optical Add/Drop Facility San Antonio Optical Regeneration Facility Baton Rouge Houston
  • Phase 1 Optical Build - COMPLETE
  • Phase 1 Progress – 100% Complete• All fiber acquired and accepted• All Optical Equipment Installed• BER Testing complete• System Commissioning – EMS system installed and populated with nodes – Internet2 NOC database population nearly complete• First 100GigE IP circuit between New York and Washington DC live and integrated into the Internet2 IP Network• First Trans-Continental 100G configured and passing pings for Internet2 Fall Member Meeting in early October
  • Phase 2 – Complete52 – 6/17/2012, © 2009 Internet2
  • Phase 3 – January 2012 – July 201353 – 6/17/2012, © 2009 Internet2
  • Current Status Seattle Spokane Olympia Bozeman Miles City Portland Missoula Bismarck Billings Fargo Eugene Dickinson Albany Boise Minneapolis Boston Madison Buffalo Eau Claire Chicago Detroit New York Milwaukee Salt Lake City Reno Denver Cleveland Philadelphia Sacramento Pittsburgh Indianapolis CincinattiSunnyvale Ashburn Washington DC St. Louis Pueblo Kansas City Las Vegas Louisville RaleighSan Luis Obispo Nashville Charlotte Memphis Chattanooga Albuquerque Tulsa Los Angeles San Diego Phoenix Tucson Jackson Atlanta El Paso Dallas Jacksonville Optical Add/Drop Facility San Antonio Baton Rouge Route Fully Operational Route Installed, Tested, Not Operational Houston Route Pending
  • Will someone please think of the routers?• Optical is great, but what about the routers?• Well, that’s a tricky story.• We’re at a crossroads• Routers are expensive, big, power hungry monsters• Did I mention expensive?• We want them everywhere• We want them cheap• We want them small• We want control• We want them powerful• Is there a white knight?
  • Openflow ?
  • History of implementation • Faster – 2002: One of the first 10G IP national backbones deployed on Juniper T640 platform – Mid-2009: Architectural Directions document cites need for 100G IP backbone – Mid-2010: $96M Broadband Technologies Opportunities Program (BTOP) National Middle Mile Award – Late-2011: First national 100GigE circuit lit between New York and Sunnyvale • Smarter – 2005: Hybrid Optical Packet Infrastructure (HOPI) network deployed in first national trial of reconfigurable network – 2007: Interoperable On-demand Network (ION) deployed at 21 locations around the US – 2011: DYNES campus deployment project begins pushing dynamic deployment to the edge – Mid-2011: Open Networking Foundation formed by a coalition of research and industry members – Late-2011: 5-node NDDI deployment and development of OSE software57 – 6/17/2012, © 2011 Internet2
  • Why now? • Demand from Internet2 Members to Innovate • Equipment Refresh Cycle • Available Funding58 – 6/17/2012, © 2011 Internet2
  • Engagement Plan • This is a very high-level restructure of the Internet2 network, so it’s important that stakeholders at all levels participate in the discussion – Researchers – Campuses – Regionals – International Peers – Funding Agencies – Governance Councils – Technical Bodies • Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking59 – 6/17/2012, © 2011 Internet2
  • Implementation possibilities • Goal: Integrate SDN functionality into the Internet2 Network – Preferred Goal: platform integration between two sets of services to provide initial network effect and migration path • A few different models to consider • Nothing is set in stone, but we do have to move aggressively60 – 6/17/2012, © 2011 Internet2
  • Separate Infrastructure: What we do today61 – 6/17/2012, © 2011 Internet2
  • Combined Infrastructure: What we might do tomorrow62 – 6/17/2012, © 2011 Internet2
  • Upgraded Layer2 Network I2 Draft Topology gcb 5/8/12 v.1 SeattlePortland Missoula Albany Minneapolis Boise Boston Sacramento Salt Lake City Chicago New York 1 Cleveland Pittsburgh New York 2 Sunnyvale Kansas City Indy Cincy Denver St Louis Washington DC Tulsa Nashville Raleigh Phoenix Albuquerque Los Angeles El Paso Atlanta Jackson Dallas Jacksonville Baton Rouge Houston NDDI Node
  • Lessons Learned• Pre-project – Know your goals – Know your limitations, then ignore them – Community Input• Negotiations – Have a good lawyer – Understand what you’re buying – Budget a lot of time • Remember: this is for 20+ years• Implementation – Documentation is critical – Inventory management needs to be worked out ahead of time – It isn’t plug and play – Budget a lot for space and power
  • QUESTIONS?