Why Not 100GbE as Service
~ JPIX Perspective ~
Yoshiki Ishida, Takejiro Takabayashi
JPIX
(Japan Internet eXchange Co., Ltd.)
What Motivates 100GbE Service?
• Customer (Member) Requirement
• Traffic
• Technology & Operation
• Cost
Traffic Background(1) ~ Europe ~
• IX points Traffic are
keep growing
• The peak traffic of three
global IX points are over
2 tera bit per second.
– AMS-IX(NL)
– DE-CIX(DE)
– LINX(UK)
https://ams-ix.net/technical/statistics
https://www.de-cix.net/about/statistics/
https://www.linx.net/pubtools/trafficstats.html
Source:
Captured date: Jan. 28, 2015
100GbE in Europe ~ No Surprise ~
• 100G services are available at these major IX
Points
– AMS-IX and Limelight Networks Field-Test Brocade
100 Gigabit Ethernet Technology to Stay Ahead of
Internet Growth - June 2011 (*1)
– DE-CIX Completes 100G Optical Layer Build for New
Flagship Apollon Platform - May 2013 (*2)
– BT expands its 21st Century Network as the London
Internet Exchange deploys 100 Gigabit Ethernet across
their leading edge IP/MPLS Internet Exchange - July
2012 (*3)
Traffic Background(2) ~ ASIA ~
http://www.hkix.net/hkix/stat/aggt/hkix-aggregate.html
http://www.mfeed.co.jp/jpnap-tokyo-i/traffic.html
Source:
• HKIX ≈400Gbps peak
– 100GbE available
• JPNAP ≈400Gbps peak
– 100GbE available
Captured date: Jan. 28, 2015
Traffic Background(3) JPIX
• JPIX’s traffic is keep
growing ≈300Gbps
• Some our customers,
especially Hypergiants
and FTTH providers, use
multiple 10G Interfaces.
And finally…
http://www.jpix.ad.jp/en/technical/traffic.html
Source:
Captured date: Jan. 30, 2015
Press Release - July 9, 2014
Press Release - July 9, 2014
• Japan Internet Exchange Co., Ltd., (hereafter referred to as “JPIX”) an IX
provider that has a largest number of customers in Japan, will launch a new IX
port service, 100 Gigabit Ethernet Port, in July. Brocade Communications
Systems, K.K., (hereafter referred to as Brocade) announces that it provides the
Brocade MLXe Series, deployed as switch to support the new service.
• Dwango Co., Ltd., a company that provides Japan’s largest video sharing service,
“niconico”, and a wide range of entertainment contents and distributes vast
amounts of content, will become the first user of the service.
• The equipment for the new IX port service, 100 Gigabit Ethernet Port, are the
Brocade MLXe Series provided by Brocade.
• Brocade MLXe Series, which include a 100 Gigabit Ethernet (GbE)-enabled
model released for the first time in the industry, were selected to equip the
100 GbE Port Service. As a result of Brocade’s feasibility testing for various
interfaces, it was proven that the 100 GbE Port Service can be provided
through main interfaces, including 10×10, 100G Base-LR4, and 100G Base-ER4.
• JPIX continues to strive to provide effective services and quality leading to
stable operation, which can satisfy customers, such as ISPs and content
providers, in order to respond to continuously increasing Internet traffic.
100GbE motives PROS & CONS
• Higher Speed
– Faster the better
• Less operation
– Less LAG, Simplex
• Less consideration
– Once you upgrade, you
are free from concerning
upgrading the
bandwidth
• Opens Opportunity
• Stability/Reliability/RISK
– 100G affects too much
traffic
• Need experience…
• COST
– CAPEX/OPEX may higher
than 10x 10G
• Opportunity Loss
PROS CONS
Issues for deploying 100G Service
~ burdens ~
• Upgrading the IX Architecture
– Needed to upgrade our BIG 100GbE supported switch act
as spine and 40GbE supported switch act as leaf switch
• Verifying 100GbE Interface
– Optical Switch testing needed
• What if one wavelength drops optical power…
• Selection of the Optical Interface and the Optical
Module
– Shall we wait or go?
• 100G Ethernet Tester
– What can we do about it…? Is it CFP or Backplane…?
Upgrading IX Architecture
• Distributed Architecture
• Spine and Leaf Architecture
Verifying 100G Interface
100G Ethernet uses Multiple Waves on a Fiber
Verifying 100G Interface for our Architecture
( eg. Optical A/B Switch)
*4
Optics and Optical Module
• Optics
– LR4(IEEE)
• 10Km SMF/4x25Gbps
– ER4(IEEE)
• 40Km SMF/4x25Gbps
– 10x10(10x10MSA)
• 2KmSMF/10x10Gbps
• 10KmSMF/10x10Gbps
• Optical Module
– CFP
– CFP2/CFP4
– CPAK
*5
*6
CFP image courtesy of Finisar
100G Ethernet Tester
• Purpose for 100G
Ethernet Tester
– Isolate the cause of
failure on a 100GbE port
– Sanity check of a CFP
– Health Check of IX
Switch
– Throughput
Test(RFC2544)
• Specification
– Hardware
• Throughput Test with
Multiple 100GbE (at least
2) Ports
• No limitation for CFP
vendors
• Portability and Battery-
powered
– Software
• Multiple Stream
• CFP Check
• ReportingTester IX Switch
CFP
CFP
CFP
CFP
courtesy of EXFO
Technology Myth & Truth
• 10x10 MSA is very attractive; cost effective
– Really?
• With CFP2 10x10MSA is unsupported… can be dead-stock
• Big Chassis that supports 100GbE can be in the
middle
– Can’t live with old GbE interface… will drop packets… not
enough buffer…, can’t be in the middle
• Vendor lock-in…
– As always
• Timing
– If we wait for another year, we could have install double
or quadruple capacity line-card…
Cost Myth & Truth
• 100GbE is getting less expensive
– Yes it has. In our case, we see the CFP cost dropped by
50%...
• Comparison year 2012 and 2015
• But Still:
– For us, it cost 21 times higher cost…. 21X
• 21x SFP+ = 1x 100G CFP PRICE…
• Is it really less expensive?
Why not 100G now?
Shall we
deploy 100G
Now?
New modules
coming
There may be
less
expensive
better
product in
market
Expensive
interface you
buy today
can be legacy
Let’s just wait
Vicious Circle
Summary
• 100G Ethernet is still “on going” technology.
– Indecisiveness of making investment decision
• Technology(Optics/Wave Multiplexity)
• Cost…
• Cutting Edge Players have already started
adopting to 100G because of the necessity
– Hypergiant, IX, R&E Community
– They needed it, we needed it
• It works fine
• When will do?
> Timing is everything < ~ mark your calendar ~
Reference
*1
http://newsroom.brocade.com/press-releases/ams-ix-and-limelight-
networks-field-test-brocade-1-nasdaq-llnw-0765435#.U-rzNGMTNW_
*2
http://press.de-cix.net/press-releases/pressemitteilung/article/de-cix-
completes-100g-optical-layer-build-for-new-flagship-apollon-platform/
*3
https://www.linx.net/publicity/2012releases/pr2012-04.html
*4
http://www.ntt-at.com/product/optical-switch/
*5
http://www.cisco.com/web/JP/product/hs/optical/ons15454/prodlit/whit
e_paper_c11-727398_031813.html
*6
http://www.epsglobal.com/downloads/Finisar/Finisar40G_100G_Presenta
tion.pdf

Why Not 100GbE as Service ~ JPIX Perspective ~

  • 1.
    Why Not 100GbEas Service ~ JPIX Perspective ~ Yoshiki Ishida, Takejiro Takabayashi JPIX (Japan Internet eXchange Co., Ltd.)
  • 2.
    What Motivates 100GbEService? • Customer (Member) Requirement • Traffic • Technology & Operation • Cost
  • 3.
    Traffic Background(1) ~Europe ~ • IX points Traffic are keep growing • The peak traffic of three global IX points are over 2 tera bit per second. – AMS-IX(NL) – DE-CIX(DE) – LINX(UK) https://ams-ix.net/technical/statistics https://www.de-cix.net/about/statistics/ https://www.linx.net/pubtools/trafficstats.html Source: Captured date: Jan. 28, 2015
  • 4.
    100GbE in Europe~ No Surprise ~ • 100G services are available at these major IX Points – AMS-IX and Limelight Networks Field-Test Brocade 100 Gigabit Ethernet Technology to Stay Ahead of Internet Growth - June 2011 (*1) – DE-CIX Completes 100G Optical Layer Build for New Flagship Apollon Platform - May 2013 (*2) – BT expands its 21st Century Network as the London Internet Exchange deploys 100 Gigabit Ethernet across their leading edge IP/MPLS Internet Exchange - July 2012 (*3)
  • 5.
    Traffic Background(2) ~ASIA ~ http://www.hkix.net/hkix/stat/aggt/hkix-aggregate.html http://www.mfeed.co.jp/jpnap-tokyo-i/traffic.html Source: • HKIX ≈400Gbps peak – 100GbE available • JPNAP ≈400Gbps peak – 100GbE available Captured date: Jan. 28, 2015
  • 6.
    Traffic Background(3) JPIX •JPIX’s traffic is keep growing ≈300Gbps • Some our customers, especially Hypergiants and FTTH providers, use multiple 10G Interfaces. And finally… http://www.jpix.ad.jp/en/technical/traffic.html Source: Captured date: Jan. 30, 2015
  • 7.
    Press Release -July 9, 2014
  • 8.
    Press Release -July 9, 2014 • Japan Internet Exchange Co., Ltd., (hereafter referred to as “JPIX”) an IX provider that has a largest number of customers in Japan, will launch a new IX port service, 100 Gigabit Ethernet Port, in July. Brocade Communications Systems, K.K., (hereafter referred to as Brocade) announces that it provides the Brocade MLXe Series, deployed as switch to support the new service. • Dwango Co., Ltd., a company that provides Japan’s largest video sharing service, “niconico”, and a wide range of entertainment contents and distributes vast amounts of content, will become the first user of the service. • The equipment for the new IX port service, 100 Gigabit Ethernet Port, are the Brocade MLXe Series provided by Brocade. • Brocade MLXe Series, which include a 100 Gigabit Ethernet (GbE)-enabled model released for the first time in the industry, were selected to equip the 100 GbE Port Service. As a result of Brocade’s feasibility testing for various interfaces, it was proven that the 100 GbE Port Service can be provided through main interfaces, including 10×10, 100G Base-LR4, and 100G Base-ER4. • JPIX continues to strive to provide effective services and quality leading to stable operation, which can satisfy customers, such as ISPs and content providers, in order to respond to continuously increasing Internet traffic.
  • 9.
    100GbE motives PROS& CONS • Higher Speed – Faster the better • Less operation – Less LAG, Simplex • Less consideration – Once you upgrade, you are free from concerning upgrading the bandwidth • Opens Opportunity • Stability/Reliability/RISK – 100G affects too much traffic • Need experience… • COST – CAPEX/OPEX may higher than 10x 10G • Opportunity Loss PROS CONS
  • 10.
    Issues for deploying100G Service ~ burdens ~ • Upgrading the IX Architecture – Needed to upgrade our BIG 100GbE supported switch act as spine and 40GbE supported switch act as leaf switch • Verifying 100GbE Interface – Optical Switch testing needed • What if one wavelength drops optical power… • Selection of the Optical Interface and the Optical Module – Shall we wait or go? • 100G Ethernet Tester – What can we do about it…? Is it CFP or Backplane…?
  • 11.
    Upgrading IX Architecture •Distributed Architecture • Spine and Leaf Architecture
  • 12.
    Verifying 100G Interface 100GEthernet uses Multiple Waves on a Fiber Verifying 100G Interface for our Architecture ( eg. Optical A/B Switch) *4
  • 13.
    Optics and OpticalModule • Optics – LR4(IEEE) • 10Km SMF/4x25Gbps – ER4(IEEE) • 40Km SMF/4x25Gbps – 10x10(10x10MSA) • 2KmSMF/10x10Gbps • 10KmSMF/10x10Gbps • Optical Module – CFP – CFP2/CFP4 – CPAK *5 *6 CFP image courtesy of Finisar
  • 14.
    100G Ethernet Tester •Purpose for 100G Ethernet Tester – Isolate the cause of failure on a 100GbE port – Sanity check of a CFP – Health Check of IX Switch – Throughput Test(RFC2544) • Specification – Hardware • Throughput Test with Multiple 100GbE (at least 2) Ports • No limitation for CFP vendors • Portability and Battery- powered – Software • Multiple Stream • CFP Check • ReportingTester IX Switch CFP CFP CFP CFP courtesy of EXFO
  • 15.
    Technology Myth &Truth • 10x10 MSA is very attractive; cost effective – Really? • With CFP2 10x10MSA is unsupported… can be dead-stock • Big Chassis that supports 100GbE can be in the middle – Can’t live with old GbE interface… will drop packets… not enough buffer…, can’t be in the middle • Vendor lock-in… – As always • Timing – If we wait for another year, we could have install double or quadruple capacity line-card…
  • 16.
    Cost Myth &Truth • 100GbE is getting less expensive – Yes it has. In our case, we see the CFP cost dropped by 50%... • Comparison year 2012 and 2015 • But Still: – For us, it cost 21 times higher cost…. 21X • 21x SFP+ = 1x 100G CFP PRICE… • Is it really less expensive?
  • 17.
    Why not 100Gnow? Shall we deploy 100G Now? New modules coming There may be less expensive better product in market Expensive interface you buy today can be legacy Let’s just wait Vicious Circle
  • 18.
    Summary • 100G Ethernetis still “on going” technology. – Indecisiveness of making investment decision • Technology(Optics/Wave Multiplexity) • Cost… • Cutting Edge Players have already started adopting to 100G because of the necessity – Hypergiant, IX, R&E Community – They needed it, we needed it • It works fine • When will do? > Timing is everything < ~ mark your calendar ~
  • 19.