Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura


Published on

Published in: Internet
  • Be the first to comment

  • Be the first to like this

High Speed Fiber Services and Challenges to the Core Network by Seiichi Kawamura

  1. 1. Copyright  ©  2014  BIGLOBE  Inc.1 High  speed  fiber  services and  challenges  to  the  core  network August  2014  @  MyNOG4 BIGLOBE  Inc.   Seiichi  Kawamura
  2. 2. Copyright  ©  2014  BIGLOBE  Inc.2 Some  figures  about  BIGLOBE  (AS2518) l 3 million consumer ISP users in Japan l 250Gbps total internet traffic p including enterprise, data center, transit traffic San  Jose  [vPOP]  -‐‑‒Equinix   IX(10G) LA  [vPOP]  -‐‑‒Coresite  Any2(10G) Japan  [Core]    -‐‑‒  3  million  broadband  customers    -‐‑‒  110  colocation  customers      -‐‑‒  2  DCs  fully  owned  by  BIGLOBE    -‐‑‒  Great  connectivity    -‐‑‒  Equinix(20G),  JPIX(10G),                                  BBIX(30G),    JPNAP(30G)    -‐‑‒  PNI  with  ALL  major  ISPs Hong  Kong[POP]  -‐‑‒HKIX(10G) Singapore  [POP]  -‐‑‒Equinix   IX(10G)  -‐‑‒  SGIX  (1G)
  3. 3. Copyright  ©  2014  BIGLOBE  Inc.3 Network  Topology  in  Japan Metro Network Metro Network Subscribers Peering Subscribers Peering
  4. 4. Copyright  ©  2014  BIGLOBE  Inc.4 Growth  of  Traffic Metro Network (Tokyo) Metro Network (Tokyo) Subscribers Peering Subscribers Peering Peering traffic grows 1.37x every year (Mobile Traffic 1.8x, broadband 1.2x)
  5. 5. Copyright  ©  2014  BIGLOBE  Inc.5 1.6x growth! The  impact  of  high  speed  services 1.2x growth
  6. 6. Copyright  ©  2014  BIGLOBE  Inc.6 Metro Network Metro Network Challenge1  :  Keeping  the  metro  growth  down 28*10G Metro Network Metro Network 10*10G 5 years Core growth 1.2x every year Keeping growth down by •  Hot potato routing •  ECMP tuning (enhanced hash) •  Higher thresholds in capacity growth •  Studying flows and moving peers to a closer exit No MPLS Pure IP
  7. 7. Copyright  ©  2014  BIGLOBE  Inc.7 Challenge  2:  Streaming     l 30% of FTTH traffic is streaming l We better have a cost effective way of handling streaming traffic (and all video traffic) l Cost effective peering was our answer p  Does not mean we love caches Tactics •  Use netflow and PeeringDB to find where we can pick up the traffic •  Remote peering (unfortunately) to keep costs low •  Extensive talks with content providers •  Disclosed peering policy to pick up peering traffic more effectively(more on this later)
  8. 8. Copyright  ©  2014  BIGLOBE  Inc.8 backbone Streaming  traffic  basic  principle Minimize total cost of delivery and keep it from going to other links IX ports, transit ports Keep traffic from overflowing into ports with high utilization. Can be used as backup links, but should control traffic Try to peer at router closest to end user Lower backhaul cost Lower peering costsCosts are fixed But actual networks don’t look like this…
  9. 9. Copyright  ©  2014  BIGLOBE  Inc.9 Actual  Network Access network concentrator Core routing Peering Ideally, we want video traffic to be here, but can we peer where the access concentrators are? No. Can we have caches here? Yes, but GGC, AANP, Open Connect, and what next? A horrible cost model More flexible DC interconnects may help
  10. 10. Copyright  ©  2014  BIGLOBE  Inc.10 Challenge3  :  Peering l Domestic peering was REALLY an issue p IX port costs Ø 2 years ago, 10G ports cost 20,000 usd per month p the burden of peering, mistrust, lack of communication between peers Ø No place to talk about peering l We wanted a better relationship with content providers
  11. 11. Copyright  ©  2014  BIGLOBE  Inc.11 Changes  coming  from  Mobile  and  Cloud  area l More mobile and cloud traffic p Broadband at 1.3x growth while mobile at 1.8x growth l Content providers have different requirements than ISPs p semi-full route p simple route servers p latency aware p mtu9000 p fast detection using BFD p DoS protection as a service
  12. 12. Copyright  ©  2014  BIGLOBE  Inc.12 Evolution  of  peering  community l Japan did not have a place to talk about peering until a few years ago p PAST: People disliked talking about peering at JANOG l Peering relied heavily on ISPs and Telcos •  BoFs, study councils, talks at NOGs, and joint efforts by the IXPs and the community –  more focus on content/dc and its requirements
  13. 13. Copyright  ©  2014  BIGLOBE  Inc.13 community  activities l Google Groups : Peering in Japan p local Japanese language only p discussion on latest peering issues p no IX personnel on list p host Peering BoFs l Tutorials p IXP provides low cost tutorials (available regularly) p free tutorials at JANOG (not always available) l CloudIX Study Council p Group of members(BGP operators) in BBIX doing technical experiments p sharing skills and helping out each other to peer
  14. 14. Copyright  ©  2014  BIGLOBE  Inc.14 Challenge4  :  Automation[  a  work  in  progress  ] l Our data center is well automated p in the past dc config change >>>>>>>>>> peering p dc requires orchestration with servers and other virtual devices l Since we started peering more (and more globally) we are now needing better automation tools l Current implementation p Simple html(form) +mysql+ jinja2 to generate configs via Web UI Ø Trying to shift to Django p Exscript to push configs to router
  15. 15. Copyright  ©  2014  BIGLOBE  Inc.15 Other  misc  challenges l Unnecessary traffic avoidance p P2P services that have security problems are controlled at the edge p BCP38 to stop sending unnecessary traffic Ø  still in initial deployment phase l Higher density with better power efficiency p Working to keep PUE around 1.2-1.3 p Switching to higher density line cards l Measuring actual user experience p “1Gbps service” is more marketing talk than actual experience p Measuring response to popular sites (alexa) p Python script to crawl command and filter out noise data “hping -p 80 -c 5 --syn [url]” works well with unix, PSPing works well with Windows
  16. 16. Copyright  ©  2014  BIGLOBE  Inc.16
  17. 17. Copyright  ©  2014  BIGLOBE  Inc.17 Next  steps  for  evolution l MPLS, TE, auto-bandwidth p Not necessary for domestic consumer service backbone, but need TE to control global network p We did not need this in the past since we did not provide VPN services (we may in the future) p we have a vxlan network running, so considering EVPN as one of the solutions l Better network management and security p BMP and other tools for better peering management p APIs to the router p BGP Flow Spec l Differentiation of services inside the backbone p Last mile QoS and internet fast lane?
  18. 18. Copyright  ©  2014  BIGLOBE  Inc.18 Summary l Higher speed services to the consumers, means new challenges to the core network l Peering really helps, but it requires a good ecosystem to be helpful p diversity in IX services, fair pricing, carrier neutrality, open talks about peering l Handling streaming and video, is still a big headache p is there a better way than everybody has their own cdn? l Automation and tools are key to handling bigger traffic in a cost effective way l Measurements make a difference in user experience. “1Gbps marketing talks” don’t