From Jisc's campus network engineering for data-intensive science workshop on 19 October 2016. https://www.jisc.ac.uk/events/campus-network-engineering-for-data-intensive-science-workshop-19-oct-2016

1. Science DMZ at Imperial
Phil Mayers, Campus network engineering workshop
19/10/2016
1

2. Science DMZ at Imperial
Phil Mayers <p.mayers@imperial.ac.uk>

3. About Imperial
● 14,700 students, 8,000 staff
● Focused on science, engineering, medicine and business
● 6 major campuses in London, also Silwood Park, and medical sites
● Perhaps more centralised IT than many universities?
● Dual 2x10G connections to JANET
● Various sponsored a.k.a. BCE customers (NHM, Science Museum, NHS trust)
● GridPP / HEP work - close relationship with researchers

4. Campus network
● Decent size network - ~2400 switches, ~2300 APs, 15k simultaneous wifi users,
>60k devices on-net including PCs, wifi/BYOD, SCADA, VoIP, etc.
● Campus to internet throughput ~2Gbit/s average, ~6Gbit/s peak (Oct 2016)
● Fully dual-stack network - 20-40% IPv6 by throughput, 15% by flows
● Typical architecture - switch, dist, router, core, firewall, wan

5. HEP group
● Main HEP grid cluster processes data for the LHC experiments, other
physics experiments/projects & non-physics communities
○ CMS, LHCb, ATLAS, LZ, COMET, biomed & pheno are the main users
● 275 compute nodes (~4000 cores) connected on 1GbE
● 55 storage nodes (~3.7PB of disk) connected on 10GbE
● Simple stacked top-of-rack switches for connectivity
● Majority of WAN traffic is CMS local-storage <-> remote-storage
○ Popular datasets are automatically placed at CMS sites
○ Users can also request data: 50TB+ dataset requests not uncommon
● Local compute nodes can read remote storage over WAN (and vice versa)
○ Generally low rates compared to storage-storage transfers

6. HEP growth - 1gig
April
2007

7. HEP growth - 10gig
Oct
2011

8. HEP growth - 20gig
Oct
2016

9. Issues faced
● Firewalls
○ Straight throughput
○ TCP window checking and other stateful inspection
○ Latency and jitter interfering with throughput
○ Impact on other traffic e.g. Office 365 is quite latency-sensitive with the Outlook client
● Equipment costs
○ Need the right size pipe at every forwarding hop
○ Building edge -> dist -> router -> core -> firewall -> WAN edge
○ A lot of those devices are of a class where fast ports are disproportionately costly
■ “Typical” campus router - approx. £1-2k for a 10gig port
■ 1U 48-port 10G switch - approx. £200 for a 10gig port

10. Solution - Science DMZ
● Had no idea it had a name when we built it!
● Separate L3 switch, outside firewall, routes HEP traffic straight onto core and
onward to JANET
● Simple stateless ACLs for outer tier of security
● Fewer hops, shallower buffers, cheaper kit, wider pipes
● HEP @ Imperial - 4x10G ports to HEP, dual 2x10G ECMP to JANET
○ Split HEP into two subnets, use BGP communities outbound to split inbound traffic
○ Necessitates HEP managing which node IPs are used for transfer

11. Results - recent past
● Quite capable of driving 4x10G at >99.5% utilisation
● Apologies for the graph - low resolution and hourly averages hiding peaks
○ Don’t be fooled - 30-second and 5-minute averages on all 4 10G links to JANET were >99% load

12. Architecture
Janet
Border
Firewall
CoreDatacentre
Science DMZ
Possible

13. Benefits
● Works - capable of driving campus connectivity to capacity
● Cheap - equipment cost on our side manageable
○ As long as upstream connectivity exists, of course
● Easy - no need to poke at firewalls or building edge to improve throughput

14. Issues
● Works too well!
● At capacity, it can drive other traffic off the campus links
○ 64 bytes from ...: icmp_seq=856 ttl=49 time=104 ms
○ from a typical 2ms to the same site
○ Have seen 10gig links running at essentially 100% for >1 hour
● Need to ensure enough spare capacity for other uses
○ Rate-limiting port channels (shudder)
○ Rate-limit $here - sure it’ll be hashed to the same bundle members at $nexthop?

15. Results - Thu 13 Oct
Latency across one leg of default route, versus throughput on same
Noticeable to customers… not great. But very impressive throughput!

16. Issues - Mark 2
● Cheap switches are cheap for a reason
● Doesn’t solve distance and fibre issues
○ Want to run in excess of 10G at distances of >10km? Get ready for a lot of zeroes
○ Fibre capacity on inter-site links (install & recurrent costs)
○ Or use DWDM (skills & training, tools, monitoring) - we do this
● Question mark over dual-use systems - is it appropriate to attach to DMZ
○ Can you do a windows domain login from a DMZ?
● Our implementation requires HEP team to split transfer nodes across two
subnets, to make use of both inbound paths
● Security policy - speak to your IT Security team first!

17. Thoughts
● We are considering making Science DMZ a core part of network architecture
○ 100G still not cost-effective for widespread campus deployment - particularly if you are
geographically distributed
○ Build parallel cheap/fast DMZ network, hook together at JANET & datacentre?
○ Present DMZ where needed (distance & fibre issues though…)
● Considerations
○ Equipment in normal office/lab locations e.g. high-throughput gene sequencers
○ Separate switches in wiring closets - have to manage patching, labelling, training
○ Spurious requests - people think they can drive 10gig and cannot
● Only applicable for mature research efforts with good tooling, IMO
○ Took GridPP community many years to be able to drive these speeds

18. Recommendations
● Speak to researchers!
● Consider appropriate cost/benefit of implementation
○ Cheap vs. high-end routers
○ Fixed versus expandable
● How will you scale, monitor, manage
○ Counters, API, routing/switching capability
● Consider your upstream capacity

19. LHCONE - if we have time
● Overlay L3VPN - used to steer HEP traffic down separate links
○ Funding reasons
● Imperial already do L3VPN internally for network segmentation
○ JANET presented LHCONE as 802.1q-tagged subint & BGP peering, into L3VPN on core
○ Core presents as 2x “peerings” (internet & LHCONE) to Science DMZ router
○ DMZ router follows routing table (401 IPv4 & 146 IPv6 BGP routes)
● Basically works
○ Very impressive throughput
● Reservations internally about ultimate scalability of this model
○ If we had a multi-researcher Science DMZ - how would that work?
○ Policy routing? Shoot me now please...