The University of Edinburgh is undergoing a large project to reprocure its campus networking infrastructure. The existing network, which has grown organically over many years, contains equipment that is up to 20 years old and no longer meets the university's needs. After an internal review in 2014 recommended a new network be procured, the university embarked on a multi-stage competitive dialogue procurement process that is still ongoing. The process involves pre-market engagement, shortlisting bidders, and multiple rounds of dialogue and evaluation to refine solutions before selecting a final vendor. The procurement has proven to be a large undertaking but may result in a network solution tailored to the university's unique requirements.

1. Campus
networking
Chair: Rob Evans,
Chief network architect, Jisc

2. Construction,
disruption and
fibre moves
Mark Franklin,
University of Sheffield

3. IPv6 @ the STFC
Philip Garrad,
Science and Technology
Facilities Council (STFC)

4. • What is STFC
• What IPv6 work had we completed before Sept 2016
• What happened in September 2016
• How we deployed IPv6 in 6 months
• What have we learnt from this
Overview

5. One of Europe’s largest multi-disciplinary scientific research organisations
Science and Technology Facilities Council
World class research, innovation and skills
• Broad range of physical, life and computational sciences
• Around 1,700 scientists in particle and nuclear physics, and astronomy
• Access for 7,500 scientists to world-leading, large-scale facilities
• Science and Innovation Campuses at Daresbury and Harwell
• Globally-recognised capabilities and expertise in technology R&D
• Inspiring young people to undertake STEM

6. One of Europe’s largest multi-disciplinary scientific research organisations
Science and Technology Facilities Council
UK Astronomy Technology Centre
Edinburgh, Scotland
Polaris House
Swindon, Wiltshire
Chilbolton Observatory
Stockbridge, Hampshire
Daresbury Laboratory
Sci-tech Daresbury Campus, Liverpool City Region
Rutherford Appleton Laboratory
Harwell Didcot, Oxfordshire
Boulby Underground
Laboratory
North Yorkshire

7. Large Hadron Collider @CERN
Science and Technology Facilities Council
Credit: CERN
Recreating the conditions a billionth of a second after the Big Bang
LHC research also delivers wide benefits for society through new
technologies, including cancer therapies, medical and industrial
imaging and manufacturing processes

8. IPv6 Address plan, devised by Robin Tasker for STFC.
STFC’s IPv6 address plan
IPv6 Status in Sept 2016
2001 0630 0058 a b c d 0000 0000 0000 0000: : : : : : :
a = STFC Address plan version (16 chances)
b = Network Type
c = Network Subtype
d = Locally assigned
NETWORK HOST
JANET RAL a b c d 0000 0000 0000 0000: : : : : :

9. • “Cheap” IPv6 firewall connected to border router
• 3x IPv6 hosts connected directly to the firewall
• Provided connectivity to allow interested parties
access to an IPv6 network.
• Take up was limited to the RAL Tier 1 group.
Small IPv6 Test installation
IPv6 Status in Sept 2016

10. • 20th September 2016 - WLCG management board
accepted the HEPiX IPv6 working group proposal to
allow sites to deploy IPv6 only CPU by April 2017.
• This mandated LHC Tier 1s to provide some services
as dual stack by April 2017
• STFC needed to deploy production quality IPv6
network at RAL within 6 months.
Worldwide Large Hadron Collider Computing Group (WLCG) meeting.
What happened in Sept 2016?

11. • Step 1 – Wait 2 months…
• Allowed for some scheduled network upgrades to complete
• Various pieces of non IPv6 capable hardware replaced
• Step 2 – Realise Christmas is coming
• Don’t make significant network change - site shuts for 10 days
• Prepare by upgrading the firmware on all devices
• Step 3 – Plan how to deploy IPv6 in 3 months
How to deploy IPv6 in 6 months?

12. Starting Point – September 2016
Starting point – Limited IPv6 in the RAL network Philip Garrad - STFC

13. Firmware Upgrades
Firmware Upgrades Philip Garrad - STFC

14. 18th January 2017
Create a new IPv6 External Core network Philip Garrad - STFC

15. 25th January 2017
Bring IPv6 into the firewall Philip Garrad - STFC

16. 1st February 2017
Connect our IPv6 testbed to the main site firewall Philip Garrad - STFC

17. 8th February 2017
Create an IPv6 subnet on an internal router Philip Garrad - STFC

18. 15th February 2017
Add DNS services to the network Philip Garrad - STFC

19. 22nd February 2017
Create an IPv6 subnet for the Tier 1 links to CERN Philip Garrad - STFC

20. 1st March 2017
Create a second IPv6 subnet for the internal IPv6 subnet Philip Garrad - STFC

21. 8th March 2017
Retire our IPv6 testbed Philip Garrad - STFC

22. The future….
How the network should look in the end… Philip Garrad - STFC

23. • LHC Tier 1 @ RAL have been running IPv6 for a year
• IPv6 is being deployed to LHC Tier 2 @ RAL now
• STFC Network group need to run their own IPv6
subnet to gain better understanding of IPv6
• Auto configuration, Security, logging, auditing.
March 2018
Still to do… (Quite a lot)

24. • Don’t get hung up on a numbering scheme
• Clearly it would have been less stressful to deploy
IPv6 to our timetable
• Moving through the network top to bottom simplified
the problem
• Starting small made things easier
• Even without an IPv6 capable IPAM system you can
deploy IPv6 – use the IPv4 address
Lessons Learnt

25. Philip Garrad
STFC Network Technical Architect
philip.garrad@stfc.ac.uk
STFC Rutherford Appleton Laboratory
www.stfc.ac.uk

26. Any questions? /
Thank you

27. Reprocuring a
large campus
network, part one
Sam Wilson,
University of Edinburgh

28. Sam Wilson
Communications Infrastructure Section
IT Infrastructure Division
Information Services
The University of Edinburgh
Sam.Wilson@ed.ac.uk

29. Agenda
>University
>History and Current Situation
>Procurement
>Lessons

30. > The University of Edinburgh has a large computer network
> The network has grown organically over the years
– We’ve tried to be economical with equipment
– “Technical debt” - network needs to be refreshed
> We’re part way through the reprocurement
– This may be the most frustrating talk you
hear today
Summary

31. A slide from our Bidders Conference

32. Oops! That was 2017.
In 2018 we're 23rd=.
A slide from our Bidders Conference
Perhaps we should
buy a new network or
something.

33. >…apparently this kind of marketing makes network
providers give you things cheap. Who knew?
I’m a techy, but…

34. > It’s not all about technology
> “Digital Transformation” – everything is digital
– Data networking is fundamental
> Data Driven Innovation
– High capacity
– Rapid change
> Student Experience
– High quality WiFi
– Stability and performance
Business drivers

35. History – How did we get here?

36. > Plan installation of LAN across campus
> 10 Mbps Ethernet, 100 Mbps FDDI, 2 Mbps
serial
– Thick and thin ethernet
– Extensive fibre installation
> 1000 hosts, growing to 3000 hosts over
~5 years
> One of first two UK Universities to buy
Cisco routers
> IP(v4), IPX, AppleTalk, DECNet Phase IV,
bridging (for Pink Book)
Indulge me – 1988-1990

37. > 2014: New director of IT Infrastructure
– First task: infrastructure review
– Some kit up to 10 (20?) years old
– No budget for renewal
> Network has grown to >100K outlets
– Integral to University business, operations
and strategy
– Phones, WiFi, BMS, CCTV, Things…
> In the interim network budget -> Estates
– Building refurb cycle (10-20 yr) != network
replacement cycle (3-5 yr)
Fast forward to 2014
Oops!

38. Let’s buy a new network!

39. We’re a University; we have committees and projects and such
Well maybe not quite yet...
2014: internal infrastructure review
2016: PTS Consultants – consider network strategy
Nov 2016: PTS report – “buy a new network”
“we need a strategy” – set up a project
May 2017: Budget approved

40. > 5 main campuses
– University estate approx. 4 x 10 miles (or 10 x 10)
– 220+(ish) buildings
– Mainly owned or leased fibre
> Over 100,000 outlets
– 30K+ active ports
– 10M, 100M, 1G, 10G, 40G
– 2700 switches, 2700 WAPs
– 600+ racks
– ~13 core and distribution routers
> Constant churn
– 10 year building programme
Scale

41. > Wired and wireless networks
> Data centre firewalls
> Wireless firewall
> Other security features
> Management and monitoring
> Automation/Orchestration
> Future expansion
> Professional services
> 5+2+2+2+2 year contract
Out of scope
Scope
In scope
> Data centre networking
> Upgrade or replacement of
cabling unless essential for
solution
> Wireless p2p links
> Student residences
> (Perimeter firewall)

42. > Budget: [REDACTED]
> Project Board: senior representatives across UoE
> Project Team: ~7 core members
– Director ITI
– 2 x Communications Infrastructure Section
– Procurement Officer
– Project Manager
– 2 x PTS Consultants
> SMEs: Security, Academic computing, Lawyer (~10)
Now can we start the procurement, Daddy?

43. > Multiple rounds of dialogue – not just an ITT
> “Down select” at each round to reduce field
– Kind of knockout stages
> Drive down price
> Co-design?
– collaborate on what’s possible/available
– range of potential solutions
– beware of steering bidder and then rejecting
Procurement by Competitive Dialogue

44. > There are things I can’t tell you:
– because we’re only part way through the
process and we don’t know yet;
– because they’re confidential –
commercially sensitive or NDA;
– because we’ve decided not to tell anyone
– like the budget
Frustrations

45. > Premarket engagement
> OJEU Contract Notice
> ESPD
– Like PQQ – select top 10
> ITPD
– 10 provide outline solution – select 5
> ITCD
– Lengthy dialogue
– 5 provide developed solution – select 3
> ITCD 2
– Select 2
> ITT
– Select 1
Programme
YOU ARE
HERE!

46. > June-July 2017: Premarket engagement
– 6 manufacturers selected by consultants
– Alcatel Lucent, Cisco, Dell EMC, Extreme, HPE, Huawei
– ½ day presentations
– Own research
> What's on the market?
– Pure L2 or pure L3 backbone? Y
– VLANs or no VLANs? Y
– Overlays, tunnels, data/control plane...? Y
– NAC for security or configuration? Y
– Automation and orchestration? Y
– SDN (some or all of the above)? Y
– Firewalls and security features? Y
– You name it…
Pre-market engagement
?!

47. > September 2017:
– Backward looking scored Qs
– Includes draft ITPD document so they know what's coming
> About 20 responses
> Mid October: Choose top 10 to receive ITPD
– Formal scoring
> No, I can’t tell you who any of them were
Contract Notice and ESPD

48. > 3 Nov: ITPD Published
> 7 Nov: Bidders conference
> ITPD documents
– 91 page project brief - simplified network model
– MRs, scored Qs (4 pages), pricing spreadsheet
> Responses required
– MRs
– Outline solution (10 pages)
– Scored Qs & As (4+10 pages)
– Pricing spreadsheet
ITPD

49. > Responses received early December
– 10 x ((10 + 14 pages) + MRs + pricing spreadsheet)
> ½ day meetings with bidders to present their outline solutions
and answer questions
– Includes manufacturers
– NOT scored
> Answers scored
> Sorry letters and ITCD invitations issued 22 Dec
– We spoiled everyone’s Christmas
> You think that sounds like a lot of work?
ITPD Responses

50. > Mid-Jan to late Feb 2018
> 2 + 1 day with each of 5 bidders
– Discuss outline solution
– Questions arising from solution
– (Some) ways that the University is different from the model
– Bidders may change original solution and costings
> Generate ITCD document with scored Qs
– Expanded from ITPD brief and Qs
– But ask original Qs again - make sure the answers
haven't changed
> Responses required
– 10 page developed solution
– 30 page scored questions (8+22)
– Pricing spreadsheet
> Issued 21 Feb 2018
> Responses 14 March 2018
ITCD

51. > ITCD 2 (3->2) – March-June 2018
> ITT (2->1) – July-September 2018
> Selection and award – September-November 2018
> Implementation – November 2018 for 18 months
> Contract – 5+2+2+2+2 years
And next...?

52. > Budget: [REDACTED]
> 20 initial responses: [REDACTED]
> 10 ITPD bidders: [REDACTED]
> 5 ITCD bidders: [REDACTED]
> 3 ITCD bidders: [REDACTED]
> Solutions we’ve been offered? [REDACTED]
> Leaning toward any particular outcome? Not redacted - because
we have to judge all bids on their merits.
The frustrating bit summarised

53. Lessons?

54. > Competitive dialogue is a **lot** of work
– ITCD had 7 staff almost full time for 6 weeks, not
including scoring
– Also for the bidders
> We might have too many stages
– Each stage takes a lot of effort (info pack, dialogue,
evaluation, down selection)
– 10 - 4 - 2 - 1 rather than 10 - 5 - 3 - 2 - 1?
> The process can change your views
– I can’t tell you how. Sorry
Is there anything we'd do differently next time?

55. End of part 1
(If you want to see part 2 you’ll have to invite us back
next year)

56. Spare Slides

57. > ESPD: European Single Procurement Document
– Like PQQ – backward looking “have you done this before?”
> ITPD: Invitation To Participate in Dialogue
– Outline solution based on model design
> ITCD: Invitation To Continue in Dialogue
– After down selection
> ITT: Invitation To Tender
Alphabet soup

58. Another Bidders Conference slide. I’ve included it because…