Presentation from Networkshop46. Brunel’s experience of designing networks and systems to use the Jisc shared data centre - by Simon Furber, Brunel University Developing an internal business case, and the associated benefits and savings, to support use of the Jisc shared datacentre - by Mike Cope, UCL Hyper convergence - our journey to the future - George Ford, University of Creative Arts

1. Data centre
networking
Chair: Guy Sudron,
Data centre manager, Jisc

2. Brunel’s experience of
designing networks
and systems to use the
Jisc shared data centre
Simon Furber,
Brunel University

3. Design Networks and Systems to
use the JISC Shared Datacentre
Simon Furber
Network and Infrastructure Manager

4. Brunel University London 4
Build site for our new Teaching and Leaning Centre
The John Crank Building

5. Brunel University London
The issues we had
• The primary data centre DC1
• One of the main network nodes half of the fibre end points (the original node 0)
• Janet connections
• BT Connections
• Virgin Media Connections ( ** are still in there ** )
• The Computer Centre Staff and IT work areas
• There was no space on campus
• There was no time
• The construction of the New Learning and Teaching Centre was a high level priority
5

6. Brunel University London
The constraints
• No new data centre on site
• Confusion over the cloud and why that didn’t fix all our problems
• Scale of the project
• The timing and pressure to demolish
• Separating the PoP from the Data Centre
• Planning and funding
6

7. Brunel University London
The John Crank Decant Project
• There is a program board that has oversight of all the aspects of this project
• Re-site all DC1 assets into DC2 or Move to SDC
• retire old hardware or virtualise
• Find a new home for the core network and it’s services
• Limited to 55M radius to preserve as much existing fibre and re-terminate
• Design and implement a new fibre network to remove single dependencies on John Crank
• Find a new home for the Computer Centre
• Staff
• Teaching IT Workareas
• DR Facilities
• Equipment and Storage
• New Storage and VMWare Host procurement
7

8. Brunel University London
We had some luck
• We already had a second data Centre which gave us resilience for BC and DR
• We became part of the Janet Optical Network (we are very lucky)
• Link Between from Brunel to IC and Brunel via RHUL to UCL - 2 x 40G bearers
• We have two nodes and avoided putting one of them in DC1
• The JISC Shared Data Centre at Infinity now Virtus was just down the road
• This had the same Janet fibre infrastructure.
• The timing of the VMWare hardware replacement and Storage replacement
• We were also able to avoid using DC1 for new services
8

9. Brunel University London
We Built a Schneider cube at SDC
• Pod B DH3 Slough SDC
• 12 Racks with enclosed cold isle
and doors DC2
• Utilising the Janet Optical
Network we ordered direct
circuits to SDC
• Designed the layouts and
monitoring
• Environment
• CCTV
• Operate it as a lights out
secondary datacentre
• New VMWare hosts and
Storage were deployed straight
to SDC
9

10. Brunel University London 10
Pod B DH3 Slough SDC

11. Brunel University London 11

12. Brunel University London
We needed a bridge built
12

13. Brunel University London 13
So we built one
• 4 x10G dual path links via Janet Network point to point
• Dual Ciena Node at either end
• 4 resilience layer 3 PtP links with Cisco OTV to transport the VLANs
• 4 x Nexus 7702, 2 x Nexus 5500’s and 4 x MDS 9250
• Layer 2 stretched and Layer 3 Separation
• FCIP for Storage

14. Brunel University London 14
We Built a new PoP
• Modular containerised solution
• Generator and UPS backed
• Designed and built to our specification
• 7 Racks with separate hot and cold isle with free cooling most of the time
• Not intended as a DC but will host infrastructure service
• We will be moving the Second Janet node into it
• Designed everything
• Operate it as a lights facility
• New VMWare hosts and Storage will be not deployed into it
• Phased migration/withdrawal of service about to begin

15. Brunel University London 15

16. Brunel University London 16

17. Brunel University London 17

18. Brunel University London 18
The John Crank Decant Project – Strategic Outcome
• Moved one Data Centre off site into shared JISC Data centre
• Virtualised and retired old server hardware (reduced migration)
• Moving services where applicable to the Cloud
• Procured new Sever Hardware and deployed strategically (reduced migration)
• Procured new storage and deployed strategically (reduced migration)
• Moved from VMWare 5.x to 6x
• New Fibre infrastructure designed and being built
• Ability to optimise and reorganise infrastructure minimising disruption
• Did not achieve Janet Connectivity at SDC - Future

19. Brunel University London 19
The Experience
• Very complex
• Planning is key and knowing that plans don’t survey contact with reality
• Good partners who are willing to be flexible
• JISC / Virtus
• ON365 / Schneider / Kinetic IT
• MAVIN / Eaton / Sensorium
• HSL / EMCORE / Brandrex / Commscope / Excel / Prism
• Cisco / BT / Dell
• Strategically designs need attention to detail
• Programme Management – We had a good one who kept us honest and kept his head

20. Brunel University London 20
Thank You
simon.Furber@brunel.ac.uk

21. Developing an internal
business case, and the
associated benefits and
savings, to support use of
the Jisc shared datacentre
Mike Cope, UCL

22. Mike Cope, CIO at UCL (mike.cope@ucl.ac.uk)
Making the internal case for the Shared Datacentre
27th March 2018

23. Agenda
 Historical context at UCL
 Datacentre strategy at UCL
 Making the case for the Shared Datacentre
 Advantages of the Shared Datacentre contract
 Current situation
28-Mar-18 © UCL 2018 23

24. What is PUE
 PUE (Power Usage Effectiveness) is a measure of datacentre efficiency
 PUE = Total power into the DC / Power into actual computing kit
 The difference between power into the DC and power into computing kit is
mainly energy to run the cooling but there can be other losses as well
 Highly efficient modern DCs can achieve a PUE around 1.2
– DC needs to be almost fully utilised to achieve low PUEs
– Waste energy recover from heat generated by IT kit can reduce PUE further
28-Mar-18 © UCL 2018 24

25. Historical context at UCL
 Three significant centrally managed on-campus datacentres
 Plus 50+ server rooms in academic departments
– Many small and poorly maintained, about 200 racks in total
– About 840m2 of space used in academic buildings for server rooms
– Server rooms in odd places, one in the top of a lift shaft and one required stepladders to
access
 UCL’s central London location means space is highly constrained
 Poor PUEs:
– Often around 2.2 for small server rooms
– Central datacentres around 2.0
– Modern facilities should achieve 1.25 or less
 All due to history and a lack of strategy
28-Mar-18 © UCL 2016 25

26. The good, the bad and the ugly
28-Mar-18 © UCL 2018 26
One of the better Server Room examples
and how it looks in the Shared datacentre

27. Have a strategy and for UCL this was…
 Have more than one but less than three datacentres
– Centrally managed, large and flexible
– Roughly equal sized, at least 20 miles apart
– Independent active-active with globally load balancing and no L2 between them
 One needs to be on-site due:
– Nature of some computing related research
– Consolidating dispersed academic server rooms into a modern shared facility was a big
prize. Seeking to do this and also move offsite was too much for one step
 HPC benefits from modern, efficient and scalable DC facilities
– Focus central HPC in offsite DC
– Local departmental HPC based in on-site DC
 …but this will be the last on-site DC we ever build…
28-Mar-18 © UCL 2018 27

28. Making the case to use the Shared Datacentre
 Financials:
– Space savings
– Power savings
– … these offset the rack monthly costs
 Flexibility and agility:
– One shared space is much more flexible than many smaller spaces
– Shared DC better supports collaboration and consortium research grants
 Maintenance and reliability:
– Dispersed academic server rooms not maintained to a good standard due low investment
and general Estates maintenance approach meant poor reliability
 The world has moved on:
– DC space now a utility like power and water
– Allows the organisation to focus on more strategic objectives
28-Mar-18 © UCL 2018 28

29. Making the case – power savings
 Assessed the power taken by the existing computing kit
 Assessed the overall power consumption of the existing DCs and Server
Rooms containing this computing kit (high PUE)
 Calculated the power that would be taken at the Shared DC for the same
power taken by the existing computing kit (at a PUE of 1.25)
 Power consumption differences drive savings since each MW reduction saves
about £1m pa
 Don’t forget sustainability and reduced CO2 emissions from reduced power
consumption
28-Mar-18 © UCL 2018 29

30. Shared DC contract tailored to HEI needs
 KISS principle (Keep It Simple Stupid)
– Only co-location facilities and smart hands
– Other higher level support services can be layered on top
– Institutions able to trade between each other
 Shaped to meet HE needs including research intensive
– Racks up to 30kW each, not typical of the commercial world
 Strong SLAs and competitive prices
– Set prices and indexation
– Contractually capped PUE of 1.25
– Collective discount with more volume
 Easy contracting
– Each university contracts under the JISC framework
– No long procurement, turn around in weeks
 Long term stability
– 15 year contract (3 x 5 years)
28-Mar-18 © UCL 2018 30

31. Current situation
 Southern Shared DC
– Institutions have taken 294 racks and 2.96MW of usable power
– Average power per rack of 10kW. Largest 19kW and smallest 4kW
– Largest installation for one institution - 71 racks
– 21 institutions in total
 UCL
– Great success
– Most HPC now located in Shared DC
– Academic departments keener than expected to use the new facilities
– UCL first order in late 14, now have 46 racks
28-Mar-18 © UCL 2018 31

32. Summary
 Have a clear strategy
 Making the case:
– Power savings
– Space savings
– Agility, scalability and professionalism
– Reliability
– Allowing the organisation to focus on more strategic objectives
 There are alternatives but the Shared DC contract is shaped to meet HEI
needs
– 21 institutions using the Southern Shared DC with 294 racks
– New Northern Shared DC established in Leeds
28-Mar-18 © UCL 2018 32

33. 28-Mar-18 © UCL 2018 33

34. Hyper convergence -
our journey to the
future
George Ford,
University of Creative Arts

39. UNIVERSITY FOOTPRINT
Remote Learning
Business School
International
Research
Shape

41. Farnham
Canterbury
Rochester
Epsom
Backups
Security
Site Bandwidth
Disaster Recovery
Increasing Storage Requirements
SERVICE DISTRIBUTION

42. Storage Fabric Compute Network
TRADITIONAL ARCHITECTURE

43. LEGACY BUILD
P2V
Hypervisor
Virtual
Machines
Physical
Machines

44. NETWORK BANDWIDTH

45. NEW TECHNOLOGY

46. Performance
Availability
Capacity
Function
Security
Service
15
30
45
60
III
VI
IX
XII
Backup
Upgrade
Patching
Break / Fix
Maintenance
SERVICE DELIVERY

48. DESIGN – DATA CENTRES
Managed Environment
Feature Rich Service
Secure Facilities
Flexibility Service

49. DESIGN - CONNECTIONS
JANET
Internet
NetPath
1
2
3
4
Primary 10gbps
Connection
Primary 10gbps
Connection
Secondary 10gbps
Connection
Secondary 10gbps
Connection
VLANs stretched across two data centres
(Border Gateway Protocol)

50. DESIGN - CONNECTION DROP
JANET
Internet
NetPath
1
2
3
4
Primary 10gbps
Connection
Primary 10gbps
Connection
Secondary 10gbps
Connection
Secondary 10gbps
Connection
VLANs stretched across two data centres
(Border Gateway Protocol)

51. DESIGN – DATA CENTRE DROP
JANET
Internet
NetPath
1
2
3
4
Primary 10gbps
Connection
Primary 10gbps
Connection
Secondary 10gbps
Connection
Secondary 10gbps
Connection
VLANs stretched across two data centres
(Border Gateway Protocol)
HSRP Default Gateway

52. Resource Unit Network
DESIGN - HYPER-CONVERGANCE

53. DESIGN - CAPACITY
Raw Capacity
FTT = 1
RAID 1
Replication
Backup
1000tb
500tb
250tb
125tb
63tb

54. PRE-PRODUCTION BUILD
1
2
1
2
SAN
Blades
Network
Legacy 1gbps
connections
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
LegacyHyper-converged
Campus
Public Services
10gbps
connections

55. NETWORK INTERCONNECT
1
2
1
2
SAN
Blades
Network
Legacy 1gbps
connections
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
LegacyHyper-converged
Campus
Public Services
10gbps
connections

56. MIGRATE CAMPUS VPNs
1
2
1
2
SAN
Blades
Network
Legacy 1gbps
connections
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
LegacyHyper-converged
Campus
Public Services
10gbps
connections

57. MIGRATE PUBLIC NETWORK
1
2
SAN
Blades
Network
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
LegacyHyper-converged
Campus
Public Services
10gbps
connections

58. MIGRATE VIRTUAL MACHINES
1
2
SAN
Blades
Network
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
LegacyHyper-converged
Campus
Public Services
10gbps
connections

59. SHUTDOWN LEGACY
1
2
SAN
Blades
Network
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
LegacyHyper-converged
Campus
Public Services
10gbps
connections

60. LESSONS LEARNED
Design Before You Start
Factor in Disk Space Resilience
Factor in transfer method
Factor in transfer speed.
Hidden costs to build method
Solution Testing
Test, Test, Test
Plan for Expansion

61. PLANNING NEXT STEPS
1
2
Backup SAN
New Services
Network
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Resource Unit
Network
Hyper-converged
Campus
Public Services
10gbps
connections

62. CC0 Creative Commons | Free for commercial use | No attribution required
TRUSTED SUPPLIERS

63. CC0 Creative Commons | Free for commercial use | No attribution required
SAFETY / SECURITY BY DESIGN