Lean data center, a telco experience by Andrea Pinnola, Telecom Italia

Copyright © Institut Lean France 2012

3 & 4 October, 2013
Paris, France

European Lean IT Summit 2013

Lean Data Center: A Telco Experience
Andrea Pinnola - Telecom Italia


Agenda
•
•
•
•

TI Background
Bologna Data Center Operations Pilot
Extending to the Delivery Value Streams
Lessons Learned

Lean Data Center: A Telco Experience - Andrea Pinnola

2


Telecom Italia Group Figures and
Organization (*)

Telecom Italia is the leading Italian Telecommunications and ICT
provider, one of the most important Telcos players in LATIN
AMERICA and one of the Top Ten global telcos players
As of March 31th, 2013

3


Domestic IT Infrastructure
Milano
(Virtual
Data Center)

IDC Cesano
Maderno
IDC Rozzano

VDCN Network
2,5 Gbps
155 Mbps

DC Padova

Torino

DC Bologna
Firenze
DC Oriolo
Romano
DC Bari
IDC Pomezia

National Data
Center Sites
Local Service
Centers

Palermo

•
•
•
•
•
•

7 National Data Center Sites
4 Local Service Centers
24.000 Servers
21.000 TB Storage
20.000 TB Backup
12.000 TB Backup
Transferred/month

Napoli

Organization
Engineering & Delivery coordination
plus 3 Data Centers Areas (North
East, North West & Center South)


4


Lean DC Bologna Pilot
The case
• Difficulty to answer to all the incoming requests
• Feeling overwhelmed
• Not clear work loads
• Customers complains

Focus on Operations activities: Reducing
Problems and Improve Effectiveness


5


Bologna Data Center Delivery &
Operations Teams
• Data Management:
 Storage
 Backup

• Systems:
 Windows e Linux
 UNIX

• Networking
• Plant & Logistics

Sponsorship from the Head of Infrastructure
Head of DC Bologna willing to improve

6

The Improvement Cycle
Process
Identification
Activity
Tracing
Data
Analysis
Improvement
Identification
and Proposal

Improvement
Execution and
Check


7


Make Process & Problems Visible:
e.g. Management of Backup Policies
Over 1200 Backup Policies for Systems and Applications
Requests to know
Policies in place
Request
Documentation
Rework

Problems
No access to the
Policies DB
Difficult Password
Retrieval

OOPS … No Data ?
Problem!


8

Make process visible: tracking requests
Operation requests Tracking

Track
every
Backup
Policy
Request received by Mail, IT
System, Phone call, coffee chat …

From 7-8 main activities / month
to 31 activities tracked / week !
100%

10

80%

8

60%

6

40%

4

Impl. Policy

Informazioni

0%
Modifica
SW

20%

0
Fault

2

Percentuale cumulata

120%

12
Numerosità

14

Review of backup controls in order
to prevent faults incidence

9

Make process visible: tracking problems
Disk Drivers Problems

• Backup Faults: highlighted
recurring errors on 4 drives
• New (type of) Fault not
described within the Fault
Management procedures


10


Gemba Walk: Critical cabling

In one room:
• Longer cable plug in
• Old cables not removed
• Difficulty and risk of faults
during interventions
Why ?
• Continue escalating requests of
new connections done in a
hurry


11


Flow of improvements proposals
40

By Team

30
20
10
0

106
40
30
20

By Type

• All teams involved in a 5 months
timeframe
• Created one sharing point for
problems, countermeasures
proposal and approval
• Weekly review meeting with the
head of DC, Team Leaders and
Bologna Facilitator

10
0


12


Actions and Results (Examples)
Actions

Results

• Storage size controls contemporary to • Effectiveness of storage assignment and
storage assignment
reduced monthly reporting (4d to 2h)
• Semi-automated procedure to OS
patching

• Reduction of OS patch time and of global
delivery time of an updated server

• Simplification of systems /servers
password management

• Eliminated password renewals delay
• Reduced time to interventions

• Backup policy requests simplified
templates

• Reliable policies update and reduction of
info request

• Enriched scripts to better signaling
backup failures

• Reduced errors and increased reliable
operations

People feeling change and the effects of
their improvements
13


Results: Critical cabling recovery

• Clean situation restored
• New interventions simplified
• Recovered fibers for 1 year
of consumption (variations)


14


Moving Forward: involving the other DCs
•
•
•
•

Lean DCs Steering Committee
Biweekly meetings to share improvements
3 Lean DCs Facilitators appointed
Involved all the DCs Team Leaders

• … move to a wider Lean initiative and joint DCs
responsibility


15


Moving Forward: the Delivery case
• Motivation
 Growing number of installations & requests

 Long time to deliver
 Need for better delivered quality

• Three Cases on:
 Infrastructure Creation
 Virtual Machines Delivery
 Market Infrastructure Delivery

Workshops on Lean Basics and VSM

16


Teamworking & Common Knowledge: the
actual Value in Value Stream Mapping
Rome

Infrastructure
Creation

Virtual
Infrastructure

Delivery to External
Market

Main Goals: Lead Time Reduction and reliable end date


17


5S in physical spaces

•
•
•
•

Before
Time spent to look for materials
Disordered Workspace
No space to act
Access forbidden to the customer

•
•
•
•

After
Phase out of obsolete and not used
Space to mount racks & shelfs
Visual control of quantities
Better Customer perception


18

5S on IT Shared Servers

•

- 50% of the “space quota” assigned to the Shared Server

•

- 83% of documents space, complete and accurate
information mantained


19


OPL: standardization and knowledge transfer

People able to manage the application since their first access

20

Results: Visual Mng & Server Farm Creation
•Physical Flow

•Information Flow

• Organization ‘Flow’
Lead Time reduced by 15%
Effort reduced by 5%

21

Results: Virtual Server Delivery
SERVICE REQUEST BACKLOG TREND

SERVICE REQUEST AGING TREND

Improvement
Lead Time

- 40%

Savings

4 hours for each activity

Quality

- 50 % reworks on docs


22


Results: External Market

Improvement
Lead Time

- 50%

Operation Tests Definition

- 50% Elapsed Time

Rework on project docs

- 75% Effort Spent

Organization

One single person as DC reference
assigned for each delivery


23


Lessons Learned
• Make Process and problems visible
• Adapt Lean to IT (e.g. 5S in physical and ‘virtual’ spaces)
• 80% of the knowledge already there (people are experts)
• Act readily on problems to get people involvement

• Management commitment from the start
• Customers involvement after a while


24


Acknowledgements
• To all the colleagues of the IT Infrastructure
Department that initiated and participated to the
Lean DC initiative


25


andrea.pinnola@telecomitalia.it


26


More Lean IT videos and presentations on
www.lean-it-summit.com

Lean data center, a telco experience by Andrea Pinnola, Telecom Italia

More Related Content

What's hot

Viewers also liked

Similar to Lean data center, a telco experience by Andrea Pinnola, Telecom Italia

More from Institut Lean France

Recently uploaded

Lean data center, a telco experience by Andrea Pinnola, Telecom Italia