VMworld 2013 Venkat Gopalakrishnan, VMware Kurt Milne, VMware Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare

1. Moving Enterprise Application Dev/Test to VMware’s
Internal Private Cloud- Operations Transformation
Venkat Gopalakrishnan, VMware
Kurt Milne, VMware
OPT5194
#OPT5194

2. 2
Executive Summary
Key Lesson Learned
Invest in Agility, and Service Quality and Cost will Improve
AppOps Team
Deploy integrated, complex SDLC instances to support 600 developers.
Challenge
Process is manual, siloed, slow, unreliable. Reduces developer efficiency. Increases risk.
Two Fundamentally Different Options
1. Fix the “human middleware” on traditional infrastructure
2. Replace and automate on private cloud SDDC
Results From Choice to Replace and Automate on SDDC
 Process time – dropped from 4 weeks to 36 hours
 Developer productivity – increased 20% or more
 Project schedule risk - eliminated
 Annual infrastructure and operating costs - reduced by $6M annually

3. 3
Agenda
 The Challenge
 The Options
 The Solution
 Transforming Operations
 The Results
 Key Takeaways

4. 4
Corporate IT Application Group
Manage portfolio of enterprise
applications used by global
business functions
AppOps team
27 engineers
Customer
600 developers
Role
Provision 16 different dev/test instances
that include 80+ app components.
Infrastructure footprint
 ~4,000 non-production VMs
 ~500 production VMs
Enterprise Application Portfolio
SaaS 65
IT tools 50
Business 100
Total 215

6. 6
Project “OneCloud” – Explosive Tenant Growth
 Corp IT AppOps = Tenant #4
 Very low cost per VM
 “Cloud first” policy in IT
AppOps
SDLC
provisioning
Hands
On Labs
Hol.vmware.com
Services &
Support
Customer environment
reproduction
Sales
Engineering
Demo Pods
VMworld 2013
Management
BU Field
Testing
TechSummit
2013
Tech Ops
Mini R&D
Cloud
Training
LiveFire
Private Cloud IaaS
Software Defined Data Center
June
2012
Jan.
2013
Today End
2013
2014
Launched
Built on
vCloud Suite
4 tenants
10,000 VMs
9 tenants
38,000 VMs
12 tenants
50,000 VMs
More
services
Timeline

7. 7
Private Cloud IaaS
Software Defined Data Center
Bring Your Own – Application Ops
Three-tier Ops Model
Different Tenants
Different Application Ops
Application Ops
(Provided by Tenant)
Now an infrastructure service consumer. Provisioning.
Monitoring. Configuring. Upgrades. Maintenance.
Many typical ops tasks still required.
Infrastructure Ops
(Provided by OneCloud infrastructure team)
Network, storage, compute availability.
Deliver to SLA.
Tenant/Service Ops
(Provided by OneCloud service team)
Common service definitions, SLA,
tenant onboarding, tenant management
Private Cloud IaaS
Software Defined Data Center

8. 8
Application Ops – Requirements Vary By Workload Type
Ops requirements for this App…
…are different than Ops
requirements for this App
Corp IT– App Dev/Test
Provision: Complex App Stack
Manage: Yes
Duration: 6+ months
Provision: VM
Manage: No
Duration: 3 weeks
Sales – Demo Pod

9. 9
Bring Your Own – Cloud Automation and Management
Tenant – needs automation and
management capabilities
IaaS – needs automation and
management capabilities
Service Manager
Decides what goes in service catalog.
Service Catalog
Mechanism to request service.
Policy
Logic used to guide automation.
Cloud Automation and Management
Manage workloads and underlying services.
Tenant 1 Tenant 2 Tenant 3
Private Cloud IaaS
Software Defined Data Center

10. 10
Phased Project Approach
 Deploy automation and management
capabilities
 Create 5+1 vDCs
 Blueprints for 80+ applications
 Service catalog with 16 instances
 Transition 2,800 VMs – Dev, Test, UAT
 Key Milestone – 4 months
• 1st automated instance @ 172 hours
 Expand service profiles – using
expanded virtual network and
storage in IaaS
 Financial transformation – chargeback
 Advanced analytics, performance
management
 Transition 1,200 VMs Stage, LoadTest
Phase 2 - H1 2014
ProductionDev Test UAT Stage
Load
Test
Phase 1 - Completed

11. © 2013 VMware Inc. All rights reserved
Automation and Ops Transformation
Venkat Gopalakrishnan – Director IT Operations

12. 12
Traditional Operations Functions – Provided by AppOps
PeopleProcess Governance
Extension
via
API and SDK
3rd Party
Components
Cloud Automation and Management
vCloud Suite
Private Cloud IaaS
Software Defined Data Center

13. 13
Human Middleware Problem – AppOps Team View
Global Team
Management
Project manage
around PTO,
holidays,
variable skills
Capacity
Constrained
Only 4-6 projects
in parallel
Slow and
Error Prone
Many manual steps.
Ticketing systems.
Human error
Handoffs
Silos. Globally
distributed teams.
Multiple application
experts

14. 14
100% Task Automation – Not Going to Meet Needs
Request Infrastructure
Verification
Hardware
Setup
Build VMs –
New or Clone
DNS Entries
Install, Setup,
Configure
Workload Database
Refresh
Latest Code
Deployment
Load Balancer
Entries
Web Server
Configuration
Firewall
Changes
External Interface &
Integration
PPM Tasks
Workload
Monitoring Setup
Security – VM
access control
Testing
1- 2 days 3- 5 days 2 – 4 weeks 3 – 5 days
1 – 2 days 4 – 7 days 2 – 3 days 2 – 5 days
2 – 5 days 1 -2 days 2 – 4 days 1 – 2 days
3 – 7 days 2 – 3 days 1 day 5 – 6 days
Task timeWait time

15. 15
SDDC Eliminates Steps
Request Infrastructure
Verification
Hardware
Setup
Build VMs –
New or Clone
DNS Entries
Install, Setup,
Configure
Workload Database
Refresh
Latest Code
Deployment
Load Balancer
Entries
Web Server
Configuration
Firewall
Changes
External Interface &
Integration
PPM Tasks
Workload
Monitoring Setup
Security – VM
access control
Testing
1- 2 days
1 – 2 days 4 – 7 days 2 – 3 days 2 – 5 days
2 – 5 days 1 -2 days 2 – 4 days 1 – 2 days
3 – 7 days 2 – 3 days 1 day 5 – 6 days
Task timeWait time

16. 16
Automation Eliminates Wait Time…
Request Infrastructure
Verification
Hardware
Setup
Build VMs –
New or Clone
DNS Entries
Install, Setup,
Configure
Workload Database
Refresh
Latest Code
Deployment
Load Balancer
Entries
Web Server
Configuration
Firewall
Changes
External Interface &
Integration
PPM Tasks
Workload
Monitoring Setup
Security – VM
access control
Testing
Task timeWait time

17. 17
… and Manual Work
Request Infrastructure
Verification
Hardware
Setup
Build VMs –
New or Clone
DNS Entries
Install, Setup,
Configure
Workload Database
Refresh
Latest Code
Deployment
Load Balancer
Entries
Web Server
Configuration
Firewall
Changes
External Interface &
Integration
PPM Tasks
Workload
Monitoring Setup
Security – VM
access control
Testing
Task timeWait time

18. 18
Why Standardize and Automate Service Provisioning?
Service
Definition
Blueprint
Policy
POC1 POC2 To Catalog
Provision QA Staging Release
40 work weeks effort – Per Release…
20 work weeks effort – Once!
Run Book
36 hours
Service
Request
4 weeks
Virtual Data Center
Virtual Server
It takes less effort/time to convert the runbook into blueprints
than it takes to “run” the runbook...

19. 19
Transformation – Process
Challenges
 First version of automation solution did not meet all needs
 Actively deploying instances while building machines
 Difficulty in managing integration with SaaS apps
 High inflow of demand
Action
 Automation capability – parallel provisioning
 Testing suite additional functions getting automated – environmental
and functional
 Continuous process improvement in place, root cause action after every cycle
 Instance provisioning being treated as a ‘release’
 Documentation is key to achieve predictability

20. 20
Total Cycle Time – Improvements
1. Re-provision instead of repair,
and cross-training teams
2. Improve blueprints to drive down
defects, automate functional and
environmental testing
3. More automation and management
capabilities
Plan to get to 24 hour goal
• Even more automation and
management changes
• Improve QA testing process
Improvements
Provision – 16 hours
QA – 8 hours
2013 Goal
0
20
40
60
80
100
120
140
160
180
200
Test13 Dev14 Test14 Dev15 Test15 Dev16 Test16
1.
2.
3.
Provisioningtime(hours)
SDLC Instance - Oracle ERP with Portal (date)
05/07 05/22 05/27 06/19 06/25 07/22 08/05

21. 21
Process – Details
Results
 4 weeks to 36 hours.
 24 hours (Provisioning 16 hours, Testing 8 hours) by Q4’13
 Streamline demand intake process
 Created bandwidth to provision an instance per week
Key
Takeaways
 Automate end to end process, not focus on individual tasks
 Empower global team
 Don’t skimp on Blueprints

22. 22
Transformation – People
Challenges
New People Roles and Change in Skill Sets
 New role for Blueprint creation and management
 Automation requires global coverage to manage process
 Scarcity of skilled resources to perform new role
 Most top skills were in one location
Action
 IT resources obtained vCloud certification
 Technical skills assessment, create a well balanced global team
 Created subject matter expertise in installation and configuration
of tech stack/application
 Team got solid shell scripting, tasks automation and trouble shooting skills

23. 23
Ticketing Systems and Late Night Phone Calls – Gone!

24. 24
People – Details
Results
 27 – now 22 – goal 5 (old instances still in use)
 Provisioning can be initiated and executed from any
of global location
 Employees performing high value work like blueprint
management
Key
Takeaways
 Promote and help people internalize vision to get in lock step
 Mental shift – fix blue print and re-provision vs fix problem

25. 25
Transformation – Governance
Challenges
 Functional test failures
 Blueprint changes resulting in manual work
 Lack of service definition and process to track cost per service
Action
 Avoided changes during provisioning cycle
 Re-provision instead of repair
 Initiated programs to transform IT-as-a-Service (ITaaS)

26. 26
Blueprint Work – Governance Before Service Request

27. 27
Governance: Details
Results
 Predictable delivery of 36 hours, targeting 24 hours by Q4’13
 Improvement in functional testing, lower defect count
 15 instances provisioned in 4 months
Key
Takeaways
 Spend time in Blue printing all apps, no shortcut
 “Disposable Infrastructure” reduce IT Capex

28. 28
Results
Phase 1 Phase 2
Cycle Time
Hours per SDLC instance
172
36
Today
Phase 1 Phase 2
Virtual Machines Transitioned
To Private Cloud
Phase 1 Phase 2
AppOps Team
# of Engineers
Goal – 4000
2,800
2,200
Goal - 5
27
22
Reduced provision time
95%
(4 weeks to 36 hours)
Improved productivity
of 600 developers
20%
Reduced
IT operations costs
$1.5M /year
Able to say
“yes”
to developer requests
Reduced the cost
of a VM/month
80%
($133 to $20)
Reduced
infrastructure costs
$4.5M/year
672 hours (4 weeks)
Goal – 24 hours
Today
Today

29. 29
Phase 1 Phase 2
Cycle Time
Hours per SDLC instance
172
36
Today
Phase 1 Phase 2
Virtual Machines Transitioned
To Private Cloud
Phase 1 Phase 2
AppOps team
# of Engineers
Goal – 4000
2,800
2,200
Goal - 5
27
22
672 hours (4 weeks)
Goal – 24 hours
Today
Today
Reduced provision time
95%
(4 weeks to 36 hours)
Improved productivity
of 600 developers
20%
Reduced
IT operations costs
$1.5M /year
Able to say
“yes”
to developer requests
Reduced the cost of
a VM/month
80%
($133 to $20)
Reduced
infrastructure costs
$4.5M/year
Bottom Line
Agility is Self-Sustaining

30. 30
Key Takeaways (Advice)
Share results of early automation with developers (customers)
Show how the effort will help them
Training is key. Blueprint management role become key SME
Help them become experts
Don’t try to automate individual tasks
Take holistic approach – system’s footprint view
SDDC provides greater flexibility not possible with server virtualization
Software controlled infrastructure

31. 31
Additional Resources
Expert Lounge – Tues 12-1pm. Sign up in person at Expert Lounge.
Group Discussion – Wed 11am – 12pm. OPT 1002-GD
Related Architecture Session- Wed 3:30 – 4:30 (Full - look for repeat)
VSVC4948 - Moving Enterprise Application Dev/Test to VMware’s internal Private Cloud –
Architecture, Implementation and Integration
Blogs.vmware.com/cloudops
White paper and video/demo of this presentation
IT Transformation webpage on vmware.com
 http://www.vmware.com/solutions/vmware-it-journey/
@vmwarecloudops #cloudops

32. THANK YOU

34. Moving Enterprise Application Dev/Test to VMware’s
Internal Private Cloud- Operations Transformation
Venkat Gopalakrishnan, VMware
Kurt Milne, VMware
OPT5194
#OPT5194