This document discusses how a large telecommunications provider implemented continuous delivery for a cloud deployment. It defines continuous delivery as automating the process of software delivery through techniques like continuous integration, automated testing, and continuous deployment. It then describes the specific components and tools used in the telecom provider's implementation, including adopting agile methodology, integrating rational team concert, automated testing with a REST API, and using SmartCloud Orchestrator for automated builds and deployment.

1. © 2013 IBM Corporation
Continuous Delivery of a Cloud
Deployment at a Large
Telecommunications Provider
M Kevin McHugh @mkmchu
Eduardo Patrocinio @patrocinio
1717 – February 25, 2014

2. 1
Agenda
Introduction
Continuous Delivery – what is it, how did we do it
Wrap up
Questions

3. 2
Agenda
Introduction
Continuous Delivery – what is it, how did we do it
Wrap up
Questions

4. 3
Solution Architecture

5. 4
Agenda
Introduction
Continuous Delivery – what is it, how did we do it
Wrap up
Questions

6. 5
Definition
Continuous Delivery (CD) is a design practice used in software development to
automate and improve the process of software delivery. Techniques such as automated
testing, continuous integration and continuous deployment allow software to be
developed to a high standard and easily packaged and deployed to test environments, resulting in the
ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk
and with minimal manual overhead. The technique was one of the assumptions of extreme
programming but at an enterprise level has developed into a discipline of its own, with job descriptions
for roles such as "buildmaster" calling for CD skills as mandatory.
(Source: Wikipedia 1/2014 - http://en.wikipedia.org/wiki/Continuous_delivery)
The ability to build a releasable asset with nearly trivial effort
What about Continuous Deployment?

7. 6
Components
Method
Continuous Integration
Process
Assets
Automated Test
Automated Build of deployable asset
Tools

8. 7
Method: Agile
7
Team Member
Team Member
Customer
Team Member
Team Lead
Agile
Value-driven lifecycle with regular production of working software
Small, co-located team developing straightforward software
Goal is to develop a high-quality system in an evolutionary, collaborative, and self-organizing manner
Pre-requisite: can build a product backlog
Agile
Scrum Roles (pigs)
Product Owner
– Owns the product backlog
– Chooses what work to do next
Team Member
– Creates the solution
– Owns estimation of difficulty of items in the backlog
– Constrains how much work can be done
Team Lead (Scrum Master)
– Removes obstacles
– Status reporting

9. 8
Agile terms and how the team operates
Backlog Grooming
Sprint Planning
Scrums
Velocity
Sprint Demo
Sprint Retrospective
Sample User Stories and point assignments:
8
Story
Points Summary
13 pts As a Bank Customer, I want to apply for a loan, so that I can…
8 pts As a Bank Customer, I want to deposit cash
5 pts As a Bank Customer, I want to change my PIN.
3 pts As a Bank Customer, I want to withdraw cash
1 pt As a Bank Customer, I check my account balance
5 pts As a Bank Customer, I want to buy postage stamps
Agile
Method: Tenants of Agile and its iterative cycle
Sprint 1 Sprint 2
Story Points DeliveredStory Points Delivered Story Points DeliveredStory Points Delivered
Velocity (rolling average of points delivered)Velocity (rolling average of points delivered)

10. 9
Agile adoption – adoption by execution
Two-week sprints
Single Product Owner
Scrum Master
Rank user stories
Set up the meetings
Method: What we did Agile
Sprint 1 Sprint 2
Story Points DeliveredStory Points Delivered Story Points DeliveredStory Points Delivered
Velocity (rolling average of points delivered)Velocity (rolling average of points delivered)

11. 10
10
Sprint by Sprint view of demonstrated progress of feature delivery
Agile
Release 1.0 - Functional Story Point Burndown
0
50
100
150
200
250
300
350
400
450
Sprint1Sprint2Sprint3Sprint4Sprint5Sprint6Sprint7Sprint8
Time
StoryPoints
Backlog Size
Remaining Backlog
Sprint Velocity
Projected Pts
Planned Pts
Method: Sample Burndown – Measured delivery

12. 11
Continuous Integration
Continuous integration (CI) is the practice, in software
engineering, of merging all developer working copies with a
shared mainline several times a day.
(Source: Wikipedia 1/29/2014 - http://en.wikipedia.org/wiki/Continuous_integration)
Develop features or fix defects on isolated branches
Merge branches onto the main
Identify points in time on the main that represent a viable code drop
This all drives the ability to have a stable, automatically tested,
“main” which can be sent forward for Test, QA, and Production
deployment

13. 12
Continuous Integration – what we did
Initially
Work space isolation obtained via Business Process Manager (BPM)
locking of a business process
The team is working on the main
This limited / prevented concurrent engineering
Led to Sprint-fall
Evolved into
Paired programmers working on separate tool kits, and backend
processes

14. 13
Process: 30,000 ft view
Development Test QA Production
Main
Feature 455
Unit Test
Asset Mgmt deploy deploy deploy
Build
Customer E2E
Automated Test
- REST API
Customer
Manual Test
- UAT
Automated Test
IBM Automated
Test
- REST API
- UI
IBM Automated
Test
- REST API
- UI
Customer
Manual Test
- UAT

15. 14
Assets
Assets include any element that the team can change on any of
the systems supporting the project. For example:
A WAR file
An executable
An environment variable
A database definition

16. 15
Assets – what we did
BPM process application and tool kit
TWX files
Pattern
The topology of an application
Patch
Those changes to SCO software and/or associated configuration changes
Configuration
Those changes to the systems supporting the SCO product
Configuration of the VMs internally and from a VM mgmt standpoint
Images
The building block for a pattern

17. 16
Rational Team Concert
Assets – what we did
Development
Exported as .twx
file, then checked
into RTC along with
a change
description
Import .twx from
RTC
QATest Production

18. 17
Automated Test
Automated test provides the means to assess the quality of the
main branch or the deployed release
Through the use of Automated Test, the Continuous Delivery
framework can then decide of a particular release candidate is of
sufficient quality to be promoted to the next level

19. 18
Automated Test – what we did
Test Architecture
Automated REST API Test
Deliver a solution that can be accessed via REST API
Verify that, through the REST API, all features work
Automated User Interface (UI) Test
Confirm that the solution works with the developed user interface

20. 19
Automated Build of Deployable Asset
Automated build provides the means to collect the set of assets
from the development system such that they can later be
deployed to a target system
Automated deploy provides the means to place the assets onto a
target system for test and use

21. 20
Automated Build of Deployable Asset – what we did
Use SmartCloud Orchestrator (SCO), REST based export/import
feature
Implemented a persistence layer to encapsulate differences
between deployment environments
Leverage build and version control tools
Control asset versions
Initiate automated testing
Track test results

22. 21
Tools to integrate and support
Integrate SCM and Build management tools to
Speed the development
Deliver to the customer control systems
Automated Test
Test the User Interface – record and replay UI tool
Test the REST Interface – test harness to invoke API, record results
Automated Build
BPM – REST API based extract from SCO
Patches, etc handled more conventionally
Automated Deploy
BPM – REST API based import to SCO
Patches, etc handled more conventionally

23. 22
Tools to integrate and support – what we did

24. 23
Tools to integrate and support – what we did

25. 24
Agenda
Introduction
Continuous Delivery – what is it, how did we do it
Wrap up
Questions

26. 25
Components
Method
Continuous Integration
Process
Assets
Automated Test
Automated Build of deployable asset
Tools

27. 26
…In Retrospect
Defect zone vs. Story zone (defect backlog vs. story backlog)
Team commitment to Agile and CD – No boycotts
Sprint granularity of CD vs. Story level of CD
Deliver both Product and Services
Difficulty grasping task ownership and delivery

28. 27
Questions

29. 28
Thank you!!