The document describes Broadcast Music Inc.'s automation of their software release process using IBM Rational tools. It discusses: 1. BMI's goals for automated release management including assembly, deployment, rollback, and redeployment. 2. How different IBM Rational tools like Team Concert, Quality Manager, and Build Forge are used to automate builds, testing, and releases of various BMI systems like WebSphere, Portal, and DataPower. 3. The technical details of setting up automated builds and deployments using Ant scripts for various components, promoting changes between environments, and storing assembled artifacts.

1. Broadcast Music – Release Automation
Rockstars!
Jim Harvey
Senior Director of Business Analysis and Quality Assurance, Broadcast Music
JHarvey@bmi.com
Greg Hodgkinson
Practice Director - Lifecycle Tools and Methodology, Prolifics
ghodgkin@prolifics.com
1874A
© 2013 IBM Corporation

2. Session Objectives
 Describe how to use Rational to automate your release
process
 Demonstrate flexibility of the Rational architecture in
automating a range of different DevOps tasks
 Share a bit of the technical stuff on how to achieve
automation on these different tasks
 Share some of our “release stories”
 Give you some ideas to get started
2
One! Two! Three! Four!

3. 3
This is BMI – Who We Are and What We Do
• Broadcast Music, Inc. - 1939
• Performing Rights Organization (PRO)
• Pay public performance royalties
• Operate on a non-profit-making basis
• 7 locations: Nashville, New York, Los Angeles,
Atlanta, Miami, Puerto Rico, London
• 600 employees
• 7.5 million works
• 500,000 songwriters and composers

4. IBM Portal
2013 Jan

5. Where We Were
 Development and Support Silos
 Several Development Languages
 Roles – Many Hats
 Inconsistent Development Life
Cycle
5
Where We Are Going
 Business Process Management(BPM)
 IBM Portal
 Service Oriented Architecture(SOA)
 Dedicated Roles
 Agile

6. Overview of our Tooling
IBM Focal Point
– Product and portfolio management
IBM Rational Requirements Composer
– Maintains all user stories, use cases and
associated artifacts
IBM Rational Quality Manager
– Test management tool
IBM Rational Team Concert
– Work item tracking linking development work to
backlog items, source control
HP Unified Functional Tester(UFT)
- Application and service test automation
Build Forge
6

7. SOAP/HTTPSOAP/HTTP
Federated ESB Architecture
WebSphere Service Registry and Repository(WSRR)
Data transformations
Portal, BPM and Mobile Consumers
SOAP/HTTP (S)SOAP/HTTP (S)
ReST/HTTP (S)ReST/HTTP (S)
JDBCJDBC
SOAP/HTTP (S)SOAP/HTTP (S)
Security (authentication and authorization)

8. Prioritization of Automation Needs
 WESB – Service Development
 WSRR – support service consumers
 Data Power
 Portal
8

9. Choosing a Partner
 IBM Rational Expertise
 Best Practices
 Guidance
 Implementation
 Enablement
9

10. What Do We Want Out of Automated Release
Management?
 Automated assemble and deploy steps
 Either trigger automation on-demand or based on schedule
 Notification of failure
 Easy troubleshooting if assemble or deploy fails
 Ability to track a release
 Control what gets included in the release
 Roll-back a deployment of a release
 Redeploy a release at any time in future
10

11. The Secret to Achieving a Top-10 Hit: A Good Producer
Release Engineer
(Music Producer)
Development
Team
(Band)
Team Concert
(Recording
Studio)
• Readies for distribution
• Controls recording/distribution
• Creative – arranges songs
• Catches bad notes
11

12. The Rational Team Concert Recording Studio!
The Rational Team
Concert Recording
Studio!
Rational Team Concert
 The core of the Rational CLM solution – provides
CCM capabilities
 Capabilities
– Planning, Work Item Management, Source Control, Builds
 Key use cases
– Agile planning
– Component-based configuration management
– Continuous integration
12

13. The Rational Team Concert Recording Studio!
The Rational Team
Concert Recording
Studio!
Jazz Build Engine
 Light-weight build engine that ships with
Rational Team Concert
 Ant-based build engine – runs Ant build scripts
 Allows build engines to be distributed
 Light-weight install – supports Windows, Linux,
z/OS and IBM i platforms
 Single-threaded build engine
 Easy to get up and running
13

14. A Simple Automated Release Management Process
Team deliver code
Automation stores
assembly for deployment
Automation assembles
(builds) the code
Release
engineer
triggers
assembly Release
engineer
promotes
release
Release
engineer
triggers
deployment
Automation
deploys
assembly
14

15. Promoting Through Streams
 Configurations managed in streams
 Streams can be linked together to form a chain – one per environment
 Changes promoted through Pending Changes view
15

16. Teeing Up a Recording – the Build Definition
 Build definition defines:
1. What…
…it is that will be
assembled/deployed
(scope)
2. Where…
…it is to be deployed to
…to get the source/assembly
from (i.e. which stream)
…it will run (which build
engine)16
Each unique set of actions
that you want to run is set up
as a build definition
3. When…
…it will run for scheduled builds
4. How…
…describes what actions to apply
5. Who…
…’s user/password to use for
permissions

17. Defining Scope – What Tracks to Include?
 Scope is defined using Properties on
the build definition –
–makes it easy to configure,
–easy to see what was included in a
build.
 Some examples:
17
WebSphere Portal
•portlets.list
•themes.list
•propertyFiles.list
WebSphere ESB
•modules.list
WebSphere DataPower
•classes.list
•fileProjects.list
WebSphere Service Registry and Repository
•services.list
HP Service Test
•testsuites.list

18. And Out Pops … a Build Record
18

19. What Was Recorded?
19

20. Assembled Items – Ready for Download
20

21. Audit-Ready Proof of Quality
21

22. Troubleshooting When Things Go Wrong
22

23. Release Automation Needs – Very Similar, Very Different
 WebSphere ESB
– Basic build of source code, and then deploy to
target nodes
– Also need to start application once deployed
 WebSphere DataPower
– Application “source” is on DP device
– Also deploy files stored in source control
– Need to merge selected objects from DEV into
 WebSphere Portal
– Application consists of source code as well as
configuration in server
 WebSphere Service Registry and Repository
– Deploying to repository as opposed to runtime
server
 HP Service Test
– Executing a set of tests
23

24. Automation Powered by “Ant”
24
External
Libraries
External
Executables
Shell
Scripts
Ant
ScriptsXSL
Transformations
Adam and the Ants

25. Ant Tasks for Publishing to our Build Record
 Publish Activities
– startBuildActivity
25
 Publish Logs
– logPublisher
 Publish Artifacts
– artifactFilePublisher
 Publish Test Results
– junitLogPublisher

26. WebSphere ESB Automation
26
IBM Integration
Developer Source
Files
EAR files
 Actions Required
1. Build application modules
2. Assemble application EARs
3. Deploy application EARs
4. Store application EARs
5. Start application EARs

27. WebSphere ESB Automation
 Ant tasks
– buildAppModules, assembleAppEARs, deployAppEARs, deliverAppEARs, startAppEARs
27
 runAntWid.sh
– projectImport ant task
– projectBuild ant task
– zip ant task
 ws_ant.sh
– wsUninstallApp ant task
– wsInstallApp ant task
– wsStopApp ant task
– wsStartApp ant task
 scp

28. WebSphere DataPower Automation
28
Service Objects e.g. XML
Firewall Service, Multi-Protocol
Gateway
Files e.g. Certificates
Deployment Policy
Base 64 Configuration
File
 Actions Required
1. Export service objects from device’s development domain into B64 file
2. Import files into device’s development domain
3. Import (merge) service objects and files into device’s integration domain
4. Take backup of merged integration domain
5. Deploy merged domain into test and production domains

29. WebSphere DataPower Automation
29
 Ant tasks
– assembleServiceObjects, assembleDomain, mergeDeltasToDomain, deployDomain,
deleteDomain, deleteServiceObjects, deployServiceObjects, deployFiles
29
 xslt ant task
 curl
 WDP XML Management
– dp:do-action
– CreateDir
– RestartDomain
– SaveConfig
– dp:del-config
– dp:do-backup
– dp:do-export
– dp:do-import
– dp:set-file

30. WebSphere Portal Automation
30
Portlets Source Code
Themes Source Code
Portal Configuration e.g. Pages
etc.
Application Property Files
Portlets WAR file
Themes WAR file
Portal Configuration File
 Actions Required
1. Create portlet and theme WARs
2. Deploy WARs to development, integration, test and production
3. Export portal configuration (pages etc) from development Portal server
4. Import (merge) portal configuration into integration Portal server
5. Import merged portal configuration to test and production

31. WebSphere Portal Automation
31
 Ant tasks
– assemblePortlets, assembleThemes, assembleConfig, deployApp (deployPortlets,
deployThemes, deployConfig, deployProperties), mergeConfig
 javac ant task
 zip ant task
 xmlaccess.sh
 releasebuilder.sh
 scp
 expect
 wsadmin.sh

32. WebSphere Service Registry and Repository Automation
32
Service files e.g. WSDL, XSD
Other
Publish ID
Version Number and Description
 Actions Required
1. Move files from source project
2. Change port bindings
3. Publish documents
4. Create links
5. Add publish ID property

33. WebSphere Service Registry and Repository Automation
33
 Ant tasks
– synchArtifacts, changePortBindings, publishArtifacts,
 curl
 REST API
– POST Content/…
– GET MetaData/…
– DELETE MetaData/…
– PUT MetaData/…
 xmltask.jar
– replace task
 move task

34. HP Service Test Automation
34
Service Tests
 Actions Required
1. Execute tests
2. Package up and publish test logs
3. Package up and publish test results
test

35. HP Service Test Automation
35
 Ant tasks
– runTests, zipAndPublishReport
 ServiceTestExecuter.exe
 zip ant task
 propertyregex ant task
 junitLogPublisher

36. Some Additional Tweaks: Storing Assemblies
 Store assemblies to include in release snapshot and use in deployments
 Were doing this manually in beginning – but decided to automate – save lots of additional
time
 Use the RTC command-line SCM client (scm.sh) to do this
– Execute the following commands
login
share (first time only)
status
checkin
changesetcomment
deliver
36

37. Some Additional Tweaks: Password Encoding
 There are a number of passwords stored in build
engine and build definition
– Runtime platform API access
– RTC SCM command-line client access
– Access to target server machines for copy across files
 Do not store these in plain text!
 Solution is to encode them using the native WAS
password encode/decode functionality
– Java class: com.ibm.ws.security.util.PasswordDecoder
37
Some thoughts on encoding…

38. Some Additional Tweaks: Preview Mode
 It is often useful to do a trial-run build just to
check that the build is processing the correct
scope – but you don’t want to actually run
before confirmed.
 Built-in preview capability – instead of
executing commands it writes them out to the
logs – and otherwise populates activities log as
normal.
38

39. Automation Release stories – How has the team benefited?
39
Automation is the
way to go! Sense of
urgency / invested
interest to complete
automation
1
Greater
transparency
in processes =
Less Drama!2
Having a feedback
process (via nightly
scheduled CI build)
helps tackle “build
failures” right away
3
Resolve dev
problems
sooner
- Improves code
quality drastically
- Error messages /
warning with
automated build
- Helps identify
missing code
Helps avoid
“It works on my
laptop”
“That’s Ok. Let’s fix
that later” mindset
4
Dev and QA benefits
from easy rollback to
use previous
snapshot
5
Control
changes via
“pending
changes” view.
Report generated
to see what’s
pending to move
from DEV to TST
& TST to PROD
6
QA- Helps control
things that were
flying “fast and
loose”
Opportunity: Can
work on stored
procedure
automation
7
QA gained better
control of what’s
in any
environment
8
QA - “Build Tasks”
email notifications
on successful runs
helps them out of
the blocks sooner
9
Improved
solution
quality
11
Smoke test helps
approve
changes
10

40. This Is Good So What Is Missing?
• Scripts have grown in complexity and sharing
and maintaining them is becoming an issue
• Hard to reuse parts of scripts
• How to combine scripts easily?
• Not trivial to combine scripts e.g. build then test
• Single-threaded build engines
• Although we have a number of build machines and
build engines, the more the automation is used
the more we encounter wait conditions
• Application deployment is no longer a bottleneck
– but what about environments?
• sometimes we hit an environment provisioning
bottleneck, esp. for new projects
40

41. Introducing an Enterprise Automation Solution
 Rational Build Forge can plug into Rational Team
Concert as a replacement for the simple Jazz build
engine
 Still get the integration between build data and SCM
and work item data
 Now also benefit from….
1. No more single threaded limitation on a build engine
2. Automation scripts are more easily managed
3. Can share common script pieces easily between scripts
4. Get better reuse of script components
5. Can easily combine scripts to create superscripts!
41

42. Going Gold – Automated Environment Provision
 Added the Rational Automation Framework to add
automated provisioning of environments
 Reduced bottleneck of setup time for new environments
 Ability to quickly configure environment based on predefined
configurations
 Can now quickly stand up a totally new test environment from
the ground up – all automated
 Vastly less effort required to stand up new environments –
frees up team for more important work
 Environment plans are a lot more predictable
42

43. You Too Can Be A Release Rockstar!
5 Tips to Get Started
1.Identify and automate the activities that take up time
in your release process – either because of complexity,
or because time consuming to do, or because cause
many problems that are time consuming to fix.
2.Start simple with the Jazz Build Engine – and move to
Build Forge once your automation increases in scale
and complexity.
3.Prioritize automation requirements using RTC agile
planning – and track defects the same way.
4.Deliver automation scripts using RTC’s SCM – and
use component-and-stream-based structure to deliver
scripts alongside code.
5.Work with team to assess effectiveness of
automation, and identify improvements for next
release/sprint.

44. Release Rockstar Recap!
1. “Everybody Needs Assemble and Deploy Automation”
2. “Publish It Up! (Service Contracts)”
3. “Sultans of Automated Service Testing”
4. “Welcome to the Joined-Up Scripts”
5. “I Love Automated Environment Provision”
6. “Smells Like A More Confident Release Process”
7. “Working 9 to 12 (Due To Reduced Effort)”
8. “You Can’t Touch This Improved Quality”
9. “Another Brick in the End-To-End DevOps Process”
10. “Go Your Own Way”
44

45. 45

46. 46
Daily Apple TV giveaway
 Complete your session surveys online each day at a conference kiosk or on
your Innovate 2013 Portal!
 Each day that you complete all of that day’s session surveys, your name will
be entered to win the daily Apple TV!
 On Wednesday be sure to complete your full conference evaluation to receive
your free conference t-shirt!

47. 47
Acknowledgements and disclaimers
© Copyright IBM Corporation 2013. All rights reserved.
– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM, the IBM logo, ibm.com, Rational, the Rational logo, Telelogic, the Telelogic logo, Green Hat, the Green Hat logo, and other IBM products
and services are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or
both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may
also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and
trademark information” at www.ibm.com/legal/copytrade.shtml
If you have mentioned trademarks that are not from IBM, please update and add the following lines:
[Insert any special third-party trademark names/attributions here]
Other company, product, or service names may be trademarks or service marks of others.
Availability: References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries
in which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided
for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any
participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided
AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise
related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating
any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license
agreement governing the use of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may
have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is
intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue
growth or other results.

48. 48
© Copyright IBM Corporation 2013. All rights reserved. The information
contained in these materials is provided for informational purposes only, and is
provided AS IS without warranty of any kind, express or implied. IBM shall not be
responsible for any damages arising out of the use of, or otherwise related to,
these materials. Nothing contained in these materials is intended to, nor shall
have the effect of, creating any warranties or representations from IBM or its
suppliers or licensors, or altering the terms and conditions of the applicable license
agreement governing the use of IBM software. References in these materials to
IBM products, programs, or services do not imply that they will be available in all
countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion
based on market opportunities or other factors, and are not intended to be a
commitment to future product or feature availability in any way. IBM, the IBM logo,
Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products
and services are trademarks of the International Business Machines Corporation,
in the United States, other countries or both. Other company, product, or service
names may be trademarks or service marks of others.

49. 49
How to use this template
 To allow all masters of your presentation to be updated correctly, download this
template to your hard drive and copy your existing slides into the new template
 See slide notes for further formatting instructions
 See below for suggested color palette. Core colors are blue and green. Additional
accent colors are purple, teal and orange, if needed.
Blue 2
R0
G176
B218
Blue 5
R0
G100
B157
Blue 6
R0
G63
B105
Blue 1
R131
G209
B245
Orange 2
R221
G115
B28
Purple 2
R127
G28
B125
Teal 1
R0
G166
B160
Green 1
R140
G198
B63
Green 2
R23
G175
B75
Green 3
R0
G138
B82