In the CM division of Amdocs Product Management Group (PBG) the various products of Amdocs were ultimately integrated manually and required over 9 weeks before a release could be announced.
WHY????
Many of the products in the suite depended on individuals to help adopt and integrate the systems together after which a team of testers could certify the integrated suite before release.
OMG!!!!
The aim of the effort we are describing is the move from the manual integration process to a fully automated process for building, integrating and certifying the products bringing the time down to 30 minutes using better engineered CI and virtualization tools.
During this journey we also reduced the time to create integrated environments from days to minutes and we INTRODUCE EAAS.
During the session we will highlight the challenges, solutions and implementations put in place to achieve this along with the preview of what is planned next.
Today we will take you through a journey that started almost a year and a half ago and we are still on it.
We’ll go from the first step which was to automate manual steps and we’ll look at the hurdles we faced and how we overcame them. (and let me tell you a secret it isn’t ROCKET SCIENCE but really simple logical steps!)
We’ll look at Environment as a Service and finally I’ll share with you our (ever changing) destination and goals.
BUT first, … a little about us…
Various roles from QA to Dev to Architect
Various product domains
Various product deployment types ,e.g., SaaS, installable, customize & install
Lead Architect for the “Customer Management” suite of products in the Amdocs portfolio
Largely agile SDLC
Amdocs is the market-leader in customer experience software solutions and services for the world’s largest communications, entertainment and media service providers
Has been largely following a well defined structure a.k.a waterfall and has been slowly transforming to Agile division by division
With that introduction…
Developer checks in
Periodically a product build is run
The chosen one (by expertise or involuntarily) would chant a few spells. Build the product
Then hand it to another Magician to integrate with the suite
Product and module teams
Delivered “drops” “email” notified to Wizard or Witch!
Integration testing team “magician”
Check in drops into source control!
Create Integrated builds
Fixed issues and follow up with the developers to get “hot-fixes”
Align products
Oh yeah, they also tested the functionality … eventually
Release time increased. Per drop 9 person-weeks, taking minimum of 6 weeks PER DROP!
In a release we would need about 3 cycles for testing. 5 MONTHS regression cycles!
Total time: 9 WEEKS effort!!!!!! 6 Weeks elapsed time!!!!
Leader: Can we bring down this time?
Expert: No WAY!!!! It has always been done this way!
Leader: We HAVE TO get this time down!
Expert: It is IMPOSSIBLE!!!!
WHY????
The process of manually ensuring that all integration points line up
Per drop
Dependencies 3 weeks
Bring up the environment 4 weeks
Management time to coordinate 2 weeks
Elapsed time: 6 weeks
Dependencies includes:
Sequence of product adoptions
Foundations provided by other divisions, not in our control
Prod 1 -> delivers on day 0
Prod 2 -> would use this delivery after “stabilization”on day 3
And so on…
Only “the Chosen One” can do it!
And he’s on vacation
And he didn’t take his mobile!
Softening of belief. Bold, underlined, italic to Italic only
IMPOSSIBLE Not us.
Loads of impediments, excuses to rationalize how IMPOSSIBLE it is!
Talk about products age. 15 years, active customer base, acquired products, out-dated technologies, multiple incompatible technologies, etc., etc…
But IMPOSSIBLE was not a word to accept!
We were lucky. We had management support who believed this needed to be changed!
They Invested in a PoC
So we grabbed the opportunity and
The goal
Each product to go through the stages we’ve outlined!
Clearly visible to everyone where we want to reach
Integration (with dependencies honored) to be automated product by product
And where we started from…
So many Supermen/Popeye …. “Adoption Magicians”
Automating is more than functional but rather the infrastructure point.
Rolling out environments that are ready for use functionally
The only way to succeed!
Automate the build process for each of the products
Integrate a Few (2 to 3) products. This time we had “Chosen products”
Not built using modern tools!
No PROBLEMO. Let’s write shell scripts to automate the build and integration process
The first step
Shell scripts + Jenkins = Automate product builds!
Have each product share a build in a common file store
Eventually replaced with Nexus
Use the last successful build not more than 3 days old
Have each product consume from the common file store
Eventually download dependencies from Nexus
Successfully tested VM snapshot saved in the Vmware catalog
Integrated environments available for testing within minutes! EaaS!
Infra I want a product environment
5 minutes
REALLY???
Yeah! Here’s the link!
Realization
It works!!!!
Its saves me so much time
From 6 weeks (9W effort) to minutes! Do the math!
Greed
Why only individual products? Why only 2 or 3 integrated products?
When can my product be delivered this fast?
Release no longer “controlled” by delivering environments only by the test cycle!
Release level savings from months to minutes (negligible)
And then we could think of having combinations and permutations (versions, products) we only dreamed off
Everything NOW!
Analyzed each product, shared the improvements and SHOWED the mind boggling benefits
Now the only constraint was “Budget” but that started flowing too
We presented our Goal to be achieved in a year! Still too long!
Work backwards:
Identify the following for each product and each integrated combination
Product Name
Blockers
Needs
Targeted Date
Identify what is missing?
Live environments without losing data refreshed with every build
P4v Streams
Per developer, maybe?
On demand CI for each stream
Click to create a new CI flow
CI as a Service!
Have a working environment to which every new build is continuously deployed
Preserves my (test) DATA?
Upgrade scripts for DB structure changes
Reduce the build cycles for each product…
Developer check in to testing in integrated environment within 6 hours! Mission Impossible – 2!
In summary, we went from a manual process that took anywhere between 3 to 9 weeks
To one that could create environments within 5 minutes using EaaS
Both for single product lines as well as permutations of integrated products
And we continue our journey with visions of CI as a service and Continuous Deployment where the latest product version is continuously upgraded and data is preserved across each continuous deploy.