In the CM division of Amdocs Product Management Group (PBG) the various products of Amdocs were ultimately integrated manually and required over 9 weeks before a release could be announced.
Many of the products in the suite depended on individuals to help adopt and integrate the systems together after which a team of testers could certify the integrated suite before release.
The aim of the effort we are describing is the move from the manual integration process to a fully automated process for building, integrating and certifying the products bringing the time down to 30 minutes using better engineered CI and virtualization tools.
During this journey we also reduced the time to create integrated environments from days to minutes and we INTRODUCE EAAS.
During the session we will highlight the challenges, solutions and implementations put in place to achieve this along with the preview of what is planned next.
Today we will take you through a journey that started almost a year and a half ago and we are still on it.
We’ll go from the first step which was to automate manual steps and we’ll look at the hurdles we faced and how we overcame them. (and let me tell you a secret it isn’t ROCKET SCIENCE but really simple logical steps!)
We’ll look at Environment as a Service and finally I’ll share with you our (ever changing) destination and goals.
BUT first, … a little about us…
Various roles from QA to Dev to Architect Various product domains Various product deployment types ,e.g., SaaS, installable, customize & install Lead Architect for the “Customer Management” suite of products in the Amdocs portfolio Largely agile SDLC
Amdocs is the market-leader in customer experience software solutions and services for the world’s largest communications, entertainment and media service providers Has been largely following a well defined structure a.k.a waterfall and has been slowly transforming to Agile division by division
With that introduction…
Developer checks in Periodically a product build is run The chosen one (by expertise or involuntarily) would chant a few spells. Build the product Then hand it to another Magician to integrate with the suite
Product and module teams Delivered “drops” “email” notified to Wizard or Witch!
Integration testing team “magician” Check in drops into source control! Create Integrated builds Fixed issues and follow up with the developers to get “hot-fixes” Align products Oh yeah, they also tested the functionality … eventually
Release time increased. Per drop 9 person-weeks, taking minimum of 6 weeks PER DROP! In a release we would need about 3 cycles for testing. 5 MONTHS regression cycles!
Total time: 9 WEEKS effort!!!!!! 6 Weeks elapsed time!!!!
Leader: Can we bring down this time? Expert: No WAY!!!! It has always been done this way!
Leader: We HAVE TO get this time down! Expert: It is IMPOSSIBLE!!!!
WHY???? The process of manually ensuring that all integration points line up
Per drop Dependencies 3 weeks Bring up the environment 4 weeks Management time to coordinate 2 weeks Elapsed time: 6 weeks
Dependencies includes: Sequence of product adoptions Foundations provided by other divisions, not in our control Prod 1 -> delivers on day 0 Prod 2 -> would use this delivery after “stabilization”on day 3 And so on…
Only “the Chosen One” can do it! And he’s on vacation And he didn’t take his mobile!
Softening of belief. Bold, underlined, italic to Italic only
IMPOSSIBLE Not us.
Loads of impediments, excuses to rationalize how IMPOSSIBLE it is! Talk about products age. 15 years, active customer base, acquired products, out-dated technologies, multiple incompatible technologies, etc., etc…
But IMPOSSIBLE was not a word to accept!
We were lucky. We had management support who believed this needed to be changed! They Invested in a PoC
So we grabbed the opportunity and
The goal Each product to go through the stages we’ve outlined! Clearly visible to everyone where we want to reach
Integration (with dependencies honored) to be automated product by product
And where we started from…
So many Supermen/Popeye …. “Adoption Magicians”
Automating is more than functional but rather the infrastructure point. Rolling out environments that are ready for use functionally
The only way to succeed!
Automate the build process for each of the products Integrate a Few (2 to 3) products. This time we had “Chosen products”
Not built using modern tools! No PROBLEMO. Let’s write shell scripts to automate the build and integration process
The first step Shell scripts + Jenkins = Automate product builds!
Have each product share a build in a common file store Eventually replaced with Nexus
Use the last successful build not more than 3 days old
Have each product consume from the common file store Eventually download dependencies from Nexus
Successfully tested VM snapshot saved in the Vmware catalog Integrated environments available for testing within minutes! EaaS!
Infra I want a product environment 5 minutes REALLY??? Yeah! Here’s the link!
Realization It works!!!! Its saves me so much time From 6 weeks (9W effort) to minutes! Do the math!
Greed Why only individual products? Why only 2 or 3 integrated products? When can my product be delivered this fast? Release no longer “controlled” by delivering environments only by the test cycle!
Release level savings from months to minutes (negligible)
And then we could think of having combinations and permutations (versions, products) we only dreamed off
Analyzed each product, shared the improvements and SHOWED the mind boggling benefits Now the only constraint was “Budget” but that started flowing too
We presented our Goal to be achieved in a year! Still too long!
Work backwards: Identify the following for each product and each integrated combination Product Name Blockers Needs Targeted Date
Identify what is missing? Live environments without losing data refreshed with every build
P4v Streams Per developer, maybe?
On demand CI for each stream Click to create a new CI flow CI as a Service!
Have a working environment to which every new build is continuously deployed Preserves my (test) DATA? Upgrade scripts for DB structure changes
Reduce the build cycles for each product… Developer check in to testing in integrated environment within 6 hours! Mission Impossible – 2!
In summary, we went from a manual process that took anywhere between 3 to 9 weeks To one that could create environments within 5 minutes using EaaS Both for single product lines as well as permutations of integrated products
And we continue our journey with visions of CI as a service and Continuous Deployment where the latest product version is continuously upgraded and data is preserved across each continuous deploy.
Agile Tour Pune 2015: Releasing a legacy product suite within 30 minutes: Savio D'souza, Rushikumar Rathod and Mitalesh Agarwal
Releasing a legacy
product suite within
17th October 2015