Successfully reported this slideshow.
Your SlideShare is downloading. ×

Agile Tour Pune 2015: Releasing a legacy product suite within 30 minutes: Savio D'souza, Rushikumar Rathod and Mitalesh Agarwal

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 16 Ad
Advertisement

More Related Content

Slideshows for you (20)

More from India Scrum Enthusiasts Community (20)

Advertisement

Recently uploaded (20)

Agile Tour Pune 2015: Releasing a legacy product suite within 30 minutes: Savio D'souza, Rushikumar Rathod and Mitalesh Agarwal

  1. 1. Releasing a legacy product suite within 30 minutes Rushikumar Rathod Mitalesh Agarwal Savio D’Souza 17th October 2015
  2. 2. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs2 Agenda Manual to automated Hurdles Work arounds EaaS Final Goal
  3. 3. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs3 About Us 11+ years building infrastructure Technologies manager Rushikumar.Rathod@amdocs.com 13+ years Quality Assurance Integration Testing Manager Mitalesh.Agarwal@amdocs.com 15+ years building products using Java Lead Architect Savio.Dsouza@amdocs.com
  4. 4. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs4 About Amdocs Leader in customer experience software solutions Portfolio of software products Customer Management is a part of this portfolio which includes a number of products for CRM, Order management, self service, etc.
  5. 5. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs5 In the beginning The Chosen One(s)
  6. 6. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs6 Realization… Why so long?
  7. 7. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs7 Mission Impossible… It cannot be done! Company X, Y & Z have done it and are practicing it! Not in this company! Division A, B & C are already there! Not with our products! Really!?! Why?
  8. 8. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs8 It cannot be done! Company X, Y & Z have done it and are practicing it! Not in this company! Division A, B & C are already there! Not with our products! Really!?! Why? Mission Impossible…
  9. 9. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs9 The Goal Stage Prod-1 Prod-2 Prod-3 Prod-4 Prod-5 Consume from Nexus Standalone build, packing & Install MAT Publish to Nexus Product Demo Env (Test Data) Product EaaS Consume from Nexus Integrated build, packing & Install MAT Publish to Nexus Integrate Demo Env (Test Data) Integrate EaaS
  10. 10. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs10 Where we started from Stage Prod-1 Prod-2 Prod-3 Prod-4 Prod-5 Consume from Nexus Standalone build, packing & Install MAT Publish to Nexus Product Demo Env (Test Data) Product EaaS Consume from Nexus Integrated build, packing & Install MAT Publish to Nexus Integrate Demo Env (Test Data) Integrate EaaS
  11. 11. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs11 Baby steps EVOLUTION
  12. 12. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs12 Mission Impossible… Request Surprise GREED
  13. 13. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs13 Success brings its own issues… Impatience
  14. 14. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs14 The Future RoadMap
  15. 15. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs15 The Journey never ends… 1 Manual Automate environment creation per product Automate environment creation - integrate products CI as a Service Continuous Deployment and beyond?
  16. 16. Information Security Level 1 – Confidential © 2015 – Proprietary and Confidential Information of Amdocs16 Thank You Savio.Dsouza@amdocs.com Rushikumar.Rathod@amdocs.com Mitalesh.Agrawal@amdocs.com

Editor's Notes

  • In the CM division of Amdocs Product Management Group (PBG) the various products of Amdocs were ultimately integrated manually and required over 9 weeks before a release could be announced.

    WHY????

    Many of the products in the suite depended on individuals to help adopt and integrate the systems together after which a team of testers could certify the integrated suite before release.

    OMG!!!!

    The aim of the effort we are describing is the move from the manual integration process to a fully automated process for building, integrating and certifying the products bringing the time down to 30 minutes using better engineered CI and virtualization tools.

    During this journey we also reduced the time to create integrated environments from days to minutes and we INTRODUCE EAAS.

    During the session we will highlight the challenges, solutions and implementations put in place to achieve this along with the preview of what is planned next.

  • Today we will take you through a journey that started almost a year and a half ago and we are still on it.

    We’ll go from the first step which was to automate manual steps and we’ll look at the hurdles we faced and how we overcame them. (and let me tell you a secret it isn’t ROCKET SCIENCE but really simple logical steps!)

    We’ll look at Environment as a Service and finally I’ll share with you our (ever changing) destination and goals.

    BUT first, … a little about us…
  • Various roles from QA to Dev to Architect
    Various product domains
    Various product deployment types ,e.g., SaaS, installable, customize & install
    Lead Architect for the “Customer Management” suite of products in the Amdocs portfolio
    Largely agile SDLC

  • Amdocs is the market-leader in customer experience software solutions and services for the world’s largest communications, entertainment and media service providers
    Has been largely following a well defined structure a.k.a waterfall and has been slowly transforming to Agile division by division

    With that introduction…
  • Developer checks in
    Periodically a product build is run
    The chosen one (by expertise or involuntarily) would chant a few spells. Build the product
    Then hand it to another Magician to integrate with the suite

    Product and module teams
    Delivered “drops” “email” notified to Wizard or Witch!

    Integration testing team “magician”
    Check in drops into source control!
    Create Integrated builds
    Fixed issues and follow up with the developers to get “hot-fixes”
    Align products
    Oh yeah, they also tested the functionality … eventually

    Release time increased. Per drop 9 person-weeks, taking minimum of 6 weeks PER DROP!
    In a release we would need about 3 cycles for testing. 5 MONTHS regression cycles!

    Total time: 9 WEEKS effort!!!!!! 6 Weeks elapsed time!!!!
  • Leader: Can we bring down this time?
    Expert: No WAY!!!! It has always been done this way!

    Leader: We HAVE TO get this time down!
    Expert: It is IMPOSSIBLE!!!!

    WHY????
    The process of manually ensuring that all integration points line up

    Per drop
    Dependencies 3 weeks
    Bring up the environment 4 weeks
    Management time to coordinate 2 weeks
    Elapsed time: 6 weeks


    Dependencies includes:
    Sequence of product adoptions
    Foundations provided by other divisions, not in our control
    Prod 1 -> delivers on day 0
    Prod 2 -> would use this delivery after “stabilization”on day 3
    And so on…

    Only “the Chosen One” can do it!
    And he’s on vacation
    And he didn’t take his mobile!
  • Softening of belief. Bold, underlined, italic to Italic only 

    IMPOSSIBLE  Not us.
  • Loads of impediments, excuses to rationalize how IMPOSSIBLE it is!
    Talk about products age. 15 years, active customer base, acquired products, out-dated technologies, multiple incompatible technologies, etc., etc…

    But IMPOSSIBLE was not a word to accept!

    We were lucky. We had management support who believed this needed to be changed!
    They Invested in a PoC

    So we grabbed the opportunity and
  • The goal
    Each product to go through the stages we’ve outlined!
    Clearly visible to everyone where we want to reach

    Integration (with dependencies honored) to be automated product by product
  • And where we started from…

    So many Supermen/Popeye …. “Adoption Magicians”

    Automating is more than functional but rather the infrastructure point.
    Rolling out environments that are ready for use functionally
  • The only way to succeed!

    Automate the build process for each of the products
    Integrate a Few (2 to 3) products. This time we had “Chosen products”

    Not built using modern tools!
    No PROBLEMO. Let’s write shell scripts to automate the build and integration process

    The first step
    Shell scripts + Jenkins = Automate product builds!

    Have each product share a build in a common file store
    Eventually replaced with Nexus

    Use the last successful build not more than 3 days old

    Have each product consume from the common file store
    Eventually download dependencies from Nexus

    Successfully tested VM snapshot saved in the Vmware catalog
    Integrated environments available for testing within minutes! EaaS!
  • Infra I want a product environment
    5 minutes
    REALLY???
    Yeah! Here’s the link!

    Realization
    It works!!!!
    Its saves me so much time
    From 6 weeks (9W effort) to minutes! Do the math!

    Greed
    Why only individual products? Why only 2 or 3 integrated products?
    When can my product be delivered this fast?
    Release no longer “controlled” by delivering environments only by the test cycle!

    Release level savings from months to minutes (negligible)

    And then we could think of having combinations and permutations (versions, products) we only dreamed off
  • Everything NOW!

    Analyzed each product, shared the improvements and SHOWED the mind boggling benefits
    Now the only constraint was “Budget” but that started flowing too

    We presented our Goal to be achieved in a year! Still too long!

    Work backwards:
    Identify the following for each product and each integrated combination
    Product Name
    Blockers
    Needs
    Targeted Date
  • Identify what is missing?
    Live environments without losing data refreshed with every build

    P4v Streams
    Per developer, maybe?

    On demand CI for each stream
    Click to create a new CI flow
    CI as a Service!

    Have a working environment to which every new build is continuously deployed
    Preserves my (test) DATA?
    Upgrade scripts for DB structure changes

    Reduce the build cycles for each product…
    Developer check in to testing in integrated environment within 6 hours! Mission Impossible – 2!
  • In summary, we went from a manual process that took anywhere between 3 to 9 weeks
    To one that could create environments within 5 minutes using EaaS
    Both for single product lines as well as permutations of integrated products

    And we continue our journey with visions of CI as a service and Continuous Deployment where the latest product version is continuously upgraded and data is preserved across each continuous deploy.

×