Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Main Author -> Snehasish Roy:- Automation Manager (Enterprise Architect)
Co-Author 1 -> Rekha Shetty:- Test Manager
Co-Aut...
Abstract
As days progress, number of software services used by human are increasing at very Rapid dynamic pace. Also today...
10 Million 1,000 Billion
1 Million 1 Billion
100,000 10 Million
10,000 100,000
1,000 1,000
100 100
1955 1975 1995 2005 201...
Software QA/QE/Testing Industry Today’s State -> Problems
Waterfall Agile DevOps
Scope
Complexity
Time to
Market
Cost
Qual...
Software QA/QE/Testing Industry Today’s State -> Problems cntd
84%
16%
Testing Level
Manual Automation
84%
16%
All Testing...
Software QA/QE/Testing Industry Today’s State -> Problems cntd
Extensive Organized Automation Process is missing in QA/QE/...
Software QA/QE/Testing Industry need and Solution
BU -> Business Unit
V -> Vertical
P -> Project
BU1-
V1-P1
Asset1
BU1-
V1...
AIM -> CONCEPTUAL POD BUSINESS ARCHITECTURE
Organization (All Accounts) - AIM
Asset List
Account
Asset List
Account Level ...
Business Unit
Business Unit ->
Verticals
BU-> V -> P -> Asset Category
AIM -> 1. Governance Model -> Add/Change Asset
QA/Q...
Reporting
Intelligent Search Engine
AIM -> 2. Governance Model -> Search Engine and Implementation
QA/QE/Testing
Org Asset...
QA/QE/Testing
Org Asset List (AIM)
AIM -> Search Engine -> AI & ML Engine -> Technical Architecture
13
11
ML
Engine
Data
H...
QA/QE/Testing
Org Asset List (AIM)
AIM -> Reporting -> ML & Report Engine -> Technical Architecture
13
12
ML
Engine
Data
H...
AIM Assets classification
QTP Versions
UFT Versions
Ginger Versions
Selenium + TestNG
Cucumber BDD
Robot
Adobe Reporting
B...
QA/QE/Testing Organized Automation Process -> Value Propositions
Cost
Efficiency
Quality
Time to
Market
Flexibility
Scalab...
AIM -> Challenge-Solution-Value Model at a Glance
THE
SOLUTION
THE
CHALLENGE
97%
• QA/QE/Testing Org High
levels of reusab...
Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement
16
Development (D) Business Values
Total Au...
Case Study -> Automation Suite/Scripts -> ROI & KPI Measurement
17
Automation Suite/Script D-> ROI* & D-> KPI*:- Developme...
Case Study-> Automation Suite/Scripts -> E->KPI -> PROD Defects Reduced drastically
18
Note:- One Release = 2 months combi...
WQR -> Main challenges to achieve desired test automation -> Solution Mitigation
19
We don’t have the right Automation Too...
Bibliography I
1. https://www.wikipedia.org/
2. https://www.sogeti.com/explore/reports/world-quality-report-201819/
3. htt...
Bibliography II
21
18.https://www.businesswire.com/news/home/20180717005888/en/Global-Software-Testing-Services-
Market-20...
Author Biography
Snehasish Roy
Automation Manager
(Enterprise Architect)
1. 13+ years of Experience in Enterprise System A...
Question & Answers
23 Back to Story Appendix V
Thank You
24 Back to Story Appendix V
Appendix I - Paper Story Elaboration
Page 1:- Thought paper title. Author details.
Page 2:- Explains the Abstract of the p...
Appendix II - Paper Story Elaboration
Page 5:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued
Fi...
Appendix III - Paper Story Elaboration
Page 9:- Fig12:- Also a very critical point explained are the Compliant Filters to ...
Appendix IV - Paper Story Elaboration
Page 16:- Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measure...
Appendix V - Paper Story Elaboration
Page 19:- WQR -> Main challenges to achieve desired test automation -> Solution Mitig...
Upcoming SlideShare
Loading in …5
×

#ATAGTR2019 Presentation "AIML Driven extensive reusable Automation Asset Management Process (AIM) for "QA/QE/Testing Org"" By Snehasish Roy, Rekha Shetty and Umashankar Reddy

21 views

Published on

Snehasish Roy who is a Test Automation Manager & Enterprise Architect at Amdocs along with Rekha Shetty who is a Program Manager at Amdocs and Umashankar Reddy who is a Software Test Manager at Amdocs took a Session on "AIML Driven extensive reusable Automation Asset Management Process (AIM) for "QA/QE/Testing Org"" at Global Testing Retreat #ATAGTR2019

Please refer our following post for session details:

https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-snehasish-roy-as-our-esteemed-speaker/

https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-rekha-shetty-as-our-esteemed-speaker/

https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-umashankar-reddy-as-our-esteemed-speaker/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

#ATAGTR2019 Presentation "AIML Driven extensive reusable Automation Asset Management Process (AIM) for "QA/QE/Testing Org"" By Snehasish Roy, Rekha Shetty and Umashankar Reddy

  1. 1. Main Author -> Snehasish Roy:- Automation Manager (Enterprise Architect) Co-Author 1 -> Rekha Shetty:- Test Manager Co-Author 2 -> Umasankar Reddy:- Test Manager Amdocs 1 AIM -> Back to Story Appendix I AIML Driven extensive reusable Automation Asset Management Process (AIM) for “QA/QE/Testing Org”
  2. 2. Abstract As days progress, number of software services used by human are increasing at very Rapid dynamic pace. Also today’s Digital World is surrounded with many smart devices and complicated technologies. To cope up with, over the period of time the Software Development Life Cycle has been changed from Waterfall to Agile and DevOps now, where testing is always been the last quality gate in the cycle which has been drastically reduced with cost decline. The above points making QA/QE/Testing very challenging and complex to handle without Organized Lean Automation Testing & Quality Design Process. As per World Quality Report statistics -> Test Automation Level is below per low with several challenges been observed for both Automation and DevOps. Automation Testing mostly in silos and at project level, not leveraged across Testing Pyramid for the same project and Cross Projects within Organization. This Thought paper will cover the resolution of majority of the challenges faced today for achieving desired level of Automation and explain transformation need for QA/QE/Testing Org to have of An Extensive Reusable Organized Automation Process at QA/QE/Testing org level. QA/QE/Testing Org need:- In DevOps world Organized Intelligent Smart Automation with Increased coverage is needed for Continuous Integration and Continuous Deployment with shrinking timelines to have Quality Product delivery. These brings a driver for Quality Engineering leadership to transform existing “QA/QE/Testing Automation” process into “Automation Asset Management Framework” to overcome associated challenges in order to sustain and stay ahead with the industry. As an enabler for Quality Engineering leadership to break the silos, “AIM” is the QA/QE/Testing Organization level solution to leverage and utilize Testing Automation Assets across. “AIM” is the AIML Driven Centralized Governed Shared Cloud based Asset Management Framework at Organization Level for Business Units/Domains -> Verticals, to revolutionize and standardize the “QA/QE/Testing” Automation Process to meet the Future purpose for QA/QE/Testing organization for years to come. The Assets/Inventory can be shared within Inter Project Testing Pyramid and Cross Projects as well as per defined Compliant process. All the implemented Assets are checked-in/out to Automation Inventory Management(“AIM”) with code branch & version control mechanism. “AIM in a Nut-Shell” and Benefit Realization:-  Using “AIM” there were 93+ assets were implemented across 29+ projects.  Introduced new development ROI and KPI for code reuse:- ~50+ % of Development Time Saving.  Automation Execution ROI and KPI introduction:- ~70-80% Execution Coverage increase.  Benefit realization is in shorter time to market, quick and faster resolution, increased efficiency and reduced escaping production defects. Also Benefit realization is observed in shorter time to market, quick and faster resolution, increased efficiency and reduced escaping production defects. 2 Back to Story Appendix I
  3. 3. 10 Million 1,000 Billion 1 Million 1 Billion 100,000 10 Million 10,000 100,000 1,000 1,000 100 100 1955 1975 1995 2005 2015 2025 2040 Interoperability-NumberofDataSources ResponseTime-Seconds 1 1 Minutes Hours Seconds Months Weeks Days Software Industry Evaluation 1. Digitization Connects People, Devices and Businesses -> Exponential 6.08 6.85 7.58 8.24 8.85 0.36 1.96 5 10.9 17.15 0 5 30.73 125 200 0 500 0 20 2000 2010 2020 2030 2040 Year:- 2000 to 2040 Global Population Internet Connections Connected Devices InBillions Year:-2018 AIML Large Data IOT SOA Cloud 3. Technology Complexity ->2. Response Time Interoperable data process -> Exponential, Real Time Multiple OS Platform & Browsers Digital Fig 1 Fig 2 Fig 3 Fig 4 Year:-2018 75+ Billion Devices & Sensors by 2025 200 Billion by 2040 3 Back to Story Appendix I Analytics
  4. 4. Software QA/QE/Testing Industry Today’s State -> Problems Waterfall Agile DevOps Scope Complexity Time to Market Cost Quality Shift Left Right SDLC Methodology compare due to Industry change QA/QE/Testing Automation Today -> In Silos and Scattered Fig 5.1 Fig 6Market Size:- $12.85 Billion by 2025 DevOps is Future CAGR 18.6% Fig 5.2 4 Back to Story Appendix I
  5. 5. Software QA/QE/Testing Industry Today’s State -> Problems cntd 84% 16% Testing Level Manual Automation 84% 16% All Testing Life Cycle Activities Manual Automation Lies between 14% – 18 % Software Testing CAGR 13-15% by 2023 Size:- $35 Billion to $67 Billion by 2023 Testing CAGR & Market SizeWorld Quality Report Stats Automation Testing CAGR 18-20% by 2023 Size:- $9 Billion to $22 Billion By 2023 Automation Testing Market Size corresponding to Software Testing Market size increase -> 26% to 32% by 2023 Fig 7.1 Fig 8.1Fig 7.2 Fig 8.2 5 Automation Testing Stats is Below par Back to Story Appendix I Executed using Automation Tools:- • 16% overall • 15% end-to-end business scenarios
  6. 6. Software QA/QE/Testing Industry Today’s State -> Problems cntd Extensive Organized Automation Process is missing in QA/QE/Testing Org. Automation reusability is not leveraged within project and across projects to handle complete coverage within Org. Automation is done in Silos.1. 2. 3. QA/QE/Testing Org Today’s state -> Consolidated Problem reasons Duplication of Automation Inventory across organization:- In-efficient utilization of assets and existing Skilled resources. 4. Fig 9 6 Back to Story Appendix II Reference taken from:- Main challenges in achieving desired level of test automation:- • Statistics calculation done with interviewing CIOs/Executives and plotting % • 3, 4 stats started in 2018. 5, 6, 7 continued as of previous year. 1. Lack of skilled and experienced test automation resources 2. We don’t have the right Automation Tools 3. We have too many different automation tools 4. We started too late with testing and Test automation 5. We don’t have the right automation testing process/method 6. We find it difficult in automating because we use multiple development life cycle 7. Find it difficult to Integrating test automation into a DevOps process
  7. 7. Software QA/QE/Testing Industry need and Solution BU -> Business Unit V -> Vertical P -> Project BU1- V1-P1 Asset1 BU1- V1-P2 Asset2 BU2- V2-P2 Asset n BU2- V1-P1 Asset1 BU4- V1-P1 Asset1 BU3- V3-P3 Asset2 BU3- V3-P1 Asset8 BUn- Vn-Pn Asset n BU4- V2-P1 Asset2 BU5- V1-P4 Asset 4 BU8- V3-P5 Asset 2 AIM Solution:- Brain Sketch Legend:- “AIM” is the Centralized Governed Shared Cloud based Automation Asset Management Framework at Organization Level for Business Units/Domains. Automation Assets leveraged across Project Testing Pyramid and Across Projects within or across Business unit and Verticals. Due to Time Shrink and Cost decline Extensive Reusable Organized Automation Process is required at QA/QE/Testing org level. QA/QE/Testing Organization transformation need to overcome Problems:- QA/QE/Testing Organization transformation Solution:- Fig 10 7 Back to Story Appendix II
  8. 8. AIM -> CONCEPTUAL POD BUSINESS ARCHITECTURE Organization (All Accounts) - AIM Asset List Account Asset List Account Level Implementation Account Level AIM DashboardMeasurement Intelligent Search Engine BU->V->P-> Assets category BU –> V -> P -> Hierarchy Compliant Filter Profile Security Access Management Governance Tool/Utility Change Tool/Utility Add Capability Change Capability Add Automation Suite Change Automation Suite Add Framework Change Framework Add • AIM:- Automation Inventory Management • One Account can have multiple Projects BU -> Business Unit, V -> Vertical, P -> Project • D -> ROI* :- One time Development -> Return on Investment • D -> KPI*:- Development -> Key performance Indicator • E -> ROI:- Multiple Execution -> Return on Investment • E -> KPI:- Multiple Execution -> Key performance Indicator Cloud Infrastructure Reporting DeleteChangeAdd Asset Evaluation Legend:- Fig 11 Development/ Maintenance E -> KPIE -> ROID -> KPID -> ROIDashboardMeasurementReporting D-> KPID-> ROIE-> KPIE-> ROIExecution 8 Back to Story Appendix II
  9. 9. Business Unit Business Unit -> Verticals BU-> V -> P -> Asset Category AIM -> 1. Governance Model -> Add/Change Asset QA/QE/Testing Org Asset List (AIM) Add/Change Request Normal Change Risk Assessment Management Approval High Risk? CAB Approval Implement Add/Change Script Add/Change Standard Add/Change Access = Yes NO YES 1 2 3 4 5 6 7 8 8 8.1 8.2 8.3 9 9 Automation Framework Automation Suite/Scripts Automation Capability Automation Tool/Utility Compliant = Yes Verticals -> Projects Add/Change Fig 12 Compliant Filter 9 Back to Story Appendix II
  10. 10. Reporting Intelligent Search Engine AIM -> 2. Governance Model -> Search Engine and Implementation QA/QE/Testing Org Asset List (AIM -> Cloud) Search Engine AIML Engine -> Assets Search Result 1. NLP Prediction Model. 2. Pattern Search 3. Behavior Driven search 4. Update Engine for successful & other results Recursive -> Update AIML Engine Add/Change Asset Implement Existing Assets from AIM Follow 1. Governance Model -> Add/Change Asset Access = Yes Users 12 13 14 15 16 Search Asset 32 1 4 5 6 7 8 9 101112 13 14 15 16 Compliant Filter Business Unit Business Unit -> Verticals Verticals -> Projects Fig 13 Dashboard & Measurements ROI & KPI 10 Back to Story Appendix III
  11. 11. QA/QE/Testing Org Asset List (AIM) AIM -> Search Engine -> AI & ML Engine -> Technical Architecture 13 11 ML Engine Data Hub AI Engine Standard Change Normal Change Service Invoker/Bus 1. ML_Get_Config_Template 1.1. ML_Supervised_Aut_ Framework 1.2. ML_Supervised_Aut_Script_Suite 1.3. ML_Supervised_Aut_Capability 1.4. ML_Supervised_Aut_Tools_Utilities Data ingestion:- 1. ML_Unsupervised_Aut_ Framework 2. ML_Unsupervised_Aut_Script_Suite 3. ML_Unsupervised_Aut_Capability 4. ML_Unsupervised_Aut_Tools_Utilities Search 1. AI_Refer_Previous_Search_Result 1.2. AI_Predict_Return_Update_Search 1.2.3. AI_Return_Search_Aut_Framework 1.2.4. AI_Return_Search_Aut_Script_Suite 1.2.4.1 AI_Return_Search_Function_Activity_Method 1.2.5. AI_Return_Search_Aut_Capability 1.2.6. AI_Return_Search_Aut_Tools_Utilities 1.2.6. AI_Update_Search_Results Desktop & Mobile App Recursive Recursive • NLP techniques (Python/R/Matlab) • Exact Match and AIM Match • Probability & Prediction • N-gram • Bayes techniques • Other AI techniques and rules First time Supervised configuration Un-supervised learn Assets Add/Change/Delete BU –> V -> P -> AIMs AI Rules Engine Legend:- Major Services are mentionedFig 14 Back to Story Appendix III
  12. 12. QA/QE/Testing Org Asset List (AIM) AIM -> Reporting -> ML & Report Engine -> Technical Architecture 13 12 ML Engine Data Hub Report Engine Standard Change Service Invoker /Bus Data ingestion:- 1. ML_Unsupervised_Aut_ Framework 2. ML_Unsupervised_Aut_Script_Suite 3. ML_Unsupervised_Aut_Capability 4. ML_Unsupervised_Aut_Tools_Utilities Dashboard/ ROI/KPI 1.1 Report_Return_Development_ROI_Project 1.2 Report_Return_Development_KPI_Project 1.1.1 Report_Return_Development_ROI_BU_Account 1.2.1 Report_Return_Development_KPI_BU_Account 1.1.1.1 Report_Return_Development_ROI_Org 1.2.1.1 Report_Return_Development_KPI_Org 2.1 Report_Return_Execution_ROI_Project 2.2 Report_Return_Execution_KPI_Project 2.1.1 Report_Return_Execution_ROI_BU_Account 2.2.1 Report_Return_Execution_KPI_BU_Account 2.1.1.1 Report_Return_Execution_ROI_Org 2.2.1.1 Report_Return_Execution_KPI_Org 3. Report_Return_Dashboard_Usage Desktop & Mobile App Recursive Recursive • Development estimate calculation • Execution utilization calculation • Release wise Defects calculations • ROI & KPI Calculations Un-supervised learn Assets Implementation BU –> V -> P -> AIMs Report Analytics Engine Legend:- Major Services are mentionedFig 15 Back to Story Appendix III
  13. 13. AIM Assets classification QTP Versions UFT Versions Ginger Versions Selenium + TestNG Cucumber BDD Robot Adobe Reporting BI TOSCA Hoverfly (Virtualization) AppliTool Appium Katalon Sikuli Automation Framework Sanity:- Customer relationship management X,X+I,.. Customer service management Y, Y+J,.. Regression:- Customer relationship management X, X+I,.. Customer service management Y, Y+J,.. Order management system Z, Z+K,.. mS (micro Services) Data creation:- Order management system Z, Z+K,.. API Progression:- Customer relationship management X,X+I,.. Customer service management Y, Y+J,.. Order management system Z, Z+K,.. Check environment DB validation DB scan tool Auto DB fetch Auto file simulator Single console execution KIBANA -> Elastic search Idle time Usage tracking Disk clean up QC audit Monitor defects Toad network tool Automation Suite/Scripts Automation Capability Automation Tool and utilities Suites:- BU = Telecom -> Vertical = Client A -> Project1, Project 2,.., Project N. Script, Script-less, Intelligent Smart 13 Back to Story Appendix III Reuse within same Project Reuse with other Project Reuse across Projects Reuse across Projects Legend :-Reuse post 1st time development -> High (71-100%) Medium (31-70%) Low (1-30%) Note-> Only some of the assets Are mentioned as examples
  14. 14. QA/QE/Testing Organized Automation Process -> Value Propositions Cost Efficiency Quality Time to Market Flexibility Scalability Consolidation Innovations QA/QE/Testing Challenges AIM Project 1 Automation Project 2 Automation Project 3 Automation ROIs KPIs Continuous Improvement Automation Maturity Model Innovation Management Reuse Pool of automation experts Defined Governance processes Common Defined Infrastructure:- 1.Frameworks, 2.Capabilities, 3.Tools & Utilities, 4.Automation Suites Defined Metrics Established Processes Combined Solution Accelerations Fig 16 Back to Story Appendix III Automation Infrastructure Management at QA/QE/Testing org 14
  15. 15. AIM -> Challenge-Solution-Value Model at a Glance THE SOLUTION THE CHALLENGE 97% • QA/QE/Testing Org High levels of reusability of existing automation assets across • DevOps -> Development & testing working together for CICD with Asset leverage • QA/QE/Testing Org -> Non-Organized automation • DevOps -> Shorter Coverage at Project level for short cycle • Silo Automation • Shorter delivery cycles • Higher defect identification in pre-production • Reduction in defect leakage to Production • Remove Asset duplicity -> Waste removal • Increased Automation coverage at Project and Org • Asset Duplication • Effort Waste • Organized extensive automation at Project level 93+ Assets implement in 29+ Projects:- AIM:- THE BUSINESS VALUE 15 Back to Story Appendix III
  16. 16. Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement 16 Development (D) Business Values Total Auto Frameworks:- ~16+ (Left shows example of 3 frameworks for 3 BUs) 26 Months 12.20 Months With Reuse effort required Development effort required With Reuse ~13.80 Man Months Saved. D -> ROI & Savings:- D -> KPI - Reuse:- ~ 213% Coverage D -> KPI – Automation coverage Fig 17 Fig 19 Automation Framework, Capability, Tools D-> ROI* & KPI*:- Fig 18 3 26 Total Auto Capabilities:- ~9+ (Left shows example of 5 capabilities for 3 BUs) 16.50 Months 10.65 Months With Reuse effort required Development effort required With Reuse ~5.85 Man Months Saved. D -> ROI & Savings:- D -> KPI - Reuse:- ~ 155% Coverage D -> KPI – Automation coverage 5 15 Total Auto Tools/Utilities:- ~26+ (Left shows example of 19 Tools/Utilities for 3 BUs) 64.15 Months 17.20 Months With Reuse effort required Development effort required With Reuse ~46.95 Man Months Saved. D -> ROI & Savings:- D -> KPI - Reuse:- ~ 373% Coverage D -> KPI – Automation coverage 19 90 Back to Story Appendix III
  17. 17. Case Study -> Automation Suite/Scripts -> ROI & KPI Measurement 17 Automation Suite/Script D-> ROI* & D-> KPI*:- Development(D) Business Values 54% Overall Release over Release:- Developed = 46% Reused = 54% Total Auto Development of Flows/Scenarios/TCs:- ~47632 2415 Months 987 Months With Reuse effort required Development effort required With Reuse ~1428 Man Months Saved. D -> ROI & Savings:- D -> KPI - Reuse:- ~ 244% Coverage D -> KPI – Automation coverage Development Reuse 47632 25266 22% Overall Regression to Progression and vice versa:- Developed = 78% Reused = 22% Total Auto Execution of Flows/Scenarios/TCs:- ~188490 4222 Months Execution (E) Business Values 1681 Months Automation Execution time Manual Execution time With Automation ~2541 Man Months Saved. E -> ROI & Savings:- 20 Days 10 Days E -> KPI - Delivery Cycle:- ~ 251% Coverage E -> KPI – Execution coverage Automation Suite/Script E-> ROI & E-> KPI:- Fig 20 Fig 21 Back to Story Appendix IV BU2 start
  18. 18. Case Study-> Automation Suite/Scripts -> E->KPI -> PROD Defects Reduced drastically 18 Note:- One Release = 2 months combined data. R1 shows state before Inventory sharing. R5 & R6 shows post Inventory Sharing over period of time with Increased Coverage. There are transformation projects as well in this Statistics. Also the Release size varies based on Scope. Fig 22 Fig 23 Fig 24 Fig 25 Fig 26 Fig 27 Early Defect Identification rate increased significantly Back to Story Appendix IV
  19. 19. WQR -> Main challenges to achieve desired test automation -> Solution Mitigation 19 We don’t have the right Automation Tools. We started too late with testing and Test automation. 1 2 3 4 Lack of skilled and experienced test automation resources. We have too many different automation tools. We don’t have the right automation testing process/method.5 We find it difficult in automating because we use multiple development life cycle.6 Find it difficult to Integrating test automation into a DevOps process.7 Inventory reuse allows to free up existing skilled resources. Coverage increased. Automation Infrastructure management with history allows to choose the right Tools. For referring existing similar automation user is able to select existing utilized Tools. Early Automation with defined Automation Strategy along with leveraging assets. AIM is the required “QA/QE/Testing Organized Automation Process”. AIM supports all SDLC Cycles along with multiple life cycles. AIM allows to move Shift left and Right along with DevOps CICD pipeline. Main challenges as WQR Solution Mitigation Back to Story Appendix V
  20. 20. Bibliography I 1. https://www.wikipedia.org/ 2. https://www.sogeti.com/explore/reports/world-quality-report-201819/ 3. https://en.wikipedia.org/wiki/Global_Internet_usage 4. https://royal.pingdom.com/2010/10/22/incredible-growth-of-the-internet-since-2000/ 5. https://www.quantumrun.com/future-timeline/2020/future-timeline-subpost-technology 6. https://www.quantumrun.com/future-timeline/2030/future-timeline-subpost-technology 7. https://www.quantumrun.com/future-timeline/2040/future-timeline-subpost-technology 8. https://www.infoplease.com/world/population-statistics/total-population-world-decade-1950-2050 9. https://www.statista.com 10.https://www.statista.com/statistics/471264/iot-number-of-connected-devices-worldwide/ 11.https://www.forbes.com/sites/louiscolumbus/2016/11/27/roundup-of-internet-of-things-forecasts-and- market-estimates-2016/#6efe14d2292d 12.https://iot-analytics.com/state-of-the-iot-update-q1-q2-2018-number-of-iot-devices-now-7b/ 13.https://futurism.com/by-2020-there-will-be-4-devices-for-every-human-on-earth/ 14.http://singularity.com/charts/page67.html 15.http://singularity.com/charts/page70.html 16.https://www.grandviewresearch.com/press-release/global-development-to-operations-devops-market 17.https://www.marketscreener.com/CIGNITI-TECHNOLOGIES-LTD-18554003/news/Software-Testing-Market- to-Grow-at-a-CAGR-of-13-by-2023-According-to-New-Research-For-Key-players-26167366/ 20 Back to Story Appendix V
  21. 21. Bibliography II 21 18.https://www.businesswire.com/news/home/20180717005888/en/Global-Software-Testing-Services- Market-2018-2022-Increasing 19.https://www.prnewswire.com/news-releases/global-automation-testing-market-2018-2023-expected-to- grow-at-a-cagr-of-17-7-during-the-forecast-period-to-reach-19-27-billion-300652566.html 20.https://www.businesswire.com/news/home/20180610005033/en/Global-Automation-as-a-service- Market-Post-20-CAGR-2018-2022 21.https://globenewswire.com/news-release/2018/05/11/1500942/0/en/Global-19-Billion-Automation- Testing-Market-2018-2023-Increasing-Adoption-of-the-Devops-Methodology.html Back to Story Appendix V
  22. 22. Author Biography Snehasish Roy Automation Manager (Enterprise Architect) 1. 13+ years of Experience in Enterprise System Architecture, Design, Development, Testing & Production Support. 2. Currently working in Amdocs. Carry prior experience from Syntel. 3. Worked in BNFS, Telecom, Insurance and other domains. 4. Automation Manager & Enterprise Architect. 5. Education:- BE in Electronics & Communication from Visvesvaraya Technological University. Certifications:- TOGAF 9.1, PRINCE2. 6. Currently Responsible for NAM, ATT and Other regional account’s Automation. 1. 14+ experience in Testing of different domains. 2. Currently working as S/W Test Manager in Amdocs. 3. Education: B.E in Computer Science from Karnataka University Certifications: CP-DOF. 4. Responsible for NAM region account E2E testing deliverables. Rekha Shetty S/W Test Manager 1. 14+ years of experience in Testing of different domains. 2. People management and End to End project governance of DevOps projects. 3. Led several testing programs with the teams spread across multiple global delivery locations. 4. Supports both Transformation + BAU projects. 5. Education: B.Tech in Electronics and Communication from Jawaharlal Nehru Technological University. Certification: DevOps certification. 6. Responsible for NAM and Canada regional accounts testing. Umasankar Reddy S/W Test Manager 22 Back to Story Appendix V
  23. 23. Question & Answers 23 Back to Story Appendix V
  24. 24. Thank You 24 Back to Story Appendix V
  25. 25. Appendix I - Paper Story Elaboration Page 1:- Thought paper title. Author details. Page 2:- Explains the Abstract of the paper. QA/QE/Testing Organization high level -> problem statements, solution and benefits. Page 3:- Software industry evaluation. Fig1:- The Trend of Global population, Internet connections, Connected device plot. Fig2:- How the World will look alike with Connected Devices and sensors with the predicted numbers in future. Fig3:- The Trend of Exponential increase of Interoperability of the Data with Exponential decrease in processing time due to continuous software industry change. (Moore’s law) Fig4:- Due to the changes seen in Fig1, Fig2, Fig3 shows Software Technology evolution towards complexity. Page 4:- Software QA/QE/Testing Industry Today’s Problems -> Automation Fig5.1:- Shows due to the Software industry change, the SDLC Methodology change and comparison of SDLC Models. With DevOps Scope & Complexity increased while Time to market and Cost decreased however the Quality to be sustained at High. Fig5.2:- Market study shows that DevOps is the Future based on CAGR & market size and currently under Adoption. Fig6:- Shows QA/QE/Testing Automation Today’s state where Automation is performed in Silos and scattered even within same project. Page 5:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued Fig7.1:- World Quality Report (WQR) was created by interviewing 1700 executives from 32 countries and benchmark for QA/QE/Testing industry. Based on WQR shows the Automation Level %. Fig7.2:- Based on WQR shows All Testing Activities Automation %. 25 Back to Story Appendix V
  26. 26. Appendix II - Paper Story Elaboration Page 5:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued Fig8.1:- Market study for Total Software Testing Market size and CAGR increase. Fig8.2:- Market study for Software Automation Testing Market size and CAGR increase. After comparing both markets, Automation Testing Market size increases due to bigger CARG rate shows that future of QA/QE/Testing work without extensive Automation is not sustainable. Page 6:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued Fig9:- As per WQR explains further challenges in achieving desired level of automation. Finally QA/QE/Testing organization Consolidated problem reasons are thought, consolidated and articulated. Page 7:- Software QA/QE/Testing Industry need and solution based on articulated problem reasoning. Explanation of QA/QE/Testing Org Transformation is explained. Solution “AIM” is explained. Fig10:- Solution Brain Sketch is designed along with Correct Organization Hierarchy required in QA/QE/Testing Org for Assets/Inventory Reuse. QA/QE/Testing Org hierarchy -> Business Unit (BU) -> Vertical (V) -> Project (P). Business Units will be BNFS, Telecom, Insurance, Healthcare, Retail etc on the same business domain. Verticals will be defined by the same Line of business (LOBs) for same or different customers without business conflict of interest according to Compliant regulations of Clients and Vendor providing QA/QE/Testing Service. Projects will be under the same LOBs of the verticals. Page 8:- AIM -> CONCEPTUAL POD ARCHITECTURE defining the transformed Reusable Organized Automation Process. Fig11:- Detailed Component Level Architecture of AIM along with 4 types of assets/inventory classification. Page 9:- AIM -> Further detailing the POD from page 8 to show -> 1. Governance Model -> Add/Change Asset. Fig12:- How the Governance model works for Addition or changing Assets/Automation Inventory ? 26 Back to Story Appendix V
  27. 27. Appendix III - Paper Story Elaboration Page 9:- Fig12:- Also a very critical point explained are the Compliant Filters to be Compliant as per Client Non- Disclosures, Copy rights and Vendor Compliant process applicable for Products and Intellectual Properties (IP). Also explains the Management Approval, Risk Management and Change Advisory Board (CAB) Approval process for AIM. Page 10:- Continue detailing the POD from page 8 to show AIM -> 2. Governance Model -> Search Engine and Implementation process capitalizing and reusing existing assets. Fig13:- How the Governance model works for Asset/Inventory search and implementation model with Compliant process in place ? Explained the Search engine and Artificial Intelligence Machine Learning (AIML) engine interactions at high level. Reporting, Dashboard, Measurements, ROI & KPI model iterations are depicted. Page 11:- Fig14:- AIM -> Search Engine -> AI & ML Engine -> Technical Architecture along with core services. Page 12:- Fig15:- AIM -> Reporting -> ML & Report Engine -> Technical Architecture along with core services. Page 13:- AIM Assets categorization and some examples provided under each category. Page 14:- QA/QE/Testing Organized Automation Process -> Value Propositions. Fig16:- QA/QE/Testing Organized Automation Process brings the value propositions and balance to overcome Today’s QA/QE/Testing challenges. Page 15:- AIM -> Challenge-Solution-Value Model at a Glance. Page 16:- Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement. This paper introduces new ROI & KPI parameters and calculation factors for Development. * refers new. 27 Back to Story Appendix V
  28. 28. Appendix IV - Paper Story Elaboration Page 16:- Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement All the ROI & KPI calculations are done after considering few BUs under Org, few projects under each BU and few assets. It is observed that actual QA/QE/Testing & Org level ROI & KPI factors increases even significantly as the reusability increases much more while accounting more implementations. D -> ROI* & D -> KPI* follows in correspondence to below for multiple projects under 3 BUs. Fig17:- Same 3 Automation framework implementation -> Coverage & Savings. Fig18:- Same 5 Automation capabilities implementation -> Coverage & Savings. Fig19:- Same 19 Automation tools/utilities implementation -> Coverage & Savings. Page 17:- Case Study -> Automation Suite/Scripts -> ROI & KPI Measurement. (Development & Execution) D -> ROI* & D -> KPI* follows in correspondence to below for multiple projects under 2 Bus for past 1 year stats. Fig20:- Automation Suite/Script Development ROI & KPI-> Reuse (%), Coverage (%) & Savings. Fig21:- Automation Suite/Script Execution ROI & KPI-> Delivery Cycle shrink for DevOps, Coverage(%) & Savings. Page 18:- Case Study -> Automation Suite/Scripts -> E-> KPI -> PROD Defects Reduced drastically Pre Production release wise defects:- Fig22:- Project Wise -> For 6 Projects under BU3 over 5 releases. Fig23:- Vertical Wise -> For 2 Verticals/Accounts under BU3 over 5 releases. Fig24:- Business unit and Organization wise -> For 2 BUs under Org and at Org over 6 releases. Post Production release wise defects (Reduction observed for increased coverage):- Fig25:- Project Wise -> For 6 Projects under BU3 over 5 releases. Fig26:- Vertical Wise -> For 2 Verticals/Accounts under BU3 over 5 releases. Fig27:- Business unit and Organization wise -> For 2 BUs under Org and at Org over 6 releases. 28 Back to Story Appendix V
  29. 29. Appendix V - Paper Story Elaboration Page 19:- WQR -> Main challenges to achieve desired test automation -> Solution Mitigation explanation. Page 20:- Research references in Bibliography I. Page 21:- Research references continued in Bibliography II. Page 22:- Author Biography details. Page 23:- Questions and Answers ? Page 24:- Thank You. Page 25:- Appendix I - Paper Story Elaboration. Page 26:- Appendix II - Paper Story Elaboration. Page 27:- Appendix III - Paper Story Elaboration. Page 28:- Appendix IV - Paper Story Elaboration. Page 29:- Appendix V - Paper Story Elaboration. Note:- All the details shared as per being compliant to Amdocs process. BU, Vertical, Automation Framework, Capability, Tool & Utility details are mentioned generically as per protocols. 29 Back to Story Appendix V

×