Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Struggles and Challenges in STLC in Ques No.13

304 views

Published on

Presentation in Ques No.13

Published in: Engineering
  • Be the first to comment

Struggles and Challenges in STLC in Ques No.13

  1. 1. Struggles and challenges in STLC May 17, 2019 (令和元年) Leisure Product Dept. Rakuten, Inc.
  2. 2. 2 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  3. 3. 3 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  4. 4. 4 [Career Summary] 2004-2010 : Experienced 2 internet based companies Aug 2010 : Joined Rakuten (3rd company for me) 2010-2013 : Lead Developer for Ad-tech platform 2013-2016 : Manager for Social tool platform dev team 2017~ : Manager for SQA Group for leisure services Self-Introduction @freddiefujiwara
  5. 5. 5 Product Quality Assurance By Testing Reaching ZERO Customer Dissatisfaction Our Mission Best Experience Assurance By Data Science Our Team Reliability Assurance By Stress Test
  6. 6. 6 Product Quality Assurance By Testing Reaching ZERO Customer Dissatisfaction Our Mission Best Experience Assurance By Data Science Our Team Reliability Assurance By Stress Test Today’s scope
  7. 7. 7 Our Team
  8. 8. 8 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  9. 9. 9 Our Product *As of Feb 28 2018 *As of Feb 28 2018 *As of Feb 28 2018 *As of Feb 28 2018
  10. 10. 10 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  11. 11. T1(Unit Test)Implementation DevSpec T2(Integration Test) 1.Test Plan 2.Test Design 3.Test Implement ation 5.Test Execution 6.Release Judgement PRD Affected Area/ Test Environment Information System Test(QA) • Test Strategy • Test Scope • Estimation(Draft ) • RTM • HLS • Estimation(Fix) • Case • Data • Env Setup SQA Task Output For stakeholders Legend • Incident Report • Final Report Signoff for Reliability , Maintainability…. 4.Smoke Test Sharing • Smoke Test Cases Smoke Test Result Output w/o sharing Input from stakeholders STLC PRD(Requirement)
  12. 12. 12 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  13. 13. 13 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  14. 14. 1.Test Plan 2.Test Design 3.Test Implement ation 5.Test Execution 6.Release Judgement • Test Strategy • Test Scope • Estimation(Draft ) • RTM • HLS • Estimation(Fix) • Case • Data • Env Setup • Incident Report • Final Report 4.Smoke Test Sharing • Smoke Test Cases Struggle#1
  15. 15. • Test Strategy • Test Scope • Estimation(Draft ) • RTM • HLS • Estimation(Fix) • Case • Data • Env Setup • Incident Report • Final Report • Smoke Test Cases Struggle#1 1.Difficult to manage both of Requirement and Test cases 2.Difficult to utilize our activity. ----- ---- ----- ----- ----- ----- ---- ---- ---- ----- ---- ----- --------- ------ 1.Test Plan 2.Test Design 3.Test Implement ation 5.Test Execution 6.Release Judgement 4.Smoke Test Sharing RTM
  16. 16. • Test Strategy • Test Scope • Estimation(Draft ) • RTM • HLS • Estimation(Fix) • Case • Data • Env Setup • Incident Report • Final Report • Smoke Test Cases Struggle#2 1.Test Plan 2.Test Design 3.Test Implement ation 5.Test Execution 6.Release Judgement 4.Smoke Test Sharing
  17. 17. • Test Strategy • Test Scope • Estimation(Draft ) • RTM • HLS • Estimation(Fix) • Case • Data • Env Setup • Incident Report • Final Report • Smoke Test Cases Struggle#2 1.Test Plan 2.Test Design 3.Test Implement ation 5.Test Execution 6.Release Judgement 4.Smoke Test Sharing
  18. 18. • Test Strategy • Test Scope • Estimation(Draft ) • RTM • HLS • Estimation(Fix) • Case • Data • Env Setup • Incident Report • Final Report • Smoke Test Cases Struggle#2 1.Difficult to make it on time about automation task 1.Test Plan 2.Test Design 3.Test Implement ation 5.Test Execution 6.Release Judgement 4.Smoke Test Sharing
  19. 19. 19 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  20. 20. 20 Self-Introduction My Name is Sonal Singh ソナル シング I’m from India I have more than 10 years of experience as QA I have worked into Telecom, Investment Banking, Insurance, Re-insurance domains I was into mobile game testing for 3 years I’m ISTQB Foundation level certified I have worked into Service based as well as product based companies
  21. 21. 21 Have you ever experienced Production Issues? I never had 
  22. 22. 22 Now?
  23. 23. 23 And Production issue is always scary!
  24. 24. 24 Retrospective result showed some facts:  Many defects were missed because some specifications were missed in test design. Requirement Test Design Requirement A  Requirement B  Requirement C  Requirement D  Requirement E  Requirement F  Requirement G  Requirement H 
  25. 25. 25 Retrospective result showed some facts:  There were many specification sheets kept at different locations.  Multiple requirements were changed after Test design was finished. Req v1.0 Req v1.1 Req v1.2
  26. 26. 26 Major Problem What if requirement changes multiple times after test design is finished and test implementation is in process?
  27. 27. 27 Is there any solution? RTM YES 
  28. 28. 28 What is RTM 
  29. 29. 29 RTM – Requirement traceability Matrix  It a matrix which is used to ensure test coverage. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Coverage 1 Login Req#1 - Login & logout 1 Successful login. TC001- TC010  1 Incorrect credentials not able to login. TC011- TC015  2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025  3 RP form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045  <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085  4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. TC086- TC095  5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110  6 Cancel a Hall <Req#7 - <Example 3 Successful cancel a hall. TC111- 
  30. 30. 30 RTM – Requirement traceability Matrix  You can create it in Microsoft excel or any test management tool which support Requirements.
  31. 31. 31 RTM – Requirement traceability Matrix  Each requirement is linked with a high level scenarios(HLS). S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Coverage 1 Login Req#1 - Login & logout Successful login. Incorrect credentials not able to login. 2 New Member Req#2 - New Member creation New member registration successfully. 3 RP form <Req#3 -Example 3> Non member can submit RP form. <Req#4 - Example 4> Member can withdraw. 4 Create a Hall <Req#5 - <Example 5> Hall can be created successfully. 5 Book a Hall <Req#6 - <Example 6> Successful book a hall. 6 Cancel a Hall <Req#7 - <Example 7> Successful cancel a hall.
  32. 32. 32 RTM – Requirement traceability Matrix  Each HLS is mapped to the corresponding test cases. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Coverage 1 Login Req#1 - Login & logout 1 Successful login. TC001-TC010  1 Incorrect credentials not able to login. TC011-TC015  2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016-TC025  3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026-TC045  <Req#4 - Example 4> 2 Member can withdraw. TC046-TC085  4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. TC086-TC095  5 Book Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096-TC110  6 Cancel Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111-TC156 
  33. 33. 33  For test case if there is any defect then link that as well. S.No. Features/Module Requirements HLS – High level scenario Test case # Defect Id # Status 1 Login Req#1 - Login & logout Successful login. TC001-TC010 Defect#001-010 Fail Incorrect credentials not able to login. TC011-TC015 Defect#0011 Pass 2 New Member Req#2 - New Member creation New member registration successfully. TC016-TC025 Defect#0012 Pass 3 Form <Req#3 -Example 3> Non member can submit form. TC026-TC045 <Req#4 - Example 4> Member can withdraw. TC046-TC085 4 Create a Hall <Req#5 - <Example 5>Hall can be created successfully. TC086-TC095 5 Book a Hall <Req#6 - <Example 6>Successful book a hall. TC096-TC110 6 Cancel a Hall <Req#7 - <Example 7>Successful cancel a hall. TC111-TC156 RTM – Requirement traceability Matrix
  34. 34. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id # Status URL 1 Login Req#1 - Login & logout 1 Successful login. TC001- TC010 Defect#00 1 Fail 1 Incorrect credentials not able to login. TC011- TC015 Pass 2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025 Pass 3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045 Pass <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085 Pass 4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. TC086- TC095 Pass 5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110 Pass 6 Cancel a Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111- TC156 Pass To be provided by PDM Not Required RTM – Requirement traceability Matrix
  35. 35. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id # Status URL 1 Login Req#1 - Login & logout 1 Successful login. TC001- TC010 Defect#00 1 Fail 1 Incorrect credentials not able to login. TC011- TC015 Pass 2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025 Pass 3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045 Pass <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085 Pass 4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. TC086- TC095 Pass 5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110 Pass 6 Cancel a Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111- TC156 Pass To be updated by QA TeamRTM – Requirement traceability Matrix
  36. 36. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id # Status URL 1 Login Req#1 - Login & logout 1 Successful login. TC001- TC010 Defect#00 1 Fail 1 Incorrect credentials not able to login. TC011- TC015 Pass 2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025 Pass 3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045 Pass <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085 Pass 4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. TC086- TC095 Pass 5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110 Pass 6 Cancel a Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111- TC156 Pass Updated by offsiteUpdated by onsite Handling of RTM by Onsite & Offsite
  37. 37. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id # Status URL 1 Login Req#1 - Login & logout New Requirement 1 Successful login. TC001- TC010 Defect#00 1 Fail 1 Incorrect credentials not able to login. TC011- TC015 Pass 2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025 Pass 3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045 Pass <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085 Pass 4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. TC086- TC095 Pass 5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110 Pass 6 Cancel a Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111- TC156 Pass More Advantage of RTM If any requirement changes then its easier to find out the test cases id which needs modification.
  38. 38. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id # Status URL 1 Login Req#1 - Login & logout 1 Successful login. TC001- TC010 Defect#00 1 Fail 1 Incorrect credentials not able to login. TC011- TC015 Pass 2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025 Pass 3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045 Pass <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085 Pass 4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. 5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110 Pass 6 Cancel a Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111- TC156 Pass More Advantage of RTM It ensures coverage, so easy to find out if any requirement is without test cases.
  39. 39. S.No. Features/Module Requirements Priority HLS – High level scenario Test case # Defect Id # Status URL 1 Login Req#1 - Login & logout 1 Successful login. TC001- TC010 Defect#00 1 Fail 1 Incorrect credentials not able to login. TC011- TC015 Pass 2 New Member Req#2 - New Member creation 1 New member registration successfully. TC016- TC025 Pass 3 Form <Req#3 -Example 3> 2 Non member can submit form. TC026- TC045 Pass <Req#4 - Example 4> 2 Member can withdraw. TC046- TC085 Pass 4 Create a Hall <Req#5 - <Example 5> 1 Hall can be created successfully. 5 Book a Hall <Req#6 - <Example 6> 2 Successful book a hall. TC096- TC110 Pass 6 Cancel a Hall <Req#7 - <Example 7> 3 Successful cancel a hall. TC111- TC156 Pass More Advantage of RTM It shows which requirement had major no of defects.
  40. 40. 40 Suggestion
  41. 41. 41 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  42. 42. 42 Who am I Name : Sadaaki Emura (nickname M) Join in Rakuten : 2007 Career : Embedded engineer (2000-2002) Web engineer (2002-2015) Product manager (2015-2016) QA (2016~) Role: Test Automation Engineer Lead Birthplace : Kanazawa-city Hobby: jog , climbing , horse racing
  43. 43. 43 How do we use automation?
  44. 44. 44 How do we use automation testing for project Regression test • Current function • Affect • Not affect • New function
  45. 45. 1.Test Plan 2.Test Design 3.Test Implementatio n 5.Test Execution 6.Release Judgement • Incident Report • Final Report 4.Smoke Test Sharing should be all green used for release judgement How do we use automation testing for project STLC
  46. 46. 1.Test Plan 2.Test Design 3.Test Implementatio n 5.Test Execution 6.Release Judgement 4.Smoke Test Sharing Difficult to make it on time about automation task Struggle#2 (再掲)
  47. 47. 47 Reason for struggle 1. Web design is changeable include important functions 2. Coding is available after application is ready in test environment 3. Release cycle is short and automation engineer resource is less
  48. 48. 48 How do we handle?
  49. 49. 49 Manual & automation scope Manual test Automation test Current function New function △ ◎ ◎ △
  50. 50. 50 Manual & automation QA process Manual Team Automation Team Test Design Catch up automation scope execute manual test Test Implementation Modify regression script Test design for automation Script run by CI New functionCurrent function Create script Marge to regression script QA done Fixed by project done Use next project 1-2 weeks
  51. 51. 51 Project & Regression test timeline time project1 project2 project3 project4 Project1 current function Automation cover project Project1 new function Project2 current function Project2 new function Project3 current function Project3 new function Project4 current function
  52. 52. 52 Project scale ratio 62% 38% Small Project Big Project ※Small Project = Release within 2 weeks
  53. 53. 53 Automation test coverage 20% 16% 64% new function current function need modification current function ※ target : SMALL project ※ scope : function to be automated Focus on this
  54. 54. 54 Result 1. Changeable design can be support 2. Regression test is done on schedule 3. Green Jenkins when release
  55. 55. 55 1.About Our Team 2.About Our Product 3.STLC 4.RTM(Struggle& Challenge#1) 5.Flow(Struggle& Challenge#2) 6.Conclusion Agenda
  56. 56. 56 1.RTM(Struggle& Challenge#1) 2.Flow(Struggle& Challenge#2) Conclusion
  57. 57. 57 1.RTM(Struggle& Challenge#1) 2.Flow(Struggle& Challenge#2) Conclusion ----- ---- ----- ----- ----- ----- ---- ---- ---- ----- ---- ----- --------- ------ RTM
  58. 58. 58 Conclusion Manual Team Automation TeamTest Design Catch up automation scope execute manual test Test Implementation Modify regression script Test design for automation Script run by CI New functionCurrent function Create script Marge to regression script QA done Fixed by project done Use next project 1-2 weeks 1.RTM(Struggle& Challenge#1) 2.Flow(Struggle& Challenge#2)

×