Agile process improvements with Quality Control circle
(QCC) and DevOps Automations.
(Quality Control circle (QCC) と DevOps Automationsにより
アジャイルプロセス改善)
Apr 16th, 2021
Mandloi Hitesh
Rakuten Group, Inc.
DevOpsDays Tokyo 2021
3
Section 1
• Self Introduction.
Section 2
• What is QCC?
• QCC approaches.
• Case study with QCC.
• Problems, analysis, actions & results.
Section 3
• Automations.
• CI-CD.
Agenda
4
Hello!
I’m Hitesh Mandloi.
• I am from India.
• SRE engineer in CCBD department.
• ~ 4.5 years in Japan.
• ~ 6Years in IT with multiple roles (Full stack engineer, SRE
engineer with DevOps and best Agile practices).
• Love to check & study new tools and technologies.
About Me
Twitter: @mandloihitesh
Email: mandloihitesh@yahoo.in
LinkedIn: https://www.linkedin.com/in/hitesh-mandloi-b4535889/
5
Before moving forward….
Common Issues in aTeam
Information
loss in geo-
located team
Inefficient
team
collaboration
Superficial
effort and
task
estimation
Sprint
spillovers
Silent Sprint
planning and
Grooming
meetings
Time
consuming
Test cycle.
Manual
deployment
6
QCC
What is QCC?
7
QCC (Quality Control Circle)
A Kaizen activity
Identify problems
& solutions
Quality Oriented
Product.
Rakuten’s Group-
Group-Wide
Initiative
8
QCC Objective and Target
Objectives
Improve
Productivity
Improve
Problem
Solving Skills.
Targets
Reduce Cost
and Effort.
Foster Kaizen
(Improvement)
Culture.
Frequency
1 ~ 3
Hours/Week
9
QCC steps
Objective
Targets
Actions / Kaizen
Problems
NewTask /
Activity
10
QCC approaches in Rakuten
Problem Solving procedure
Task-Achieving procedure
11
QCC Problem Solving Procedure
Problems
Solutions
Further Kaizen & Standardization (Shikumika)
Validate results
ImplementAction
Identify & CreateAction Plan
Analyze &Verify Root Causes
SetTarget
Analyze Current Situation
Decide the Problem
12
QCC Task-Achieving Procedure
Task Objective
ExpectedTargets
Further Kaizen & Standardization (Shikumika)
Validate Results
Implement the Optimal Action
Follow Optimal Action
Identify & CreateAction Plan
SetTarget
Clarify Issues
Decide the Objective
13
QCC case study
14
Example of QCC Case in Rakuten group
Objective
Increase participation on scrum meetings.
Target
Estimate >=25% of stories per grooming
meeting.
Voluntarily pick stories during planning
meetings.
Provide >=1 feedback per restrospective
meeting.
15
Root Cause Analysis (Fishbone Analysis)
16
Grooming meeting Planning meeting Retrospective meeting
Poker cards for pointing stories Identify similar stories
Retrospective page should be ready and cleanup by day 1 of
day 1 of sprint
Do polls/discussions on points
Define how to tackle dependency between UI and
and API
Do separate sprint retrospective
(when multiple sprints run in parallel)
Decouple story presentation and pointing Define a plan to obtain a healthy backlog
Prepare action points with PIC and deadline as tickets in
in JIRA
Set up preparation time before meetings
Capacity per developer per sprint to be calculated
calculated beforehand
Include stories retrospective
(challenging, complex, overestimate ...)
Keep meeting minutes Revisit current end-of-sprint schedule
“Fun Retro”
(theme-based, voting action points …)
Update information on stories during discussion
discussion
Include aknowledgement and recognition section
Define a common template for story creation
Countermeasures/Actions
17
Validating Results
Estimate >=25% of stories per
grooming meeting.
Sprints Target Actual
Sprint01 10% 8%
Sprint02 15% 12%
Sprint03 25% 20%
Sprint04 25% 25%
Voluntarily pick stories during planning
planning meetings.
Sprints Target Actual
Sprint01 8SP 5SP
Sprint02 8SP 8SP
Sprint03 10SP 8SP
Sprint04 13SP 10SP
Provide >=1 feedback per
restrospective meeting
Sprints Target Actual
Sprint01 1/developer 1/3 developer
Sprint02 1/developer 1/ 2developer
Sprint03 2/developer 1/1 developer
Sprint04 3/developer 2/1 developer
18
Next steps
Grooming meeting Planning meeting Retrospective meeting
Try new tools / poker tools Automate template generation in Jira Increase more participants
Avoid constant screen share switch
Prepare and submit (if applicable) proposal for studied
studied Jira plugins acquisition
Identify new ways to get healthy feedbacks
Sometimes new voting after point discussion is
is needed
Update information on the spot
19
Shikumika
/
Automations
20
Automation test execution - Pipeline
Batch One click Deployment
CI-CD using pipelines-as-a-code
Server ENV One Click sync automation
DB scriptsOne Click migration
Further Kaizen - Shikumika / Systemization / Automations
Automations identified usingQCC Cycle
And many more …
21
Automation test
execution - Pipeline
22
Automation Test Execution - Targets
Merge & Deployment
Sprint
Release
• After each branch merge and
and deployment.
• Execution after each Sprint
Sprint completion
• Execution after each release
release completion
23
Automation Test Execution - Pipeline
24
CI-CD
using
pipelines-as-a-code
25
CI-CD using pipelines-as-a-code
Strategy
Fail Quick Recover Fast
Repositories, Branches,Tagging naming structure
Branches creation / merge strategies
Project release versioning
Identify trigger point for each action (commit, push,
tagging, merge etc)
Static CodeAnalysis
Methodology
Poll SCM
WebhookTrigger
26
Strategy (Repositories, Branches, Tagging naming structure)
RBAC
Pattern based
branches
Tagging
pattern
27
Strategy - Static Code Analysis (SonarQube)
Monitor code-
coverage reports
Duplication /
Security checks
Code smells
28
CI-CD using pipeline-as-a-code
Groovy
based
library
Groovy support in
IntelliJ.
Jenkinsfile
validation with
Jenkins Linter.
Helper
Plugins
Jenkins Pipeline
plugins
Generic webhook
trigger plugin
Remote Jenkinsfile
provider plugin
Email extension
plugin
29
CI-CD Flow
30
Which processes should be automated?
Analyze
Plan &
set Goal
Automate
31
References
Jenkins https://www.jenkins.io/
Docker https://www.docker.com/
Terraform https://www.terraform.io/
Kubernetes https://kubernetes.io/
MicrosoftTeams https://www.microsoft.com/en-ww/microsoft-teams/group-chat-software
Bitbucket https://bitbucket.org/product
Jfrog artifactory https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifactory
32
Thank you
Please join us.
https://japan-job-jp.rakuten.careers/search-jobs

[Hitesh mandloi] DevOpsDays Tokyo 2021 presentation

  • 1.
    Agile process improvementswith Quality Control circle (QCC) and DevOps Automations. (Quality Control circle (QCC) と DevOps Automationsにより アジャイルプロセス改善) Apr 16th, 2021 Mandloi Hitesh Rakuten Group, Inc. DevOpsDays Tokyo 2021
  • 2.
    3 Section 1 • SelfIntroduction. Section 2 • What is QCC? • QCC approaches. • Case study with QCC. • Problems, analysis, actions & results. Section 3 • Automations. • CI-CD. Agenda
  • 3.
    4 Hello! I’m Hitesh Mandloi. •I am from India. • SRE engineer in CCBD department. • ~ 4.5 years in Japan. • ~ 6Years in IT with multiple roles (Full stack engineer, SRE engineer with DevOps and best Agile practices). • Love to check & study new tools and technologies. About Me Twitter: @mandloihitesh Email: mandloihitesh@yahoo.in LinkedIn: https://www.linkedin.com/in/hitesh-mandloi-b4535889/
  • 4.
    5 Before moving forward…. CommonIssues in aTeam Information loss in geo- located team Inefficient team collaboration Superficial effort and task estimation Sprint spillovers Silent Sprint planning and Grooming meetings Time consuming Test cycle. Manual deployment
  • 5.
  • 6.
    7 QCC (Quality ControlCircle) A Kaizen activity Identify problems & solutions Quality Oriented Product. Rakuten’s Group- Group-Wide Initiative
  • 7.
    8 QCC Objective andTarget Objectives Improve Productivity Improve Problem Solving Skills. Targets Reduce Cost and Effort. Foster Kaizen (Improvement) Culture. Frequency 1 ~ 3 Hours/Week
  • 8.
    9 QCC steps Objective Targets Actions /Kaizen Problems NewTask / Activity
  • 9.
    10 QCC approaches inRakuten Problem Solving procedure Task-Achieving procedure
  • 10.
    11 QCC Problem SolvingProcedure Problems Solutions Further Kaizen & Standardization (Shikumika) Validate results ImplementAction Identify & CreateAction Plan Analyze &Verify Root Causes SetTarget Analyze Current Situation Decide the Problem
  • 11.
    12 QCC Task-Achieving Procedure TaskObjective ExpectedTargets Further Kaizen & Standardization (Shikumika) Validate Results Implement the Optimal Action Follow Optimal Action Identify & CreateAction Plan SetTarget Clarify Issues Decide the Objective
  • 12.
  • 13.
    14 Example of QCCCase in Rakuten group Objective Increase participation on scrum meetings. Target Estimate >=25% of stories per grooming meeting. Voluntarily pick stories during planning meetings. Provide >=1 feedback per restrospective meeting.
  • 14.
    15 Root Cause Analysis(Fishbone Analysis)
  • 15.
    16 Grooming meeting Planningmeeting Retrospective meeting Poker cards for pointing stories Identify similar stories Retrospective page should be ready and cleanup by day 1 of day 1 of sprint Do polls/discussions on points Define how to tackle dependency between UI and and API Do separate sprint retrospective (when multiple sprints run in parallel) Decouple story presentation and pointing Define a plan to obtain a healthy backlog Prepare action points with PIC and deadline as tickets in in JIRA Set up preparation time before meetings Capacity per developer per sprint to be calculated calculated beforehand Include stories retrospective (challenging, complex, overestimate ...) Keep meeting minutes Revisit current end-of-sprint schedule “Fun Retro” (theme-based, voting action points …) Update information on stories during discussion discussion Include aknowledgement and recognition section Define a common template for story creation Countermeasures/Actions
  • 16.
    17 Validating Results Estimate >=25%of stories per grooming meeting. Sprints Target Actual Sprint01 10% 8% Sprint02 15% 12% Sprint03 25% 20% Sprint04 25% 25% Voluntarily pick stories during planning planning meetings. Sprints Target Actual Sprint01 8SP 5SP Sprint02 8SP 8SP Sprint03 10SP 8SP Sprint04 13SP 10SP Provide >=1 feedback per restrospective meeting Sprints Target Actual Sprint01 1/developer 1/3 developer Sprint02 1/developer 1/ 2developer Sprint03 2/developer 1/1 developer Sprint04 3/developer 2/1 developer
  • 17.
    18 Next steps Grooming meetingPlanning meeting Retrospective meeting Try new tools / poker tools Automate template generation in Jira Increase more participants Avoid constant screen share switch Prepare and submit (if applicable) proposal for studied studied Jira plugins acquisition Identify new ways to get healthy feedbacks Sometimes new voting after point discussion is is needed Update information on the spot
  • 18.
  • 19.
    20 Automation test execution- Pipeline Batch One click Deployment CI-CD using pipelines-as-a-code Server ENV One Click sync automation DB scriptsOne Click migration Further Kaizen - Shikumika / Systemization / Automations Automations identified usingQCC Cycle And many more …
  • 20.
  • 21.
    22 Automation Test Execution- Targets Merge & Deployment Sprint Release • After each branch merge and and deployment. • Execution after each Sprint Sprint completion • Execution after each release release completion
  • 22.
  • 23.
  • 24.
    25 CI-CD using pipelines-as-a-code Strategy FailQuick Recover Fast Repositories, Branches,Tagging naming structure Branches creation / merge strategies Project release versioning Identify trigger point for each action (commit, push, tagging, merge etc) Static CodeAnalysis Methodology Poll SCM WebhookTrigger
  • 25.
    26 Strategy (Repositories, Branches,Tagging naming structure) RBAC Pattern based branches Tagging pattern
  • 26.
    27 Strategy - StaticCode Analysis (SonarQube) Monitor code- coverage reports Duplication / Security checks Code smells
  • 27.
    28 CI-CD using pipeline-as-a-code Groovy based library Groovysupport in IntelliJ. Jenkinsfile validation with Jenkins Linter. Helper Plugins Jenkins Pipeline plugins Generic webhook trigger plugin Remote Jenkinsfile provider plugin Email extension plugin
  • 28.
  • 29.
    30 Which processes shouldbe automated? Analyze Plan & set Goal Automate
  • 30.
    31 References Jenkins https://www.jenkins.io/ Docker https://www.docker.com/ Terraformhttps://www.terraform.io/ Kubernetes https://kubernetes.io/ MicrosoftTeams https://www.microsoft.com/en-ww/microsoft-teams/group-chat-software Bitbucket https://bitbucket.org/product Jfrog artifactory https://www.jfrog.com/confluence/display/JFROG/JFrog+Artifactory
  • 31.
    32 Thank you Please joinus. https://japan-job-jp.rakuten.careers/search-jobs

Editor's Notes

  • #3 Please join us for awesome devops journey, get experience in kaizen culture.
  • #4 Here, I will talk about myself, QCC , it’s approaches & one of the simple case study at Rakuten. Later little bit of automations and how QCC is connected with automations.
  • #6 Have you faced issues / problems like below? Being part of global team, not getting time to share information to all members? Members are not taking part actively in story planning, grooming? Not able to complete stories in a sprint? Having problems with Time / Task management? Regression testing takes so much time and manual work? Not confident to do operations / trouble recovery in production? As a developer, not sure how to deploy on Staging to test story? Deployment is taking time?
  • #8 Based on PDCA Run PDCA cycle several times to figure out objective / scenario and deliver the results. A Kaizen activity led by small team/group of Employees. QCC is started as Group-wide Initiative inside Rakuten which focuses on Quality Oriented Company
  • #9 Main aim is to improve productivity and quality of work. Objective – Theme of this initiative (High level) Target – actions & scope Period – depends 3-4 months
  • #11  Now, As we got to know, QCC is an Activity that focuses on Better quality More productivity Solving problems New Learnings Improvements. So, Based on this, two approaches / processes are =>
  • #12 Divided in 8 sections but focuses on 4 phases. These 8 steps are Often called QC story Plan & Track Analyze Implement Evaluate
  • #16 Can use any analysis tool For e.g. Histogram to visualize data variations Cause and effect (Fishbone) to focus on root cause of problem Pareto diagram to identify crucial problem Tree or matrix diagram
  • #21 To reduce manual work and mistakes and save time, We identified several necessary automations in our QCC approach.
  • #23 One of the important reason to include regression test cases in CICD is to check integration issues within Microservices.
  • #26 We have implemented CICD considering specific git management process - "gitflow"
  • #30 Yesterday’s session from Ranking team Roll your product with Kaizen Culture
  • #31 Here, you can choose automation proceeses like deployment, testing But you need to be aware and check what kind of automation need careful analysis and discussion.