Agile Gurugram 2016 | Conference | Continuous Agile Testing @ Naukri | Meetu Arora

www.agilegurgaon.com
Meetu Arora
Sr V.P. Quality Assurance
Naukri.com

Testing Team Journey @Naukri.com
2006
2012
2007
2008
2010
2015
Automation
WATIR
Agile testing team
Automation
Selenium
Entire team: Manual &
Automation (Selenium)
Automation
QTP
Inception of
Testing Team

• Identify Need
• Identify Metrics/What
• Identify Path/How
• Prerequisites
• Implementation
• Our Measurements
• Results

Identify Metrics/What

Identify Path/How

Prerequisites
• Testing Team capable of doing automation
• Test case consolidation & management
• Dedicated scrum teams v/s shared resources
• Metric Baselines
Post live defect seepage
Test case coverage
Automation coverage – UT, IT, FT
Automation test flakiness
Automation Execution time
Velocity/Delivery
Planning Efficiency
Build Quality

Implementation
Implementation

Transition to Automation
35 Manual testers & 3 Automation testers
TO
38 Manual + Automation = POWER testers

Transition to Automation - Challenges
• Skill mismatch
• Team dynamics
• Lack of inclination to move towards automation
• High investment in terms of time and effort

Transition to Automation - Path
• Perseverance
• Don’t fall back
• Very Small Steps
• Low hanging fruit first - ROI
• Tester empowerment through automation
• Focus on frameworks
• We created Selenium POM code generator, which has been open
sourced: https://github.com/naukri-
engineering/SeleniumCodeGenerator
• TestNG and XSLT for reporting
• Generic Function library
• Coding Guidelines and Code Review Process
• Contests for motivation
• Provide migration channels

Test Case Consolidation and Management
To know more about how to integrate automation scripts with Jenkins visit our blog:
http://engineering.naukri.com/2015/05/integrate-your-automation-with-jenkins/

Testing Throughout
• Testing individual stories and Integration testing as
you go along
• Progressive/Parallel automation testing
• Minimizing Automated Tests Flakiness
• Reducing our automated tests execution time
• Run automated regression suites periodically using Jenkins
To know more about Continuous Testing @ Naukri read our blog:
http://engineering.naukri.com/2016/03/continuous-testing-naukri/

Automation Coverage
596 554 611 876 929 1228 1354 1508 1859 2249 2636 3047
7746 8153 8318 8679 8925 9313 10062 10283 10491 10846 10542 10641
68.76%
74.32%
75.67%
81.56% 84.55% 86.29% 90.82%
92.10%
92.72%
95.96%
96.65%
96.99%
0%
20%
40%
60%
80%
100%
0
2000
4000
6000
8000
10000
12000
14000
16000
April May June July August Septemebr October November December January February March
AutomationCoverage
No.ofTestcases
Naukri India
Not Automatable Automatable %Automation Coverage out of Automatable
124 309 347 317 336 339 422 417 402 410 415 417
1818
3072 3077
3859
4436 4576
5145 5658 5784 5861 5885 6242
25.00%
27.80%
29.29%
46.30%
58.00%
71.50%
76.80%
84.60% 88.10% 92.90% 93.00% 93.20%
0%
20%
40%
60%
80%
100%
0
1000
2000
3000
4000
5000
6000
7000
April May June July August Septemebr October November December January February March
AutomationCoverage
No.ofTestcases
Naukri Gulf
Not Automatable Automatable %Automation Coverage out of Automatable

Automation Flakiness Reduction
• 15% to 5% in Naukri
• 45% to 7% in Mobile Apps
• 17% to 7% in NaukriGulf
To know more about how we were able to optimize our tests and reduce
flakiness visit our blog: http://engineering.naukri.com/2016/03/reduce-
test-automation-flakiness/

Automation Execution Time Reduction
By selenium grid implementation we were
able to reduce our execution time to 1/5th
– From 25Hrs to 5Hrs
To know more about Selenium Grid implementation visit our blog:
http://engineering.naukri.com/2015/10/parallel-testing-at-naukri/

Preventing Bugs
• Tester, developer, product owner, architect are all part
of the backlog grooming
• Entire team focuses on defining what and how
• Product backlog grooming is done one iteration in
advance
• Testers contribute test cases upfront during this
period and add them to the user stories in form of
acceptance criteria or alternate paths

Preventing Bugs
• Peer testing at developer end
• Automated Build verification tests have been created
and are run before providing builds to testers
• Progressive Automation testing approach is used.

Testing Understanding
• Put yourself in the customer shoes
• We encourage our scrum teams to have direct interaction with actual customers
– proactive & reactive
• Effective feedback loops
• Our tech support team regularly shares reports on issue patterns which are used
by scrum teams as inputs to design/test/improve systems
• Measure the customer usage pattern and use it to design test cases
• We regularly analyse user data patterns to come up with and refine our test
strategy

Building the best system
• Build Implicit Requirements
• Focus more on building positive product scenarios
• Focus on Bug Causal Analysis
• Peripheral testing: Focus v/s Defocus
To know more about peripheral testing please visit our blog
http://engineering.naukri.com/2016/03/peripheral-testing/

Team responsibility for quality
• Everybody tests as and when needed
• Measure Quality at various levels
• Build Quality Meter
• Post Production Issue Seepage

Build Quality Meter
Bug Severity
Bug Type: Functional, UI, product design,
Implicit, Validation, Incomplete requirement,
Insufficient impact analysis, Integration
Environment, Configuration, Inadequate
testing,DOA

Our Measurements
Sprint15-28Dec Sprint29Dec-11Jan Sprint12-25Jan Sprint26Jan-8Feb Sprint9-22Feb Sprint23Feb-7March Sprint8-21Mar
Commitment 39 40 44 42 44 42 46
Completed Unplanned P1/P2 2 3 2 5 3 4 2
Completed Unplanned 6 5 1 2 4 2 5
Planned Live/Staging 6 1 2 10 5 3 4
Completed Planned 19 27 39 25 30 32 34
Target 41 41 41 41 41 41 41
0
5
10
15
20
25
30
35
40
45
50
Velocity Report
Indicative Data has been used for illustration purpose

Our Measurements
0
4 4
11
0 1 2
9
0 1 0
35 7
0
5
0 0 1 00 0 0 1
93
80.86
70.68
60.76
0
10
20
30
40
50
60
70
80
90
100
0
10
20
30
40
50
60
70
80
90
100
26th Jan'16-08th Feb'16 9th Feb'16-22nd Feb'16 23rd Feb'16-7th March'16 8th March'16-21st March'16
BUILDQUALITY%
BUGS
ITERATION
Build Quality Trend
Functional Bugs UI Bugs Validation Live Implicit Integration Build Quality
Bad
[Less than 70]
Average
[70-80]
Good
[80-90]
Excellent
[90 +]
Excellent
Good
Average
Bad

Our Measurements
2
14
9
17
1
11
4
13
1
11
4
13
0
2
4
6
8
10
12
14
16
18
26th Jan'16-08th Feb'16 9th Feb'16-22nd Feb'16 23rd Feb'16-7th March'16 8th March'16-21st March'16
NO.OFSTORIES
Functional Automation Coverage
Total Stories Automatable Automated

Our Measurements
26 Jan - 8 Feb 9 -22 Feb 23 Feb-7 Mar 8-21 March'16
Client Reported Issues (Data Fixes) 2 4 3 1
Client Reported Issues (Code Fixes) 1 2 0 1
Live issues (Except client issues) 2 1 3 3
2
1
3 3
1
2
0
1
2
4
3 1
0
1
2
3
4
5
6
7
8
9
10
BUGCOUNT
Bugs Reported on Live

Results
Post live defect seepage: 50% reduction
Test Cases: 10K increase
Automation Coverage: 44% increase
Automation Scripts Execution time: 20% reduction
Automation flakiness: 20% reduction
Velocity: 25% increase
15% Build quality improvement
30 % Planning Efficiency improvement

Key Take Away’s
• Defining the “Right Metric”
• Measure, Review, Improve ….
• For Agile testing through out focus should be on progressive/parallel
automation testing along with creating reliable tests that take minimal time
to execute.
• Defect Prevention is the Key
• Team v/s Individual mind-set

Agile Gurugram 2016 | Conference | Continuous Agile Testing @ Naukri | Meetu Arora

Recommended

Recommended

More Related Content

What's hot

What's hot (11)

Similar to Agile Gurugram 2016 | Conference | Continuous Agile Testing @ Naukri | Meetu Arora

Similar to Agile Gurugram 2016 | Conference | Continuous Agile Testing @ Naukri | Meetu Arora (20)

More from AgileNetwork

More from AgileNetwork (20)

Recently uploaded

Recently uploaded (20)

Agile Gurugram 2016 | Conference | Continuous Agile Testing @ Naukri | Meetu Arora