When Anurag joined Honeywell in 2014, it was an industrial company on a mission to transform itself into a Digital industrial company. Development was done in a traditional waterfall method, testing was only done manually, resulting in very few software releases with below average quality. He is helping transform the organization by driving (scaled) agile development, test automation and effective DevOps. The big question faced by the company now is how to best leverage the army of manual test resources in this automated DevOps world. In this talk, he will explain how he helped answer that question.

1. Are Manual Testers
Needed in
Automated DevOps
world?
From:
Anurag Sharma,
Director Quality and DevOps
Software Solutions, Honeywell Home and Building Technologies

2. 2
Mobile app testing:
1. Functional App testing
2. Functional System testing
3. System performance testing (normal load)
Cloud testing:
1. Functional API scenario-based
testing
2. Load/stress/performance testing
Hardware:
DV/CQ, Pre compliance, Certification and
Reliability
Firmware:
Functional test, Product Acceptance Test
(Sub system integration tests, NFR tests)
Configuration - Various:
1. Different OS (iOS, Android, etc)
2. Different signal strengths (especially for
negative testing)
3. Different device types
4. Different localization
5. Different user account (single/multi locations
single/multi Honeywell devices, >1 users)
.
.
System test automation
Setup
End to End System Test:
Verification (Customer use cases,
performance, capacity,
compatibility, system stability,
Killer tests), Security
CHIL
TCC
LCC
WLD

3. Key to DevOps: Automation - Test, CI/CD pipeline.
• Past:
• Many legacy products, primarily reliant on manual testing
• Products continue to become more complex.
• Present:
• Making a rapid shift towards test automation. Buy, Borrow, Build
• Integration and System-level manual test cases -> Automated test cases

4. Continuous Testing
• Fail early, Fail often
• Behavior Driven Development,
Test Driven Development
• Test Automation part of
Definition of Done. Equally
important as source code.

5. Manual Testers can add value by:
• More Edge-case and Exploratory testing
• Better understand user environment
• Gracefully error out, secure and usable
• Destructive testing
• VMs/Nodes/Scale Units turned OFF
• Visual testing
• Usability testing
• Getting more value out of Alpha/Beta testing
• Performance and Stress testing
• Security testing
• Compatibility testing
Testing done by Manual Testers is more crucial than ever!

6. Mobile App Testing challenges
• Typical DevOps principles like deploying fast, deploying many times a day, redu
• Delay is imposed by App stores in publishing apps – CD not possible.

7. Help better manage mobile testing challenges
• Identify and continuously tweak the combination that automated
tests should run on.
• Strike balance between simulator/emulator and real mobile device
testing
• Drive test strategies for compatibility against newer version of OS,
device and other third party libraries/SDKs used
• Perform Visual testing and drive usage of automation test tools to
automate some of the critical visual elements

8. Be the Test Design Architect, Test strategist
• Review automated test cases - ensure scenarios automated are
relevant.
• Prioritize backlog of automatable test cases
• Participate in Design reviews, System Architecture reviews - ensure
testability. Identify tools, resources, devices needed to test each
component and debug root cause.
• Work with Developers and Operation resources to build automated
Continuous Delivery pipeline.

9. Direct involvement in all aspects of test automation ecosystem
• Analyze automated test failures
• Identify product issues, test environment issues, sporadic issues, script issues
• Train developers, product owners etc. in similarly analyzing test results and
taking corrective actions in a timely manner
• Increase test automation coverage
• Test automated tests
• Identify false positives and sporadic test failures. Gather trends and help
resolve.
• Eliminate unwanted latencies
• Optimize test case organization – combine/split test cases
• Ensure maximum parallelism, test bed maintenance
• Device ‘source code to test case’ mapping

10. • Contribute to creation of automated test scripts
• Action based testing – Create and leverage parameterized cross-platform
modules (or keywords), page objects
• Create specs for keywords and help them evolve
• Ensure test logs are readable, verbose for quick debugging
• Measure test automation effectiveness
• Analytics on test pass/fail ratio over time/teams. Help improve process, Built
In Quality, code architecture/refactoring.
• Ensure automated tests are part of CI/CD. Address failures immediately.
• Correlate test failures with results from other ALM tools
• Reduce overall test cycle time