This document summarizes key topics from the GTAC 2015 conference on test automation and innovative development. It discusses trends in test automation like component testing, mock testing, and production monitoring. The top five talks are summarized, focusing on continuous deployment, transforming monolithic code to microservices, and using production traffic for regression testing. Statistical sampling techniques are discussed for reducing large production datasets. Measuring efficiency, success versus failure, and challenges of startups versus corporations are also addressed.
Marketing Management 16th edition by Philip Kotler test bank.docx
GTAC 2015 And Innovative Development 30 min
1. GTAC 2015 and
Innovative Development
Igor Goulko,
Independent Consultant
e-mail: goulko@gmail.com
17-Mar-2016 (ANZTB)20
2. Agenda
1. Innovation inTest Automation
2. TOP 5Talks
3. Efficiency and measurements
4. Experiments and Learning
5. Success and Failure
6. Start Up vs Corporation
3. Innovation inTest Automations
General Direction and Best Practice
◦ ComponentTesting
◦ Mock everything (LinkedIn, Google and others)
◦ Production DataTesting
◦ Production Monitoring
◦ More advanced testing tools
◦ Custom tools
4. Top 5 talks
Testing Tube (Jurgen Allgayer)
◦ Unified dev/test environment
◦ Everyone writes tests
◦ Dev=integration
◦ No surprises = Borg
◦ Daily pushes
◦ 200 services
Large-Scale AutomatedVisual Testing
(Adam Carmi, Applitools)
◦ Very sophisticated image comparison
◦ Cross browsers and device testing
◦ Monitoring dynamic applications
◦ Automate a maintenance
5. Hands Off RegressionTesting
Continuous deploy
Transformation from Huge Monolithic
code to micro services
Unit tests and Component tests
5 Code Paths for 6 Methods ~ 15KTC
Diffy uses production traffic for
regression tests
Candidate (new), Primary and Secondary
deployments
7. Statistical Data Sampling
Celal Ziftci and Ben Greenberg
Smart Production Sample Data (1.6B->78K)
14 hours -> 2 mins and it runs for EVERY single submit
Data
Sampler
Data
Sampler
Data
Sampler
Data
Sampler
Complete Production Data
...
Data
Sampler
Final Sample
Sample Sample Sample Sample
200 Delegates, 3 from Australia
Alister Scott (WorldPress, Brisbane)
Fernanda Rocket Gomes (Atlassian, Sydney)
Me (Independent Consultant, Melbourne)
Linkedin use mock for internet, Google Fake Backends with RpcReplay (Matt Garett, Google)
Component testing: Google, Google, google…
Production data testing: almost all talks
Production monitoring: Google
More advanced testing tools: all
Custom tools: GOOGLE!!! All (99.99%) tools are custom
Continuous upgrades: all
Coverage: Woterloo University
Robots: OptoFidelity
Testing TubeChallenges:
Unpredictable launch intervals
Mobile/Devices: manual heavy
Culture change
Results:
Unified dev/test environment
Everyone writes tests
Dev=integration
No surprises = Borg
Daily pushes
200 services
Hands Off Regression Testing Continuous deploy
Transformation from Huge Monolithic code to microservices
Unit tests and Component tests
5 CodePaths for 6 Methods ~ 15K TC
Diffy uses production traffic for regression tests
Candidate (new), Primary and Secondary deployments
Statistical Data Samplings
Covers expected and edge cases
Tests run much faster
Diffs are a lot smaller and non-repetitive
150<Age = 0, but if age=null
Smart Production Sample Data (1.6B->78K)
Good samples, expected and boundaries cases
Size of Random sample and Chance to miss specific data
In case of 10% of sample we have only 7% chance to catch
Time of test execution and sample sizes
Learn from samples – add or go
14 hours -> couple minutes and run for every single submit
Large-Scale Automated Visual Testing
Bitmap comparison and false positives
Very sophisticated image comparison
Cross browsers testing
Cross device testing
Monitoring dynamic applications
Automate a maintenance
Your tests are not Flaky
“Kill Bill.” Kill Flaky Tests
A re-run culture is toxic
Trust the tests.
Testable Application and Design“Toaster is under smoke alarm. Is smoke alarm flaky?”
Complexity
Testable app == usable app; usable != flaky
Flaky are not useless
Efficiency, effectiveness and testing confidence
In addition just couple words about Fake Backends with RpcReplay
Problem:
Maintain these implementations until the end of time.
You’re exposed to bugs, test implementations differ from production implementations.
RPC Log and RPCReplay Fake
Good samples, expected and boundaries cases
Size of Random sample and Chance to miss specific data
In case of 10% of sample we have only 7% chance to catch
Time of test execution and sample sizes
Learn from samples – add or go
14 hours -> couple minutes and run for every single submit
Covers expected and edge cases
Tests run much faster
Diffs are a lot smaller and non-repetitive
150<Age = 0, but if age=null
Your tests are not Flaky
“Kill Bill.” Kill Flaky Tests
A re-run culture is toxic
Trust the tests.
Testable Application and Design“Toaster is under smoke alarm. Is smoke alarm flaky?”
Complexity
Testable app == usable app; usable != flaky
Flaky are not useless
Efficiency, effectiveness and testing confidence
When we have diffusion team out the productivity is lower than when we have a solid team in the same open space area.
Technical Mastery
Business Mastery
Personal Mastery
It is easier to create new green field product in right way, a refactoring is always difficult and required more skills and resources.It is applied to development processes as well.
Tunning, refactoring and rebuilding of products have to be Business as usual. The same pattern has to be applied to process management. Nothing can be static in our very dynamic world.
A lot of time I heard from different people – “We are not start up. We are not SEEK, We are not Google”, so why we are not? We can be so successful as Start Up, SEEK, Google … The question is – what we have to do for it?!Do you have deck at home? Did you learn to build it? Did you plant tree? Did you get sex lessons? NO!!!But we can do it pretty well and successfuly. Maybe not from first time but after several experiments.Why?Because we want!!!It is the main point - every one can do everything in the best way when... they willing to do it.