SlideShare a Scribd company logo
1 of 29
Testing the Ministry Of Justice
Biometric Identification System
Gerlof Hoekstra (NL)
| Gerlof Hoekstra| © Atos
Chain testing, Acceptance testing,
End-To-End Business Processes
▶ Multiple stakeholders / applications
▶ Complex
▶ Publicly visible
▶ Politically sensitive
▶ Previous attempt(s) failed
Introduction and context
Introducing myself…
Gerlof
Developer
Designer
Tester
2
| Gerlof Hoekstra| © Atos
▶ Context
▶ The Ministry Of Justice Intelligence Department
▶ The identification system
▶ The situation I faced when I joined the project
▶ What makes this chain test challenging
▶ Aspects:
▶ Team
▶ Test specification
▶ Formal / informal testing
▶ Test data
▶ From waterfall to iterative
▶ Findings
▶ Lessons learnt
3
| Gerlof Hoekstra| © Atos
Introduction and context
Ministry Of Justice Intelligence Department
4
| Gerlof Hoekstra| © Atos
▶ Police
▶ Prosecutors
▶ Judges
▶ Investigation services
▶ Prisons
▶ Lawyers
▶ Rehabilitation
▶ Customs offices
▶ Debt collection agency
▶ …
▶ …
The judicial chain
5
| Gerlof Hoekstra| © Atos
The Identification Console
Installed in:
- every Police Office
- central immigration offices
- prisons
- Schiphol airport
6
| Gerlof Hoekstra| © Atos
Hot topic !
7
| Gerlof Hoekstra| © Atos
The Identification Console
Functionality overview
8
| Gerlof Hoekstra| © Atos
The Identification System
Technology
▶ Biometric middleware
▶ Photo-software
▶ NIST
▶ ebXML
▶ Jubes
▶ ESB
▶ External Police Broker
▶ NORA
▶ JAB
▶ Ministry of Justice security standards
▶ Photo standards
▶ Fingerprint standards
▶ GBA character set
▶ 7x24
▶ 30.000 users
9
| Gerlof Hoekstra| © Atos
The Identification System
High level application architecture & test scope
Crime Immigration
Central message broker
10
| Gerlof Hoekstra| © Atos
Initial situation
When I joined the project
▶ Phase 1
▶ Goal: connect to Havank (crime evidence traces)
▶ Earlier attempt failed
▶ New project leader
▶ Well documented business process
▶ Integrated test environment available
▶ Phase 2:
▶ Build & implement a completely new front-end
▶ Connect to foreigner/immigration chain
▶ Validate ID-documents
▶ Optimize business process (scan once)
11
| Gerlof Hoekstra| © Atos
Challenges
What makes this chain special?
▶ Complicated business process (many legal rules, privacy)
▶ Many registrations: de facto inconsistent
▶ Most “customers” not willing to co-operate (try to fraude)
▶ Chain-consistent test data (esp. fingerprints)
▶ Many different stakeholders, physical distance, living on their own islands
▶ Uncertainty / problems in the Police organization
▶ Heavy message transfer, complex message broker functionality
12
| Gerlof Hoekstra| © Atos
Assembling a Test Team
Testers vs subject matter experts, formal vs informal
▶ Subject matter matters!
▶ Process knowledge
▶ Know the organization
▶ Middleware & message
transfer
▶ Involved in implementation
▶ Police officers
▶ Dactylocopists
▶ Application manager
▶ Justice chain expert
▶ Free thinkers
▶ Not afraid to experiment
▶ No dogmas
▶ Excellent observers
▶ Easy making contact
13
| Gerlof Hoekstra| © Atos
Test specification: formal part
Keep it 'simple' and compact; connect to the world of the stakeholders
▶ Process description(test basis)
▶ Classification tree (test overview)
▶ User scenarios (details)
14
| Gerlof Hoekstra| © Atos
Test specification: informal part
Make optimum usage of subject matter experts: test charters
Theme based:
▶ Chain components not available
▶ Attribute validation
▶ Unstructured addresses
▶ (Partly) unknown birth dates
▶ Message transfer anomalies
▶ False ID-documents
▶ Bad quality fingerprints
▶ Residential address unknown
▶ Inconsistencies between registrations
Role based:
▶ Test with customs employee
▶ Test met prison employee
15
| Gerlof Hoekstra| © Atos
Formal vs informal
Dealing with the quality 'watchdog'
Dilemma:
 versus 
“How Dare You Apply Exploratory Testing”
Solution:
▶ Have a small, but well thought-out predefined test set
▶ Identify exploratory test charters in advance
▶ Use simple check lists
▶ Have a good relation with QA management
▶ Explain why you are doing it like this
▶ Report what you have done
16
| Gerlof Hoekstra| © Atos
Test data
Accepting & dealing with restrictions
▶ Which fingerprints do I
use?
▶ How to load these into
a far far away foreign
test database?
▶ How to clean up test
data?
▶ How to get ID
documents for test?
▶ Solution: puzzling,
know what you want,
preparation, know the
right people!
Direct
database
inserts
Insert data
with
application
Use
available
test data
Cleanup
possible?
17
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Initial plan phase 2
▶ Newly build front-end
▶ Monthly iterations
▶ Back-end system changes
▶ Every party has his own process & implementation schedule
▶ E2E test starts when all components are ready
▶ mitigate late integration bugs by:
▶ Gerlof reviews/monitors all supplier tests
▶ 1:1 interface tests (peer-to-peer)
▶ stub usage
18
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Project in trouble!
▶ Front-end
▶ SCRUM in name only
▶ No business value delivered yet
▶ Back-end system changes
▶ DONE ! (?)
▶ Some even deployed in production
▶ Reviews, peer-to-peer testing, stubs:
▶ I did what I could ....
▶ No time, complex, error prone, ...
19
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Re-organizing the project
▶ Front-end
▶ Monthly sprints, now with business focus
▶ My struggle to realize an early & iterative E2E test
▶ 'No time/resources for linking to integrated test environment'
▶ 'Trust the stubs'
▶ 'You disturb the developers'
▶ 'In the end, Iterative E2E testing takes more time'
▶ 'No need, it should work, we reviewed everything’
20
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Finally, from sprint 4 we were able to E2E test, AND HECK WE DID !!!!!
21
| Gerlof Hoekstra| © Atos
Some interesting findings
Typical integration problems
▶ Simple 'No Hits' leading to fatal errors at the front-end application
▶ Legacy systems not following agreed standards
▶ Un- or incorrectly documented interfaces
22
| Gerlof Hoekstra| © Atos
Getting defects fixed
Many independent parties, legacy systems
▶ Sometimes very hard & frustrating
▶ “Not our fault”
▶ “On our side, everything works fine!”
▶ “Yeah, that’s exactly how it is supposed to work”
▶ “Please call our service desk, they will help you”
▶ “Next release we will fix the problem (in 6 months)”
▶ Process
▶ Responsibility for defect fixing !!
▶ Diagnose (zie processchema)
▶ Solution alternatives
▶ Decision making
▶ Fixing & retesting
▶ Follow-up
23
| Gerlof Hoekstra| © Atos
What did we learn
Co-operation
▶ Getting into contact / building relations / acquire business chain
knowledge is crucial
▶ (From time to time) working from 1 location adds tremendous value
▶ End users / monitoring / troubleshooting / architect
24
| Gerlof Hoekstra| © Atos
What did we learn
Early integration
▶ Perform E2E chain testing during the
sprints
▶ Do not postpone difficult challenges;
if something seems difficult, do it as
early as possible!
▶ If you really want something, you can
arrange it!
▶ Stubs can be useful, but “Nothing
beats the real thing”
25
| Gerlof Hoekstra| © Atos
What did we learn
Test specification
▶ Some carefully specified test scenarios are
useful, but do not completely rely on pre-
defined test cases
▶ Make a mix between formal techniques
and exploratory testing and let coincident
happen.
▶ Do not underestimate the value of human
observation (we revealed many bugs by
observing a bit deeper than the test script)
▶ Don't trust application designs, even if
reviewed 100 times;
▶ for E2E testing, a application design is
no more than a supporting document
▶ Business process = test basis
?
A collection of applications that
all work as designed does NOT
automatically mean a working
chain !
26
| Gerlof Hoekstra| © Atos
What did we learn
Others
▶ Effective reporting
▶ Frequent
▶ Visual
▶ In the stakeholders' language
▶ Forecast
▶ Directly linked to the implementation scenario
▶ Implement testability
▶ Log’s and traces
▶ Alternative input (esp. for fingerprints)
▶ Police officers are great testers!
27
| Gerlof Hoekstra| © Atos
Result
Status summer 2016
▶ 2.5 years extensive use
▶ Much more asylum seekers than planned
▶ More locations than planned
▶ Users are happy – less work – quicker business process
▶ No messing with ink fingerprints anymore
▶ More crimes solved
▶ Improved maintainability
▶ No more dependency on 1 external supplier
▶ Various enhancements realized
▶ Finally, the maintenance contract is signed
28
Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline,
BlueKiwi, Canopy the Open Cloud Company, Yunano, Zero Email, Zero
Email Certified and The Zero Email Company are registered
trademarks of Atos. January 2015. © 2015 Atos. Confidential
information owned by Atos, to be used by the recipient only. This
document, or any part of it, may not be reproduced, copied, circulated
and/or distributed nor quoted without prior written approval from
Atos.
Thank You !
Contact:
M+ 31 6 512 88 478
gerlof.hoekstra@atos.net

More Related Content

Viewers also liked

test plan
test plan test plan
test plan
rosh26
 
Testing strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorks
Testing strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorksTesting strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorks
Testing strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorks
Thoughtworks
 

Viewers also liked (18)

Implementing Test Automation: What a Manager Should Know
Implementing Test Automation: What a Manager Should KnowImplementing Test Automation: What a Manager Should Know
Implementing Test Automation: What a Manager Should Know
 
test plan
test plan test plan
test plan
 
E2E-тестирование мобильных приложений
E2E-тестирование мобильных приложенийE2E-тестирование мобильных приложений
E2E-тестирование мобильных приложений
 
Strategic Testing (CodeMash 2016)
Strategic Testing (CodeMash 2016)Strategic Testing (CodeMash 2016)
Strategic Testing (CodeMash 2016)
 
End-to-End Quality Approach: 14 Levels of Testing
End-to-End Quality Approach: 14 Levels of TestingEnd-to-End Quality Approach: 14 Levels of Testing
End-to-End Quality Approach: 14 Levels of Testing
 
Test Planning_Arsala
Test Planning_ArsalaTest Planning_Arsala
Test Planning_Arsala
 
End to End Test Management Test Strategy Estimation and Metrics Workshop
End to End Test Management Test Strategy Estimation and Metrics WorkshopEnd to End Test Management Test Strategy Estimation and Metrics Workshop
End to End Test Management Test Strategy Estimation and Metrics Workshop
 
Saksham Sarode - Building Effective test Data Management in Distributed Envir...
Saksham Sarode - Building Effective test Data Management in Distributed Envir...Saksham Sarode - Building Effective test Data Management in Distributed Envir...
Saksham Sarode - Building Effective test Data Management in Distributed Envir...
 
E2E Life Cycle Intelligence & Channel Strategy @ TPC
E2E Life Cycle Intelligence & Channel Strategy @ TPCE2E Life Cycle Intelligence & Channel Strategy @ TPC
E2E Life Cycle Intelligence & Channel Strategy @ TPC
 
Workshop - E2e tests with protractor
Workshop - E2e tests with protractorWorkshop - E2e tests with protractor
Workshop - E2e tests with protractor
 
Preparing and Implementing a Comprehensive ICD-10 Testing Strategy
Preparing and Implementing a Comprehensive ICD-10 Testing StrategyPreparing and Implementing a Comprehensive ICD-10 Testing Strategy
Preparing and Implementing a Comprehensive ICD-10 Testing Strategy
 
e2e frameworks - czyli kij ma dwa końce
e2e frameworks - czyli kij ma dwa końcee2e frameworks - czyli kij ma dwa końce
e2e frameworks - czyli kij ma dwa końce
 
11 software testing_strategy
11 software testing_strategy11 software testing_strategy
11 software testing_strategy
 
Testing strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorks
Testing strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorksTesting strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorks
Testing strategies for micro services - Ketan Soni, Jesal Mistry, ThoughtWorks
 
Configuration Management
Configuration ManagementConfiguration Management
Configuration Management
 
End-to-End Test Automation for Both Horizontal and Vertical Scale
End-to-End Test Automation for Both Horizontal and Vertical ScaleEnd-to-End Test Automation for Both Horizontal and Vertical Scale
End-to-End Test Automation for Both Horizontal and Vertical Scale
 
Agile testing principles and practices - Anil Karade
Agile testing principles and practices - Anil KaradeAgile testing principles and practices - Anil Karade
Agile testing principles and practices - Anil Karade
 
Test data management a case study Presented at SiGIST
Test data management a case study Presented at SiGISTTest data management a case study Presented at SiGIST
Test data management a case study Presented at SiGIST
 

Similar to QA Fes 2016. Gerlof Hoekstra. E2E Testing the Ministry Of Justice Biometric Identification System

Iwsm2014 importance of benchmarking (john ogilvie & harold van heeringen)
Iwsm2014   importance of benchmarking (john ogilvie & harold van heeringen)Iwsm2014   importance of benchmarking (john ogilvie & harold van heeringen)
Iwsm2014 importance of benchmarking (john ogilvie & harold van heeringen)
Nesma
 

Similar to QA Fes 2016. Gerlof Hoekstra. E2E Testing the Ministry Of Justice Biometric Identification System (20)

Utilizing Technology to Design Efficient Law Firm Processes
Utilizing Technology to Design Efficient Law Firm ProcessesUtilizing Technology to Design Efficient Law Firm Processes
Utilizing Technology to Design Efficient Law Firm Processes
 
A Tester's Life
A Tester's LifeA Tester's Life
A Tester's Life
 
apidays LIVE Paris 2021 - Boavitza, Year 2 by Laurent Eskenazi
apidays LIVE Paris 2021 - Boavitza, Year 2 by Laurent Eskenaziapidays LIVE Paris 2021 - Boavitza, Year 2 by Laurent Eskenazi
apidays LIVE Paris 2021 - Boavitza, Year 2 by Laurent Eskenazi
 
Install PRESTO KPI in 5 weeks
Install PRESTO KPI in 5 weeksInstall PRESTO KPI in 5 weeks
Install PRESTO KPI in 5 weeks
 
Innovations in Test Automation: It’s Not All about Regression
Innovations in Test Automation: It’s Not All about RegressionInnovations in Test Automation: It’s Not All about Regression
Innovations in Test Automation: It’s Not All about Regression
 
Module 8: Purchasing & Project Management Essentials
Module 8: Purchasing & Project Management EssentialsModule 8: Purchasing & Project Management Essentials
Module 8: Purchasing & Project Management Essentials
 
SystemQuest Ltd - Simple Maintenance Systems
SystemQuest Ltd - Simple Maintenance SystemsSystemQuest Ltd - Simple Maintenance Systems
SystemQuest Ltd - Simple Maintenance Systems
 
Environmental and social impacts Across supply chains - LCA conference 4 Nov ...
Environmental and social impacts Across supply chains - LCA conference 4 Nov ...Environmental and social impacts Across supply chains - LCA conference 4 Nov ...
Environmental and social impacts Across supply chains - LCA conference 4 Nov ...
 
Document Control in FDA Regulated Environments - When and how to automate
Document Control in FDA Regulated Environments - When and how to automateDocument Control in FDA Regulated Environments - When and how to automate
Document Control in FDA Regulated Environments - When and how to automate
 
Are project tracking tools helping or complicating Continuous Improvement Pro...
Are project tracking tools helping or complicating Continuous Improvement Pro...Are project tracking tools helping or complicating Continuous Improvement Pro...
Are project tracking tools helping or complicating Continuous Improvement Pro...
 
Peter Verstraeten (Proceedix): Digital Work Instructions and Inspections Powe...
Peter Verstraeten (Proceedix): Digital Work Instructions and Inspections Powe...Peter Verstraeten (Proceedix): Digital Work Instructions and Inspections Powe...
Peter Verstraeten (Proceedix): Digital Work Instructions and Inspections Powe...
 
Mistakes Outsourcing Customers Make by Kees Blokland
Mistakes Outsourcing Customers Make by Kees BloklandMistakes Outsourcing Customers Make by Kees Blokland
Mistakes Outsourcing Customers Make by Kees Blokland
 
Deciding what and when to automate in testing: Experience from multiple projects
Deciding what and when to automate in testing: Experience from multiple projectsDeciding what and when to automate in testing: Experience from multiple projects
Deciding what and when to automate in testing: Experience from multiple projects
 
Supplier Assessment Sample Report: What You Can Expect
Supplier Assessment Sample Report: What You Can ExpectSupplier Assessment Sample Report: What You Can Expect
Supplier Assessment Sample Report: What You Can Expect
 
2015 02 24 lmtv baselining
2015 02 24 lmtv baselining2015 02 24 lmtv baselining
2015 02 24 lmtv baselining
 
EDM101: Implementation Practices - Project Management
EDM101: Implementation Practices - Project ManagementEDM101: Implementation Practices - Project Management
EDM101: Implementation Practices - Project Management
 
Legal Technology Vision (Singapore) 2018 update v 2
Legal Technology Vision (Singapore) 2018 update v 2Legal Technology Vision (Singapore) 2018 update v 2
Legal Technology Vision (Singapore) 2018 update v 2
 
Monika_Resume
Monika_ResumeMonika_Resume
Monika_Resume
 
Improve Your Test Process from the Bottom Up
Improve Your Test Process from the Bottom UpImprove Your Test Process from the Bottom Up
Improve Your Test Process from the Bottom Up
 
Iwsm2014 importance of benchmarking (john ogilvie & harold van heeringen)
Iwsm2014   importance of benchmarking (john ogilvie & harold van heeringen)Iwsm2014   importance of benchmarking (john ogilvie & harold van heeringen)
Iwsm2014 importance of benchmarking (john ogilvie & harold van heeringen)
 

More from QAFest

QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QAFest
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QAFest
 

More from QAFest (20)

QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилинQA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
QA Fest 2019. Сергій Короленко. Топ веб вразливостей за 40 хвилин
 
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The FutureQA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
QA Fest 2019. Анна Чернышова. Self-healing test automation 2.0. The Future
 
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
QA Fest 2019. Doug Sillars. It's just too Slow: Testing Mobile application pe...
 
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
QA Fest 2019. Катерина Спринсян. Параллельное покрытие автотестами и другие и...
 
QA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать большеQA Fest 2019. Никита Галкин. Как зарабатывать больше
QA Fest 2019. Никита Галкин. Как зарабатывать больше
 
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiledQA Fest 2019. Сергей Пирогов. Why everything is spoiled
QA Fest 2019. Сергей Пирогов. Why everything is spoiled
 
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгораниемQA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
QA Fest 2019. Сергей Новик. Между мотивацией и выгоранием
 
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
QA Fest 2019. Владимир Никонов. Код Шредингера или зачем и как мы тестируем н...
 
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
QA Fest 2019. Владимир Трандафилов. GUI automation of WEB application with SV...
 
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium ClusterQA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
QA Fest 2019. Иван Крутов. Bulletproof Selenium Cluster
 
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
QA Fest 2019. Николай Мижигурский. Миссия /*не*/выполнима: гуманитарий собесе...
 
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
QA Fest 2019. Володимир Стиран. Чим раніше – тим вигідніше, але ніколи не піз...
 
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automationQA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
QA Fest 2019. Дмитрий Прокопук. Mocks and network tricks in UI automation
 
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
QA Fest 2019. Екатерина Дядечко. Тестирование медицинского софта — вызовы и в...
 
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
QA Fest 2019. Катерина Черникова. Tune your P’s: the pop-art of keeping testa...
 
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях ITQA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
QA Fest 2019. Алиса Бойко. Какнезапутаться в коммуникативных сетях IT
 
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложенииQA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
QA Fest 2019. Святослав Логин. Как найти уязвимости в мобильном приложении
 
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
QA Fest 2019. Катерина Шепелєва та Інна Оснач. Що українцям потрібно знати пр...
 
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
QA Fest 2019. Антон Серпутько. Нагрузочное тестирование распределенных асинхр...
 
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
QA Fest 2019. Петр Тарасенко. QA Hackathon - The Cookbook 22
 

Recently uploaded

The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
heathfieldcps1
 
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdfFinancial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
MinawBelay
 

Recently uploaded (20)

How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17
 
Software testing for project report .pdf
Software testing for project report .pdfSoftware testing for project report .pdf
Software testing for project report .pdf
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
 
demyelinated disorder: multiple sclerosis.pptx
demyelinated disorder: multiple sclerosis.pptxdemyelinated disorder: multiple sclerosis.pptx
demyelinated disorder: multiple sclerosis.pptx
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
How to Manage Closest Location in Odoo 17 Inventory
How to Manage Closest Location in Odoo 17 InventoryHow to Manage Closest Location in Odoo 17 Inventory
How to Manage Closest Location in Odoo 17 Inventory
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024
 
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community PartnershipsSpring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
 
MOOD STABLIZERS DRUGS.pptx
MOOD     STABLIZERS           DRUGS.pptxMOOD     STABLIZERS           DRUGS.pptx
MOOD STABLIZERS DRUGS.pptx
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the life
 
Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment
 Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment
Envelope of Discrepancy in Orthodontics: Enhancing Precision in Treatment
 
IPL Online Quiz by Pragya; Question Set.
IPL Online Quiz by Pragya; Question Set.IPL Online Quiz by Pragya; Question Set.
IPL Online Quiz by Pragya; Question Set.
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceutics
 
Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"
Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"
Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"
 
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdfFinancial Accounting IFRS, 3rd Edition-dikompresi.pdf
Financial Accounting IFRS, 3rd Edition-dikompresi.pdf
 
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
 
REPRODUCTIVE TOXICITY STUDIE OF MALE AND FEMALEpptx
REPRODUCTIVE TOXICITY  STUDIE OF MALE AND FEMALEpptxREPRODUCTIVE TOXICITY  STUDIE OF MALE AND FEMALEpptx
REPRODUCTIVE TOXICITY STUDIE OF MALE AND FEMALEpptx
 

QA Fes 2016. Gerlof Hoekstra. E2E Testing the Ministry Of Justice Biometric Identification System

  • 1. Testing the Ministry Of Justice Biometric Identification System Gerlof Hoekstra (NL)
  • 2. | Gerlof Hoekstra| © Atos Chain testing, Acceptance testing, End-To-End Business Processes ▶ Multiple stakeholders / applications ▶ Complex ▶ Publicly visible ▶ Politically sensitive ▶ Previous attempt(s) failed Introduction and context Introducing myself… Gerlof Developer Designer Tester 2
  • 3. | Gerlof Hoekstra| © Atos ▶ Context ▶ The Ministry Of Justice Intelligence Department ▶ The identification system ▶ The situation I faced when I joined the project ▶ What makes this chain test challenging ▶ Aspects: ▶ Team ▶ Test specification ▶ Formal / informal testing ▶ Test data ▶ From waterfall to iterative ▶ Findings ▶ Lessons learnt 3
  • 4. | Gerlof Hoekstra| © Atos Introduction and context Ministry Of Justice Intelligence Department 4
  • 5. | Gerlof Hoekstra| © Atos ▶ Police ▶ Prosecutors ▶ Judges ▶ Investigation services ▶ Prisons ▶ Lawyers ▶ Rehabilitation ▶ Customs offices ▶ Debt collection agency ▶ … ▶ … The judicial chain 5
  • 6. | Gerlof Hoekstra| © Atos The Identification Console Installed in: - every Police Office - central immigration offices - prisons - Schiphol airport 6
  • 7. | Gerlof Hoekstra| © Atos Hot topic ! 7
  • 8. | Gerlof Hoekstra| © Atos The Identification Console Functionality overview 8
  • 9. | Gerlof Hoekstra| © Atos The Identification System Technology ▶ Biometric middleware ▶ Photo-software ▶ NIST ▶ ebXML ▶ Jubes ▶ ESB ▶ External Police Broker ▶ NORA ▶ JAB ▶ Ministry of Justice security standards ▶ Photo standards ▶ Fingerprint standards ▶ GBA character set ▶ 7x24 ▶ 30.000 users 9
  • 10. | Gerlof Hoekstra| © Atos The Identification System High level application architecture & test scope Crime Immigration Central message broker 10
  • 11. | Gerlof Hoekstra| © Atos Initial situation When I joined the project ▶ Phase 1 ▶ Goal: connect to Havank (crime evidence traces) ▶ Earlier attempt failed ▶ New project leader ▶ Well documented business process ▶ Integrated test environment available ▶ Phase 2: ▶ Build & implement a completely new front-end ▶ Connect to foreigner/immigration chain ▶ Validate ID-documents ▶ Optimize business process (scan once) 11
  • 12. | Gerlof Hoekstra| © Atos Challenges What makes this chain special? ▶ Complicated business process (many legal rules, privacy) ▶ Many registrations: de facto inconsistent ▶ Most “customers” not willing to co-operate (try to fraude) ▶ Chain-consistent test data (esp. fingerprints) ▶ Many different stakeholders, physical distance, living on their own islands ▶ Uncertainty / problems in the Police organization ▶ Heavy message transfer, complex message broker functionality 12
  • 13. | Gerlof Hoekstra| © Atos Assembling a Test Team Testers vs subject matter experts, formal vs informal ▶ Subject matter matters! ▶ Process knowledge ▶ Know the organization ▶ Middleware & message transfer ▶ Involved in implementation ▶ Police officers ▶ Dactylocopists ▶ Application manager ▶ Justice chain expert ▶ Free thinkers ▶ Not afraid to experiment ▶ No dogmas ▶ Excellent observers ▶ Easy making contact 13
  • 14. | Gerlof Hoekstra| © Atos Test specification: formal part Keep it 'simple' and compact; connect to the world of the stakeholders ▶ Process description(test basis) ▶ Classification tree (test overview) ▶ User scenarios (details) 14
  • 15. | Gerlof Hoekstra| © Atos Test specification: informal part Make optimum usage of subject matter experts: test charters Theme based: ▶ Chain components not available ▶ Attribute validation ▶ Unstructured addresses ▶ (Partly) unknown birth dates ▶ Message transfer anomalies ▶ False ID-documents ▶ Bad quality fingerprints ▶ Residential address unknown ▶ Inconsistencies between registrations Role based: ▶ Test with customs employee ▶ Test met prison employee 15
  • 16. | Gerlof Hoekstra| © Atos Formal vs informal Dealing with the quality 'watchdog' Dilemma:  versus  “How Dare You Apply Exploratory Testing” Solution: ▶ Have a small, but well thought-out predefined test set ▶ Identify exploratory test charters in advance ▶ Use simple check lists ▶ Have a good relation with QA management ▶ Explain why you are doing it like this ▶ Report what you have done 16
  • 17. | Gerlof Hoekstra| © Atos Test data Accepting & dealing with restrictions ▶ Which fingerprints do I use? ▶ How to load these into a far far away foreign test database? ▶ How to clean up test data? ▶ How to get ID documents for test? ▶ Solution: puzzling, know what you want, preparation, know the right people! Direct database inserts Insert data with application Use available test data Cleanup possible? 17
  • 18. | Gerlof Hoekstra| © Atos From waterfall to iterative Initial plan phase 2 ▶ Newly build front-end ▶ Monthly iterations ▶ Back-end system changes ▶ Every party has his own process & implementation schedule ▶ E2E test starts when all components are ready ▶ mitigate late integration bugs by: ▶ Gerlof reviews/monitors all supplier tests ▶ 1:1 interface tests (peer-to-peer) ▶ stub usage 18
  • 19. | Gerlof Hoekstra| © Atos From waterfall to iterative Project in trouble! ▶ Front-end ▶ SCRUM in name only ▶ No business value delivered yet ▶ Back-end system changes ▶ DONE ! (?) ▶ Some even deployed in production ▶ Reviews, peer-to-peer testing, stubs: ▶ I did what I could .... ▶ No time, complex, error prone, ... 19
  • 20. | Gerlof Hoekstra| © Atos From waterfall to iterative Re-organizing the project ▶ Front-end ▶ Monthly sprints, now with business focus ▶ My struggle to realize an early & iterative E2E test ▶ 'No time/resources for linking to integrated test environment' ▶ 'Trust the stubs' ▶ 'You disturb the developers' ▶ 'In the end, Iterative E2E testing takes more time' ▶ 'No need, it should work, we reviewed everything’ 20
  • 21. | Gerlof Hoekstra| © Atos From waterfall to iterative Finally, from sprint 4 we were able to E2E test, AND HECK WE DID !!!!! 21
  • 22. | Gerlof Hoekstra| © Atos Some interesting findings Typical integration problems ▶ Simple 'No Hits' leading to fatal errors at the front-end application ▶ Legacy systems not following agreed standards ▶ Un- or incorrectly documented interfaces 22
  • 23. | Gerlof Hoekstra| © Atos Getting defects fixed Many independent parties, legacy systems ▶ Sometimes very hard & frustrating ▶ “Not our fault” ▶ “On our side, everything works fine!” ▶ “Yeah, that’s exactly how it is supposed to work” ▶ “Please call our service desk, they will help you” ▶ “Next release we will fix the problem (in 6 months)” ▶ Process ▶ Responsibility for defect fixing !! ▶ Diagnose (zie processchema) ▶ Solution alternatives ▶ Decision making ▶ Fixing & retesting ▶ Follow-up 23
  • 24. | Gerlof Hoekstra| © Atos What did we learn Co-operation ▶ Getting into contact / building relations / acquire business chain knowledge is crucial ▶ (From time to time) working from 1 location adds tremendous value ▶ End users / monitoring / troubleshooting / architect 24
  • 25. | Gerlof Hoekstra| © Atos What did we learn Early integration ▶ Perform E2E chain testing during the sprints ▶ Do not postpone difficult challenges; if something seems difficult, do it as early as possible! ▶ If you really want something, you can arrange it! ▶ Stubs can be useful, but “Nothing beats the real thing” 25
  • 26. | Gerlof Hoekstra| © Atos What did we learn Test specification ▶ Some carefully specified test scenarios are useful, but do not completely rely on pre- defined test cases ▶ Make a mix between formal techniques and exploratory testing and let coincident happen. ▶ Do not underestimate the value of human observation (we revealed many bugs by observing a bit deeper than the test script) ▶ Don't trust application designs, even if reviewed 100 times; ▶ for E2E testing, a application design is no more than a supporting document ▶ Business process = test basis ? A collection of applications that all work as designed does NOT automatically mean a working chain ! 26
  • 27. | Gerlof Hoekstra| © Atos What did we learn Others ▶ Effective reporting ▶ Frequent ▶ Visual ▶ In the stakeholders' language ▶ Forecast ▶ Directly linked to the implementation scenario ▶ Implement testability ▶ Log’s and traces ▶ Alternative input (esp. for fingerprints) ▶ Police officers are great testers! 27
  • 28. | Gerlof Hoekstra| © Atos Result Status summer 2016 ▶ 2.5 years extensive use ▶ Much more asylum seekers than planned ▶ More locations than planned ▶ Users are happy – less work – quicker business process ▶ No messing with ink fingerprints anymore ▶ More crimes solved ▶ Improved maintainability ▶ No more dependency on 1 external supplier ▶ Various enhancements realized ▶ Finally, the maintenance contract is signed 28
  • 29. Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline, BlueKiwi, Canopy the Open Cloud Company, Yunano, Zero Email, Zero Email Certified and The Zero Email Company are registered trademarks of Atos. January 2015. © 2015 Atos. Confidential information owned by Atos, to be used by the recipient only. This document, or any part of it, may not be reproduced, copied, circulated and/or distributed nor quoted without prior written approval from Atos. Thank You ! Contact: M+ 31 6 512 88 478 gerlof.hoekstra@atos.net

Editor's Notes

  1. Do you know this game: whisper in each others ears and the last person has to speak out the message.