Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Testing the Ministry Of Justice
Biometric Identification System
Gerlof Hoekstra (NL)
| Gerlof Hoekstra| © Atos
Chain testing, Acceptance testing,
End-To-End Business Processes
▶ Multiple stakeholders / appli...
| Gerlof Hoekstra| © Atos
▶ Context
▶ The Ministry Of Justice Intelligence Department
▶ The identification system
▶ The si...
| Gerlof Hoekstra| © Atos
Introduction and context
Ministry Of Justice Intelligence Department
4
| Gerlof Hoekstra| © Atos
▶ Police
▶ Prosecutors
▶ Judges
▶ Investigation services
▶ Prisons
▶ Lawyers
▶ Rehabilitation
▶ ...
| Gerlof Hoekstra| © Atos
The Identification Console
Installed in:
- every Police Office
- central immigration offices
- p...
| Gerlof Hoekstra| © Atos
Hot topic !
7
| Gerlof Hoekstra| © Atos
The Identification Console
Functionality overview
8
| Gerlof Hoekstra| © Atos
The Identification System
Technology
▶ Biometric middleware
▶ Photo-software
▶ NIST
▶ ebXML
▶ Ju...
| Gerlof Hoekstra| © Atos
The Identification System
High level application architecture & test scope
Crime Immigration
Cen...
| Gerlof Hoekstra| © Atos
Initial situation
When I joined the project
▶ Phase 1
▶ Goal: connect to Havank (crime evidence ...
| Gerlof Hoekstra| © Atos
Challenges
What makes this chain special?
▶ Complicated business process (many legal rules, priv...
| Gerlof Hoekstra| © Atos
Assembling a Test Team
Testers vs subject matter experts, formal vs informal
▶ Subject matter ma...
| Gerlof Hoekstra| © Atos
Test specification: formal part
Keep it 'simple' and compact; connect to the world of the stakeh...
| Gerlof Hoekstra| © Atos
Test specification: informal part
Make optimum usage of subject matter experts: test charters
Th...
| Gerlof Hoekstra| © Atos
Formal vs informal
Dealing with the quality 'watchdog'
Dilemma:
 versus 
“How Dare You Apply E...
| Gerlof Hoekstra| © Atos
Test data
Accepting & dealing with restrictions
▶ Which fingerprints do I
use?
▶ How to load the...
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Initial plan phase 2
▶ Newly build front-end
▶ Monthly iterations
▶ ...
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Project in trouble!
▶ Front-end
▶ SCRUM in name only
▶ No business v...
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Re-organizing the project
▶ Front-end
▶ Monthly sprints, now with bu...
| Gerlof Hoekstra| © Atos
From waterfall to iterative
Finally, from sprint 4 we were able to E2E test, AND HECK WE DID !!!...
| Gerlof Hoekstra| © Atos
Some interesting findings
Typical integration problems
▶ Simple 'No Hits' leading to fatal error...
| Gerlof Hoekstra| © Atos
Getting defects fixed
Many independent parties, legacy systems
▶ Sometimes very hard & frustrati...
| Gerlof Hoekstra| © Atos
What did we learn
Co-operation
▶ Getting into contact / building relations / acquire business ch...
| Gerlof Hoekstra| © Atos
What did we learn
Early integration
▶ Perform E2E chain testing during the
sprints
▶ Do not post...
| Gerlof Hoekstra| © Atos
What did we learn
Test specification
▶ Some carefully specified test scenarios are
useful, but d...
| Gerlof Hoekstra| © Atos
What did we learn
Others
▶ Effective reporting
▶ Frequent
▶ Visual
▶ In the stakeholders' langua...
| Gerlof Hoekstra| © Atos
Result
Status summer 2016
▶ 2.5 years extensive use
▶ Much more asylum seekers than planned
▶ Mo...
Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline,
BlueKiwi, Canopy the Open Cloud Company, Yunano, Zero Ema...
Upcoming SlideShare
Loading in …5
×

QA Fes 2016. Gerlof Hoekstra. E2E Testing the Ministry Of Justice Biometric Identification System

225 views

Published on

One of the core processes at the Ministry Of Justice is the identification of crime suspects and foreigners. For this purpose, all police offices are equipped with so called identification consoles, to scan id documents, take finger prints and photos. Based on these information, the system queries many national and European databases to verify if this person is the person he claims to be, check the residence status and if he was involved in any crime. This presentation shares the experiences of testing this entire chain of systems, end-to-end.

Aspects covered are:
- How to assemble a solid test team
- How to specify test scenarios
- How to manage test data
- How to effectively report to stakeholders
- Lessons learnt
All is illustrated with real-world examples and anecdotes.

Published in: Education
  • Be the first to comment

  • Be the first to like this

QA Fes 2016. Gerlof Hoekstra. E2E Testing the Ministry Of Justice Biometric Identification System

  1. 1. Testing the Ministry Of Justice Biometric Identification System Gerlof Hoekstra (NL)
  2. 2. | Gerlof Hoekstra| © Atos Chain testing, Acceptance testing, End-To-End Business Processes ▶ Multiple stakeholders / applications ▶ Complex ▶ Publicly visible ▶ Politically sensitive ▶ Previous attempt(s) failed Introduction and context Introducing myself… Gerlof Developer Designer Tester 2
  3. 3. | Gerlof Hoekstra| © Atos ▶ Context ▶ The Ministry Of Justice Intelligence Department ▶ The identification system ▶ The situation I faced when I joined the project ▶ What makes this chain test challenging ▶ Aspects: ▶ Team ▶ Test specification ▶ Formal / informal testing ▶ Test data ▶ From waterfall to iterative ▶ Findings ▶ Lessons learnt 3
  4. 4. | Gerlof Hoekstra| © Atos Introduction and context Ministry Of Justice Intelligence Department 4
  5. 5. | Gerlof Hoekstra| © Atos ▶ Police ▶ Prosecutors ▶ Judges ▶ Investigation services ▶ Prisons ▶ Lawyers ▶ Rehabilitation ▶ Customs offices ▶ Debt collection agency ▶ … ▶ … The judicial chain 5
  6. 6. | Gerlof Hoekstra| © Atos The Identification Console Installed in: - every Police Office - central immigration offices - prisons - Schiphol airport 6
  7. 7. | Gerlof Hoekstra| © Atos Hot topic ! 7
  8. 8. | Gerlof Hoekstra| © Atos The Identification Console Functionality overview 8
  9. 9. | Gerlof Hoekstra| © Atos The Identification System Technology ▶ Biometric middleware ▶ Photo-software ▶ NIST ▶ ebXML ▶ Jubes ▶ ESB ▶ External Police Broker ▶ NORA ▶ JAB ▶ Ministry of Justice security standards ▶ Photo standards ▶ Fingerprint standards ▶ GBA character set ▶ 7x24 ▶ 30.000 users 9
  10. 10. | Gerlof Hoekstra| © Atos The Identification System High level application architecture & test scope Crime Immigration Central message broker 10
  11. 11. | Gerlof Hoekstra| © Atos Initial situation When I joined the project ▶ Phase 1 ▶ Goal: connect to Havank (crime evidence traces) ▶ Earlier attempt failed ▶ New project leader ▶ Well documented business process ▶ Integrated test environment available ▶ Phase 2: ▶ Build & implement a completely new front-end ▶ Connect to foreigner/immigration chain ▶ Validate ID-documents ▶ Optimize business process (scan once) 11
  12. 12. | Gerlof Hoekstra| © Atos Challenges What makes this chain special? ▶ Complicated business process (many legal rules, privacy) ▶ Many registrations: de facto inconsistent ▶ Most “customers” not willing to co-operate (try to fraude) ▶ Chain-consistent test data (esp. fingerprints) ▶ Many different stakeholders, physical distance, living on their own islands ▶ Uncertainty / problems in the Police organization ▶ Heavy message transfer, complex message broker functionality 12
  13. 13. | Gerlof Hoekstra| © Atos Assembling a Test Team Testers vs subject matter experts, formal vs informal ▶ Subject matter matters! ▶ Process knowledge ▶ Know the organization ▶ Middleware & message transfer ▶ Involved in implementation ▶ Police officers ▶ Dactylocopists ▶ Application manager ▶ Justice chain expert ▶ Free thinkers ▶ Not afraid to experiment ▶ No dogmas ▶ Excellent observers ▶ Easy making contact 13
  14. 14. | Gerlof Hoekstra| © Atos Test specification: formal part Keep it 'simple' and compact; connect to the world of the stakeholders ▶ Process description(test basis) ▶ Classification tree (test overview) ▶ User scenarios (details) 14
  15. 15. | Gerlof Hoekstra| © Atos Test specification: informal part Make optimum usage of subject matter experts: test charters Theme based: ▶ Chain components not available ▶ Attribute validation ▶ Unstructured addresses ▶ (Partly) unknown birth dates ▶ Message transfer anomalies ▶ False ID-documents ▶ Bad quality fingerprints ▶ Residential address unknown ▶ Inconsistencies between registrations Role based: ▶ Test with customs employee ▶ Test met prison employee 15
  16. 16. | Gerlof Hoekstra| © Atos Formal vs informal Dealing with the quality 'watchdog' Dilemma:  versus  “How Dare You Apply Exploratory Testing” Solution: ▶ Have a small, but well thought-out predefined test set ▶ Identify exploratory test charters in advance ▶ Use simple check lists ▶ Have a good relation with QA management ▶ Explain why you are doing it like this ▶ Report what you have done 16
  17. 17. | Gerlof Hoekstra| © Atos Test data Accepting & dealing with restrictions ▶ Which fingerprints do I use? ▶ How to load these into a far far away foreign test database? ▶ How to clean up test data? ▶ How to get ID documents for test? ▶ Solution: puzzling, know what you want, preparation, know the right people! Direct database inserts Insert data with application Use available test data Cleanup possible? 17
  18. 18. | Gerlof Hoekstra| © Atos From waterfall to iterative Initial plan phase 2 ▶ Newly build front-end ▶ Monthly iterations ▶ Back-end system changes ▶ Every party has his own process & implementation schedule ▶ E2E test starts when all components are ready ▶ mitigate late integration bugs by: ▶ Gerlof reviews/monitors all supplier tests ▶ 1:1 interface tests (peer-to-peer) ▶ stub usage 18
  19. 19. | Gerlof Hoekstra| © Atos From waterfall to iterative Project in trouble! ▶ Front-end ▶ SCRUM in name only ▶ No business value delivered yet ▶ Back-end system changes ▶ DONE ! (?) ▶ Some even deployed in production ▶ Reviews, peer-to-peer testing, stubs: ▶ I did what I could .... ▶ No time, complex, error prone, ... 19
  20. 20. | Gerlof Hoekstra| © Atos From waterfall to iterative Re-organizing the project ▶ Front-end ▶ Monthly sprints, now with business focus ▶ My struggle to realize an early & iterative E2E test ▶ 'No time/resources for linking to integrated test environment' ▶ 'Trust the stubs' ▶ 'You disturb the developers' ▶ 'In the end, Iterative E2E testing takes more time' ▶ 'No need, it should work, we reviewed everything’ 20
  21. 21. | Gerlof Hoekstra| © Atos From waterfall to iterative Finally, from sprint 4 we were able to E2E test, AND HECK WE DID !!!!! 21
  22. 22. | Gerlof Hoekstra| © Atos Some interesting findings Typical integration problems ▶ Simple 'No Hits' leading to fatal errors at the front-end application ▶ Legacy systems not following agreed standards ▶ Un- or incorrectly documented interfaces 22
  23. 23. | Gerlof Hoekstra| © Atos Getting defects fixed Many independent parties, legacy systems ▶ Sometimes very hard & frustrating ▶ “Not our fault” ▶ “On our side, everything works fine!” ▶ “Yeah, that’s exactly how it is supposed to work” ▶ “Please call our service desk, they will help you” ▶ “Next release we will fix the problem (in 6 months)” ▶ Process ▶ Responsibility for defect fixing !! ▶ Diagnose (zie processchema) ▶ Solution alternatives ▶ Decision making ▶ Fixing & retesting ▶ Follow-up 23
  24. 24. | Gerlof Hoekstra| © Atos What did we learn Co-operation ▶ Getting into contact / building relations / acquire business chain knowledge is crucial ▶ (From time to time) working from 1 location adds tremendous value ▶ End users / monitoring / troubleshooting / architect 24
  25. 25. | Gerlof Hoekstra| © Atos What did we learn Early integration ▶ Perform E2E chain testing during the sprints ▶ Do not postpone difficult challenges; if something seems difficult, do it as early as possible! ▶ If you really want something, you can arrange it! ▶ Stubs can be useful, but “Nothing beats the real thing” 25
  26. 26. | Gerlof Hoekstra| © Atos What did we learn Test specification ▶ Some carefully specified test scenarios are useful, but do not completely rely on pre- defined test cases ▶ Make a mix between formal techniques and exploratory testing and let coincident happen. ▶ Do not underestimate the value of human observation (we revealed many bugs by observing a bit deeper than the test script) ▶ Don't trust application designs, even if reviewed 100 times; ▶ for E2E testing, a application design is no more than a supporting document ▶ Business process = test basis ? A collection of applications that all work as designed does NOT automatically mean a working chain ! 26
  27. 27. | Gerlof Hoekstra| © Atos What did we learn Others ▶ Effective reporting ▶ Frequent ▶ Visual ▶ In the stakeholders' language ▶ Forecast ▶ Directly linked to the implementation scenario ▶ Implement testability ▶ Log’s and traces ▶ Alternative input (esp. for fingerprints) ▶ Police officers are great testers! 27
  28. 28. | Gerlof Hoekstra| © Atos Result Status summer 2016 ▶ 2.5 years extensive use ▶ Much more asylum seekers than planned ▶ More locations than planned ▶ Users are happy – less work – quicker business process ▶ No messing with ink fingerprints anymore ▶ More crimes solved ▶ Improved maintainability ▶ No more dependency on 1 external supplier ▶ Various enhancements realized ▶ Finally, the maintenance contract is signed 28
  29. 29. Atos, the Atos logo, Atos Consulting, Atos Worldgrid, Worldline, BlueKiwi, Canopy the Open Cloud Company, Yunano, Zero Email, Zero Email Certified and The Zero Email Company are registered trademarks of Atos. January 2015. © 2015 Atos. Confidential information owned by Atos, to be used by the recipient only. This document, or any part of it, may not be reproduced, copied, circulated and/or distributed nor quoted without prior written approval from Atos. Thank You ! Contact: M+ 31 6 512 88 478 gerlof.hoekstra@atos.net

×