This document describes a test suite for OAuth and OpenID Connect protocols. It discusses the need for conformance testing to ensure interoperability between authorization servers, clients, and protected resources. The test suite is designed to test the protocols in a multi-party manner using a structured configuration, logging, and modular execution units. It aims to test both normal and error cases to avoid a false sense of security from only testing happy paths. The architecture involves conditions, modules, plans, environments, and logging to API endpoints. The goal is to make the testing fully scriptable and transparent.

1. @justin__richer
FAPI/OB Test Suite
Justin Richer
July 2018
1

2. @justin__richer
Who am I?
• Independent consultant in Boston, USA
• Direct contributor to OAuth2 and OIDC
• Editor of OAuth RFCs 7591, 7592, and 7662
• Software architect for Authlete and Fintechlabs
• Author of OAuth2 In Action
2

3. @justin__richer
Why conformance testing?
3

4. @justin__richer
Interoperability
4
AS
AS
AS
Client
Client
Client
Client

5. @justin__richer
Interoperability?
5
AS
AS
AS
Client
Client
Client
Client

6. @justin__richer
Conformance
6
AS
AS
AS
Conformance
Test Suite

7. @justin__richer
Testing these protocols is tricky
7

8. @justin__richer
Resource
Owner
Authorization
Server
Protected
Resource
Client
Resource owner’s
credentials
Client’s
credentials
Authorization
code
Access token
8

9. @justin__richer
Resource
Owner
Authorization
Server
Protected
Resource
Client
Resource owner’s
credentials
Client’s
credentials
Authorization
code
Access token
How do we fit the test harness in here?
9

10. @justin__richer
End User
Session at the
Relying Party
Identity Provider
Identity Profi le APIRelying Party
(Application)
End User’s Credentials,
Authorization of the Relying Party
ID Token and
Access Token
Access Token and User Information
10

11. @justin__richer
Design goals
• Multi-party protocol testing
• Structured configuration
• Structured logging and results
• Deterministic, modular execution units
• Protect sensitive configuration and results data
• Transparent process
11

12. @justin__richer
We need to handle special cases
• Front-channel requests that may never return
• How things react to intentionally bad requests
– Testing only the happy path leads to a false sense of
security
12

13. @justin__richer
What we test
• UK Open Banking
• FAPI
• HEART
• AS, Client, and RS
13

14. @justin__richer
Architecture
14
Condition
Configuration Environment Event Log
Module Plan

15. @justin__richer
Code structure
15
Plan
Module
Condition

16. @justin__richer
Runtime structure
16
Configuration Environment Event Log
Instance

17. @justin__richer
Using the environment
17
Condition 1
Environment
Condition 2
Write value “foo”
Read value “foo”
Write value “bar

18. @justin__richer
18

19. @justin__richer
19

20. @justin__richer
20

21. @justin__richer
21

22. @justin__richer
22

23. @justin__richer
23

24. @justin__richer
https://gitlab.com/fintechlabs/
fapi-conformance-suite
24

25. @justin__richer
Everything through an API
• Create, start, stop tests
• Retrieve test logs
• Retrieve test plan information
• Fully scriptable
25

26. @justin__richer
Questions?
26

27. @justin__richer
Backup Slides
27

28. @justin__richer
Condition
• Simple
• Reusable
• Deterministic
• Not built on existing OAuth/OIDC libraries
– Easily isolate functionality
– Better for testing for negative behaviors
28

29. @justin__richer
Module
• String a set of conditions together in order
• Manage the state between condition calls
• Determine how condition results map to test results
– E.g., optional conditions can fail in some circumstances
29

30. @justin__richer
Plan
• Allows you to run several related modules with the
same configuration
• Tracks history of module run results
30

31. @justin__richer
Environment
• Holds the full current state of a test run
• Modules and conditions can read and write to it
• Entirely in JSON
31

32. @justin__richer
Configuration
• Anything the test module needs to run
– Server locations
– Secrets and keys
– Certificates
• Can’t be changed once test starts
• Changes for different tests
• Entirely in JSON
32

33. @justin__richer
Event log
• Records results as tests run
• Made of many individual entries
– Timestamp, source, data
• Stored in MongoDB
• Entirely in JSON
33

34. @justin__richer
Image upload
• Stored in the event log (as JSON)
• Capture what happens in the user’s browser
– Error pages
– User interaction
34

35. @justin__richer
Protecting data
• Use OpenID Connect for all logins
• All test instances have an “owner”
• All log entries have an “owner”
35

36. @justin__richer
Open source
• Publicly available on GitLab
• Code can be fully audited (no black boxes)
• Enhancements from several groups to date
• Contributions are welcome!
36