The document outlines the audit update from Bryan Horne regarding the regulatory framework for vocational qualifications and apprenticeships. It describes the methodologies used to assess compliance among awarding organizations, including annual statements of compliance, audits, and technical evaluations, while prioritizing higher-risk organizations for scrutiny. The spring audit program and feedback from stakeholders are discussed, highlighting the ongoing commitment to improving compliance and responsiveness in regulation.

1. Audit update
Bryan Horne - Associate Director Standards for Vocational Qualifications and
Apprenticeships
April 2015

2. Areas to cover
What we are aiming to achieve
How we use our risk model
Statement of compliance 2015
Commissioning process
Audits’ place amongst our regulatory ‘tool kit’
Spring audit programme - feedback
What next?

3. What we are aiming to achieve
We use a range of tools to gain assurance about whether
awarding organisations are meeting our requirements and
therefore delivering sufficiently valid qualifications.
Our tools include:
– annual statement of compliance
– audit
– technical evaluation, and
– investigation.
Through a risk-based programme of audit work - we gather
evidence about how well an awarding organisation’s systems,
processes and controls are meeting our regulatory
requirements.
Building a baseline evidence base – allowing for increasingly
focussed activity in areas of highest risk.

4. Lifecycle

5. How we use our risk model
Our risk framework allows us to identify risks to the provision of
valid qualifications from:
– an organisation’s governance, financial structure, internal
processes, people and systems, and/or
– external factors.
We assess risk through:
– qualification portfolio analysis,
– risk assessments against defined risk indicators,
– risk intelligence,
– market and systemic risk analysis, and
– audit, research and investigation outcomes.
Targeting regulatory compliance activity - highest risk
organisations are subject to stronger scrutiny. All have some
form of scrutiny, but it is progressively less intense for lower-
risk organisations.

6. Statement of compliance - 2015
Your annual statement is in accordance with the requirements
set out in Condition B2.
To make your submission refer to the instructions we issued on
2nd April.
The submission window opens 1st June and closes 30th
September.
Your annual statement must be submitted during this window.
We will tell you next month how to access the online form used
to make your statement.
We will use your statement in conjunction with the other
information we hold about you to inform our regulatory
approach.

7. Declaration

8. Details of Non compliance….

9. Commissioning process
Risk model outputs
Standards
directorates determine
priorities
Audit programme
determined
Audits commissioned
and agreed with audit
team
Audits carried out
Audit outputs fed back
to commissioner
Next steps determined

10. Audits’ place amongst our regulatory ‘tool
kit’
Annual statement of compliance
– annual statement of compliance with the Conditions of
Recognition.
Audit
– gathers evidence about how well an awarding organisation’s
systems, processes and controls are meeting our regulatory
requirements.
Technical evaluation
– appropriate research and evaluation methods to assess and make
expert judgements about how well awarding organisations apply
their processes so that qualifications are sufficiently valid.
Investigation
– we may investigate where there is information that leads to a
suspicion of non-compliance with our Conditions of Recognition.

11. Audit approach
We start with the assumption that an awarding organisation is
fully compliant with the Conditions.
An audit tests how this is achieved by examining processes,
systems and resources.
Because our Conditions are ‘outcome focussed’ – we may
sometimes wish to examine products as well as the systems in
place to produce them.
A risk driven approach.
Where we have a concern, we may provide feedback – an
awarding organisation can then consider how to act upon this.
In some cases – we may not provide feedback.

12. Testing systems and products
Technical
evaluation of
assessment
Tests:
- systems
- processes
- resources
Credible?
Used?
Assessment
procedure?
Used?
Will do this Will do this
Yes
Fit for
purpose
qualification
Yes
Invalid
assessment
Invalid
assessment
NoYes
Unlikely to
deliver valid
assessment
No
Unlikely to
deliver valid
assessment
No
Evaluates
- assessment
procedures
NoYes

13. Spring audit programme - 1
Approximately 25 organisations selected
Testing how organisations comply with:
– processes for obtaining User support (Condition E1.3),
– levels of resources and arrangements (Condition A5), and
– assessment design (Condition E4.1 and E4.2).
When the Regulatory Compliance team receives a commission
to undertake an audit - awarding organisation will be notified of
the following:
– notification Regulatory Compliance has received the
commissioned work,
– the Commissioner,
– the regulatory purpose of the audit,
– the key lines of enquiry (KLOE) for the audit,
– the relevant Condition(s) to be audited, and
– the timetabling for the audit activity.

14. Spring audit programme - 2
Audit work
Supplementary
information
provided
(+ 5 days)
Factual
checking
(+ 10 days)
Evidence
record agreed
(+ 15 days)
Findings
record to
Commissioner
(+ 25 days)
Next steps
communication
(+ 30 days)
Representation
Report agreed
Findings analysed
Audit
Checking
Next steps

15. Spring audit programme – feedback so far
Good points
Audits were handled well - two/three weeks notice of an audit.
AOs invited to select two qualifications for audit - in this
programme we selected two, but provided notice.
The audit teams were friendly and approachable - AOs noted
that the audits took less time than they had anticipated.
To consider
 Audits still ‘work in progress’ with a concern about
inconsistency – feedback ‘on-site’/written.
 Ofqual to provide written report sooner.
 AOs gathered together more evidence than necessary in some
cases.
 AOs would like the opportunity to feedback on their experience
of the audit.

16. What next?
Audit programme will continue – building a baseline evidence
base about how Conditions are complied with.
Supported by outcomes of annual statement of compliance.
Technical evaluation programme will begin.
In some cases – organisations can expect a greater intensity of
audit work.
Where evidence builds and perceived risks become lower – the
intensity of scrutiny will be lower.
If we determine that there has been a breach of our
requirements, we will decide what regulatory action to take.
In some cases we will review our regulatory requirements to see
if they can be improved. When we propose to change
requirements, we will consult awarding organisations and the
public.

17. Questions
Bryan Horne
Associate Director Standards for
Vocational Qualifications and
Apprenticeships, Ofqual