This document provides an update on the formative and impact evaluations of the National Collaborative Outreach Programme (NCOP). It summarizes the following key points: 1) The formative evaluation has included capacity building activities for consortia and a survey of consortia that found new partnerships have been formed and activities would not have been delivered without NCOP support. 2) The impact evaluation participant survey has been finalized and rolled out to over 120,000 students across consortia. Two randomized controlled trials are also confirmed. 3) Work is underway to derive statistical controls for the evaluation through quasi-experimental methods and linking data to secondary sources. 4) Early observations from the consortia

1. DR. ABIGAIL DIAMOND
Managing Director CFE Research
Emerging findings from the Impact and
Formative evaluations of the NCOP

2. Outline of the work undertaken
in both strands of the NCOP
evaluation

3. Formative evaluation – progress to date
◦ Capacity building activities (two webinars, best practice
guidelines on surveying participants).
◦ Gathering together evaluation frameworks/logic chains
ahead of the third Webinar. This will inform the meta-analysis
of local evaluation evidence.
◦ Consortia survey some headline findings which I will be
sharing with you shortly.
◦ Some capacity building and process evaluation work
complete.

4. Impact evaluation – Participant survey update
◦ Participant Survey – baseline survey finalised (and optional Part B
question bank) and consortia roll-out has commenced. Baseline
surveying should be taking place when your participants engage
and or in September. At least 12 consortia have their surveys
underway now. The survey will be repeated every September and
the baseline findings will inform our 12-month report to HEFCE.
◦ 10 consortia opted for Option A so will be administering the survey
themselves, 19 opted for Option B that is the CFE administered
survey. At least 17 consortia are either using a paper based version
or a combination of paper and online.

5. Impact evaluation –update continued
◦ Collectively you have so far estimated that we can expect the
participant survey to reach at least 120,000 young people.
◦ 9 consortia have requested or made use of a flyer that we
have developed to explain the participant survey to schools.
Let us know if you want it too.
◦ Quasi-experimental methods: good practice guide prepared
for consortia to advise on how to derive comparison groups.
Please get in touch with us if you are planning to derive
comparison groups and undertake quasi-experimental
activity.

6. Update on the quasi-
experimental and experimental
work

7. Update on quasi-experimental work
◦ Guidelines for constructing counterfactual groups have been
shared with consortia now, please get in touch if you intend
to do this.
◦ Quasi-experimental work isn’t planned across the whole
national impact evaluation – our intention will be to derive a
statistical control in the second phase of the NCOP to link
with other secondary data in the form of HESA data, as it is
unlikely that UCAS data will be available – this will only be
possible for Year 12-13.
◦ Consortia quasi-experimental work will inform impact of the
softer/proxy measures of impact via the participant survey.

8. Update on experimental work
RCTs – two flagship RCTs confirmed.
1) Light-touch text-messaging NCOP outreach activity
involving the NEACO consortium. Despite the intervention
being light-touch, it will be sustained over time so therefore
meets the NCOP requirements of intensive targeting.
2) Intensive outreach activity in the form of e-Mentoring
(Brightside) with the SUN consortium.
A third RCT will be planned for the next year in relation to a
summer school.

9. National overview and early
observations

10. Early observations – consortia survey findings
• We received 325 full responses to the consortia survey. All
consortia leads responded. The number of responses per
consortium varied between 2 and 32. Respondent roles:
29
6
10
135
143
2
0 20 40 60 80 100 120 140 160
NCOP Lead/Director at Lead Higher Education Institution
NCOP Chair
Member of the consortium's governing board
Employee of the Lead Higher Education Institution
Employee of a partner institution or organisation
Other

11. New posts created as part of NCOP in lead
organisations
Lead organisations
• Consortia have created between 2 and 29 new full-time equivalent
positions that have been specifically created and or funded by NCOP.
• Most posts in lead organisations are new posts: Average is 10.1 FTE
new posts, compared with 10.7 FTE posts overall (= 87% new posts)
Partner organisations
• Consortia have created between 1 and 35 new full-time equivalent
positions.
• Most posts in partner organisations are also new posts: Average 8.4
FTE new posts, compared with 9.2 FTE posts overall (= 91% new
posts)

12. The roles of those within NCOP consortia include the
following.
28
26
25
26
28
9
Management Delivery Evaluation Monitoring Administration Other role

13. Strategic priorities of NCOP consortia
6.3
6.2
5.7
5.7
5.5
5.4
5.4
5.4
5.4
1 2 3 4 5 6 7
Understand NCOP aims & objectives
Understand consortium aims & objectives
Consortium delivery plans outline how aims & objectives will be met
Consortium aims & objectives align with local area needs
NCOP will contribute to achievement of organisation's wider strategic priorities
Clear targets have been set to measure progress
Consortium aims & objectives align with those of organisation
All partners support the consortium vision
Other consortium members understand consortium aims & objectives

14. Ensuring partners understand NCOP strategic priorities
Consortia report that better
communication and collaboration is
needed:
“More guidance from HEFCE
around where NCOP slots into
the WP narrative and
complements OFFA”
“More support to promote the
programme at a national level,
particularly to schools and
colleges”
“Sharing of information and
working more collaboratively.”
“…continued open dialogue
and communication are vitally
important to ensure continued
engagement both at a
consortium and national level.”
“Clearer communication
between Operational and
Strategic groups”

15. Perceptions of positive impact of NCOP
2.4
2.4
2.3
2.3
2.0
1 2 3
The quality of IAG for pupils in target wards
Access to IAG for pupils in target wards
The likelihood that able pupils will consider progressing to HE who
would have otherwise only considered alternative options
Teachers' knowledge and awareness of the options for students in HE
Parents' knowledge and awareness of the options for students in HE

16. Perceptions of other impact of NCOP
6.0
5.7
5.6
5.4
5.4
3.3
1 2 3 4 5 6 7
NCOP adds value to existing activities within my organisation
We have developed new partnerships with schools/colleges
We have developed new partnerships with institutions in FE & HE
We have developed new partnerships with local organisations
We would have delivered WP activities without support from NCOP
NCOP duplicates existing activities within my organisation

17. Consortia’s perceptions of the impact of NCOP
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
The consortium is on track to achieve its goals overall
NCOP is on track to achieve its goals overall at a national level
1 2 3 4 5 6 7 I don't know It's too early to say

18. Relationships with consortia and
work taking place locally

19. Relationships with consortia
• We have a case management approach established now
and hopefully you are all aware of who within the evaluation
team is your case manager. The case managers are in
regular contact with their consortia. If you are not aware
then please do ask a member of the team today or email us
at NCOP@cfe.org.uk
• The NCOP Jiscmail NCOP@JISCMAIL.AC.UK also seems
to be working effectively as a means to seek feedback.
• We are also in the process of creating a password protected
area of the website for consortia to share materials online.

20. Across all consortia, there is a total of 345 core members
and 504 wider members
109
110
47
20
10
28
9
12
24
72
296
21
11
23
42
15
0 50 100 150 200 250 300 350
HEIs (including Lead Organisation)
Further Education Colleges
Schools
Businesses
Local Enterprise Partnerships
Local Authorities
Community or voluntary organisations
Other
Core members
Wider members

21. To what extent do you agree with the following statements
about partnership working in your consortium?
6.2
6.1
6.0
5.6
5.3
5.3
5.2
5.1
4.9
1 2 3 4 5 6 7
I understand how my role contributes to the work of the consortium
I understand how my organisation contributes to the achievement of the
consortium's aims and objectives
The consortium is comprised of appropriate partner organisations
I have a positive and trusting relationship with other partners within my
consortium
The roles and responsibilities of the partner organisations are clear
Tasks are appropriately assigned to organisations within the consortium
The workload is shared fairly between the partners, based on the level of
funding provided
Duplication between partners is avoided
Partners effectively communicate with each other

22. To what extent do you agree with the following
statements about the leadership of your consortium?
6.0
5.8
5.7
5.6
5.6
5.5
5.1
1 2 3 4 5 6 7
Senior leaders within my organisation are supportive of my role
within NCOP
Senior leaders within my organisation understand the goals of
NCOP
Effective measures are in place to evaluate the impact of the
consortium's activities
Other colleagues within my organisation are supportive of NCOP
activities
The Lead Organisation is providing effective leadership
The Lead Organisation communicates effectively with consortium
members
Senior leaders within my organisation prioritise NCOP activities

23. Work taking place locally–consortia survey findings
• All 29 consortia plan to use student interns/ambassadors (or
the like) to deliver or support part of NCOP.
• Numbers of consortia who envisage that student
interns/ambassadors will get involved in:
10
11
5
6
1
Mentoring
Delivering/supporting outreach
Planning and administration support
Evaluation support
Other

24. Next Steps

25. Next steps
• Six consortia case-study visits forthcoming in Oct-Nov 2017.
We will be in touch shortly with participating consortia.
• Third Webinar on supporting consortia in developing their
evaluation frameworks/logic chains for their local
evaluations will take place in November 2017.
• Just a reminder that consortia need to share their data-
sharing agreements for signing.
• Please get in touch if you are planning to construct local
control / comparison groups.

26. Any questions?