1. The evaluation strategy outlines DEEWR's plan to evaluate the effectiveness, efficiency and accessibility of the new Disability Employment Services (DES) model that began in 2010. 2. The evaluation will compare the new DES model to the previous model (DEN/VRS) and assess whether the key changes in DES have improved outcomes. It will also assess the overall impact of DES on labour market participation of people with disability. 3. The strategy identifies four main questions that will guide the evaluation: 1) Whether all eligible job seekers receive timely and appropriate services; 2) Whether access to services is similar across different groups; 3) Whether DES has led to increased skills/sustainable employment and satisfaction; 4)

1. EVALUATION STRATEGY FOR DISABILITY EMPLOYMENT
SERVICES 2010–2012
Employment Monitoring and Evaluation Branch
1 October 2010
1

2. Evaluation Strategy for
Disability Employment Services 2010–2012
A new model for disability employment services.......................................11
Employer Incentives.........................................................................12
Evaluation requirement....................................................................13
1. Participation in Disability Employment Services...................................14
2. Services for participants...............................................................15
3. Helping people from culturally and linguistically diverse backgrounds........16
4. Helping Indigenous Australians........................................................17
5. Supporting the transition from school to work.....................................17
6. Responding to the needs of employers...............................................17
7. Servicing remote areas.................................................................18
8. Effectiveness of service provision ....................................................18
9. Efficiency of service provision.........................................................19
10. Administrative load....................................................................20
Program take-up and usage................................................................21
Sustainable employment...................................................................21
Service quality...............................................................................21
Effectiveness and efficiency ..............................................................21
Program outcomes...........................................................................22
Dynamics of Australian Income Support and Employment Services Survey........26
Post Program Monitoring Survey...........................................................26
Service Quality Monitoring Program......................................................26
DEEWR and Centrelink administrative data.............................................27
Special purpose data collections..........................................................27
Other data sources..........................................................................27

3. Version Control
Date Status Version Author Version Details
28/06/201
0
Draft 0.1 SA0307 First draft for Working Group.
08/07/201
0
Draft 0.2 SA0307 Second draft for Working Group.
22/07/201
0
Draft 1.0 SA0307 Incorporates Working Group comments. For
clearance to circulate.
31/08/201
0
Draft 1.1 CH2669 Incorporates comments from DES Reference
Group and government agencies.
1/10/2010 Final 1.2 CH2669 Approved by Employment Steering
Committee.
3

4. Glossary
Most entries in this glossary are taken from the Disability Employment Services Deed 2010-
2012.
Commencement
Disability Management Service
Effectiveness
Efficiency
The time at which the DES Provider has recorded the
completion of the initial interview with the participant
on DEEWR information systems.
Services provided to a Program A participant under
Chapter 5 of the Deed.
The extent that services help the major subgroups of
participants noted in this strategy to achieve Job
Placements and Outcomes, and remain in Employment
through the provision of Ongoing Support (see also
Disability Employment Services Deed 2010-2012: Key
Performance Indicator 2).
Time taken from commencement to the achievement
of Employment Outcomes (see also Disability
Employment Services Deed 2010-2012: Key
Performance Indicator 1).
Employer An entity that has the legal capacity to enter into a
contract of employment with a participant.
Employment Support Service Services provided to a Program B participant under
Chapter 5 of the Deed.
Job Capacity Assessment A holistic assessment of a participant’s participation
barriers and current work capacity, undertaken by a
JCA Provider.
Job seeker ‘Job seeker’ is used in this document to refer
specifically to participants in the Employment
Assistance phase (looking for work). Where used in
reference to a survey, for example, ‘job seeker survey’,
the meaning is the same as for ‘participant’.
National Disability Recruitment
Coordinator
A service under the DES program that strengthens links
between large employers and DES providers to
promote the employment of people with disability
Ongoing Support Services that a participant may receive while they are
in employment, unsubsidised self-employment, an
apprenticeship or a traineeship after a 26 Week
Employment Outcome or Job in Jeopardy Outcome and
until they exit. There are three levels of Ongoing
Support: Flexible, Moderate, and High.
4

5. Ongoing Support Assessment
Open employment
An assessment by an Ongoing Support Assessor as to a
participant’s need for Ongoing Support
Employment in the open (competitive) labour market
Participant
Post Program Monitoring
A person who is registered with a Disability
Employment Services provider, either Disability
Management Service or Employment Support Service
as the context requires. A participant can be
unemployed (i.e. a job seeker) or employed
A survey of program participants that produces post-
program outcome and satisfaction measures.
Provider
Referrral
Registration
A provider of services under the Disability Employment
Services Deed 2010–2012.
The referral of a person to a DES Provider by
Centrelink, DEEWR or a JCA Provider.
The act of registering the creation or activation of a
participant’s record on DEEWR information systems.
5

6. Abbreviations
ABS
CALD
DEEWR
Australian Bureau of Statistics
Culturally and linguistically diverse
Australian Government Department of Education, Employment
and Workplace Relations
DEN Disability Employment Network
DES Disability Employment Services 2010-2012
DMS Disability Management Service
EAF Employment Assistance Fund
EPP Employment Pathway Plan
ESS Employment Support Service
JCA Job Capacity Assessment
NDRC National Disability Recruitment Coordinator
OSA
PPM
Ongoing Support Assessment
Post Program Monitoring Survey
SDAC Survey of Disability, Ageing and Carers (ABS survey)
VRS Vocational Rehabilitation Services
6

7. Executive summary
From March 2010, Disability Employment Services are provided under a new model,
governed by the Disability Employment Services Deed 2010–2012. Key changes include:
1 removal of the previous cap on services, so that all eligible job seekers with disability
can access services to help them get and maintain a job
2 providing the right assistance as early as possible, including for school leavers, to
ensure a successful transition to work
3 increased resources for job seekers in remote areas
4 greater emphasis on education, training and skills development
5 three levels of assistance for people who need ongoing support in the workplace
after commencing employment, including those who need irregular or intermittent
support
6 measures to improve responsiveness to employers.
The changes are designed to strengthen a service system founded on the principle of social
inclusion through economic participation.
Government requires an evaluation of the effectiveness, efficiency and accessibility of
Disability Employment Services 2010–2012. This document outlines the strategy to be
followed by the Department of Education, Employment, and Workplace Relations (DEEWR)
in conducting the evaluation. The evaluation will be guided by four overarching questions:
1. Do all eligible job seekers receive timely and appropriate service? The evaluation will
assess whether the removal of caps has enabled services to become demand-driven
and the extent to which participants are satisfied with the quality and intensity of
assistance.
2. Are levels of access to services similar across different groups of job seekers? This
question concerns the use of services by young and mature job seekers, school
leavers, women and men, Indigenous job seekers and job seekers from diverse cultural
and linguistic backgrounds, in different geographic locations, and with different
primary disabilities?
3. Has DES led to increased skills development and skills transfer and sustainable
employment outcomes for participants? This should lead to high levels of satisfaction
with the program as reported by participants and employers.
4. Is the new service model more effective and efficient than the previous model?
The strategy is to evaluate the impact on both program and target populations. First, in
viewing Disability Employment Services as a milestone in program design, the new model
will be compared with the one it replaced to assess whether key changes have indeed made
a difference. Second, the evaluation will attempt to assess the overall impact of Disability
Employment Services on the labour market participation and employment of people with
disability.
7

8. The main information sources are: administrative data from DEEWR systems for
administering the employment services market; Centrelink administrative data relating to
income support; job seeker surveys, including but not limited to the ongoing Post Program
Monitoring Survey and the longitudinal Dynamics of Income Support and Employment
Services survey; and surveys of employers and DES providers. Information gathered through
the department’s service quality monitoring program will also be used. Contextual
population data will be sourced from the Australian Bureau of Statistics Survey of Disability,
Ageing and Carers and related publications.
Twenty key performance indicators from a larger set of performance measures will be used
to summarise the results (see below).
As the evaluation will be conducted over three years, the approach will be sufficiently
flexible to respond to issues that may emerge over the course of the project. New areas of
investigation may be indicated and existing areas could need significant redesign. Another
important point is that it will take time for an accurate picture of the longer-term outcomes
of participants to emerge due to the often lengthy periods in employment assistance.
Preliminary outcomes based on relatively few job placements achieved under the new
program may not be a reliable indication of effectiveness in the area of sustainable
employment. In reporting, the evaluation will need to carefully balance the need for timely
information against the risk of misinformation—caveats may need to be placed on interim
findings.
Findings will be disseminated to relevant areas within the department to inform 2012
contracting arrangements. The evaluation will be completed with a final report in the
2012–13 financial year. Subject to Ministerial agreement the report may be made publicly
available on the DEEWR internet site.
The evaluation strategy was developed in consultation with the Disability Employment
Services Reference Group. The department thanks members of the Reference Group,
particularly the Disability Employment Services Evaluation Strategy Working Group
subcommittee, for advice during strategy development.
8

9. Evaluation Key Performance Indicators
Objective Key Performance Indicator Key comparisons
Timely and
appropriate service
1. Numbers of referrals and commencements
per month
2. Proportion of referrals that result in the
participant commencing in the
recommended service
 DES v DEN/VRS
 Primary disability
groups
(ESS v DEN)
Equity of access to
services
3. Program participants as a proportion of
income support population
 DES v DEN/VRS
 DES participant
subpopulations
Participants receive
skills development
and skills transfer
Program meets the
needs of employers
4. Proportion of DES participants who receive
training/skills development from or
through their DES provider
5. Level of participant satisfaction with
training and skills development provided
by their DES provider
6. Level of employer agreement that DES
providers refer job applicants with relevant
work skills and abilities
 Funding levels
 Primary disability
groups
 Eligible school leavers
More effective
services
7. Proportion of participants who commence
in employment during their time in the
program
8. Rate of attrition, total and service-related
9. Proportion of exited participants who are
employed 3 months after leaving the
program
10.(i) Number and proportion of Ongoing
Support or Maintenance participants who
exit as independent workers and remain
employed 3 months after exit; (ii) Number
and proportion of Ongoing Support or
Maintenance participants who remain
employed with support at 15 months after
entering Ongoing Support/Maintenance
11.Level of participant satisfaction with
services
12.Level of employer acceptance of services
13.Proportion of employed participants who
would like to work more hours
 DES v DEN/VRS
 DES participant
subpopulations
14.26 Week Full Outcome rate  DES participant
subpopulations
More efficient
services
15.Proportion of referrals that result in service
commencement within 4 weeks of referral
16.Mean duration between first job placement
and 26 Week Full Outcome
17.Mean duration to exit as independent
worker
18.Mean number of employers that contribute
to 26 Week Full Outcomes
19.Mean cost (program payments) per 26
Week Full Outcome
 DES v DEN/VRS
 Primary disability
groups
9

10. 20.Extent to which providers agree or disagree
that administrative load is lower under
DES.
10

11. Introduction
The Australian Government Department of Education, Employment and Workplace Relations
(DEEWR) funds over 200 organisations to provide employment services for people with
disability, injury or ill health (hereafter shortened to ‘people with disability’). These services
are part of a wider system of services that assist people with disability to produce and
consume and participate as valued members of their communities.
After consideration of public submissions on two discussion papers (Review of Disability
Employment Services and The Future of Disability Employment Services in Australia) the
Government announced a new service model to replace the Disability Employment Network
and Vocational Rehabilitation Services 2006–2009 (DEN/VRS). Changes in the form of
Disability Employment Services 2010-2012 (DES) thus mark the latest stage in the evolution
of employment services under the Disability Services Act 1986.
This strategy describes an approach for assessing the effectiveness, efficiency and
accessibility of Disability Employment Services 2010–2012 both as a milestone in program
design—where the emphasis is on comparing DES to DEN/VRS—and in its totality as an
instrument of the Disability Services Act 1986. While the impact of changes that took effect
from 1 March 2010 is an important focus, elements of the strategy also consider
participation and outcomes in the wider working-age population with disability. In other
words, the strategy is to evaluate impact on the program and target populations.
The evaluation strategy was developed in consultation with the Disability Employment
Services Reference Group. The department thanks members of the Reference Group
subcommittee, the Disability Employment Services Evaluation Strategy Working Group, for
their advice during strategy development.
A new model for disability employment services
Services purchased for the period 1 March 2010 to 30 June 2012 operate under the
Disability Employment Services Deed 2010-2012. The overarching objective is to help
individuals with disability, injury or ill health to secure and maintain sustainable
employment. DES aims to increase the focus on the needs of the most disadvantaged job
seekers and achieve greater social inclusion, to boost employment participation and the
productive capacity of the workforce, and to address skills shortage areas and better meet
the needs of employers (Disability Employment Services Deed 2010-2012, Section 5A.74).
Key features of the new model include:
7 removal of the previous cap on services, so that all eligible job seekers with disability
can access services to help them get and maintain a job
8 providing the right assistance as early as possible, including for school leavers, to
ensure a successful transition to work
9 increased resources for job seekers in remote areas
10 greater emphasis on education, training and skills development
11 three levels of assistance for people who need ongoing support in the workplace
after commencing employment, including those who need irregular or intermittent
support
11

12. 12 measures to improve the responsiveness to employers.
With the introduction of case-based funding in 2005, financial assistance for the provision of
disability employment services was tied to measurable outcomes for people who use the
services. Changes to the counting rules for employment outcomes in DES build on this
outcomes focus by rewarding more sustainable job placements. If achieved, this would
strengthen the program’s ability to help people with disability “achieve positive outcomes,
such as increased independence, employment opportunities and integration in the
community” (Disability Services Act 1986).
DES comprises two distinct demand-driven programs, illustrated in Figure 1:
13 Disability Management Service (DMS), for job seekers with a disability, injury or
health condition who require the assistance of a disability employment service but
are not expected to need long-term support in the workplace
14 Employment Support Service (ESS), for job seekers with a permanent disability and
with an assessed need for more long term regular ongoing support in the workplace.
A single funding level applies to all participants in DMS, while participants in ESS can be
funded at one of two levels depending on the intensity of support they require. For more
information on the services provided under DES, please refer to the Disability Employment
Services Deed 2010-2012.
Figure 1: Overview of Disability Employment Services, 2010-2012
Centrelink Registration
JSCI/JCA
DMS ESS
Employment Pathway Plan
For job seekers with
disability injury or ill
health
For job seekers with permanent disability plus long term ongoing
support requirements
Single funding level Funding Level 1 Funding Level 2
Flexible Ongoing
Support or exit as
independent
worker
Flexible Ongoing
Support or exit as
independent worker
Moderate Ongoing
Support
High Ongoing Support
Employer Incentives
Employer Incentives
The DES programs are supported by a number of employer incentives, including: the
Supported Wage System (SWS) and Wage Subsidy Scheme (WSS); Ongoing Support
12

13. Assessments (OSA); National Disability Recruitment Coordinator (NDRC); the Employment
Assistance Fund (EAF); and JobAccess information service. Employer incentives which are
new or have undergone significant change in DES are OSA and the EAF.
Ongoing Support Assessments
For participants in ESS, Moderate and High Ongoing Support in the workplace will be
available for as long as a participant is assessed by an OSA assessor as requiring it. Ongoing
Support Assessors will independently assess the post-placement support needs of
participants to determine whether and which level of Ongoing Support will best suit the
needs of the participant and their employer. Flexible Ongoing Support is available in both
DMS and ESS. A participant can receive up to six instances of Flexible Ongoing Support in any
26 calendar week period.
Employment Assistance Fund
EAF incorporates the previous Auslan for Employment service and the Workplace
Modifications Scheme into a single fund. The EAF provides assistance to employers and
employees (DES participants) to modify physical work environments and purchase adaptive
technology and Auslan interpreting services. Assistance under the EAF extends to providing
specialist support for employees with mental illness and learning difficulties.
Evaluation requirement
Program evaluation helps to drive continuous improvement and innovation. The
Government requires the development of an evaluation strategy for Disability Employment
Services 2010-2012 and for the new program to be progressively monitored from
implementation. Program evaluation is to consider:
15 how effectively and efficiently job seekers engage with DES and with employers
16 the long and short term outcomes for job seekers including employment, education,
training, reduction in reliance on income support and social inclusion
17 accessibility and availability of services.
In particular, the Government requires an assessment of: the satisfaction of job seekers and
employers with services; outcomes across the different service streams and for groups of
participants including Indigenous Australians and people from culturally and linguistically
diverse backgrounds; pathways to outcomes; and the level of job satisfaction experienced by
job seekers who achieve employment.
A final evaluation report will be delivered in 2012-13. Public release of the report will be at
the Minister’s discretion.
Evaluation objective
The objective is to determine how effectively and efficiently the DES program reaches its
target population and achieves sustainable employment for participants. This can be
expressed in four overarching evaluation questions.
1. Do all eligible job seekers receive timely and appropriate service? The evaluation will
assess whether the removal of caps has enabled services to become demand-driven
13

14. and the extent to which participants are satisfied with the quality and intensity of
assistance.
2. Are levels of access to services similar across different groups of job seekers? This
question concerns the use of services by young and mature job seekers, school
leavers, women and men, Indigenous job seekers and job seekers from diverse
cultural and linguistic backgrounds, in different geographic locations, and with
different primary disabilities?
3. Has DES led to increased skills development and skills transfer and sustainable
employment outcomes for participants? This should lead to high levels of
satisfaction with the program as reported by participants and employers.
4. Is the new service model more effective and efficient than the previous model?
These four main themes will be explored by addressing an expanded set of evaluation
questions contained in this strategy.
Scope of the evaluation
The evaluation covers the two DES programs, DMS and ESS. While it will not
comprehensively review the NDRC, the evaluation will examine its effectiveness in
facilitating sustainable employment for job seekers, raising employer awareness of DES, and
assisting large employers to develop strategies to recruit and retain workers with disability.
Employer incentives which are new or substantially changed, the Employment Assistance
Fund and Ongoing Support Assessment, are included in the evaluation.
Functions not in scope because they are not strictly components of DES or are subject to
other evaluative processes are: Job Capacity Assessment, JobAccess, the Supported Wage
System, the Wage Subsidy Scheme, and the Capacity Building Fund. However, any findings of
import on the interfaces of these components and the DES program will be reported.
This is a program level evaluation and as such will not be concerned with the performance of
individual service providers. Information collected from DES providers and employers will
contribute only to program-level investigations. Performance management (Star Ratings)
and compliance monitoring are conducted separately from evaluation.
Evaluation questions
The Disability Employment Services Evaluation Strategy Working Group agreed on a set of
themes and related questions to address the evaluation objective. Both qualitative and
quantitative data will be used as evidence.
1. Participation in Disability Employment Services
Participation in the two DES programs will be examined using administrative data. Specific
questions for this area of investigation include:
18 Demand driven servicing—the removal of service caps is intended to ensure that all
eligible job seekers receive the most appropriate service—has this been achieved?
14

15. 19 Are participation rates higher in DES compared with the previous DEN/VRS?
Participation rates will be based on an appropriate target—income support—
population.
20 Which referral pathways do people take to engage with DES?
21 Do job seekers commence in the services to which they are referred?
22 How do services received relate to the identified needs of participants?
23 How does target group take-up of specialist providers compare with that of general
DES providers?
The evaluation will construct a time series, commencing in 2003-04, to identify any change
in participation with respect to the target income support population that may be associated
with the introduction of Disability Employment Services. An interrupted time series
approach will allow the evaluation to discern any systematic change in participation distinct
from underlying trend or irregular perturbation.
The evaluation will also examine participation in DES in the context of labour force
participation and employment of people with disability, by age group, primary disability, and
location, using the Australian Bureau of Statistics Survey of Disability, Ageing and Carers
(ABS SDAC) conducted in 2003 (ABS 2004) and 2009 (planned release in April 2011).
A different perspective on participation is whether DES providers are employers of choice for
people with disability. The Deed requires providers to produce and implement a Disability
Employment Strategy—the evaluation will examine progress in this area.
Primary data sources: Administrative data; ABS SDAC.
2. Services for participants
Participants in DES engage in a range of activities designed to provide individualised
assistance to address their employment barriers and to find and maintain employment in
the open labour market. Forms of assistance and the quality of those services will be
examined with reference to specific questions:
24 What outcomes are associated with DMS, ESS Funding Levels 1 and 2 and Ongoing
Support categories of Flexible, Moderate, and High?
25 Are Ongoing Support Assessments correctly identifying participants who require
post-placement support to achieve sustainable employment?
26 Do initial funding level classifications correlate with Ongoing Support Assessments?
27 Is the Ongoing Support model sufficiently flexible for participants, employers and
DES providers?
28 How is the Employment Assistance Fund being used to assist participants?
29 To what extent are participants referred to education and training and in what types
of training do job seekers participate? What governs access to training?
30 Are participants satisfied with the nature of the working alliance established with
the DES provider?
15

16. 31 Are participants satisfied that services are sufficiently tailored to their individual
needs?
32 To what extent do services recognise and act upon factors relevant to the individual
(for example, type of disability, caring responsibilities, age, cultural diversity) when
developing Employment Pathway Plans and delivering assistance?
33 Are participants satisfied with the range and quality of services from their DES
providers? What types of assistance do participants value most?
34 Do DES providers engage with other services to address significant non-vocational
barriers faced by some participants, such as homelessness or mental illness?
Primary data sources: Administrative data, job seeker surveys, DES Provider Survey, in-depth
interviews with providers to be conducted by the evaluation team.
3. Helping people from culturally and linguistically diverse backgrounds
Cultural competence can greatly assist people from diverse backgrounds into education and
employment, who might otherwise not participate. A culturally competent agency is much
better placed to assist people from non-English speaking backgrounds to find and retain
employment.
The evaluation will assess the extent to which DES providers operate culturally competent
services in order to ensure the accessibility and relevance of services to job seekers from
diverse backgrounds. Evaluation questions include:
35 What barriers prevent people from culturally and linguistically diverse (CALD)
backgrounds from engaging with services?
36 To what extent do participants from culturally and linguistically diverse backgrounds
access training and skills acquisition opportunities?
37 How do employment outcomes for this group of participants compare with those for
other participants in the same ESA?
38 What information is available to participants in other languages?
39 How many bilingual or multilingual staff members do DES providers employ?
40 Do employment consultants and support staff have access to cultural awareness
training?
41 What means are available to assist job seekers from CALD backgrounds to engage
with service providers (and exercise choice of provider)?
42 How do service providers support this group of people in the workplace?
43 Are participants satisfied with their level of access to interpreter services?
44 What can be done to increase participation in services by people from CALD
backgrounds?
Primary data sources: job seeker surveys, in-depth interviews with providers to be
conducted by the evaluation team.
16

17. 4. Helping Indigenous Australians
It is thought that many Indigenous Australians with disability are unlikely to engage with
disability services because they do not identify as having disability. High rates of disability
and unemployment are experienced by Australia’s Indigenous people but relatively few
working-age Indigenous Australians use disability employment services.
45 What barriers prevent Indigenous Australians engaging with services?
46 To what extent do Indigenous participants access training and skills acquisition
opportunities?
47 How do the employment outcomes of Indigenous participants compare with those
for non-Indigenous participants in the same ESA?
48 Are DES providers developing Service Level Agreements with Community
Development Employment Projects (CDEP) and Indigenous Employment Program
(IEP) providers to improve servicing for Indigenous job seekers in remote locations?
(The Deed requires them to do so by 31 July 2010). What is being achieved under
these agreements?
The Deed requires providers to produce and implement an Indigenous Employment Strategy
—the evaluation will examine progress in this area.
Primary data sources: discussion forums with Indigenous people, in-depth interviews with
providers to be conducted by the evaluation team, administrative data; DES provider survey.
5. Supporting the transition from school to work
DES is intended to provide the right assistance as early as possible to ensure successful
transitions from school to work. The evaluation will report on the use of DES by school
leavers with disability and the nature of assistance provided to them. A special focus report
on school leavers with disability will deal in depth with the issues surrounding transition
from school to work and the effectiveness of DES in addressing those issues. For example:
49 What outcomes are achieved by school leavers?
50 What support is available for school students to undertake part-time work while
studying?
51 What linkages do DES providers have with state and territory-based transition to
work programs?
Primary data sources: Administrative data; DES Provider survey; ABS SDAC.
6. Responding to the needs of employers
The achievement of employment outcomes depends on strong networks and sound working
relationships between DES providers and employers. The issues covered by this area of
investigation reflect the factors which contribute to the effectiveness of this relationship.
52 How do employers engage with Disability Employment Services, for example, is it
typically that an employment service provider approaches the employer or the other
way around?
53 Once a participant has commenced employment, what forms of assistance are
provided to employers?
17

18. 54 Is the NDRC successful at creating job opportunities for people with disability? The
evaluation will identify job placements and outcomes facilitated by the NDRC and
examine how the NDRC and DES Providers work together.
55 Do DES providers demonstrate approaches to service delivery that are tailored to
the business and workforce needs of individual employers?
56 Do (and how do) DES providers conduct job analysis?
57 How effectively do DES providers convey a job seeker’s capabilities to a prospective
employer?
58 Do employers regard the job seekers referred to them as suitably skilled and with
appropriate capacity for work?
59 Are job placements related to the area of training provided through DES? The
evaluation will look at how often participants enter a field of employment that is
related to training received through a DES provider.
60 Which of the current suite of employer incentives provide the strongest incentive to
employers to hire a candidate with disability?
61 Are employers satisfied with the support they and their employees receive? What
other types of support would they like to receive, including other types of employer
incentives?
Primary data sources: Employer Survey; DES provider survey; administrative data.
7. Servicing remote areas
DES has increased the fee loading for services to remote locations, which is intended to
facilitate the provision of services in areas of significantly higher costs due to geographic
remoteness.
62 To what extent do the service fee multipliers and outcome fee bonuses contribute
to improved participation in DES and to employment outcomes in remote areas?
63 Do participants in remote areas report similar levels of satisfaction with services to
those in other locations?
64 To what extent do participants in remote areas access training and skills acquisition
opportunities?
Primary data sources: Administrative data, job seeker surveys, in-depth interviews with
providers to be conducted by the evaluation team.
8. Effectiveness of service provision
Summary measures of the effectiveness of new services include:
65 Outcomes for participants (see Performance Measures, below).
66 Program attrition.
67 Number and proportion of employed participants who remain in Ongoing Support or
exit Ongoing Support as Independent Workers.
18

19. 68 The quality of employment placements, in terms of earnings, hours worked, job
satisfaction of workers and participants’ views of their long term employment
prospects.
69 Retention in employment with post-placement support under DES Ongoing Support
(compare with DEN Employment Maintenance).
70 Reason for exit from DES.
Standard disaggregation of summary measures of effectiveness:
71 primary disability (five categories: psychiatric; physical/diverse; intellectual; learning
disability/developmental delay/autism spectrum disorder; and sensory/speech)
72 service level (DMS, ESS Funding level 1, ESS Funding level 2)
73 age group
74 sex.
Where possible, summary measures will also be reported by:
75 severity of disability
76 Indigenous status
77 CALD background
78 homeless status.
Statistical modelling will be used to compare the performance of DES with that of DEN/VRS
and examine the correlates of effective service provision.
A logistic regression methodology similar to the one used to assess the impact of the Active
Participation Model1
will be used. Dependent variables (performance measures) will include:
13 and 26 Week Full Outcomes, Education Outcomes, and reduced reliance on income
support. Conversion rates—rates at which job placements convert to employment outcomes
—will also be examined. Independent variables, or covariates, include: person characteristics
(for example, age, sex and primary disability), service characteristics (for example, program
and funding level, geographic location), and exogenous factors such as labour market
conditions. It may not be possible to include some covariates such as CALD background,
Indigenous status, and homelessness because of low numbers of participants.
The study of comparative program performance will be completed when there is sufficient
data on employment outcomes under the new programs, most likely towards the end of
2012.
Primary data sources: Administrative data.
9. Efficiency of service provision
Efficiency can be considered both in terms of the time taken to achieve a result and the cost
involved. The evaluation will examine:
1 See Active Participation Model Evaluation (DEEWR 2008) at www.workplace.gov.au.
19

20. 79 Proportion of referrals that result in service commencement within a specified
timeframe.
80 Mean duration between referral and registration.
81 Mean duration between referral and commencement in services.
82 Proportion of participants who obtain employment over a defined period, for
example, 12 or 18 months.
83 Mean duration between commencement in services and first job placement.
84 Mean duration between first job placement and achievement of 26 Week Full
Outcome.
85 The mean cost in terms of program payments per 26 Week Full Outcome.
86 Cost per outcome.
The need for cautious definition and interpretation of time-based measures is also discussed
below (see Performance Measures).
Primary data sources: Administrative data.
10. Administrative load
Many of the submissions to the Review of Disability Employment Services commented that
administrative load of arrangements under DEN and VRS reduced the capacity of
employment services to deliver assistance. The new model is intended to simplify fee
structures and assessment and referral processes, introduce a less complex and
administratively burdensome funding instrument and provide greater flexibility of assistance
to participants and employers. The evaluation will examine whether DES is achieving these
aims.
87 To what extent (and in what areas, for example, payment structure, DEEWR IT) is the
administrative load for providers changed under DES 2010-12 compared with
DEN/VRS?
88 Has there been an effect on service activity and service quality from any change in
the administrative load?
Primary data sources: DES Provider Survey, in-depth interviews with providers to be
conducted by the evaluation team.
Performance measures
The evaluation questions provide a basis for identifying performance measures and the
required data items. In addressing the above questions the evaluation will produce and
analyse a large set of performance measures using a number of different data sources,
including administrative data, surveys, and qualitative data (see Appendix A).
Because it is unrealistic to expect employment outcomes in the first few months following
referral, reasonable periods of assistance must elapse in advance of determining the time
taken to achieve outcomes. It will be some time before enough data are available on DES
participants who have received an adequate period of assistance to accurately represent the
new program’s comparative performance. Performance measures based on relatively few
20

21. observations are susceptible to the effects of unusual cases, or outliers, particularly for this
type of program. This may limit the level of data disaggregation that is possible in the early
stages of the evaluation. The timing of existing data collections, both internal and external to
the department, will have an impact on the availability of data for evaluation.
The evaluation will produce a comprehensive set of performance measures, covering
program take-up and usage, sustainable employment, service quality, program
effectiveness, and program efficiency. Program-defined outcomes will contribute to the set.
Program take-up and usage
Data on referrals, commencements and length of time in assistance provide measures of
take-up and usage. These measures will be available on the number and share of
participants in the different forms of assistance (DMS, ESS Funding Level 1 and ESS Funding
Level 2). It will be possible to compare take-up and usage with services delivered previously
and between participants with and without participation requirements relating to the
income support they receive. Participation rates will be calculated with reference to the
target income support population in the reference period.
Sustainable employment
Sustainable employment will be measured through the Post Program Monitoring Survey
(PPM) and income support data. The PPM is a sample survey of active participants and
participants who have left the program (see Appendix A). It records the employment status
of people at around three months after exit.
In addition, a cohort of income support recipients who achieve an Employment Outcome in
DES will be followed for the duration of the evaluation to determine how continuing
employment alters the amount and type of income support payments that they receive. This
will give an indication of long-term employment of program participants (apart from income
support data, once a person has left employment services it is not possible to track their
employment status through DEEWR information systems).
Service quality
Data on the types of services delivered to participants together with participant and
employer perspectives on service quality will be collected. Many of these data items will be
comparable with data for earlier employment service contracts and new data designed
specifically for the DES evaluation will be produced. Measures of service quality include
participants’ perceptions of the degree of individual tailoring of assistance by their DES
provider and satisfaction with various aspects of services. Employer satisfaction with
services will also be measured.
Effectiveness and efficiency
Standard measures of effectiveness and efficiency include the distribution of employment
outcomes across participant groups and the proportion of clients who attain an employment
commencement or a defined duration of employment after a reasonable period of
assistance. Care needs to be taken when defining this reasonable period of assistance. Care
also needs to be taken when interpreting length of time in assistance. Relatively longer
periods of assistance do not necessarily imply inefficiency. The program costs of services and
21

22. program costs per employment outcome are relevant to determining the cost-effectiveness
of the new programs compared to previous programs.
 The purchasing cost (to Government) of DES services will be reported in total and per
participant, calculated by dividing the sum of all the fees by the number of
participants. DEEWR costs incurred in program administration will not be calculated.
 Cost per outcome type will be calculated by dividing the total purchasing costs of
assistance by the number of outcomes of that particular type.
Changes over time in these measures will need to be interpreted within the context of any
changes to the economy and any change in the overall profile mix of program participants.
Changes in definitions of outcome variables across programs will be accommodated or
limitations due to non-concordance will be identified.
Program outcomes
Program-defined outcomes generate a set of performance measures that have
conventionally formed the backbone of DEEWR employment program evaluation. These will
play an important role in the evaluation of DES, particularly for comparing the outcomes of
subpopulations of DES participants. Caution should be exercised in comparing the 13 and 26
Week Full Outcome rates of DES with those of DEN/VRS because stricter rules on the
anchoring of job placements in DES mean that these outcomes are not directly comparable.
 Placements in employment—the number of people who commence employment
during their participation in the program.
 Employment status during and following program participation (number employed as
a proportion of total); employment status at three months after leaving the program
is collected by the Post Program Monitoring Survey.
 Paid Full Employment Outcomes, where a provider receives a payment because a
participant remains in employment for 13 or 26 weeks.
 Paid Full Education Outcomes, where a provider receives a payment because a
participant remains in qualifying education for 13 or 26 weeks.
 Paid Pathway Employment Outcomes, where a provider receives a payment because a
participant makes substantial progress toward the achievement of sustainable
employment, such as through substantial part-time work.
 Paid Pathway Education Outcomes, where a provider receives a payment because a
participant makes substantial progress toward the achievement of sustainable
employment, such as through substantial part-time education.
 Exits from employment assistance. This refers to participants who cease to be
registered as requiring assistance.
 Off-benefit outcomes. This refers to the proportion of income support recipients in
the DES program who leave income support. These off-benefit rates are measured,
for example, each month after commencement in assistance.
 Income support reliance, expressed as the proportion of a participant’s total income
(earned income plus income support payments) which is derived from income support
payments, where possible measured monthly during the 12 months before and after
assistance.
22

23.  Paid Bonus Outcomes, where a provider receives a higher payment for a full or
pathway outcome for placing a participant in training related to the local market and
within 12 months the participant achieves an employment outcome directly related to
the training.
 Job in Jeopardy Outcomes.
Key Performance Indicators
Objective Key Performance Indicator Key comparisons
Timely and
appropriate service
1. Numbers of referrals and
commencements per month
2. Proportion of referrals that result in the
participant commencing in the
recommended service
 DES v DEN/VRS
 Primary disability
groups
(ESS v DEN)
Equity of access to
services
3. Program participants as a proportion of
income support population
 DES v DEN/VRS
 DES participant
subpopulations
Participants receive
skills development
and skills transfer
Program meets the
needs of employers
4. Proportion of DES participants who receive
training/skills development from or
through their DES provider
5. Level of participant satisfaction with
training and skills development provided
by their DES provider
6. Level of employer agreement that DES
providers refer job applicants with
relevant work skills and abilities
 Funding levels
 Primary disability
groups
 Eligible school leavers
More effective
services
7. Proportion of participants who commence
in employment during their time in the
program
8. Rate of attrition, total and service-related
9. Proportion of exited participants who are
employed 3 months after leaving the
program
10. (i) Number and proportion of Ongoing
Support or Maintenance participants who
exit as independent workers and remain
employed 3 months after exit; (ii) Number
and proportion of Ongoing Support or
Maintenance participants who remain
employed with support at 15 months after
entering Ongoing Support/Maintenance
11. Level of participant satisfaction with
services
12. Level of employer acceptance of services
13. Proportion of employed participants who
would like to work more hours
 DES v DEN/VRS
 DES participant
subpopulations
23

24. 14. 26 Week Full Outcome rate  DES participant
subpopulations
More efficient
services
15. Proportion of referrals that result in
service commencement within 4 weeks of
referral
16. Mean duration between first job
placement and 26 Week Full Outcome
17. Mean duration to exit as independent
worker
18. Mean number of employers that
contribute to 26 Week Full Outcomes
19. Mean cost (program payments) per 26
Week Full Outcome
20. Extent to which providers agree or
disagree that administrative load is lower
under DES.
 DES v DEN/VRS
 Primary disability
groups
Evaluation governance
This strategy was developed in consultation with the Disability Employment Services
Reference Group. A subcommittee of the Reference Group was established to give advice
during strategy development (terms of reference at Appendix B). The Reference Group will
be informed of progress throughout the life of the project.
DEEWR consulted the departments of Prime Minister and Cabinet, Finance and
Deregulation, and Families, Housing, Community Services and Indigenous Affairs during
strategy development.
The evaluation is to be undertaken by the DEEWR Employment Monitoring and Evaluation
Branch.
Oversight of the evaluation of Disability Employment Services 2010-2012 rests with the
department’s Executive. The Executive approves the Evaluation Strategy, monitors progress,
and accepts evaluation reports.
Reporting
Reporting milestones are shown in Figure 2.
The image of Figure 2 has been removed and replaced with a description of the image to
ensure that the information is available to people with various information accessibility
needs. The image is included in the PDF version of this publication.
Figure 2 depicts the program and evaluation timelines as follows:
 Program Period extends from 1 March 2010 to 30 June 2012
 development of a draft evaluation strategy occurs from 1 June to 30 August 2010
 finalisation of the evaluation strategy occurs by 30 September 2010
 a report on transition to work report (DES Eligible School Leavers) is planned for
December 2011
 the final DES evaluation report is planned for December 2012
24

25. Findings from the evaluation will be progressively disseminated within the department as
they become available.
Preliminary findings will be used by the department to inform contracting from 1 July 2012,
to the extent possible with data available at the time. A final evaluation report will be
delivered in the 2012-13 financial year.
Where appropriate the evaluation will draw on other reviews of labour market interventions
and related programs. This may involve using findings and sharing sources of data. Relevant
reviews include the Job Services Australia evaluation.
Resourcing
Just under $1.6 million has been allocated over 3 years for DES evaluation and monitoring
activities, with more than half of the amount allocated to the 2010–11 financial year.
25

26. Appendix A: Data sources
The evaluation will be supported by a range of data sources and collections which will
provide the information required to address the research questions. The main sources are
briefly described below.
Dynamics of Australian Income Support and Employment Services Survey
The Dynamics of Australian Income Support and Employment Services Survey is a
longitudinal survey of job seekers designed in part to collect data not available from other
sources for the evaluation of the new employment services, including Job Services Australia
and Disability Employment Services. The survey collects information on the experiences and
outcomes of job seekers, including their pathways after they leave assistance and go off-
benefit. It will be conducted over four years beginning in the first half of 2009, involving
eight waves of data collection six months apart with a new cohort of around 11,000
introduced every year to allow detailed analysis of job seeker groups of interest as well as
the wider working age income support population.
To support the DES evaluation, proxy interviews have been introduced to the survey so that
people who have difficulty responding to the survey have the option of choosing a
representative to answer on their behalf.
Post Program Monitoring Survey
Since the late 1980s the department has used the PPM to measure the labour force and
education status of labour market program participants around three months after ceasing
assistance. The PPM has expanded to include active participants and collects information
about study and training activities, tenure of employment, earnings, and satisfaction with
services. The PPM is a valuable source of information on program outcomes and will
continue during DES 2010-12. A mail-out with telephone follow-up is used to collect data.
Special measures have been taken to improve the presentation of the DES PPM for people
with vision impairment or cognitive impairment.
Initially, the PPM data will be a sample of participants who exit services (people are
surveyed three months after exiting). Over time, the PPM will also pick up active participants
surveyed three months after reaching either 12 months in DES assistance or a total of 12
months in assistance with at least 6 months in DES. The earliest that DES PPM results could
include responses from active participants is December 2010 (there is some additional time
for data processing). Data for the final report will include active participants (Employment
Assistance and Ongoing Support) as well as people who have exited the program.
Service Quality Monitoring Program
The Service Quality Monitoring Program collects information on the quality and
responsiveness of services delivered by the employment service and by Centrelink. The
program includes regular surveys of job seekers, employment service providers and
employers. Both computer assisted telephone interviewing and face-to face interviewing
are usually used. Key elements of the program include:
89 Employer Survey—interviews will be held with employers to examine their
relationships with Disability Employment Services. Topics include recruitment
26

27. methods, awareness, understanding and usage of the employment service and
perceptions of the quality of services.
90 Survey of DES Providers—the Survey of Employment Service Providers has been
conducted annually since 1999 and will continue during DES 2010-2012. The 2010
survey of DES Providers is substantially revised to cover the transition to DES and
key features of the new program.
DEEWR and Centrelink administrative data
As part of its purchasing, regulating and managerial role DEEWR collects information on the
activity and performance of employment service providers, including DES providers. This
includes information on fees and Employment Pathway Fund expenditure, client numbers
and characteristics (for example, age, gender, unemployment duration, educational
attainment, and equity group) as well as data on referrals, commencements and outcomes.
As business custodian (under the Protocol for the release of Social Security Information) of
social security information, DEEWR has access to income support data collected by
Centrelink in the process of administering income support, consistent with the Privacy Act
1988 and the Social Security (Administration) Act 1999.
Special purpose data collections
The evaluation budget includes an allowance for the collection of data not otherwise
available through:
91 a set of focus groups with job seekers conducted prior to the commencement of DES
2010-12;
92 an online survey of DES Providers;
93 in-depth interviews with DES Providers to be conducted by the evaluation team;
94 in-depth interviews with Employers;
95 a job seeker survey, available in English and the two most common non-English
languages spoken by DES participants, with special provision for people with
intellectual disability; and
96 forums with Indigenous people.
Other data sources
Data available from other agencies may be used to support the evaluation where
appropriate. Potential examples include the Survey of Disability, Aging and Carers from the
Australian Bureau of Statistics and data collected by the Australian Institute of Health and
Welfare.
27

28. Appendix B
Disability Employment Services
Evaluation Strategy Working Group
Terms of Reference
1 June – 30 July 2010
The Disability Employment Services Evaluation Strategy Working Group is a
subcommittee of the Disability Employment Services Reference Group, formed to
assist the Department of Education, Employment and Workplace Relations (DEEWR)
to develop an Evaluation Strategy for Disability Employment Services 2010-2012.
1 Provide advice on evaluation scope and strategy, consistent with program
objectives.
2 Review and comment on a draft Evaluation Strategy.
3 Provide advice as required to assist in the preparation of a final Evaluation
Strategy.
The DEEWR Employment Management Committee will approve a final Evaluation
Strategy.
These terms of reference cover the period of evaluation strategy development and
will expire on 30 July 2010. After that date industry stakeholders will be kept
informed of the status of the evaluation through the Disability Employment Services
Reference Group.
Chair
Justin Griffin, Branch Manager, Employment Monitoring and Evaluation Branch,
DEEWR
Membership
Industry Peaks
 Bec Jackson and Lynette May, ACE National
 Peter Davidson, Australian Council on Social Services
 Suzanne Colbert AM, Australian Network on Disability
 Nikki Brouwers, Australian Rehabilitation Providers Association
 Danielle Ballantine, Break Thru People Solutions
 Miles Davies, Job Centre Australia
 David Thompson, Jobs Australia
 Ken Baker, National Disability Services
Consumer Representatives
28

29.  Damian Griffis, Aboriginal Disability Network
 Leah Hobson, Australian Federation of Disability Organisations
 Paul Cain, National Council on Intellectual Disability
 Zeliha Iscel, National Ethnic Disability Alliance
 Michael Bleasdale, People with Disability Australia
DEEWR Evaluation
 Cathy Hales and Simon Arthur, Employment Monitoring and Evaluation
Branch
DEEWR Disability Employment Policy and Programs
 Sharon Stuart, Branch Manager, Disability Employment Services Branch
 Alison Durbin, Branch Manager, Disability Employment Policy and Support
Branch
 Susan Thomson, Director, Disability and Mental Health Policy Section
Member roles and responsibilities
The role of the DEEWR Evaluation Team is to design an evaluation to measure the
efficiency, effectiveness and quality of Disability Employment Services 2010-2012, in
consultation with stakeholders on the Evaluation Strategy Working Group. Under
direction from the DEEWR Employment Management Committee, the Evaluation
Team will conduct the Disability Employment Services evaluation and report to
Government.
The role of Industry Peaks is to advise the Evaluation Strategy Working Group of
service delivery issues that should be considered in the development of an
Evaluation Strategy.
The role of Consumer Representatives is to advise the Evaluation Strategy Working
Group of disability and cultural issues that should be considered in the development
of an Evaluation Strategy.
Representatives of DEEWR Disability Employment Policy and Programs will advise
the Evaluation Strategy Working Group on policy and program design features that
should be considered in developing an Evaluation Strategy.
Meetings
Up to two formal meetings will be held during June and July 2010. Meetings will be
via videoconference or teleconference. Out-of-session correspondence will be by e-
mail (DESevaluation@deewr.gov.au).
29