1. 1
PROGRAM EVALUATION PLAN:
Western Australian Department of Health Online
Consultation Management System
Program Background
The Western Australian Department of Health (WA Health) conducts many consultations and
is interested in which is the most effective way to engage with key stakeholders on health
reform and policy. To this end, WA Health has purchased online consultation management
software with the goal of increasing stakeholder participation in policy and program feedback
and development.
The objectives laid out for the online consultation management system include, but are not
limited to;
 Having a user friendly interface to effectively share information between WA Health
and relevant stakeholders;
 Providing a central repository of upcoming, current and previous consultations by the
Department to minimize duplication and share information; and
 Reporting feedback on the outcome of consultations to the general public and directly
to participants in the process.
The cloud-based online consultation system, CitizenSpace created by Delib, was chosen
through an open tendering process and is being tested through an initial two year trial period.
The trial involves two areas of WA Health (Public Health and System Policy & Planning) –
with possible expansion to other WA Health branches after completion of the trial evaluation
(e.g. Clinical Services, Purchasing & System Performance, and System & Corporate
Governance).
The online consultation system is intended to provide a central point of reference for health
professionals and consumers to access information on WA Health policies and programs as
well as allowing them to provide feedback.
A key component of the online consultation software is the “We asked, you said, we did”
feature that encourages consultation managers to close the feedback loop with stakeholders
regarding outcomes of consultations and information on how the data received was used to
inform decision making at WA Health. The provision of feedback to relevant stakeholders
serves to increase the amount and effectiveness of stakeholder engagement with the
Government.

2. 2
Following completion of the trial, it is expected that an evaluation on the use of an online
consultation management system will provide information about whether such a system is an
effective way for WA Health to engage with stakeholders while providing useful information
for the creation of policies and programs. This evaluation will focus on the consultations
completed during the first year of the online consultation system’s two year trial period.
Currently there is no standard approval procedure for WA Health consultations undertaken as
part of the Online Consultation Management System trial period. While there are no set
approval guidelines, there is an informal approval process that has emerged throughout the
trial. This process is outlined in the flowchart below.
Consultation Approval Process:
Purpose of Evaluation
The primary purpose of the evaluation is to establish whether an online consultation
management system is an effective method for WA Health to engage with stakeholders about
WA Health policies and programs. WA Health’s two-year trial of an online consultation
management system will be used to inform the evaluation and provide evidence to support
findings.
The evaluation aims to determine how useful information collected through online
consultation is, as well as the extent to which it influences policy and program decisions. In
addition, the representation and quality of stakeholder response as well as the experience of
WA Health staff with using an online consultation management system will help determine
the overall effectiveness of such a system.
The findings from the evaluation along with recommendations will be reported to the
Director General of WA Health and the State Health Executive Forum (SHEF) to consider
whether an online management system should be implemented across all of WA Health.

3. 3
Audiences and Stakeholders
The table below outlines the key stakeholders with an interest in the process of evaluation
and the evaluation’s findings, as well as their positions, departments and areas of interest.
Primary Stakeholders:
Secondary Stakeholders:
 the Public
 Consultation Respondents
 WA Health Employees
 Health Professionals
 Health Departments in Other States
 Other Government Departments
Stakeholder Interest Groups:
Here, the stakeholders’ interests are made explicit. The interests groups (designated with
letters A - D) are assigned to the various stakeholders in the table above in order to clearly
identify the information needs of each stakeholder.
A: Efficient use of Resources
 Cost Effectiveness (roll out, maintenance, training, etc.)
 Time Efficiency
B: Optimal Outcomes
 Information gained from consultations is useful.

4. 4
 Interest in number of consultations conducted and number of responses.
 The information gained through consultations serves to influence the creation of
policies and programs.
C: Ease of Use
 Consultations are intuitive and easy to follow, allowing them to be completed fully.
 Access to consultations available in rural areas and on older operating systems.
 Sufficient access and ease of use for people with disabilities.
 The information gained by consultations is easy to access and analyse.
 The creation of consultations is simple and straightforward.
D: Online Consultation as a Meaningful and Effective Form of Stakeholder
Engagement
 The process generates value for both sides of the consultation (WA Health and
participating stakeholders).
 The consultations provide a real voice for stakeholders, leading to effective
communication between the Government and the public.
It is important to clearly identify the target audiences of the evaluation and its findings. These
audiences are split into two groups, primary and secondary. The primary audience is the main
target of the evaluation. In this case the primary audience is made up of decision makers that
will ultimately use the information gained through the evaluation to inform whether or not
WA Health’s use of an online consultation system is effective, and whether or not its use
should be expanded. The secondary audience is made up of groups that may find the
outcomes of the evaluation beneficial, but do not have any say in the implementation of
online consultation at WA Health.
Primary Audience – Department of Health, relevant divisions and directors/coordinators, and
the WA Minister for Health, Health Departments of other states, other public sector agencies
Secondary/Potential Audiences – Public Health Organisations (Health Consumers Council,
Disability Services Commission WA, etc.) wider WA Health, Chief Medical Officer
Form of Evaluation
The evaluation of the online consultation management system fits within two of Owen’s five
forms of evaluation: the Interactive form and the Impact form (Owen 2006).
In an Interactive evaluation, the evaluation is most concerned with the process of program
delivery and whether or not the program is working efficiently (Owen 2006). Importantly for
this exercise, an Interactive evaluation looks at the ways in which program delivery can be
improved in order to foster better program outcomes (Owen 2006).
Impact evaluations can also be called summative evaluations that, “…assist with decisions
about whether to terminate a program or to adopt it in another place” (Owen 2006, p. 47).
The Impact part of the online consultation management system evaluation will look at
whether or not the goals of the program have been achieved, whether the program is effective

5. 5
and efficient in achieving these goals and whether or not the needs of WA Health have been
met by the program (Owen 2006).
Information Required – Key Evaluation Questions
Asking the right questions is crucial to the success of any evaluation. The key questions
outlined in this plan will uncover the information most useful to WA Health and will dictate
the direction that the evaluation will take. In addition to directing the evaluation, the
questions themselves will influence what information will be collected, and how it will be
collected. Most importantly, however, the key evaluation questions ensure that the client and
the evaluators are on the same page and want the same information; and that the evaluation
stays focused (Owen 2006).
In addition to these key questions, the unintended outcomes (both positive and negative) of
the online consultation management system will also be uncovered and analysed.
The key evaluation questions for the online consultation management system are as follows:
1. Is public online consultation being done efficiently?
1.1. Is there sufficient (and quality) stakeholder response to online consultations?
2. Is the online consultation system meeting WA Health expectations, and providing the
desired outcomes? (i.e. Is the online consultation system effective?) – Analysis of this
question will be based on the completed consultations within the first year of the two year
trial period
2.1. Is feedback and consultation being directed to the appropriate stakeholders?
2.2. Are the proper people within the WA Health receiving the results of these
stakeholder consultations?
2.3. Is the information gained through consultations useful and up to consultation creator
expectations?
2.4. Is the information gained being used effectively to influence policy and program
decisions
3. Can/should this online consultation be expanded into other areas of WA Health, what
adjustments need to be made in order to accommodate this?
Information Collection and Analysis
The number of completed consultations will need to be examined as well as the number of
responses and the quality of information gained. The user experience on the WA Health side
(creating consultation, approval process, usefulness of responses, etc.) will need to be
investigated, as well as the experiences of consultation respondents. These investigations will
look at each of the consultations individually. The information gained through the

6. 6
examinations of the different consultations will then be compared in order to highlight
strengths and weaknesses in the process.
It must be determined whether or not the information gained from stakeholder consultation
has been effectively used to influence policy and program development and that this
information is going to the correct people. The consultation process must also be efficient
from the perspective of both WA Health and the respondents to the consultations.
Criteria for Making Judgements
The success of this program will be judged against WA Health expectations for the
consultation process. The expectations of consultation creators and parties involved in the
consultation approval process will set the benchmarks for measuring the success of the
program. In order to be considered successful, the program must meet the needs of WA
Health while ensuring a high quality, effective means of stakeholder engagement.
These needs could include expected response rates, the number of completed trials and the
usefulness of information gained through consultations. The results gained from online
consultation must be worth the time and effort put into the creation of these consultations. If
the online consultation system is less efficient in receiving quality stakeholder feedback than
other forms of stakeholder engagement, it can be seen as less desirable. In order to assess
this, online consultations will be compared to other forms of consultation in terms of cost,
efficiency and the usefulness of information received. Comparing online consultation to
traditional forms of consultation will provide insight into how useful online consultation is as
a stakeholder engagement tool.
Methods and Activities
The way that data and information will be collected for the evaluation is outlined in this
section. The information gained through the use of these methods and activities will form the
basis of the final evaluation report.
1. Interviews:
 Stakeholders will be interviewed in an attempt to answer the key questions as laid out
in the evaluation plan.
 Stakeholders will be interviewed individually as to avoid bias and receive a more
honest view of the consultation process.
 The evaluation process will require approximately ten to fifteen interviews to be
conducted.
2. Consultation Analysis:
 Each of the consultations completed throughout the first year of the two year trial of
the online consultation management system will be analysed separately.
 This analysis will involve the views of the consultation creators, the numbers and
rates of consultation responses, and the overall quality of the consultations.

7. 7
 The information gained through this analysis will serve to identify the benefits and
possible areas of improvement in the consultation process.
3. Document Analysis:
3.1. Literature Review:
 A literature review will be conducted focusing on online consultation in
government and the area of health.
 The literature review will serve to answer the question of whether or not online
consultation is an effective form of stakeholder engagement.
 The lessons learned from the literature review can then be applied to the way that
WA Health consults with stakeholders.
3.2. Analysis of Related WA Health Documents:
 Analysis of these documents, such as the Project Plan for the online consultation
management system, will allow the evaluators to compare the planned objectives
of the system to its actual outcomes.
 These documents will also serve to provide context on the consultation system
and will highlight WA Health’s reason for its implementation.
Resources
In order to complete the evaluation, evaluators must be given access to relevant stakeholders
within WA Health, as well as access to consultation respondents. These stakeholders include
consultation creators as well as people throughout the consultation approval chain. These
stakeholders will be interviewed in order to determine their views on the efficiency and
effectiveness of the online consultation management system.
The evaluators must also be given access to WA Health information concerning the online
consultation system. This information would include the results of completed consultations,
documents relating to the relationship between Delib and WA Health, and documents
outlining the requirements for the online consultation system.
Throughout the process, WA Health should provide a mentor to the evaluators in order to
answer questions and provide guidance. The involvement of a supervisor would ensure
quality throughout the process and would also allow the evaluator access to relevant
information about the online consultation management system.
Dissemination & Engagement
Stakeholders at WA Health will be engaged throughout the evaluation process. Stakeholders
will be involved in the design and implementation of the evaluation in order to ensure its
relevance to WA Health and the accuracy of its findings. In order to achieve this,
stakeholders will be involved in the creation of the key evaluation questions as well as
helping to establish the program’s criteria for success.

8. 8
In addition to this, a report will be created that outlines the evaluation’s findings. This report
will lay out the evaluation process and will make clear the scope of the evaluation and
whether or not there are any biases or conflicts of interest that need to be taken into account.
This report will be created and delivered at the end of the evaluation process and will include
qualitative findings related to the performance of the online consultation management system
versus the expectations of WA Health.
In order to ensure the validity of the report, it must be targeted toward the correct decision
makers and must be created with stakeholder buy-in throughout the process.
Timeline
The evaluation will take place over three stages:
1. Evaluation Planning (March 2015 – June 2015)
 In this stage there will be two main deliverables:
o Evaluation Plan – 20 May 2015
o Literature Review – 18 June 2015
2. Data Collection (July 2015 – October 2015)
 This phase of the evaluation will involve the collection and organisation of
relevant data (including interviews and analysis of the completed
consultations). Important deadlines for this phase are:
o Interviews Conducted – 15 September 2015
o Consultations Analysed – 5 October 2015
o Data Organised and Reported – 31 October 2015
3. Reporting (March 2016 – June 2016)
 This phase will involve the final reporting on the outcomes of the evaluation
along with recommendations to WA Health. Important dates for this phase
are:
o Draft of Final Report – 25 May 2016
o Final Report – 20 June 2016
Quality and Other Issues
 Structural changes within WA Health
o The evaluators are unable to control or foresee possible changes in WA Health
and must adapt and cope with any changes that may occur.
o It is assumed that the findings of this evaluation will be relevant no matter
what possible changes may occur to the structure of WA Health.

9. 9
 This evaluation is being conducted in compliance with the Guidelines for the Ethical
Conduct of Evaluations published by the Australasian Evaluation Society Inc. (AES
2002).

10. 10
Table 1 – Key Questions and Data Collection
This table outlines the key evaluation questions, as well as how and where the answers to these questions will be determined.
Key Questions and Sub-Questions Data Sources Method of Collection & Analysis
Is public online consultation being done efficiently? Consultation Creators; Senior Research Officer –
Development & Reporting Team, Senior Policy
Officer – Disaster Preparedness & Management
Interview
 Is there sufficient, quality, stakeholder response to
online consultation?
Director; DDGs; Senior Research Officer –
Development & Reporting Team, Senior Policy
Officer – Disaster Preparedness & Management
Interview
Consultation Creators; Consultation Feedback
Analysts; 10 Completed Consultations
Interview; Analysis of Completed Evaluations
Is the online consultation systemmeeting WA Health
expectations, and providing the desired outcomes? (i.e. Is the
online consultation systemeffective?)
Director, DDGs Senior Research Officer –
Development & Reporting Team, Senior Policy
Officer – Disaster Preparedness & Management;
Consultation Respondents
Interview
 Is feedback and consultation being directed to the
appropriate stakeholders?
DDGs; Director; Consultation Creators; Senior
Research Officer – Development & Reporting
Team, Senior Policy Officer – Disaster
Preparedness & Management
Interview
 Are the proper people within the WA Health
receiving the results of these stakeholder
consultations?
Director, DDGs; Data Analysts; Senior Research
Officer – Development & Reporting Team, Senior
Policy Officer – Disaster Preparedness &
Management
Interview
 Is the information gained through consultations
useful and up to consultation creator expectations?
Consultation Creators; Senior Research Officer –
Development & Reporting Team, Senior Policy
Officer – Disaster Preparedness & Management;
Director; DDGs
Interview
 Is the information gained being used effectively to
influence policy and program decisions?
Director; DDGs; Senior Research Officer –
Development & Reporting Team, Senior Policy
Officer – Disaster Preparedness & Management
Interview
Can/should this online consultation be expanded into other
areas of WA Health. If so, what adjustments need to be made
in order to accommodate this?
Senior Research Officer – Development &
Reporting Team, Senior Policy Officer – Disaster
Preparedness & Management; Director; DDGs;
Director General; HIN; D/Comms; Project Lead
Interview

11. 11
Bibliography:
Australasian Evaluation Society (AES) (2002). Guidelines for the Ethical Conduct of
Evaluations. www.aes.asn.au, AES.
Department of Health. (2013). WA Health Writing Style Guide, Communications
Directorate, Department of Health, Perth.
Department of Health. (2013). Request: Consultation Management System. W. A. D. o.
Health.
Owen, J. M. (2006). Program Evaluation: Forms and Approaches. Crows Nest NSW,
Allen & Unwin.
Program Evaluation Unit (2014), Evaluation Guide, Department of Treasury, Government
of Western Australia, Perth.

12. 12
Important Documents for Phase 2:
 Completed Evaluation Plan
 WA Treasury Evaluation Guide
 Request for Tender Document for online consultation management system
 Literature Review
 AES Ethics Guidelines