Design journal public services evaluation tool

1. DESIGNING A PUBLIC SERVICES EVALUATION TOOL
FOR THE NATIONAL CHAMBER OF SOCIAL
INSURANCE MOLDOVA
2016-2017
DESIGN

2. CONTENTS
CONTEXT
PROCESS
DISCOVER –
HYPOTHESES ARE NOT REALITY
DEFINE – GENERATING IDEAS
DESIGN – OPPORTUNITIES & IDEAS
DESIGN – PROTOTYPING & TESTING
DELIVER – PILOTING
1
3
4
13
16
19
26

3. The Government of Moldova committed itself to
streamline the Public Administration Reform, pub-
lic services reform being an inherent part of the
Public Administration Reform Strategy. Neverthe-
less, there are no sustainable mechanisms or tools
for evaluating the quality of public service.
Moreover, a mechanism allowing people to involve
directly in public services quality assessment does
not exist. Lack of timely people-generated data in
monitoring and evaluation of public services nega-
tively affects their quality, a successful implemen-
tation of public services reform, and in the long run,
can undermine citizens’ trust in the government.
1
CONTEXT:

4. 2
Moldova Innovation Lab (MiLab) saw the growing
need to develop a mechanism for citizens’ feedback
on the quality of the public services they are using.
Likewise, the scope of the project was to create a
citizen-generated data flow naturally integrated in
the operational processes of public institutions,
and applied in the policy making cycle (from policy
elaboration to policy evaluation) of public institu-
tions.
For the scope of this endeavour, MiLab selected as
an implementing partner the National Chamber
of Social Insurance (NCSI). MiLab had a success-
ful cooperation with NCSI on the reengineering of
several public services with the application of hu-
man-centred design. After piloting with NCSI, we
aim to scale up the tool at other institutions.

5. PROCESS
Our team engaged with
citizens and NCSI pub-
lic servants, aiming to
identify real needs and
expectations on the
quality of public services
and potential channels
for information collec-
tion. Also, together with
NCSI management, we
mapped the current
state of services eval-
uation process and its
workflow within NCSI.
Built on the concepts
generated during the
ideation phase, we en-
gaged in an IT Boot
Camp. Together with
service users, service
providers and IT com-
munity we produced a
prototype of the evalua-
tion tool.
Together with our us-
ers – public servants
and citizens – during
an ideation workshop
- we defined the main
issues, and generated
early concepts on how
how potential tool would
work.
The prototypes were
tested and evaluated
by service users. After-
wards, the prototype was
installed within selected
branches of the NCSI,
with the purpose of pi-
loting it in real life, within
public institutions.
DISCOVER DELIVERDEVELOPDEFINE
3

6. HYPOTHESES ARE
NOT REALITY
DISCOVER
Before conducting the research, we explored various business models of service evaluation applied both in
Moldova and worldwide, predominantly the ones used in private sector as businesses are profit-oriented
entities and their profit depends on the customer’s satisfaction. We discovered that most private companies
are using all-inclusive evaluation tools to collect as extensive information as possible – telephone call back,
e-mail and mobile surveys via SMS and feedback forms. Moreover, dedicated units are working on data
analysis and translation of collected information into targeted actions.
Our situation was different: we had limited financial resources, lack of human resources at public institu-
tions, and a pressing need to define a tool which would be in demand among citizens. For this purpose, we
decided to apply human-centred approach: dig deeper and identify all aspects of the existing ecosystem
around evaluation of public services - people’s motivations, operational procedures, policy actions, etc.
4

7. MiLab carried out an ethnographic research with the engagement of citizens and service
providers, aimed to achieve deep and nuanced understandings of various aspects of user
experience, needs and expectations from the evaluation of public services.
We selected 8 NCSI branches across Moldova sit-
uated in: Chisinau (capital city), Centre, South and
North of the country; both with and without front
offices. This differentiation should have given us a
clearer and more objective picture of the process.
Along with that, our scope was to elaborate an
inclusive tool for different categories of people.
Based on the assumption that people in age (senior
people) are less literate in terms of access to infor-
mation and use of digital tools, we selected them as
a target group of the research.
5
Evolution of Electronic Communications Market in 2017,
http://anrceti.md/files/filefield/Raport%20ev.%20pieteCE%203Q_2017.pdf

8. Initially, we were assessing the possibility
of using mobile phones as communication
channels for information collection. This
hypothesis was built on the official statis-
tics data: mobile phones penetration rate in
Moldova is about 126,5%. However, during
the research it was identified that about half
of the interviewed didn’t have a personal
mobile telephone or a mobile phone in the
household.
6
40% 130%
125%
120%
115%
110%
105%
100%
35%
30%
25%
20%
15%
Pay TV
Fixed Telephony
Dedicated Mobile Internet Fixed Internet
MobileTelephony
121,0% 121,0%
122,9%
124,7% 124,9%
126,1% 126,5%
Q1 Q2 Q1 Q2Q3 Q3Q4
2016 2017
10%
5%
0%
7,7%
8,4%
7,6%
8,4%
7,6%
8,6%
7,6%
8,6%
7,5%
8,6%
7,4%
8,7%
9,0%
8,8%
15,1%
15,3%
15,4%
15,7%
15,9%
16,0%
33,8%
33,5%
33,4%
33,0%
32,9%
32,6%
16,2%
32,4%
Along with that, digital literacy level of the interviewed
people was very low. Only a few had an email address,
most of them did not even think of creating one.
Also, most of the interviewed while asked how they
used their mobile phones, answered that they used
them only for phone calls to friends, children, rel-
atives; SMS is used only for receiving the messages
and not to send them. As for the usage of mobile in-
ternet, majority of the interviewed did not possess a
smartphone and were not planning to buy one.

9. As in many other countries, Moldova faces the issue of declining peoples’ trust in public institutions , because
the state is unable to respond to their legitimate needs. Despite this fact, people do want to keep the Govern-
ment accountable and showed active interest to contribute to improvement of the services. People were keen
to evaluate the quality of delivered public services, without benefitting from any incentives.
Along with that, people were concerned by the lack of transparency on the decision-making process, and lack
of information on what is happening when citizens’ petitions or reports submitted to public authorities.
LOW TRUST IN PUBLIC INSTITUTION
IS A REAL ISSUE
7
Highest Growth Rates on Mobile Broadband Market in Moldova
http://en.anrceti.md/news29112017
http://ipp.md/wp-content/uploads/2017/12/Prezentarea-publica.pdf
-100.0%
A great deal of trust
Church
City Hall
Army
Mass-Media (TV, PRESS, RADIO)
The President
Police
Banks
Security and INTELLIGENCE SERVICES
NGOs
The Government
National Anticorruption Center
Office of the Prosecutor General
Constitutional Court
Justice
Trade Union
Central Election Commission
Parliament
Political Parties
Somewhat trustSomewhat distrustHighly distrust
PUBLIC OPINION BAROMETER OF REPUBLIC OF MOLDOVA
-60.0% -20.0% 20.0% 60.0%

10. 8
One of the obstacles to report about the
quality of public services and communica-
tion with public authorities generally was
friction costs of the process: interviewed
people had never submitted official peti-
tions or filed complaints to a public insti-
tution, demotivated by the time-consuming
process and standard procedures that are
not straightforward, transparent and us-
er-friendly.
At the same time, they expressed their will-
ingness to evaluate the services, it was im-
portant to make their voices heard. In this
vein, according to citizens’ feedback, the
mechanism should be as easy and clear as
possible, since not everybody can “grasp
how to use sophisticated equipment, ma-
chines or complicated questions about pub-
lic services quality”.
PEOPLE DO NOT WANT
TO SPEND MUCH TIME ON THE REPORTING PROCESS
„I have never filed a complaint though I regularly face
some problems. Whenever I come to get an answer
about the documents I need to submit, the only answer
I receive is to come another day. Why should I file com-
plaints, if nobody reacts?”
Stated a 56 years old man from Cahul city

11. Following, we were facing an ambiguity related to the questions to be asked about the quality of public ser-
vices. Certainly, we had some assumptions on the things we should ask or evaluate, but the best experts
on what works or not in service provisions are clients, people. We asked the people about their needs and
expectations with reference to public services quality.
Lack of clear information and transparency, long queues and deficient internal infrastructure, friction costs
(time and transportation) are the key issues that citizens experience most of the time. Politeness, clear
language of public servants were also matters of concern.
WHAT SHOULD WE ASSES IN PUBLIC SERVICES?
9
„It is extremely inconvenient
for me to come to the terri-
torial office , because it’s lo-
cated in a different town and
this involves additional costs,
especially when the file is in-
complete and I need to bring
back additional documents”
Said a 47 years old woman
from Balti city suburb.
„It is extremely inconvenien
for me to come to the ter
torial office , because it’s
cated in a different town
this involves additional costs,
especially when the file is in-
complete and I need to bring
back additional documents”
Said a 47 years old woman
from Balti city suburb.
nt
rri-
’s lo-
n and
l costs,
„Waiting in the queue con-
sumes time and energy.Usu-
ally I refuse to wait and come
back later”
said a 59 years old woman
from Chisinau.
“Very often we are redirect-
ed to other servants that ask
to bring another file of doc-
uments or to sign some pa-
pers without any explanation
on what these are for”
mentioned a 62 years old man
from Ialoveni town.

12. During the research, we also talked to public servants
who are directly involved and responsible for the quality
of public services provision. It was essential to under-
stand operational processes at public institutions, and
not just to make ground for the collection of the informa-
tion, but to make use of it translating the collected data
into policy actions.
As it turned out, due to low salaries there is a deficit of human resources, and public institutions are already
overwhelmed with the current assignments. Thus, it was crucial to identify the entry points on how citizens’
feedback could be integrated in operational processes without creating additional administrative steps and
burdens.
WHAT SERVICE PROVIDERS SAY?
10
HOW TO DEVELOP AN EFFICIENT TOOL
WITH UNDERSTAFFED PUBLIC INSTITUTIONS?
“We are understaffed with front-office employees, and
deadline for processing a request is very tight, thus put-
ting additional pressure on the staff. Another adminis-
trative issue is that sometimes we have to check with
other public institutions some documents from the file,
in order to ensure the correctness of the answer, so this
is an additional burden and time-consuming moment”.
stated the Head of department from Comrat city

13. “I’m not afraid of citizens’ negative feedback because
we are offering ansewrs to the questions they have,
and explain their rights in compliance with legislation
in force. Of course, there is a risk that negative feed-
back could affect our performance evaluation, but
this will lead to a positive change as well.”
Stated a 55-year-old NCSI employee from Chisinau
11
“Yes, I agree to have citizens feedback included in
my performance evaluation. For instance, I’m really
enjoying the moment when we manage to help peo-
ple. Sometimes, a retired person comes to our office
being very sad and disappointed, and after we have
served him with a positive attitude and answered to
all his questions it makes him feel better.”
Mentioned a 38 years old public servant from Cahul city
Along with that, public servants demonstrated their motivation to improve the quality of public services
they deliver. And almost all service providers mentioned they were open to have a tool that would collect
citizens’ feedback and help them to map problematic parts of services and even to improve them. Moreover,
they even proposed to elaborate an indicator based on these data which will be integrated into performance
evaluation process of the personnel. The main concern still was whether citizens’ feedback on the quality
of public services would be objective one.
PUBLIC SERVANTS WOULD LIKE
TO BE MORE HELPFUL WITH THEIR CLIENTS.

14. 12
“Theethnographicresearchallowedustounderstand
citizens’ expectations on the way this tool should look
like, on what communication channels should we
use, on how this evaluation will be integrated within
internal business processes. We’ve worked with the
elderly, who are among the most marginalized and
harder-to-reach populations in terms of access to in-
formation, access to Internet and electronic services.
The idea is that if we respond to the specific needs of
this category,we’ll be able to meet the needs of other
categories of people as well.”
Mentioned Laura Grecu, president of the National Chamber
of Social Insurance

15. DEFINE
GENERATING IDEAS:
ALIGN THE DIFFERENCES
FOR A COMMON GOAL.
HOW MIGHT WE ENSURE A NEWLY CREATED
PUBLIC SERVICES EVALUATION TOOL TO BE
WIDELY USED BY CITIZENS?
13

16. 14
How might we ensure that NCSI makes use of the collected data for policy-making on the reform of
public services?
How might we create accessible and clear com-
munication channels to enable people report on
the quality of public services?
How might we create a feedback loop to let
citizens know what hands-on actions are taken to
tackle the issues reported by citizens?
People say they want to report on the quality of pub-
lic services, but the process should be simple and
clear in use.
People do not want to submit official petitions to
public institutions as they do not see any tangible
outcome of this. So, they do want to see what NCSI
undertakes to respond to their reports.

17. 15
How might we frame the PSET within NCSI in
order to alleviate these concerns and make it a
resourceful tool for assisting NCSI staff and man-
agement in the improvement of public services?
During the research, NCSI staff had some concerns
about framing PSET as a performance indicator for
public servants. The main reason of this was the fact
that people are often frustrated with the amount of
social entitlements and their assessment can be a
bias one.
How might we minimize any activities or steps
that unduly creates additional administrative
steps and burdens on NCSI staff while integrat-
ing PSET into NCSI activities?
NCSI branch staff and HQ staff have indicated a
strong interest in poeple’s feedback and application
of potential tool. The main concern was not to cre-
ate a new workload burden as some NCSI branch
offices are understaffed.

18. DESIGN
OPPORTUNITIES &
IDEAS
16

19. GIVING PEOPLE AN OPPORTUNITY
TO REPORT SIMPLY AND FAST
17
People should report about the quality of public services within NCSI offices. At
the same time, we should attract people’s attention to the reporting channel. What
if we install the terminals akin to those where people can pop-up their telephones
or pay the utilities?
QUESTIONS RAISED:
What shall we ask people? How to make reporting mechanism very fast and
comprehensive? How to attract people’s attention to the terminal? How shall
we assess online services?
GIVING PEOPLE REASONS
TO TRUST PUBLIC INSTITUTIONS
People should see what they are reporting about and how other people evaluate
the quality of the NCSI services. Also, people would like to see what NCSI does
to tackle the submitted issues. What if we create a feedback loop mechanism to
show people the statistics on the reporting and information about the actions tak-
en by the NCSI. We could do it on social networks. Alternatively, we could post the
information on the information board.
QUESTIONS RAISED:
Most of the NCSI information boards are paper-based, and people usually
do not read the information placed there. Moreover, the text is written with
a small font making it unreadable for the majority of population, in partic-
ular for people in age. How could we present this information in a more us-
er-friendly and attractive manner? How frequent should we update the infor-
mation?
WHAT YOU REPORTED
WHAT NSCI DID

20. 18
CREATING A RESOURCEFUL AND EASY
IN USE TOOL FOR NCSI STAFF
NCSI management are complaining that plenty of offices are understaffed. NCSI
staff will not be able to dedicate a lot of time to the analysis and interpretation
of data. We should create a semi-automated mechanism to present data dash-
boards in a user-friendly and clear manner to NCSI branch and HQ management.
QUESTIONS RAISED:
How to naturally integrate a new dataflow into NCSI operational planning cy-
cles? How to empower NCSI branch management to deliver better services?
CREATION OF A MECHANISM
TO ESCAPE BIAS OPINIONS
Branch managers are concerned by the fact that citizens’reports could be used
for evaluation of their performance. Managers believe customer satisfaction
with the public services is primarily impacted by other factors outside of their
control - amount of social entitlements citizens receive. What if we introduce
a mitigation mechanism – some questions about the amount of social benefits
- to allow citizens blowing off their steam and frustration of low social benefits.
QUESTION RAISED:
How should we formulate the question? Will it be a good way to prevent peo-
ple’s frustration or it might deteriorate the objectiveness of the responses?
QUALITY
OFREPUBLIC
SERVICES

21. 19
DESIGN
PROTOTYPING &
TESTING

22. We’ve engaged with citizens, service providers and ICT specialists in a two week IT Boot
Camp to develop a prototype of the tool in an inclusive and collaborative manner.
Around 40 IT specialists responded to our call, and five teams were formed to design a solution to translate
the insights we had collected during the research and ideation stages.
AS SIMPLE AS POSSIBLE
FOR PEOPLE
20

23. During two weeks, the participating teams were working closely with the beneficiaries exploring
different options on how to collect information, and to identify the most appropriate channel for the
collection of people’s feedback.
DESCRIBING THE PROCESS:
21
Together with citizens, participating teams were seeking the answers for the following
questions:
How should the interface look like?
What colour spectrum should we use?
What ranking methods (figures, smiles, etc.)?
How shall we ensure the privacy of the data collection?

24. 22
KEY TAKEAWAYS:
Some beneficiaries mentioned they would like to see how other people rank the quality of public
services at the branch
One of the participating team added the option of visualising the statistics of the answers and actions
taken by the NCSI within the branch.
Other people mentioned it was rather complicated to rank services from one to five or from one to
ten.
It is better to use self-speaking emoticons as they are more clear and attractive for people.
Taking into account the fact we are working with senior people, the tool should have had simple and
colourful design, large fonts and images.

25. 23
At the same time, participating teams were working closely with the service providers, NCSI staff. It was
important to create a data disaggregation and visualisation options tailored to the needs of NCSI staff,
and in compliance with the current technical setup and capabilities of the NCSI.
Along with the NCSI staff, participants were trying to answer the questions, such as:
How should the data be presented in the back-office?
How to visualise the collected data for NCSI staff?
How to export the data collected?
What will be the role of branch office in the analysis of the collected data?
What will be the role of NCSI HQ management in the analysis of the collected data?
Who will be in charge to formulate and change the questions in the evaluation mechanism?

26. 24
SIMPLE IN USE, COMPREHENSIVE IN A NUTSHELL
OFFLINE TERMINALS installed within NCSI offic-
es which would include general evaluation of the
services, some in-depth questions on the concrete
painful parts of the services (e.g. infrastructure,
documents collection, queues, clarity of informa-
tion, etc.), and a separate comments field for fur-
ther information on the services quality.
MOBILE APPLICATION which would allow service
assessment using smartphones, and working in
compliance with the same logic as offline terminals.
EVALUATION OF ONLINE (DIGITAL) SERVICES
The winning team came up with a three-way feedback collection mechanism for the
evaluation of public services:
Complete
THE QUESTIONNAIRE
Desk Nr. 5 Desk Nr. 6
Russian
Excellent Well Bad
Romanian
FEEDBACK LOOP
(visualisation of data collected,
and actions taken by NCSI to citizens)
FEEDBACK
INTEGRATION IN
NCSI OPERATIONAL
PROCESSES
(policy planning on the public services improvement)
PEOPLE’S FEEDBACK
COLLECTION
(different channels)

27. 25
SOME FEATURES
The team which developed the best solution was awarded with 10,000 USD. The winners
got the award for further elaboration of the tool together with the service providers and
beneficiaries, and deployment of the pilot.
THE PUBLIC INSTITUTION
WILL ANALYSE
YOUR FEEDBACK
DOWNLOAD
50 25 80OFFLINETERMINALS
35 20 25MOBILEAPPICATION
45 25 35ONLINE
FEEDBACK
The winning team proposed a natural and simple integration of the tool into the operational process-
es of the NCSI;
Importantly, back-office information system is developed for the people even without special IT back-
ground and trainings;
The team proposed to develop a feedback loop through presentation of the information on the LED
screens within NCSI offices and online: this will tackle the problem of low trust, and a desire of peo-
ple to know more about the follow-up steps of the NCSI on the received reports.

28. 26
DELIVER
PILOTING

29. 27
During three months, together with the winning team we worked closely with the NCSI staff and citizens to
elaborate a hardware and software prototype of the public services evaluation tool. As a result of several
iterations, we came up with a prototype of the tool to be piloted during the following four months:
1. People will evaluate the quality of public ser-
vices selecting one of three smiles in dependence
on their general satisfaction with public services.
People will have an option to select a counter they
have attended. It will allow NCSI staff to assess
the quality of civil servants’ work.

30. 28
2. People will have an opportunity to give us more information by selecting different parts (elements)
of public services they did not like.
3. We’ll have an extended feedback field option to enable citizens to give us more detailed or
supplementary information about their experience.

31. 29
4. After people complete the survey, they will see general statistics of answers within the NCSI
branch.
5. NCSI branch will place the statistics about people’s reports (answers), as well as on the actions
taken to tackle the issues, on the LED screen installed with the NCSI pilot branches.
TO DEVELOP BETTER
PUBLIC SERVICES
ANALYZE ACTION BETTER
PUBLIC SERVICES

32. 30
6. NCSI staff will have different roles in the back office of the public services evaluation tool. Only
HQ management staff will be entitled to make changes in the evaluation mechanism (questions,
number of counter offices, etc.)

33. 31
7. NCSI branch office staff will have access to the information on the statistics data of people’s
answers, and will not be able to make any changes in the system.

34. 32
8. Also, we elaborated a prototype
for the evaluation of NCSI online
public services. We’ll pilot this
tool along with the offline termi-
nals.
The picture below will allow to easily grasp why we started to work on the development of the tool,
how we did it, what features it will have and how it will function

35. 31

36. NEXT STEPS:
Moldova Innovation Lab (MiLab) is a joint project of the United Nations Development Program (UNDP) in Moldova
and the Government of Moldova (State Chancellery).
This project is implemented with the financial support of the Government of Slovakia through Transformative Governance
and Finance Facility.
Prepared by:
Dmitri Belan Irina Frunza
dmitri.belan@undp.org irina.frunza@undp.org
www.milab.md facebook.com/MoldovaInnovationLab/
1. MiLab will pilot the prototypes till July 2018.
MiLab will M&E the tool, closely with citizens and NCSI staff, to identify weak and strong parts of the
tool, and to explore opportunities for the improvement.
2. After piloting stage, MiLab will come with recommendations to scale up the tool to other public
institutions.