Democratizing Co-Production Of Sustainable Public Services Pilots MEF VARAM Zaragoza

1. First Stakeholder Event
14 December 2022
Highlights from the 1st implementation period
Prof. Diego López-de-Ipiña, University of Deusto
Roberta Lotti, Innovation Manager – Italian Ministry of
Economy and Finance
Matīss Veigurs, Project Manager - VARAM, Latvia
Daniel Sarasa Funes, Director of the Zaragoza City of
Knowledge Foundation (FZC)

2. 2
Agenda
14/12/2022 Summary of pilots’ iteration 1
h 5.50 - 6.25 PM | Highlights from the first implementation period
● General introduction (20’):
○ Diego López de Ipiña González de Artaza, Full professor – Universidad de Deusto
● INTERLINK pilots: practical examples of co-creation (15’)
○ Roberta Lotti, Innovation Manager – Italian Ministry of Economy and Finance
○ Matīss Veigurs, Project Manager - VARAM, Latvia
○ Daniel Sarasa Funes, Director of the Zaragoza City of Knowledge Foundation (FZC)

3. 3
General introduction to
PILOT ITERATION 1
(20 minutes)

4. 4
What do we want to evaluate? CO-PRODUCTION approach
Summary of pilots’ iteration 1
Summary of pilots’ iteration 1
14/12/2022

5. 5
What do we want to evaluate? Collaborative Environment
14/12/2022 Summary of pilots’ iteration 1
• COLLABORATION & RE-USE
▪ The INTERLINK platform offers a digital environment that facilitates
co-production processes between Public Administrations, private
stakeholders and citizens and promotes the re-use of software for
delivery of public services.
• CO-DESIGN & CO-DELIVERY
▪ INTERLINK provides a step-by-step guidance for the co-production
and co-delivery of public services along with guidelines, tips and
templates that facilitate the collaboration of different actors.
• INTERLINKERs
▪ Pieces of knowledge or software that your team can re-use and
customize to deliver services.

6. 6
With whom and Why? Quadruple helix & valorization
14/12/2022 Summary of pilots’ iteration 1
learning opportunity,
social activity,
public respect, …
new revenue,
product marketing,
corportate image,
partnerships, …
improved processes,
cost savings,
less to do, …
research topics,
project funding,
publications, …
better services,
improved life,
good-will, …
Co-production
Team

7. 7
Evaluation and assessment strategy
14/12/2022 Summary of pilots’ iteration 1
• 2 iterations per pilot:
• 2 sub-phases per phase:
April - May 2022
(M16 - M17)
PRE-PILOT PILOT EXECUTION
PILOT ITERATION 2
CONSOLIDATION
PRE-PILOT PILOT EXECUTION
PILOT ITERATION 1
EXPLORATION
June - September 2022
(M18 - M21)
March- April 2023
(M27 - M28)
May - Sept 2023
(M29 - M33)
PRE-PILOT PILOT EXECUTION
GOAL: Instantiation of INTERLINK platform and
testing
TESTING GROUP: Selected group of testers
GOAL: Pilot open to all stakeholders (INTERLINKERs and services published and
available to authorized users)
TESTING GROUP: Open set of users: citizens, public administrations and local
businesses (up to 200 users are expected)
ITERATION I (M16 - M21)
April 2022 June 2022 September 2022

8. 8
Deployment features (Collaborative Environment)
14/12/2022 Summary of pilots’ iteration 1
Iteration 1 in numbers:
● 3 deployments
● 62 INTERLINKERs in INTERLINK Catalogue
○ KNOWLEDGE INTERLINKERs: 33 internal & 12 external
○ SOFTWARE INTERLINKERs: 5 internal + 1 external (internally created) + 11 INTERLINKERs pointing external
ones (e.g. Doodle)
● Co-production schemas: 3

9. 9
Deployment features (catalogue)
14/12/2022 Summary of pilots’ iteration 1

10. 10
Pilot iteration 1’s Evaluation Strategy (Dimensions)
14/12/2022 Summary of pilots’ iteration 1
Will INTERLINK co-production model and its supporting tools and co-production enablers
(INTERLINKERs) enhance the quality, quantity, and reuse of public services?

11. 11
Evaluation journey followed
14/12/2022 Summary of pilots’ iteration 1
Alpha release of INTERLINK
platform
Pre-pilot execution subphase
Pilot execution subphase
Heuristic
evaluation
Usability
tests
Focus
Groups
Cross testing
between WP2
and WP5
Training session
on apha tester
representatives
Cross testing
sessions with
alpha testers
Beta release of INTERLINK platform
Pilot execution
at each pilot
User-behavior
analysis
Assessment of
quality of co-
production
Release 1 of the INTERLINK platform

12. 12
Pre-pilot subphase evaluation (II)
Summary of Pilot 1
This subphase helped us progress from the ALPHA release into the BETA release by applying an
assortment of evaluation activities with alpha testers:
● Training sessions with alpha testers: to get pilot owners trained to guide other alpha testers and
future beta testers. User Manual was produced. Support mechanisms realized (support
questionnaire, feedback questionnaire, Redmine deployment).
● Cross-testing session with alpha testers: aimed to identify blocking usability problems from
end-user point of view and discover key missing features or improvement areas.
○ MEF: order of tasks (first team or project), information overload, selection of schema, guidance to
co-production, catalogue filtering, language support
○ VARAM: Augmenter is easier to use than Collaborative Environment, better to use purpose
specific co-production model than standard one, resource concept and its lifecycle improvable.
○ ZGZ: detected need to enable pilots to define their own co-production schemas (template
produced)
A range of usability enhancements have been performed to deliver BETA release of Collaborative
Environment (30 core issues).
● New BETA release was produced

13. 13
Assessment of quality of the co-production process
14/12/2022 Summary of pilots’ iteration 1
● Can we quantify the associated Quality to a co-production process and its resulting artefacts?
QoS co-production = average(user-based quality; value-based quality; satisfaction; trust; acceptance)
● where:
○ User-based quality = AVG (Effectiveness, Efficiency, Usefulness, Ease of Use, Flexibility).
○ Value-based quality = AVG (Inclusiveness, Security/Privacy, Democratic values, Weberian criteria)
● Automatized calculation of KPIs and
analysis of questionnaires
● QoS values per pilot are very encouraging,
not conclusive, overall 3.79/5,0 (among co-
producers)
● Moderate acceptance level perception,
3.24/5 (among co-producers)

14. 14
Pilot execution subphase evaluation
14/12/2022 Summary of pilots’ iteration 1
● Continuous improvements were performed as the pilot execution progressed
○ As result we made progress from beta to release 1 of Collaborative
Environment, integrating the following features:
■ Guided overview for co-production process administrators
■ Team assignment to whole co-production process
■ Co-creation schema structure modification
■ Introduced automatic updates of GUI (websockets)
■ Enhanced navigation and searching features of the catalogue
■ Improved documentation

15. 15
Pilot execution subphase evaluation (IV)
Summary of Pilot 1

16. 16
INTERLINK pilots: practical
examples of co-creation
(15 minutes)

17. Summary of pilots’ iteration 1 17
14/12/2022
MEF use case: Participatory Strategic Planning Module
Use case: co-design of a PSPM
Stakeholders: public administrations, civil servants, Government
stakeholders and Service companies
Goals: to test how co-design approach can be supported by Technical
and methodological solution producing potentially more highly
aceptable Solutions in the public sector

18. Summary of pilots’ iteration 1 18
14/12/2022
Latvian Ministry of Environmental Protection and
Regional Development (VARAM)
Use case: refinement of municipal digital services’ descriptions
Stakeholders: municipalities, citizens, digital agents, and state
customer service centres
Goals: to test the co-production model and usability of citizen
sourced information for policy shaping

19. 14/12/2022 Summary of pilots’ iteration 1 19
100 participants engaged (PA, digital agents and
citizens) in 10 municipalities
Suggestions provided for 4 unifiable services and
results transferred to VARAM digital policy
departments
Next step: more focused approach on refinement of
service descriptions through creation ofa reusable
service description template
1
9
Results

20. 14/12/2022 Summary of pilots’ iteration 1 20
Public admin
NGOs
Research
Entrepreneurs
Artists
Civic Tissue
Goal:Promoting collaborative governance and
engagement in Etopia_ as a factory of innovation,
creativity. → Interlink is the DIGITAL side of Etopia
Stakeholders: schools, creators, artists, researchers,
entrepreneurs, companies and citizens in general.
Project Plan:
#Iteration1: develop internal governance and
productivity tools
#Iteration2: Etopia as an innovation platform
#Exploitation: “an open source digital platform
for a planet of civic laboratories.”
Zaragoza Use Case: Interlink as eTOPIA_’s OPEN, COLLABORATIVE
& DIGITAL INNOVATION PLATFORM

21. 14/12/2022 Summary of pilots’ iteration 1 21
Zaragoza Use Case: REFLECTIONS after Iteration #1
● Social processes are flexible and liquid by nature →
Need for a digital collaborative environment
adaptative and flexible
● Social innovation are iterative → projects must be
agile and changes should be incremental in order to
be accepted
● Measuring innovation is a very subtle issue →
Innovation can not be predicted by nature.
● The HUMAN dimension is crucial, and technology
should ease the facilitator’s work, being optional.
TECHNOLOGY works for HUMANS (and only when
these need it)

22. 22
● A post-pilot workshop gave place to the following feedback:
○ Structure of the co-production working group: different usage depending
pilot, identification of three roles (administrator, contributors, observers)
○ Motivation and awareness of the process: depending on the actual value
delivered to pilots, progress dashboard considered very important
○ Structure of co-production schema: difficult to grasp, more graphical,
more plain English for terminology, better link between INTERLINKERs,
resources and tasks
○ Choice of co-production schema: positively received, further
enhancements needed to adapt existing ones into new co-production
schemas, notification mechanism to team members
○ Importance of success stories: learning by example considered as having
great potential
Post-pilot reflection phase: lessons learnt
Summary of pilots’ iteration 1
14/12/2022

23. 23
● A post-pilot workshop gave place to the following feedback:
○ Exploitation rights management: not easy to understand but seems highly
important from governance point of view
○ More functionalities of collaborative environment: possibility of creating
processes, teams and associating resources under a process and linking to
external resources.
○ More useful INTERLINKERs: more to the point recommendation of
INTERLINKERs, most useful INTERLINKERs being Google Docs, Loomio,
Jamboard and pilot-specific ones
○ Motivation for using Collaborative Environment and co-produce:
collaboration is the central theme of INTERLINK but must be incentivized
and valorized!
Post-pilot reflection phase: lessons learnt
14/12/2022 Summary of pilots’ iteration 1

24. interlink-project.eu