The document presents the welive framework, a set of tools designed to enable the co-creation of urban public service applications by integrating open innovation, data, and services, while addressing current limitations in public service design. It emphasizes the importance of stakeholder involvement and collaboration between citizens, entrepreneurs, and public administrations to enhance public service usage and efficiency. The paper also outlines the evaluation process and results from the pilot phase in Bilbao, aimed at assessing user experience and the impact of the platforms on public services.

1. Towards Citizen Co-created Public Service Apps
H2020-INSO-2014, 645845
Maspalomas, Las Palmas, Spain, 2nd December 2016
Diego López-de-Ipiña, Mikel Emaldi, Unai Aguilera, Jorge Pérez
MORElab Research Group, DeustoTech – Deusto Institute of Technology,
Faculty of Engineering, University of Deusto
dipina@deusto.es
http://paginaspersonales.deusto.es/dipina

2. Index
• WeLive Project aims
• Service co-creation approach
• WeLive platform
• WeLive platform and apps evaluation in
Bilbao pilot
• Conclusions

3. Abstract
• This paper describes the WeLive framework, a
set of tools to enable co-created urban apps by
means of bringing together Open Innovation,
Open Data and Open Services paradigms.
– Proposes a more holistic involvement of stakeholders
across service ideation, creation and exploitation 
WeLive co-creation process
• The two-phase evaluation methodology designed
and the evaluation results of pre-pilot sub-phase
are also presented.
– Including early user experience evaluation for WeLive

4. Why WeLive? (I)
Challenge #1
Current public services are built following an administration-
centric approach which drives into a low usage of those
services.
Need #1
Need to move towards a
more open model of
design, production and
delivery of public services
Need #2
Fostering the collaboration
between citizens,
entrepreneurs and public
administrations

5. Why WeLive? (II)
Challenge #2
PAs are facing key socioeconomic challenges such as
demographic change, employment, mobility, environment…
plus the squeeze on public finances
Need #1
Need for the modernisation
of PA to maximize the
public service delivered and
local economy
Need #2
Citizens want to transit
from mere consumers of
public services to providers
of those services
Citizens expectations in terms of burden reduction, efficiency
and personalisation are growing…

6. Why WeLive? (III)
Challenge #3
Cities and territories should be transformed into hubs of
welfare, innovation and economic growth giving place to
Smarter Cities or territories.
Need #1
Need to make a more
efficient management of
resources and be reactive
to socio-economic needs of
their stakeholders
Need #2
Empowerment of
stakeholders is necessary
by incentiving them to take
a more active role
Public-private partnerships and contribution of citizens have to
be fostered to produce a sustainable model of government

7. WeLive proposes…
Transform the current e-government approach into…
WeLive Open and Collaborative Government Solution = We-
government + t-government + I-government + m-government
We-
All stakeholders
are treated as
peers and
prosumers
t-
Providing
Technology tools
to create public
value
l-
To do more with
less by involving
other players
m-
Utilisation of
mobile tech. for
public services
delivery

8. How? (I)
A novel We-Government ecosystem of tools (Live) that is
easily deployable in different PA and which promotes co-
innovation and co-creation of personalised public services
through public-private partnerships and the
empowerment of all stakeholders to actively take part in
the value-chain of a municipality or a territory
Open Data Open Services Open Innovation

9. How? (II): WeLive concept
• WeLive = assembly line of co-created urban services
1. Offers tools to transform the Needs into Ideas
2. Tools to select the best Ideas and create the Building Blocks
necessary to build the envisioned solutions, and
3. A way to compose the Building Blocks into mass market
Applications which can be exploited through the WeLive
marketplace
Stakeholder Collaboration + Public-private Partnership 
IDEAS >> APPLICATIONS >> MARKETPLACE

10. How? (III): WeLive Service co-
creation approach
Inspire and involve
Challenge and Idea Generation
Idea Evaluation and Selection
Idea Refinement
Suggested
collaborators
list
Suggested
PSAs list to user
Suggested BBs and
Datasets list to
realize the PSA
Get required
BBs list
Publish PSAs
Idea Implementation
PSA Deployment
Open Service Layer
Core BB
Open Data Stack
Research Query
Get required
Datasets list
BBs and PSAs
Registry and
Execution environment
Communities
Analytics
Dashboard
Query mapper

11. Service co-creation and innovation
vs Idea lifecycle
Co- experience
Co- definitionCo- development
Co- delivery
Inspire and
involve
Idea
generation
Evaluation
and selection
Refinement
Implementation
Monitoring

12. Scenario-driven Artefact
Definition per City
25/05/2016
1 – Agree a common methodology for
stakeholders involvement and scenarios
definition (benchmarking)
2a – Activities execution for
insights gathering – stakeholder
consultation process
2b –Deep analysis of city strategy
and pilot focus, current IT and
open data infrastructure
3a – Scenario #1 definition 3b – Scenario #2 definition
4a – New
public
service #1
4b – New
public
service #2
4g – New
public
service #7
4h – New
public
service #8
5 – Set of building
blocks

13. WeLive Architecture

14. GUI to access framework
functionalities: Controller

15. WeLive Tools: Open Innovation
Area – OIA
OIA main page Ideas explorer
• Open Innovation Area:
https://dev.welive.eu/innovation-area

16. WeLive RESTful API

17. WeLive Player
• To discover, download and launch WeLive urban apps in 4
3 European cities (Bilbao, Novi Sad, Trento) and 1 region
(Helsinki-Uusimaa)

18. Bilbao: Consultation Process (I)

19. Bilbao: Consultation Process (II) –
Focus Groups
• In the second step, focus groups for each stakeholder (citizens, companies
and public administrations) were carried out.
• For each focus group people were divided in different groups of 4-5 people ,
each of the group focused on one of the topics.
• Focus Group methodology:
– Inspiration phase
– Brainstorming about ideas
– Voting for ideas by all groups
– Specific service or use case definition
– Presentation of these services
– Voting for services

20. Bilbao: Scenario-driven Services
• Sc-Bilbao-1: Collaboration in the Neighborhood and Participation in
municipal decision
– Neighbourhood Citizen Voting/Bilbozkatu: Service to foster citizens´
participation in neighbourhood decision making.
– AuzoNet : Social network for neighbourhoods for providing solutions to
the inhabitants
– Faborenet/Bilbao Volunteering Service to connect people with some
specific need to those volunteers who are available to solve these needs
• Sc-BILBAO-2: Collaborating on a more Efficient and Transparent City
– WikiWhere/BilbON: Service to locate different public utilities in the city:
bike rentals, bus station, POIs…
– Bilboenergy: Service to sensitive the citizens about energy consumption
– Public Spent Dashboard: Service to improve city’ transparency by
providing information about how citizens taxes are spent

21. Application: Bilbozkatu
• Pilot City/Region  Bilbao
• Synopsis  Proposals by citizens to improve their
neighborhoods
• Main features 
– Create proposal
– Vote in favour or against a proposal
– Comment and rate proposals
– See proposals geolocated on a map
• WeLive Building Blocks used  Feedback BB, In-app
questionnaire, AAC, Logging
• WeLive Datasets used  None

22. Application: Bilbozkatu

23. Phase I Piloting
• Divided into 2 sub-phases:
– Sub-phase 1:
• WeLive framework and all the services running are fully tested
– Controlled reduced group of alpha testers reports about their experience using
the services and infrastructure
– Trial site specific and app specific KPIs will refined and new ones defined
– Sub-phase 2:
• Launch of Pilot Phase I  beta testers access apps published on Google
Play
• Users might access apps through the WeLive and create ideas through the
Open Innovation Area
Pre-Pilot sub-phase Pilot Execution sub-phase
Aim  Full testing of WeLive framework, BBs and
services.
Testing group  Controlled and selected group of
alpha testers (5-10 users per pilot site)
Aim  Pilot open to all citizens living in a city
(services published into Google Play).
Testing group  open set of users: citizens,
developers, public administration and local
businesses (up to 500 users are expected)
March 2016 (M14)- July 2016(M18)
August 2016 (M19) – December 2016
(M23)

24. Bilbao’s Pilot Schedule for Pilot
Phase I
M14 M23
Pre-Pilot Launching Pilot Execution Launching
M19
Support
Training
Communication
Release
Monitoring
Execution
Evaluation
Support Task Force/Support Mail
Account
FAQ and Usage Documentation + mail support
Workshop
alpha testers
Selection of 5-10
alpha testers
Intensive targeted
dissemination
Release of WeLive
platform + BIO
apps
Monitoring of the
early testers
Monitoring
Innovation
Area
Monitoring
BilbOn app
Monitoring
BilBozkatu
Monitoring
AuzoNet
FaboreNet
Platform and apps
update 1
Milestones
Service specific
surveys analysis
Logs collection &
analysis
Workshops: Generic,
Enterpreneours, Citizens and
Developers (Hackathon)
Awareness
material
Questionnaires and in-
depth interviews
analysis
Platform and apps
update 2
Ms1
Ms2
Ms3

25. Evaluation Methodology (I)
• What is being evaluated?
– Acceptability of WeLive platform and public
service apps
User
experience
Acceptability
Usability

26. Evaluation Methodology (II)
• What is being measured to assess WeLive ACCEPTABILITY?
– Indicators associated to the Open Innovation Area as promoter of public service apps
– Indicators associated to the Open Data Stack as a catalyst of Broad Data exploitation
– Indicators associated to the Open Services Framework as facilitator of public service app
creation
– Indicators associated to the Personalization of Public Services
– Indicators associated to the Stakeholder Behaviour Change towards e-Government through
WeLive
– Indicators associated to the Economic and Job Market impact
– Indicators associated to Transparency and Public Administration impact
– Indicators associated to WeLive ecosystem users´ profiles
– Indicators associated to User reputation and rankings
– Indicators associated to the WeLive framework tools usage
– Indicators associated to WeLive public service apps
• Tables with KPIs:
– https://docs.google.com/spreadsheets/d/1W-
xXtGUl670qyfkWOUrbcgSHhIKsSxZuSkhMGW_L4dE/edit

27. Evaluation Methodology (III)
• For each app, the following indicators are being gathered:
– Common KPIs for all public services:
• KPI.BIO.1  Number of completed user surveys/questionnaires is more than 20
(10% of total apps’ users)
• KPI.BIO.2  Percentage of highly engaged or committed users (i.e. percentage of
users that suggest improvements to the app over the users that respond to the
survey per app): 10%
• KPI.BIO.3  Simplicity: number of users who think that it is simple to communicate
with the city through this channel over the total of users that respond to the survey
per app: 70%
• KPI.BIO.4  Percentage of users who used the app without experiencing major
issues: 90% (over the total of users that respond to the survey per app)
• KPI.BIO.5  App scores over 3.5/5
– For “Bibozkatu – Bilbao Citizen Voting”:
• KPI.BIO.6  Number of downloads is at least 47
• KPI.BIO.7  Number of times the app is started: > 150
• KPI.BIO.8  Number of WeLive registered users (active users) through this app: 20
• KPI.BIO.9  Number of voting campaigns organized: 4
• KPI.BIO.10  Number of votes per campaign received: 80% of downloads

28. Measuring Acceptability with
Public Representatives (I)
• User Experience Questionnaire (UEQ): a questionnaire to measure
User Experience (UX)
– http://www.ueq-online.org/
• Allows a quick assessment of the user experience of interactive
products
– The format of the questionnaire supports users to immediately
express feelings, impressions, and attitudes that arise when they use a
product.
– The scales of the questionnaire cover a comprehensive impression of
user experience, i.e. measure both classical usability aspects
(efficiency, perspicuity, dependability) and user experience aspects
(originality, stimulation)
• UEQ Questionnaires to assess WeLive available at:
– https://drive.google.com/drive/u/0/folders/0B70GHRZ8PwqfRXE5SW1
sRzZFWUE

29. User Experience – UEQ (I)

30. User Experience – UEQ (II)1 2 3 4 5 6 7
annoying        enjoyable 1
not understandable        understandable 2
creative        dull 3
easy to learn        difficult to learn 4
valuable        inferior 5
boring        exciting 6
not interesting        interesting 7
unpredictable        predictable 8
fast        slow 9
inventive        conventional 10
obstructive        supportive 11
good        bad 12
complicated        easy 13
unlikable        pleasing 14
usual        leading edge 15
unpleasant        pleasant 16
secure        not secure 17
motivating        demotivating 18
meets expectations        does not meet expectations 19
inefficient        efficient 20
clear        confusing 21
impractical        practical 22
organized        cluttered 23
attractive        unattractive 24
friendly        unfriendly 25
conservative        innovative 26

31. User Experience – UEQ (III)
-2
-1
0
1
2

32. • WeLive is an Open Government enabling
solution to foster the co-creation of urban
services among public administration, citizens
and componies
• Trialling of 1st phase is concluding, where
citizen engagement has resulted challenging
– 2nd phase of the project will transit from
evaluating co-ideation to assessing co-creation
(enabled by Visual Composer)
Conclusions

33. Towards Citizen Co-created Public Service Apps
H2020-INSO-2014, 645845
Maspalomas, Las Palmas, Spain, 2nd December 2016
Diego López-de-Ipiña, Mikel Emaldi, Unai Aguilera, Jorge Pérez
MORElab Research Group, DeustoTech – Deusto Institute of Technology,
Faculty of Engineering, University of Deusto
dipina@deusto.es
http://paginaspersonales.deusto.es/dipina