This work has to do with the evaluation of an advanced OGD e-Infrastructure based on value models and setting the improvement priorities.

1. University of the Aegean – Department of Information and Communication Systems Engineering
Charalabidis, Y., Loukis, E., Alexopoulos, H.
University of the Aegean, Greece

2. INTRODUCTION: THE OPEN /BIG DATA
MOVEMENT IN THE BACKGROUND
 Governments are increasingly opening to the society
important data they possess, in order to be used for
scientific, commercial and political purposes.
 Initially a first generation of Internet-based open
government data (OGD) infrastructures has been
developed in many countries, influenced by the Web
1.0 paradigm, in which there is a clear distinction
between content producers and content users.
2

3. A SECOND GENERATION OF OGD
INFRASTRUCTURES
 Recently a second generation of more advanced OGD
infrastructures is under development, which is
influenced by the principles of the new Web 2.0
paradigm:
 elimination of the clear distinction between ‘passive’
content users/consumers and ‘active‘ content producers
 They aim to support highly active users,
 who assess the quality of the data they consume and
mention weanesses of them and new needs they have
 and often become data pro-sumers‘ = both consumers
and providers of data
3

4. THE NEED FOR AN EVALUATION METHOD
 The big investments in this area necessitate a systematic
evaluation of these OGD infrastructures, in order to gain a
better understanding and assessment of the multi-
dimensional value they generate
 However, a structured and comprehensive evaluation
methodology is missing.
 This method contributes to filling this gap.
 It presents and validates a methodology for evaluating these
advanced second generation of ODG infrastructures,
 based on a ‘value model approach’,
 i.e. on the estimation of value models of these infrastructures
from users’ ratings.
4

5. INTRODUCTION
 In particular: it assesses various measures of generated
value by OGD infrastructures,
 structured in three layers (associated with efficiency,
effectiveness and users’ future behavior),
 and also the relations among them,
 leading finally to the formation of a value model of the
OGD infrastructure, which enables:
 a deeper understanding of the whole value generation
mechanism of it
 and also a rational definition of IS improvement
priorities
5

6. BACKGROUND / SYNTHESIS
6
Scoping eInfrastructures
Stakeholders Data Acquisition Data Provision Communication
Literature Review
IS Evaluation TAM IS Success Models E-Services

7. Research Streams Insights
IS Evaluation
 IS’s offer various types of benefits, both financial and
non-financial, and also tangible and intangible ones,
which differ among the different types of IS
 it is not possible to formulate one generic IS evaluation
method, which is applicable to all IS
 a comprehensive methodology for evaluating a
particular type of IS should include evaluation of both
its efficiency and its effectiveness, taking into
account its particular characteristics, capabilities and
objectives
7

8. Research Streams Insights
 TAM (Technology Acceptance Model)
 identify the characteristics and factors affecting the attitude
towards using an IS, the intention to use it and finally the
extent of its actual usage
 perceived usefulness and perceived ease of use determine an
individual's intention to use a system with intention to use
serving as a mediator of actual system use
 IS Success Models
 IS evaluation should adopt a layered approach based on the
above interrelated IS success measures (information quality,
system quality, service quality, user satisfaction, actual use,
perceived usefulness, individual impact and organizational
impact) and on the relations among them
8

9. Research Streams Insights
e-Services Evaluation
 frameworks that assess the quality of the capabilities
that the e-service provides to its users
 frameworks that assess the support it provides to users
for performing various tasks and achieving various
objectives, or users’ overall satisfaction
 the above frameworks do not include advanced ways of
processing the evaluation data collected from the
users, in order to maximize the extraction of value-
related knowledge from them
9

10. Our Evaluation Model
Approach
 (a) Efficiency layer: it includes ‘efficiency’ measures,
which assess the quality of the basic capabilities
offered by the e-service to its users.
 (b) Effectiveness layer: it includes ‘effectiveness’
measures, which assess to what extent the e-service
assists the users for completing their tasks and
achieving their objectives.
 (c) Future behaviour layer: it includes measures
assessing to what extent the e-service influences the
future behaviour of its users (e.g. to what extent they
intend to use the e-service again in the future, or
recommend it to friends and colleagues).
10

11. Value Model Definition
11
User-level Feedback Capabilities
Support for
Achieving User
Objectives
Support for
Achieving
Provider Objecti.
Future
Behaviour
Efficiency Level
Effectiveness
Level
Fut. Behavior
Level
Ease of Use
Performance
Data Processing Capabilities
Data Search & Download Capabilities
Data Provision Capabilities
Data Upload Capabilities
Provid-level Feedback Capabilities

12. Value Measures
 The total of 41 value measures (all layers) were
defined where 35 for the 1st layer
 14 common value measures
 15 value measures for users
 06 value measures for providers
 These value measures was then converted to a
question to be included in questionnaires to be
distributed to stakeholders
 A five point Likert scale is used to measure
agreement or disagreement
 2 Questionnaires have been formulated
12

13. Indicative Value Dimension – 1st Level
13
Ease of Use
1.1 Friendliness
The platform provides a user friendly and easy to use
environment.
1.2 Learning Easiness It was easy to learn how to use the platform.
1.3 Aesthetics The web pages look attractive.
1.4
Ease of performing
tasks
It is easy to perform the tasks I want in a small number
of steps.
1.5 Multilingual aspects The platform allows me to work in my own language.
1.6 Personalization
The platform supports user account creation in order
to personalize views and information shown.
1.7 Support & Training
The platform provides high quality of documentation
and online help.

14. Indicative Value Dimension – 1st Level
14
Data Processing Capabilities
7.1 Data Enrichment
The platform provides good capabilities for data
enrichment (i.e. adding new elements - fields)
7.2 Data Cleansing
The platform provides good capabilities for data
cleansing (i.e. detecting and correcting ubiquities
in a dataset)
7.3 Linking
The platform provides good capabilities for linking
datasets.
7.4 Visualisation
The platform provides good capabilities for
visualization of datasets

15. Indicative Value Dimension – 2nd Level
15
Support for Achieving User Objectives
8.1 ACC1
I think that using this platform enables me to do better
research/inquiry and accomplish it more quickly
8.2 ACC2
This platform allows me to draw interesting conclusions on
past government activity
8.3 ACC3
This platform enables me to create successful added-value
electronic services
8.4 ACC4
I am in general highly satisfied with this platform

16. Application : The ENGAGE project
 OGD system to evaluated: ENGAGE - A new multi-
country, multi-lingual open data infrastructure for
researchers, available at www.engagedata.eu
 Target user group: post-graduate students from TU
Delft and Uaegean, trained in the platfom
 Method of user input: electronic questionnaires
 Number of valid questionnaire responses processed: 42
(when the paper was submitted, now more than 100)
16

17. The ENGAGE System
Providing PSI to research
communities and citizens in
a personalised manner
Curating, Annotating,
Harmonising , Visualising
Gathering data from
governmental
organisations and systems
(the Gov Cloud)
Data Linking Semantic Annotation Anonymisation Harmonisation
Visualisation
- Analytics
Search and
Navigation tools
Social
sciences
Data Service
Provision
Infrastructure
Data Curation
Infrastructure
Public Sector Information Sources
Tailored data
services
Research and Industry Governance and
policy making
Citizens and
education
Data
analytics
Knowledge /
Data Mining
Directory services
and direct linking to
data archives
ICT
Citizens
Natural
Sciences and
Engineering
Governance
User groups
Collaboration /
Communities
Personalisation
Single point of
Access
Data Quality Knowledge Mapping
Public Organisations, Repositories, Databases
Law
Policy
Modelling
Automatic curation
algorithms

18. Value Model Estimation Algorithm
18
Value Dimensions
Internal
Consistency
Examination
Value
Dimensions
Variables
Calculation
Average Ratings
Calculation
Regression
Models
Estimation
Correlations
Estimation
Value Models’
Construction
Improvement
Priorities
Identification

19. Estimated Value Model
19
Data Provision
Capabilities
3.03
Data Search & Download
Capabilities
3.03
User-level Feedback
Capabilities
2.97
Ease of Use
3.35
Performance
2.15
Data Processing
Capabilities
3.27
Data Upload Capabilities
2.93
Provider-level Feedback
Capabilities
3.44
Support for Achieving
User Object.
3.17
Support for Achieving
Provider Obj.
3.12
Future Behaviour
3.19
0.624
0.489
0.639
0.760
0.651
0.307
0.680
0.730
0.479
0.379
0.135
0.632
0.735

20. Improvement Priorities
Identification
 Such an OGD infrastructure value model,
 Enables the identification of improvement
priorities,
 which are the first layer OGD systems
capabilities that receive low evaluation by the
users,
 and at the same time have high impact on
higher layers’ value generation

21. Mapping for decision support
22
Lower Ratings
Group
Higher Ratings
Group
data provision
capabilities
data search-
download cap.
data upload
capabilities
performance
provider-level
feedback cap.
ease of use
data processing
capabilities
user-level
feedback capabil.
Lower Impact
Group
Higher Impact
Group
data provision
capabilities
user-level feedback
capab.
performance
provider-level
feedback cap.
data processing
capabilities
ease of use
data search-
download cap.
data upload
capabilities
6-9/01/2014 HICSS 47 - University of the Aegean

22. Conclusions 1/2
23
 This paper has presented a methodology for determining the value
generation mechanism and the improvement priorities of advanced
2nd generation open government data systems,
 which are characterized by the elimination of the distinction
between providers and consumers of such data.
 The proposed methodology assesses a wide range of types of value
generated by such OGD infrastructures for data ‘pro-sumers’,
 and at the same time exploits the relations between the above
types of value (which are usually not exploited and ignored by IS
evaluation methodologies in general),
 leading to additional useful value-related information and more
insights into these advanced ODG systems,
 providing valuable support for making important ODG systems
investment, management and improvement decisions.

23. Conclusions 2/2
24
 An algorithm for advanced processing of users’ evaluation
data has been proposed,
 which leads to the estimation of the value model of the
OGD infrastructure,
 enabling a better understanding of the whole value
generation mechanism of its,
 and the identification of improvement priorities,
 which are the first layer OGD systems capabilities that
receive low evaluation by the users, and at the same time
have high impact on higher layers’ value generated.
 A first application-validation of the proposed methodology
provided interesting conclusions for the OGD systems
developed in ENGAGE infrastructure