Daidalos Evaluation ECIME 2007

Large Scale Research Project, Daidalos Evaluation Framework Frances Cleary Grant Miguel Ponce de Leon TSSG – Waterford Institute of Technology Ireland

What is Daidalos ??
FP6 IST Integrated project
Daidalos aims to provide mobile users with a diverse range of personalized services – seamlessly supported by the underlying technology and transparently provided through a pervasive interface.
The Integration of WP innovations were completed through the selected ‘Nidaros’ Scenario and demonstrator.

Daidalos Evaluation
Daidalos (49 partners), Evaluation was an immense task
co-ordinated and implemented by WP5.
Activity 5.4 Evaluation
and Recommendations
Evaluation Activities & Tasks
Initial scope of the Evaluation work
Identify and define the evaluation criteria
Define Evaluation process, and evaluation mgmt process
Analyse evaluation results :- concluding with set of recommendations

Evaluation Timeframe
Identify Evaluation profile procedures. (first 2 months)
Definition of Evaluation report scoreboard. (first 2 months)
Assessment of Individual work Packages (continuous over 8 months)
Definition and completion of achievements and Impact criteria. (continuous over 8 months)
Completion of evaluation report scoreboard (last 2 months)
Completion of recommendations and conclusions (last 3 months)

What to Capture? Where to start?

Evaluation Profile Procedures Formative Formative evaluation is an overall assessment of activities as they progress prior to their completion. Summative Summative evaluation provides information on a projects ability to complete assigned and set objectives and milestones.

Daidalos Evaluation management Structure

Daidalos Evaluation Methodology
Identification of specific work package deliverables suitable to be evaluated
Analysis of these deliverables with end aim of gathering technical recommendations
Identification of processes and result related reports.
Analysis of these reports to obtain non technical recommendations

Daidalos Evaluation Methodology
Relevance (IST objectives, Socio-Economic Aspects, Gender Action plan)
Potential Impact (Contribution to standards, Disseminating Knowledge, Training
Scientific & Technical Excellence ( Activity/deliverable Innovations Achieved, Barriers in performing S&T excellence)
Resource Mobilisation (Integration management, Integration of resources, Test bed resources)

Evaluation Report Scoreboard (ERS)
This ERS included all the criterion categories selected to be evaluated within Daidalos Phase I.
Each Criterion was awarded a score based on feedback and input from specified and targeted focus groups .
For Example for Criterion – Resource Mobilisation the following focus groups were targeted to provide feedback and scores.
Management Team
Work Package leaders
Test bed Managers
WP5 partners

Evaluation Report Scoreboard (ERS)
1. Relevance
Sub-criteria
Relevance to IST Objective: European Leadership in the Generic & Applied Technologies
Consolidated European Approach to technology, system & services.
Socio-Economic Aspects
Gender Action Plan
Mark:
2. Potential impact
Sub-criteria
Contribution to Standards & Management of the Project Impact
Disseminating Knowledge
Training
Mark:
3. S&T excellence
Sub-criteria
Activity Innovations achieved
Deliverable Innovations Achieved
Barriers in performing Scientific and Technical Excellence
Mark:
4.Resource Mobilisation
Sub-criteria
Integration management Activities
Integration of resources
Test Bed resources
Mark: Overall remarks (score out of 20) Overall score:

ERS Results
Relevance: Proactive Communication externally on the projects innovations and achievements. Liaison Managers.
Potential Impact: Increase interaction with standard bodies.
Phase II present working prototypes to public
Increase external project impact.
Scientific & Technical Excellence : Use more effectively vertical teams and scientific forums.
Resource Mobilisation: Improve integration processes.

Daidalos scenario : Qualitative Evaluation
Qualitative evaluation of Nidaros demonstrator
Daidalos Nidaros scenario video
Distribution of a User-centric Questionnaire
The questionnaire gathers factual information about the user and asks a series of questions regarding the video as well as general aspects of the Daidalos framework.
Understand more about user’s thoughts and feelings towards this technology and their likelihood of using such technologies if they become available.

Feasibility of Daidalos Evaluation Framework
1. Formative/summative Tasks: helped identify an initial idea of tasks,
surveys and project milestone documentation to be analysed during
evaluation phase.
2. Detailed Timeframe extremely important: well defined deadlines, prioritise workload.
3. Management structure must be logical and adapt to projects structure.
Clear points of contact and assigned leaders for each evaluation sub-phase.
4. Evaluation report scoreboard provided a means to evaluate the project on a topical level.
Combining focus groups as part of this process, improved the output and quality level of the recommendations and scoring.
5. ERS : Visually conveyed the level of achievement of the project.

End Of Presentation
Thank You For Your Time

Contact details : Frances Cleary Grant
Email : [email_address]
Organisation: TSSG , Waterford Institute Of Technology

Daidalos Evaluation ECIME 2007

by Miguel Ponce de Leon @ TSSG / Waterford Institute of Technology on Dec 16, 2009

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 11

Statistics

Daidalos Evaluation ECIME 2007 — Presentation Transcript

http://www.miguelpdl.com	9
http://www.tssg.org	2