Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Daidalos Evaluation ECIME 2007


Published on

Presentation of the paper "Large Scale Research Project, Daidalos Evaluation Framework" at ECIME 2007

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Daidalos Evaluation ECIME 2007

  1. 1. Large Scale Research Project, Daidalos Evaluation Framework Frances Cleary Grant Miguel Ponce de Leon TSSG – Waterford Institute of Technology Ireland
  2. 2. What is Daidalos ?? <ul><li>FP6 IST Integrated project </li></ul><ul><li>Daidalos aims to provide mobile users with a diverse range of personalized services – seamlessly supported by the underlying technology and transparently provided through a pervasive interface. </li></ul><ul><li>The Integration of WP innovations were completed through the selected ‘Nidaros’ Scenario and demonstrator. </li></ul>
  3. 3. Daidalos Evaluation <ul><li>Daidalos (49 partners), Evaluation was an immense task </li></ul><ul><li>co-ordinated and implemented by WP5. </li></ul><ul><li>Activity 5.4 Evaluation </li></ul><ul><li>and Recommendations </li></ul><ul><li>Evaluation Activities & Tasks </li></ul><ul><li>Initial scope of the Evaluation work </li></ul><ul><li>Identify and define the evaluation criteria </li></ul><ul><li>Define Evaluation process, and evaluation mgmt process </li></ul><ul><li>Analyse evaluation results :- concluding with set of recommendations </li></ul>
  4. 4. Evaluation Timeframe <ul><li>Identify Evaluation profile procedures. (first 2 months) </li></ul><ul><li>Definition of Evaluation report scoreboard. (first 2 months) </li></ul><ul><li>Assessment of Individual work Packages (continuous over 8 months) </li></ul><ul><li>Definition and completion of achievements and Impact criteria. (continuous over 8 months) </li></ul><ul><li>Completion of evaluation report scoreboard (last 2 months) </li></ul><ul><li>Completion of recommendations and conclusions (last 3 months) </li></ul>
  5. 5. What to Capture? Where to start?
  6. 6. Evaluation Profile Procedures Formative Formative evaluation is an overall assessment of activities as they progress prior to their completion. Summative Summative evaluation provides information on a projects ability to complete assigned and set objectives and milestones.
  7. 7. Daidalos Evaluation management Structure
  8. 8. Daidalos Evaluation Methodology <ul><ul><ul><li>Identification of specific work package deliverables suitable to be evaluated </li></ul></ul></ul><ul><ul><ul><li>Analysis of these deliverables with end aim of gathering technical recommendations </li></ul></ul></ul><ul><ul><ul><li>Identification of processes and result related reports. </li></ul></ul></ul><ul><ul><ul><li>Analysis of these reports to obtain non technical recommendations </li></ul></ul></ul>
  9. 9. Daidalos Evaluation Methodology <ul><ul><ul><li>Relevance (IST objectives, Socio-Economic Aspects, Gender Action plan) </li></ul></ul></ul><ul><ul><ul><li>Potential Impact (Contribution to standards, Disseminating Knowledge, Training </li></ul></ul></ul><ul><ul><ul><li>Scientific & Technical Excellence ( Activity/deliverable Innovations Achieved, Barriers in performing S&T excellence) </li></ul></ul></ul><ul><ul><ul><li>Resource Mobilisation (Integration management, Integration of resources, Test bed resources) </li></ul></ul></ul>
  10. 10. Evaluation Report Scoreboard (ERS) <ul><li>This ERS included all the criterion categories selected to be evaluated within Daidalos Phase I. </li></ul><ul><li>Each Criterion was awarded a score based on feedback and input from specified and targeted focus groups . </li></ul><ul><li>For Example for Criterion – Resource Mobilisation the following focus groups were targeted to provide feedback and scores. </li></ul><ul><ul><ul><li>Management Team </li></ul></ul></ul><ul><ul><ul><li>Work Package leaders </li></ul></ul></ul><ul><ul><ul><li>Test bed Managers </li></ul></ul></ul><ul><ul><ul><li>WP5 partners </li></ul></ul></ul>
  11. 11. Evaluation Report Scoreboard (ERS) <ul><li>1. Relevance </li></ul><ul><li>Sub-criteria </li></ul><ul><li>Relevance to IST Objective: European Leadership in the Generic & Applied Technologies </li></ul><ul><li>Consolidated European Approach to technology, system & services. </li></ul><ul><li>Socio-Economic Aspects </li></ul><ul><li>Gender Action Plan </li></ul>Mark: <ul><li>2. Potential impact </li></ul><ul><li>Sub-criteria </li></ul><ul><li>Contribution to Standards & Management of the Project Impact </li></ul><ul><li>Disseminating Knowledge </li></ul><ul><li>Training </li></ul>Mark: <ul><li>3. S&T excellence </li></ul><ul><li>Sub-criteria </li></ul><ul><li>Activity Innovations achieved </li></ul><ul><li>Deliverable Innovations Achieved </li></ul><ul><li>Barriers in performing Scientific and Technical Excellence </li></ul>Mark: <ul><li>4.Resource Mobilisation </li></ul><ul><li>Sub-criteria </li></ul><ul><li>Integration management Activities </li></ul><ul><li>Integration of resources </li></ul><ul><li>Test Bed resources </li></ul>Mark: Overall remarks (score out of 20) Overall score:
  12. 12. ERS Results <ul><ul><ul><li>Relevance: Proactive Communication externally on the projects innovations and achievements. Liaison Managers. </li></ul></ul></ul><ul><ul><ul><li>Potential Impact: Increase interaction with standard bodies. </li></ul></ul></ul><ul><ul><ul><ul><ul><li>Phase II present working prototypes to public </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Increase external project impact. </li></ul></ul></ul></ul></ul><ul><ul><ul><li>Scientific & Technical Excellence : Use more effectively vertical teams and scientific forums. </li></ul></ul></ul><ul><ul><ul><li>Resource Mobilisation: Improve integration processes. </li></ul></ul></ul>
  13. 13. Daidalos scenario : Qualitative Evaluation <ul><li>Qualitative evaluation of Nidaros demonstrator </li></ul><ul><ul><li>Daidalos Nidaros scenario video </li></ul></ul><ul><ul><li>Distribution of a User-centric Questionnaire </li></ul></ul><ul><ul><li>The questionnaire gathers factual information about the user and asks a series of questions regarding the video as well as general aspects of the Daidalos framework. </li></ul></ul><ul><ul><li>Understand more about user’s thoughts and feelings towards this technology and their likelihood of using such technologies if they become available. </li></ul></ul>
  14. 14. Feasibility of Daidalos Evaluation Framework <ul><li>1. Formative/summative Tasks: helped identify an initial idea of tasks, </li></ul><ul><li>surveys and project milestone documentation to be analysed during </li></ul><ul><li>evaluation phase. </li></ul><ul><li>2. Detailed Timeframe extremely important: well defined deadlines, prioritise workload. </li></ul><ul><li>3. Management structure must be logical and adapt to projects structure. </li></ul><ul><ul><li>Clear points of contact and assigned leaders for each evaluation sub-phase. </li></ul></ul><ul><li>4. Evaluation report scoreboard provided a means to evaluate the project on a topical level. </li></ul><ul><ul><li>Combining focus groups as part of this process, improved the output and quality level of the recommendations and scoring. </li></ul></ul><ul><li>5. ERS : Visually conveyed the level of achievement of the project. </li></ul>
  15. 15. <ul><li>End Of Presentation </li></ul><ul><li> Thank You For Your Time </li></ul><ul><li> </li></ul><ul><li>Contact details : Frances Cleary Grant </li></ul><ul><li>Email : [email_address] </li></ul><ul><li>Organisation: TSSG , Waterford Institute Of Technology </li></ul>