• Save
Daidalos Evaluation  ECIME 2007
Upcoming SlideShare
Loading in...5
×
 
  • 806 views

Presentation of the paper "Large Scale Research Project, Daidalos Evaluation Framework" at ECIME 2007

Presentation of the paper "Large Scale Research Project, Daidalos Evaluation Framework" at ECIME 2007

Statistics

Views

Total Views
806
Views on SlideShare
760
Embed Views
46

Actions

Likes
0
Downloads
0
Comments
0

3 Embeds 46

http://www.miguelpdl.com 38
http://miguelpdl.com 6
http://www.tssg.org 2

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • With the Daidalos Architecture consisting of 3 technical work packages WP2 “ Network Integration” WP3 “Service and Network Management” WP4 “ Pervasive Systems” WP5 “Integrated System Evaluation” aim was to materialise the achievements of work packages into practical assessment to allow for concept validation.

Daidalos Evaluation  ECIME 2007 Daidalos Evaluation ECIME 2007 Presentation Transcript

  • Large Scale Research Project, Daidalos Evaluation Framework Frances Cleary Grant Miguel Ponce de Leon TSSG – Waterford Institute of Technology Ireland
  • What is Daidalos ??
    • FP6 IST Integrated project
    • Daidalos aims to provide mobile users with a diverse range of personalized services – seamlessly supported by the underlying technology and transparently provided through a pervasive interface.
    • The Integration of WP innovations were completed through the selected ‘Nidaros’ Scenario and demonstrator.
  • Daidalos Evaluation
    • Daidalos (49 partners), Evaluation was an immense task
    • co-ordinated and implemented by WP5.
    • Activity 5.4 Evaluation
    • and Recommendations
    • Evaluation Activities & Tasks
    • Initial scope of the Evaluation work
    • Identify and define the evaluation criteria
    • Define Evaluation process, and evaluation mgmt process
    • Analyse evaluation results :- concluding with set of recommendations
  • Evaluation Timeframe
    • Identify Evaluation profile procedures. (first 2 months)
    • Definition of Evaluation report scoreboard. (first 2 months)
    • Assessment of Individual work Packages (continuous over 8 months)
    • Definition and completion of achievements and Impact criteria. (continuous over 8 months)
    • Completion of evaluation report scoreboard (last 2 months)
    • Completion of recommendations and conclusions (last 3 months)
  • What to Capture? Where to start?
  • Evaluation Profile Procedures Formative Formative evaluation is an overall assessment of activities as they progress prior to their completion. Summative Summative evaluation provides information on a projects ability to complete assigned and set objectives and milestones.
  • Daidalos Evaluation management Structure
  • Daidalos Evaluation Methodology
        • Identification of specific work package deliverables suitable to be evaluated
        • Analysis of these deliverables with end aim of gathering technical recommendations
        • Identification of processes and result related reports.
        • Analysis of these reports to obtain non technical recommendations
  • Daidalos Evaluation Methodology
        • Relevance (IST objectives, Socio-Economic Aspects, Gender Action plan)
        • Potential Impact (Contribution to standards, Disseminating Knowledge, Training
        • Scientific & Technical Excellence ( Activity/deliverable Innovations Achieved, Barriers in performing S&T excellence)
        • Resource Mobilisation (Integration management, Integration of resources, Test bed resources)
  • Evaluation Report Scoreboard (ERS)
    • This ERS included all the criterion categories selected to be evaluated within Daidalos Phase I.
    • Each Criterion was awarded a score based on feedback and input from specified and targeted focus groups .
    • For Example for Criterion – Resource Mobilisation the following focus groups were targeted to provide feedback and scores.
        • Management Team
        • Work Package leaders
        • Test bed Managers
        • WP5 partners
  • Evaluation Report Scoreboard (ERS)
    • 1. Relevance
    • Sub-criteria
    • Relevance to IST Objective: European Leadership in the Generic & Applied Technologies
    • Consolidated European Approach to technology, system & services.
    • Socio-Economic Aspects
    • Gender Action Plan
    Mark:
    • 2. Potential impact
    • Sub-criteria
    • Contribution to Standards & Management of the Project Impact
    • Disseminating Knowledge
    • Training
    Mark:
    • 3. S&T excellence
    • Sub-criteria
    • Activity Innovations achieved
    • Deliverable Innovations Achieved
    • Barriers in performing Scientific and Technical Excellence
    Mark:
    • 4.Resource Mobilisation
    • Sub-criteria
    • Integration management Activities
    • Integration of resources
    • Test Bed resources
    Mark: Overall remarks (score out of 20) Overall score:
  • ERS Results
        • Relevance: Proactive Communication externally on the projects innovations and achievements. Liaison Managers.
        • Potential Impact: Increase interaction with standard bodies.
            • Phase II present working prototypes to public
            • Increase external project impact.
        • Scientific & Technical Excellence : Use more effectively vertical teams and scientific forums.
        • Resource Mobilisation: Improve integration processes.
  • Daidalos scenario : Qualitative Evaluation
    • Qualitative evaluation of Nidaros demonstrator
      • Daidalos Nidaros scenario video
      • Distribution of a User-centric Questionnaire
      • The questionnaire gathers factual information about the user and asks a series of questions regarding the video as well as general aspects of the Daidalos framework.
      • Understand more about user’s thoughts and feelings towards this technology and their likelihood of using such technologies if they become available.
  • Feasibility of Daidalos Evaluation Framework
    • 1. Formative/summative Tasks: helped identify an initial idea of tasks,
    • surveys and project milestone documentation to be analysed during
    • evaluation phase.
    • 2. Detailed Timeframe extremely important: well defined deadlines, prioritise workload.
    • 3. Management structure must be logical and adapt to projects structure.
      • Clear points of contact and assigned leaders for each evaluation sub-phase.
    • 4. Evaluation report scoreboard provided a means to evaluate the project on a topical level.
      • Combining focus groups as part of this process, improved the output and quality level of the recommendations and scoring.
    • 5. ERS : Visually conveyed the level of achievement of the project.
    • End Of Presentation
    • Thank You For Your Time
    • Contact details : Frances Cleary Grant
    • Email : [email_address]
    • Organisation: TSSG , Waterford Institute Of Technology