Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain

1,063 views

Published on

Author: Kenny Meesters, Bartel van de walle
Tilburg University

Published in: Education, Technology, Business
  • Be the first to comment

ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain

  1. 1. Designing towards an impact evaluation framework for a collaborative information supply chain KENNY MEESTERS, BARTEL VAN DE WALLE ISCRAM BADEN-BADEN, MAY 2013
  2. 2. Outline Outline Domain Processes Evaluation types Systems evaluation Evaluation perspectives Scope Measurement Concept Supply Usage Findings Objectives Indicators Conclusion Results V&TC Design Approach Research Future work
  3. 3. V&TC Data collection •Media •Geo-location •SMS Data processing •Analysis •Verification Dissemenation •Information products •Maps, reports, etc Information consumers •Decision making •Monitoring Outline
  4. 4. Volunteer Training Open* SBTF Transition Impact Evaluation Decision Makers Data Scramble Data licensing Preparedness UN OCHA IM ISCRAM IMMAP, Google, Mapaction Woodrow, UN, Harvard, OSM Mapaction, GISCorp, Munster ISCRAM, ICT4Peace, Mapaction SBTF, UNV ISCRAM, Harvard, UvT TBC UNV, Munster “The challenge is to improve coordination between the structured humanitarian system and the relatively loosely organized volunteer and technical communities. ” -Valerie Amos, UN Under-Secretary-General V&TC ISCRAM, Harvard, UvT Impact Evaluation Impact Evaluation
  5. 5. What do we need?What do we want to know?What do we know? BusinessV&TC Practice Theory Impact Evaluation
  6. 6. What do we need?In general… BusinessV&TC Practice Theory (2) Measure Status quo New situation (1) Define Indicators Situations (3) Analyze Comparison Conclusion
  7. 7. In general…Applications BusinessV&TC Practice Theory ProjectA 1. Impact evaluation 2. Impact assessment 3. Program evaluation Project B
  8. 8. BusinessV&TC Practice Theory •Determine how well specific initiatives perform •Adjust and fine tune specific decisions/projects •Determine ‘best’ response •Manage provided solutions •Secure resources •AdvocateV&TC • Impact evaluation • Impact assessment • Program evaluation Use for V&TCApplications
  9. 9. BusinessV&TC Practice Theory Use forV&TC • Design principles of frameworks • Types • Measurements • Indicators • V&TC • Objective • Scope and focus • Indicators • Evaluate the framework • Case studies • Refine-able, usable tools Next steps
  10. 10. Next stepsEvaluation types Evaluation Perspective Systems General Formative Resource centered Efficiency oriented Goal centered Effectiveness oriented Summative
  11. 11. Evaluation typesSystem evaluation Evaluation Perspective Efficiency oriented Effectiveness oriented Systems Resource investment Production capability Resource consumption Organization Organizational performance User performance System performance
  12. 12. System evaluationEvaluation implementation Organizational performance Department A (Sub) Project A (Sub) Project B (Sub) Project C Department B Evaluation framework Project Management Efficiency oriented Effectiveness oriented
  13. 13. Scope Overall impact of the response to a crisis Impact of the decision making process on crisis Impact of information products on the decision making process Effect of data processing on information products Impact of data collection on data processing Soft- en hardware impact on the system performance 1. 2. 3. 4. 5. 6. SUPPLIERCONSUMER Evaluation implementation
  14. 14. level 0: Request / definition level 1: Resource allocation level 2: Team capability level 3: Investments Impact level 1: Support & information level 2: Decision making level 3: Response effectiveness ScopeMeasurements Efficiency-oriented perspective Effectiveness- oriented perspective System implementation Product generation efficiency Response effectiveness V&TC deployment
  15. 15. MeasurementsIndicators level 1: Resource allocation level 2: Team capability level 1: Support & information level 2: Decision making Impact Objective Performance measure Applied toV&TC System development Facilities allocation Availability of required (tech.) facilities Schedule compliance Time required to setup required systems Requirements definition The clarity of requested products Operational resources Data collection efforts Time/effort required to analyze data System maintenance Time/effort required to maintain system Training/support/comm Efforts for user assistance. Objective Performance measure Applied toV&TC Team capacity Productivity rate Level ofV&TC body deployment Required man-hours The total amount of hours used Operational capability Throughput Products delivered/users served Utilization rate Hours to product ratio Response time Turn-around time on specific requests Objective Performance measure Applied toV&TC System quality Usability Ease of use of information products System features Customization of information products Access / Availability Ease to reach information products Information quality Understandability Presentation of gathered information Consistency Provided information is consistent Importance / Relevance Relevance of provided information Objective Performance measure Applied toV&TC Individual impact Awareness / Recall Better situational awareness Decision effectiveness Enhanced effectiveness of job Individual productivity Increased personal productivity Organizational impact Cost-effective Information products save resources Increased capacity Increased effectiveness of operations Overall productivity Potentially improved outcomes
  16. 16. 4 suppliers, 7 consumers7 suppliers, 12 consumers IndicatorsCase Study Developpers and entry team Specific knowledge Time critical No budget Geographically seperate Users are ‘unkown’ Developpers vs data entry Expertise available Time limited Limited budget Located in 1 office Direct contact w. users = = ≈ ≈ ≠ ≠ NGODHN
  17. 17. Case StudyInformation Supply System NGO V&TC FA: SC: RD: Facilities allocation Schedule compliance Requirements definition Data collection efforts System maintenance Training, suppor t and communication Operational resources System development NGO Development V&TC Deployement Resource NGO V&TC DC: SM: TS: Productivit y Required man-hours Throughpu t Utilization rate Response time Operational capability Team capacity NGO Development V&TC Deployement System NGO V&TC PD: MH: Resource NGO V&TC TP: UR: RT: Level 2: Capabilities Level 1: Resources
  18. 18. Information SupplyInformation use System NGO V&TC SF: AV: US: Inform. NGO V&TC US: CO: IM: Individ. NGO V&TC US: AW: EF: PR: Organ. NGO V&TC OP: CI: CE: US: Level 2: Processes Level 1: Information Availability Usability System features Understandabi lity Consistency Importance NGO Development V&TC Deployement System quality Information quality Usage Awareness Effectiveness ProductivityUsage Cost-effective Capacity increase Overall… NGO Development V&TC Deployement Individual impact Organization impact
  19. 19. Information useFindings •Agile vs.Waterfall •Organizational use •Strong integration •Requirement analysis •Sample selection •Identifying population •Also for other information supply chains • Difference in system use • Increasing impact • Improving evaluation
  20. 20. Future work V&TC Feedback: Increase deployment impact Advocacy: Secure resources Manage: Improve products Findings •Historical data •Feedback loops •Add/remove variables •Scope of evaluation • Refinements • Framework design • Application Apply framework Select case Impact Evaluation Framework Select particiapants Control group Conducted Interview Apply framework Select particiapants Control group Conducted Interview Verify results Verify results Refine Framework Statistical analysis Data store Model Refinement loop Impact evaluation outcome Evaluation approach V&TC Feedback: Increase deployment impact Advocacy: Secure resources Manage: Improve products Feedback: Manage pool of resources Advocacy: Common understan- diing of IS impact Manage: Identify gaps, ensur e good fit Coordination Feedback: Manage pool of resources Advocacy: Common understan- diing of IS impact Manage: Identify gaps, ensur e good fit Coordination Feedback: Improve effectivenss by IS use Advocacy: Articulate needs, and require- ments Manage: Improve IS use in future responses Decision Makers Feedback: Improve effectivenss by IS use Advocacy: Articulate needs, and require- ments Manage: Improve IS use in future responses Decision Makers Impact Evaluation for theV&TC: Communicate, Learn, Advocate

×