ISCRAM Impact Evaluation
•Download as PPTX, PDF•
1 like•311 views
Emerging technologies provide opportunities for the humanitarian responders’ community to enhance the effectiveness of their response to crisissituations. A part of this development can be contributed to a new type of information supply chains -driven by collaboration with digital, online communities- enabling organizations to make better informed decisions. However, how exactly and to what extend this collaboration impacts the decision making process is unknown. To improve these new information exchanges and the corresponding systems, an evaluation method is needed to assess the performance of these processes and systems. This paper builds on existing evaluation methods for information systems and design principles to propose such an impact evaluation framework. The proposed framework has been applied in a case study to demonstrate its potential to identify areas for further improvement in the (online) collaboration between information suppliers and users.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to ISCRAM Impact Evaluation
Similar to ISCRAM Impact Evaluation (20)
Recently uploaded
Recently uploaded (20)
ISCRAM Impact Evaluation
- 1. Designing towards an impact evaluation framework for a collaborative information supply chain KENNY MEESTERS, BARTEL VAN DE WALLE ISCRAM BADEN-BADEN, MAY 2013
- 2. Outline Outline Domain Processes Evaluation types Systems evaluation Evaluation perspectives Scope Measurement Concept Supply Usage Findings Objectives Indicators Conclusion Results V&TC Design Approach Research Future work
- 3. V&TC Data collection •Media •Geo-location •SMS Data processing •Analysis •Verification Dissemenation •Information products •Maps, reports, etc Information consumers •Decision making •Monitoring Outline
- 4. Volunteer Training Open* SBTF Transition Impact Evaluation Decision Makers Data Scramble Data licensing Preparedness UN OCHA IM ISCRAM IMMAP, Google, Mapaction Woodrow, UN, Harvard, OSM Mapaction, GISCorp, Munster ISCRAM, ICT4Peace, Mapaction SBTF, UNV ISCRAM, Harvard, UvT TBC UNV, Munster “The challenge is to improve coordination between the structured humanitarian system and the relatively loosely organized volunteer and technical communities. ” -Valerie Amos, UN Under-Secretary-General V&TC ISCRAM, Harvard, UvT Impact Evaluation Impact Evaluation
- 5. What do we need?What do we want to know?What do we know? BusinessV&TC Practice Theory Impact Evaluation
- 6. What do we need?In general… BusinessV&TC Practice Theory (2) Measure Status quo New situation (1) Define Indicators Situations (3) Analyze Comparison Conclusion
- 7. In general…Applications BusinessV&TC Practice Theory ProjectA 1. Impact evaluation 2. Impact assessment 3. Program evaluation Project B
- 8. BusinessV&TC Practice Theory •Determine how well specific initiatives perform •Adjust and fine tune specific decisions/projects •Determine ‘best’ response •Manage provided solutions •Secure resources •AdvocateV&TC • Impact evaluation • Impact assessment • Program evaluation Use for V&TCApplications
- 9. BusinessV&TC Practice Theory Use forV&TC • Design principles of frameworks • Types • Measurements • Indicators • V&TC • Objective • Scope and focus • Indicators • Evaluate the framework • Case studies • Refine-able, usable tools Next steps
- 10. Next stepsEvaluation types Evaluation Perspective Systems General Formative Resource centered Efficiency oriented Goal centered Effectiveness oriented Summative
- 11. Evaluation typesSystem evaluation Evaluation Perspective Efficiency oriented Effectiveness oriented Systems Resource investment Production capability Resource consumption Organization Organizational performance User performance System performance
- 12. System evaluationEvaluation implementation Organizational performance Department A (Sub) Project A (Sub) Project B (Sub) Project C Department B Evaluation framework Project Management Efficiency oriented Effectiveness oriented
- 13. Scope Overall impact of the response to a crisis Impact of the decision making process on crisis Impact of information products on the decision making process Effect of data processing on information products Impact of data collection on data processing Soft- en hardware impact on the system performance 1. 2. 3. 4. 5. 6. SUPPLIERCONSUMER Evaluation implementation
- 14. level 0: Request / definition level 1: Resource allocation level 2: Team capability level 3: Investments Impact level 1: Support & information level 2: Decision making level 3: Response effectiveness ScopeMeasurements Efficiency-oriented perspective Effectiveness- oriented perspective System implementation Product generation efficiency Response effectiveness V&TC deployment
- 15. MeasurementsIndicators level 1: Resource allocation level 2: Team capability level 1: Support & information level 2: Decision making Impact Objective Performance measure Applied toV&TC System development Facilities allocation Availability of required (tech.) facilities Schedule compliance Time required to setup required systems Requirements definition The clarity of requested products Operational resources Data collection efforts Time/effort required to analyze data System maintenance Time/effort required to maintain system Training/support/comm Efforts for user assistance. Objective Performance measure Applied toV&TC Team capacity Productivity rate Level ofV&TC body deployment Required man-hours The total amount of hours used Operational capability Throughput Products delivered/users served Utilization rate Hours to product ratio Response time Turn-around time on specific requests Objective Performance measure Applied toV&TC System quality Usability Ease of use of information products System features Customization of information products Access / Availability Ease to reach information products Information quality Understandability Presentation of gathered information Consistency Provided information is consistent Importance / Relevance Relevance of provided information Objective Performance measure Applied toV&TC Individual impact Awareness / Recall Better situational awareness Decision effectiveness Enhanced effectiveness of job Individual productivity Increased personal productivity Organizational impact Cost-effective Information products save resources Increased capacity Increased effectiveness of operations Overall productivity Potentially improved outcomes
- 16. 4 suppliers, 7 consumers7 suppliers, 12 consumers IndicatorsCase Study Developpers and entry team Specific knowledge Time critical No budget Geographically seperate Users are ‘unkown’ Developpers vs data entry Expertise available Time limited Limited budget Located in 1 office Direct contact w. users = = ≈ ≈ ≠ ≠ NGODHN
- 17. Case StudyInformation Supply System NGO V&TC FA: SC: RD: Facilities allocation Schedule compliance Requirements definition Data collection efforts System maintenance Training, suppor t and communication Operational resources System development NGO Development V&TC Deployement Resource NGO V&TC DC: SM: TS: Productivit y Required man-hours Throughpu t Utilization rate Response time Operational capability Team capacity NGO Development V&TC Deployement System NGO V&TC PD: MH: Resource NGO V&TC TP: UR: RT: Level 2: Capabilities Level 1: Resources
- 18. Information SupplyInformation use System NGO V&TC SF: AV: US: Inform. NGO V&TC US: CO: IM: Individ. NGO V&TC US: AW: EF: PR: Organ. NGO V&TC OP: CI: CE: US: Level 2: Processes Level 1: Information Availability Usability System features Understandabi lity Consistency Importance NGO Development V&TC Deployement System quality Information quality Usage Awareness Effectiveness ProductivityUsage Cost-effective Capacity increase Overall… NGO Development V&TC Deployement Individual impact Organization impact
- 19. Information useFindings •Agile vs.Waterfall •Organizational use •Strong integration •Requirement analysis •Sample selection •Identifying population •Also for other information supply chains • Difference in system use • Increasing impact • Improving evaluation
- 20. Future work V&TC Feedback: Increase deployment impact Advocacy: Secure resources Manage: Improve products Findings •Historical data •Feedback loops •Add/remove variables •Scope of evaluation • Refinements • Framework design • Application Apply framework Select case Impact Evaluation Framework Select particiapants Control group Conducted Interview Apply framework Select particiapants Control group Conducted Interview Verify results Verify results Refine Framework Statistical analysis Data store Model Refinement loop Impact evaluation outcome Evaluation approach V&TC Feedback: Increase deployment impact Advocacy: Secure resources Manage: Improve products Feedback: Manage pool of resources Advocacy: Common understan- diing of IS impact Manage: Identify gaps, ensur e good fit Coordination Feedback: Manage pool of resources Advocacy: Common understan- diing of IS impact Manage: Identify gaps, ensur e good fit Coordination Feedback: Improve effectivenss by IS use Advocacy: Articulate needs, and require- ments Manage: Improve IS use in future responses Decision Makers Feedback: Improve effectivenss by IS use Advocacy: Articulate needs, and require- ments Manage: Improve IS use in future responses Decision Makers Impact Evaluation for theV&TC: Communicate, Learn, Advocate
Editor's Notes
- First step, how we can determine the impactWhy is it important? We develop a lot of tools and provide servicesHow effective is that, how can we better focus our efforts
- Impact evaluation important part to further steer the development of the V&TC.Understand how the affect the response and how this can be improved.
- Our approach: looking at business to derive theories and practicesThis requires translation
- How do evaluations work, what are they made up of?
- What are evaluations used for?
- What can we use evalautions for?
- How do we build our evaluation framework?
- We have two types:Look at the outcomes or look at the process
- Construct an evaluation framework:Ingredients:Objective (Assesment, evaluate project, evaluate program), Scope, Measurments, IndicatorsAll derived from anotherHow does it fit into a process
- IT is a small part of the entire operationLet start by examinng the scope,Main focus the use of products in the processes and organization of the desiscion makerNot including the impact on the decision made.With each level more factors (confounding variables) are introduced, blurring the measurment.
- Next look at the measurments, again two views:-The efficiency for the generation of products-The effectivenss of the use of those products-Combined they yield the impact
- From measurements we derive indicators. Various existing framework are used to find the indicators and are then translated to the V&TC community.
- Two case studies used to test framwork. It does not give general informationObjective to detect diferences between two IS deploymentsSome similarities some differences. Results are checked with interviews
- Example output of evaluation frameworkGood in data collection and facilitatiesHigh prouctivy: for limited amount of time
- Systems:High availbility and usablity in the V&TC systemsProcesses limited:Capacity increase, cost effective
- Yes, we can design an evaluation framework that provides usefull resultsThe analysis show that the framework is able to pick up on key differences between ‘regular’ and V&TC deployments. Main point: integration and sustainbilityFor framework: embed in process, easy-to-use and part of deployementStill depend on sampling, need quantifable indicators and automation
- The thesis was to demonstrate the feasibility and explore the domain. We succeededNOT to develop an accurate framework but lead by example and discussionThe biggest question: how are we going to use evaluation and wherefor.Importance: manage pool of growing options for DM, and improve quality of systems realtimeThis will determine the refinements and research efforts for further development