Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Designing towards an impact
evaluation framework for a
collaborative information supply chain
KENNY MEESTERS, BARTEL VAN D...
Outline
Outline
Domain Processes
Evaluation
types
Systems
evaluation
Evaluation
perspectives
Scope Measurement
Concept Sup...
V&TC
Data collection
•Media
•Geo-location
•SMS
Data
processing
•Analysis
•Verification
Dissemenation
•Information
products...
Volunteer Training
Open*
SBTF
Transition
Impact Evaluation Decision Makers
Data Scramble
Data licensing
Preparedness
UN OC...
What do we need?What do we want to know?What do we know?
BusinessV&TC
Practice
Theory
Impact Evaluation
What do we need?In general…
BusinessV&TC
Practice
Theory (2) Measure
Status quo
New situation
(1) Define
Indicators
Situat...
In general…Applications
BusinessV&TC
Practice
Theory
ProjectA
1. Impact evaluation
2. Impact assessment
3. Program evaluat...
BusinessV&TC
Practice
Theory
•Determine how well specific initiatives perform
•Adjust and fine tune specific decisions/pro...
BusinessV&TC
Practice
Theory
Use forV&TC
• Design principles of frameworks
• Types
• Measurements
• Indicators
• V&TC
• Ob...
Next stepsEvaluation types
Evaluation
Perspective
Systems
General Formative
Resource
centered
Efficiency
oriented
Goal cen...
Evaluation typesSystem evaluation
Evaluation
Perspective
Efficiency
oriented
Effectiveness
oriented
Systems
Resource
inves...
System evaluationEvaluation implementation
Organizational performance
Department A
(Sub)
Project A
(Sub)
Project B
(Sub)
P...
Scope
Overall impact of
the response to a crisis
Impact of the decision making
process on crisis
Impact of information pro...
level 0:
Request / definition
level 1:
Resource allocation
level 2:
Team capability
level 3:
Investments
Impact
level 1:
S...
MeasurementsIndicators
level 1:
Resource allocation
level 2:
Team capability
level 1:
Support & information
level 2:
Decis...
4 suppliers, 7 consumers7 suppliers, 12 consumers
IndicatorsCase Study
Developpers and entry team
Specific knowledge
Time ...
Case StudyInformation Supply
System NGO V&TC
FA:
SC:
RD:
Facilities
allocation
Schedule
compliance
Requirements
definition...
Information SupplyInformation use
System NGO V&TC
SF:
AV:
US:
Inform. NGO V&TC
US:
CO:
IM:
Individ. NGO V&TC
US:
AW:
EF:
P...
Information useFindings
•Agile vs.Waterfall
•Organizational use
•Strong integration
•Requirement analysis
•Sample selectio...
Future work
V&TC
Feedback:
Increase
deployment
impact
Advocacy:
Secure
resources
Manage:
Improve
products
Findings
•Histor...
Upcoming SlideShare
Loading in …5
×

of

ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 1 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 2 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 3 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 4 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 5 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 6 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 7 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 8 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 9 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 10 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 11 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 12 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 13 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 14 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 15 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 16 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 17 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 18 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 19 ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain Slide 20
Upcoming SlideShare
DRL Field Research Philippines: A Journey into the Information Disaster
Next

2 Likes

Share

ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain

Author: Kenny Meesters, Bartel van de walle
Tilburg University

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

ISCRAM 2013: Designing towards an impact evaluation framework for a collaborative information supply chain

  1. 1. Designing towards an impact evaluation framework for a collaborative information supply chain KENNY MEESTERS, BARTEL VAN DE WALLE ISCRAM BADEN-BADEN, MAY 2013
  2. 2. Outline Outline Domain Processes Evaluation types Systems evaluation Evaluation perspectives Scope Measurement Concept Supply Usage Findings Objectives Indicators Conclusion Results V&TC Design Approach Research Future work
  3. 3. V&TC Data collection •Media •Geo-location •SMS Data processing •Analysis •Verification Dissemenation •Information products •Maps, reports, etc Information consumers •Decision making •Monitoring Outline
  4. 4. Volunteer Training Open* SBTF Transition Impact Evaluation Decision Makers Data Scramble Data licensing Preparedness UN OCHA IM ISCRAM IMMAP, Google, Mapaction Woodrow, UN, Harvard, OSM Mapaction, GISCorp, Munster ISCRAM, ICT4Peace, Mapaction SBTF, UNV ISCRAM, Harvard, UvT TBC UNV, Munster “The challenge is to improve coordination between the structured humanitarian system and the relatively loosely organized volunteer and technical communities. ” -Valerie Amos, UN Under-Secretary-General V&TC ISCRAM, Harvard, UvT Impact Evaluation Impact Evaluation
  5. 5. What do we need?What do we want to know?What do we know? BusinessV&TC Practice Theory Impact Evaluation
  6. 6. What do we need?In general… BusinessV&TC Practice Theory (2) Measure Status quo New situation (1) Define Indicators Situations (3) Analyze Comparison Conclusion
  7. 7. In general…Applications BusinessV&TC Practice Theory ProjectA 1. Impact evaluation 2. Impact assessment 3. Program evaluation Project B
  8. 8. BusinessV&TC Practice Theory •Determine how well specific initiatives perform •Adjust and fine tune specific decisions/projects •Determine ‘best’ response •Manage provided solutions •Secure resources •AdvocateV&TC • Impact evaluation • Impact assessment • Program evaluation Use for V&TCApplications
  9. 9. BusinessV&TC Practice Theory Use forV&TC • Design principles of frameworks • Types • Measurements • Indicators • V&TC • Objective • Scope and focus • Indicators • Evaluate the framework • Case studies • Refine-able, usable tools Next steps
  10. 10. Next stepsEvaluation types Evaluation Perspective Systems General Formative Resource centered Efficiency oriented Goal centered Effectiveness oriented Summative
  11. 11. Evaluation typesSystem evaluation Evaluation Perspective Efficiency oriented Effectiveness oriented Systems Resource investment Production capability Resource consumption Organization Organizational performance User performance System performance
  12. 12. System evaluationEvaluation implementation Organizational performance Department A (Sub) Project A (Sub) Project B (Sub) Project C Department B Evaluation framework Project Management Efficiency oriented Effectiveness oriented
  13. 13. Scope Overall impact of the response to a crisis Impact of the decision making process on crisis Impact of information products on the decision making process Effect of data processing on information products Impact of data collection on data processing Soft- en hardware impact on the system performance 1. 2. 3. 4. 5. 6. SUPPLIERCONSUMER Evaluation implementation
  14. 14. level 0: Request / definition level 1: Resource allocation level 2: Team capability level 3: Investments Impact level 1: Support & information level 2: Decision making level 3: Response effectiveness ScopeMeasurements Efficiency-oriented perspective Effectiveness- oriented perspective System implementation Product generation efficiency Response effectiveness V&TC deployment
  15. 15. MeasurementsIndicators level 1: Resource allocation level 2: Team capability level 1: Support & information level 2: Decision making Impact Objective Performance measure Applied toV&TC System development Facilities allocation Availability of required (tech.) facilities Schedule compliance Time required to setup required systems Requirements definition The clarity of requested products Operational resources Data collection efforts Time/effort required to analyze data System maintenance Time/effort required to maintain system Training/support/comm Efforts for user assistance. Objective Performance measure Applied toV&TC Team capacity Productivity rate Level ofV&TC body deployment Required man-hours The total amount of hours used Operational capability Throughput Products delivered/users served Utilization rate Hours to product ratio Response time Turn-around time on specific requests Objective Performance measure Applied toV&TC System quality Usability Ease of use of information products System features Customization of information products Access / Availability Ease to reach information products Information quality Understandability Presentation of gathered information Consistency Provided information is consistent Importance / Relevance Relevance of provided information Objective Performance measure Applied toV&TC Individual impact Awareness / Recall Better situational awareness Decision effectiveness Enhanced effectiveness of job Individual productivity Increased personal productivity Organizational impact Cost-effective Information products save resources Increased capacity Increased effectiveness of operations Overall productivity Potentially improved outcomes
  16. 16. 4 suppliers, 7 consumers7 suppliers, 12 consumers IndicatorsCase Study Developpers and entry team Specific knowledge Time critical No budget Geographically seperate Users are ‘unkown’ Developpers vs data entry Expertise available Time limited Limited budget Located in 1 office Direct contact w. users = = ≈ ≈ ≠ ≠ NGODHN
  17. 17. Case StudyInformation Supply System NGO V&TC FA: SC: RD: Facilities allocation Schedule compliance Requirements definition Data collection efforts System maintenance Training, suppor t and communication Operational resources System development NGO Development V&TC Deployement Resource NGO V&TC DC: SM: TS: Productivit y Required man-hours Throughpu t Utilization rate Response time Operational capability Team capacity NGO Development V&TC Deployement System NGO V&TC PD: MH: Resource NGO V&TC TP: UR: RT: Level 2: Capabilities Level 1: Resources
  18. 18. Information SupplyInformation use System NGO V&TC SF: AV: US: Inform. NGO V&TC US: CO: IM: Individ. NGO V&TC US: AW: EF: PR: Organ. NGO V&TC OP: CI: CE: US: Level 2: Processes Level 1: Information Availability Usability System features Understandabi lity Consistency Importance NGO Development V&TC Deployement System quality Information quality Usage Awareness Effectiveness ProductivityUsage Cost-effective Capacity increase Overall… NGO Development V&TC Deployement Individual impact Organization impact
  19. 19. Information useFindings •Agile vs.Waterfall •Organizational use •Strong integration •Requirement analysis •Sample selection •Identifying population •Also for other information supply chains • Difference in system use • Increasing impact • Improving evaluation
  20. 20. Future work V&TC Feedback: Increase deployment impact Advocacy: Secure resources Manage: Improve products Findings •Historical data •Feedback loops •Add/remove variables •Scope of evaluation • Refinements • Framework design • Application Apply framework Select case Impact Evaluation Framework Select particiapants Control group Conducted Interview Apply framework Select particiapants Control group Conducted Interview Verify results Verify results Refine Framework Statistical analysis Data store Model Refinement loop Impact evaluation outcome Evaluation approach V&TC Feedback: Increase deployment impact Advocacy: Secure resources Manage: Improve products Feedback: Manage pool of resources Advocacy: Common understan- diing of IS impact Manage: Identify gaps, ensur e good fit Coordination Feedback: Manage pool of resources Advocacy: Common understan- diing of IS impact Manage: Identify gaps, ensur e good fit Coordination Feedback: Improve effectivenss by IS use Advocacy: Articulate needs, and require- ments Manage: Improve IS use in future responses Decision Makers Feedback: Improve effectivenss by IS use Advocacy: Articulate needs, and require- ments Manage: Improve IS use in future responses Decision Makers Impact Evaluation for theV&TC: Communicate, Learn, Advocate
  • sibosol

    Apr. 18, 2015
  • holemansp

    Nov. 27, 2013

Author: Kenny Meesters, Bartel van de walle Tilburg University

Views

Total views

1,224

On Slideshare

0

From embeds

0

Number of embeds

568

Actions

Downloads

0

Shares

0

Comments

0

Likes

2

×