IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper session 4 Giel Ton
CAPTURING CHANGES INORGANIZATIONAL SOCIAL CAPITALAN IMPACT EVALUATION OF A SMALL-GRANT INNOVATION FUNDIDS EVENT “Impact, Learning and Innovation: Towards a Research andPractice Agenda for the Future” – 26-27 March 2013 – IDS SussexGiel Ton
Key impact pathway B:Business plans improve organisational resilience (org. social capital)Key impact pathway D:Grant investments release constraints in access to financial servicesKey impact pathway C:Business plans lead to increased economic performanceINTERVENTIONKey impact pathway A:Business plans improve capacity to pay for collective actionATTRIBUTION PLAUSIBLE =NET-EFFECT MEASUREMENT FEASIBLE =QUASI-EXPERIMENTAL DESIGNAPPROPIATECONTRIBUTION STORY = NEED FORMONITOIRNG INFORMATION
Data collection for building the impactcontribution story InterventionUse operational data forimmediate outcomes Ultimate outcomesReflect on (existing)secondary data on poverty Intermediate outcomesAppropriate proxies forkey outcome areas ConfigurationsProcess tracingand monitoring ofinterdependencies
Impacts ofinnovation grantson collectivemarketing groups
1. Core design elements Time-series on key indicators:● Sales: total sales & sales of innovative product● Patrimony● Self-funding of recurrent cost● Access to funds/credit Treatment and comparison group● 30 farmer organizations that are funded● 20 that are not-funded (though eligible)● Stratified random selection Statistical tests and econometrics outcome variables:● Growth rates● Regression with year after implementation as explanatory variable
2. Contribution story and threats to validity The grant is assumed to be a vital parts of a ‘package’ of causal factorsthat together is sufficient to produce the ultimate outcomes.• E.g. banks show increasing willingness to negotiate with producerorganisations that have some patrimony• E.g. through the investment the access to governmentprocurement (nutritional programmes) is improved but still dependon parti-political dynamics However, it is possible that the grants are not necessary: do you still seetraces of processes (partly) triggered by the intervention• Most likely, the grant is necessary in some configurations withsome type of organizations/markets, but not in all (‘what works forwhom under what conditions?’)• Tackling the threats to validity to the claim of contribution: counterfactualthinking• Can we explain the realization of the outcomes without makingreference to the grant (‘alternative explanations’)?
Question 1 Question 2INHERENT TENSIONS IN COLLECTIVEMARKETING THAT NEED ORGANISATIONALMECHANISMS TO RESOLVE THEMThis tensionis present inthe activitieswe realizeHardlypresentNeverpresentWe have managed toresolve it withorganisationalagreements / internalregulationsWe arelooking forways toresolve itWe don’tneed toresolve it1- “Regulating member supply”Members sometimes protest that theorganisation does not buy all their produce?2- “Quality assurance systems”Are there members that try to deliver lowerquality products than is required?3- “Reduce the need for workingcapital”Members demand cash payment instead ofwaiting until the organisations has sold theproduct?4- “Prevention of disloyal behaviour”Are there members that sell part of theirproduce to other buyers though theypromised to sell to the organisation?5- “Ways to distribute profits”Do members accept that the organisationsdoes not distribute all its profits?6- “Differ benefits and services tomembers and non-members”Is there preferential treatment (e.g. price)when buying from members compared withnon-members?7- “Decide on investments andactivities that do not benefit all”Did the organisation projects or investmentsthat are only to the benefit of a sub-group ofmembers of the group?8- “Delegating and supervisingmarketing tasks”Do members accept that others in theorganisation take decisions on prices ofproducts sold without prior consult to theassembly?9- “Legal responsibility in contractsand loans”Do members take responsibility for eventualfines and sanctions related with salecontracts or loans that the board negotiates?10- “Manage political aspirations ofboard and staff”Do members accept that board members orteam staff take party politicalresponsibilities?Quantitativecomparative toolPregunta 1 Pregunta 2TENSIONESINHERENTES A LACOMERCIALIZACIÓN COLECTIVAEstatensión sepresentaen lasactividadesquerealizamoscomoorganizaciónSepresenta muypocono sepresenta nuncahemos logradoresolverlo conacuerdos/arreglos organizativosestamosbuscando laforma deresolverlononecesitamos resolverlo“Regular lacantidad a acopiarde los miembros”“Sistemas degarantía decalidad”“Reducir lanecesidad decapital de trabajo”“Prevenirdeslealtad en lasventas”“Maneras dedistribuir losexcedentes”“Diferenciar losbeneficios yservicios amiembros y nomiembros”“Decidir sobre
Table 3. Weighing factors used for calculating each tension-containment variable (Tx)ScoreQ1xThis tension is presentin their activitiesHardly present NeverpresentThey have managed to resolve 9 6 3ScoreQ2x They are looking for ways toresolve it6 4 2They don’t need to resolve it 3 2 0Tx=ScoreQ1 x *ScoreQ2x
Conventional statistical tricksTable.. Rotated component matrix on organizational performance indicatorsFactor 1:Economic performanceFactor 2:Organizational sizeFactor 3:Capital intensityPatrimony per member (Ln)-.025 -.006 .979Active members (Ln).189 .874 -.294Organizational Age (Ln).059 .897 .224Sales (Ln).886 .438 -.126Sales per member (Ln).991 -.043 .041Extraction Method: Principal Component Analysis.Rotation Method: Varimax with Kaiser Normalization.
Set-theoretic tricks: fQCA The visual inspection of scatterplots helped to detect anddescribe the diversity in the sector of economic farmerorganizations. To further deepen the analysis, other case-based comparativetechniques are needed to find groupings of organizations withdistinct scores on each of the tensions that compose ourconstruct, as certain organizations may perform functionswhere not all tensions are relevant and for which the absenceis not indicative of weaker capabilities. We expect to explain success with a configurational analysis:like Qualitative Comparative Analysis (Rihoux and Ragin,2008; Schneider and Wagemann, 2012), when we have the‘success-variable’ late 2013.
Practical issues:● Time-series: confidentiality and ‘contentiousproperty’● Quasi-experimental design: attrition of thecomparison group Fundamental problems● ‘Fishing’ versus ‘Iterative process to have a closerlook at cases’● ‘Success’ is multifacetal: what is the dependantvariable in fQCA?