Evaluating ABCElly Hiby (elly.hiby@dogstrust.org.uk)Dogs Trust&Lex HibyConservation Research
Evaluating ABC• Why evaluate• What indicators can we measure• Methods for measuring indicators– Dog population survey• Nov...
• Are you involvedin running anABC programme?• Do you evaluateyour programme?• Does thisevaluation gobeyond reportingnumbe...
What is evaluation?• Evaluation“the assessing and judging the value of a pieceof work, an organisation or a service. Its m...
What is evaluation?• Evaluation“the assessing and judging the value of a pieceof work, an organisation or a service. Its m...
Why evaluate?• Donors – reward their generosity and buildcommitment• Government and animal welfareorganisations – inspire ...
Indicators• Evaluation involves measuring a range ofindicators to expose important changes• When selecting indicators cons...
Indicators• Animal welfare:– Body condition score– Skin condition– Lactating females(more visible andconsistent to scoreth...
Methods of measurement• Official records• Community questionnaires/focus groups• Dog population surveys
Methods of measurement• Official records• Municipality records of nuisance complaints• Health centre records of dog bites ...
Colombo dog rabies casesDog rabies cases in CMC against elimination, vaccination andsterilisation from 1990 to end 2010020...
Methods of measurement• Community questionnaires/focus groups– WHO (2004) Guidelines for conductingcommunity surveys on in...
Colombo focus groupsa**
Methods of measurement• Dog population surveys– Index of abundance (aka ‘indicator survey’)• Quicker/cheaper• But only ind...
Jaipur Pink City counts
Example – surveying for a newABC project• Baseline1. Dog population estimate based on exhaustivesearch in small number of ...
1) Baseline population estimate• Establish smallest authority area for which informationon covariates is available (e.g. h...
Colombo, Sri Lanka
Colombo, Sri Lanka
Colombo, Sri Lanka
1) Baseline population estimate• Extrapolate to entire city:– Using sampling fraction (multiply number of dogscounted by t...
1) Baseline population estimate• This is a ‘rough’ minimum roaming dogpopulation estimate:– Small sample size– Dogs roamin...
2) Mark-resight of notched dogs• Dogs are notched during the ABC programmeproviding a sample of ‘marked’ dogs in the city....
2) Mark-resight
2) Mark-resight of notched dogs• ABC example– ABC programme sterilises 3,500 dogs in first year, with anannual survival es...
2) Mark-resight of notched dogs• Mark-resight estimates the total number ofroaming dogs rather than the number roaming att...
3) Index of abundance routes• Several standard routes of known length• Representative areas• Confounding factors– Time, pe...
What to record• Information for each dog:– Age• Adult• Puppy – agree maximum age (e.g. 4 months)– Sex• Male• Female• Unkno...
Welfare measures• Body condition score (5 point BCS)– Reduction % of dogs in score 1 BCS over time andbetween notched and ...
Using an Android phone to recorddata• An offline database plus somefree smartphone apps can helpyou to monitor your interv...
% Jaipur females pregnant051015202530Oct-08Nov-08Dec-08Jan-09Feb-09Mar-09Apr-09May-09Jun-09Jul-09Aug-09Sep-09Oct-09Nov-09D...
ADMIN:download andanalyze city dataFUNDING AGENCY:view by NGO and time periodNGO 1: uploadby month, viewNGO 2: uploadby mo...
Discussion point• Which indicator is best for dog abundance?• Total population for the city?– But cities are expanding in ...
Conclusion• Evaluation provides a measure of changein important indicators and an opportunityto improve performance
Conclusion• Animal welfare:– Body condition score– Skin condition– Lactating females(more visible andconsistent to scoreth...
Conclusion• Methods of measurement– Official records– Community questionnaires/focus groups– Dog population surveys• Datab...
Whose responsibility?“ An important job had to be done and Everybodywas sure that Somebody would do it. Anybodycould have ...
Thank you forlisteningAny questions?
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby
Upcoming SlideShare
Loading in …5
×

NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby

864 views

Published on

Dr Elly Hiby and Lex Hiby's presentation on Evaluating Animal Birth Control Programme

Published in: Lifestyle
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
864
On SlideShare
0
From Embeds
0
Number of Embeds
84
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Good morning and thank you for joining us today. We’ll be talking through some aspects of evaluating ABC today – its quite a broad topic, so apologies if we do not cover everything. You’ll notice that we have a double act today – I will be presenting for the first part of the workshop and then will be handing over to my father – Lex Hiby. Please note there are handouts available with a list of resources relevant to this topic here – if there are not enough copies, or indeed if you would like an electronic copy, please do email me afterwards and I’ll send this on. I also want to encourage you to ask questions throughout this wrkshop – I will also be stopping at times to ask for your input to certain key questions – so please spk up, I hope this can truly be a 2-way conversation
  • So today we plan to cover the following. By way of an intro looking at the definition of evaluation and why we would ever bother delaying our normal work to play with data What indicators are available that can tell us something about the aspects of the world we are hoping to change I’ll then go through some methods of collecting data that will help us measure these indicators, in particular different methods of dog population surveying Before handing over to Lex to talk through a novel tool for conducting ongoing dog surveys during the running of an ABC programme – combining a mobile phone app and database of clinic data from the ABC operations to track dog population size and coverage of ABC. I should be quick to point out that this tool has been developed with funding support from HSI.
  • I wanted to start by just checking who in the audience is involved with running an ABC programme? If you can put your hands up if this is true for you. Can you keep you hands up if you evaluate that programme in anyway
  • This is one of many definitions of the word evaluation. This time from the Evaluation Trust. But from my perspective the most important reason to define and measure your goal is so we can learn to do our job better
  • This is one of many definitions of the word evaluation. This time from the Evaluation Trust. But from my perspective the most important reason to define and measure your goal is so we can learn to do our job better
  • You may ask – why evaluate? It takes up time I could be using to ABC more dogs. Who cares? First audience is our donors – who may be government, AWBI or municipalities or state gov, or may be an animal welfare donor - raise your hand if you are responsible for raising money for your organisation. Effective evaluation can prove to your donor that their money is making a difference and secure further commitment to donate – important when we are running ABC projects that have to sustained for the long-term Government or animal welfare organisations that have yet to launch ABC – good evaluation can inspire them to start and replicate the benefits Yourself and your staff, your volunteers, your CEO. – pride in your work and motivates the team to keep dedicated to ABC And finally your end beneficiary – the animals themselves, who I am sure if they could would ask – would value seeing the benefit of ABC, they are after all the ones that have to be caught, held, anaesthetised and undergo surgery. What difference will this make to their life and the lives of other animals But I think that the most important reason is to be from my perspective the most important reason to define and measure your goal is so we can learn to do our job better
  • This is an example from Colombo – you can see on this blue line the number of dog rabies cases, measured and reported by the municipal vet office – we could plot this against the years of culling and then against the years of mass vaccination – showing neatly what worked best
  • - Explored attitude towards dogs ‘now’ and ‘in the past’ (>4 years ago prior to when intervention started) Participants were asked open questions about roaming dogs in their area, both currently and in the past – facilitators then followed up with additional questions and participatory exercises to draw from each group a list of problems presented by roaming dogs – both in 2011 and before the intervention started. On this spider web diagram you can see the 9 most commonly mentioned problems caused or experienced by roaming dogs around the outside – the blue line indicates problems in the past and the red those mentioned in 2011. We can see that all 9 focus groups mentioned rabies and breeding or the presence of puppies as a problem in the past – however only 1 group mentioned this as a problem in 2011. We can also see that the total number of problems mentioned in the past – as indicated by the area delineated by the blue line - also decreased significantly by 2011 – as indicated by the smaller area outlined in red
  • We conducted a population survey like this in Colombo Sri Lanka using ‘wards’ the Colombo city municipality split into 47 wards and we counted dogs in 7 of those. Importantly there were maps available for these wards that allowed to ensure we were counting all the streets in the city
  • Need maps available for these official boundaries
  • Use Google Earth where maps are poor
  • Jack Jack to explain why they chose the Pink City and the time it takes to complete the counts
  • 3
  • 1
  • 4
  • 2
  • Never tried it but what about behaviour? Does this describe your dogs?
  • xxx
  • So I just want to share something with you that I learnt the hard way. Have a read of this quote. Ever experienced this? I most certainly have and specifically on this issue of evaluation. I have been part of workshops were over a period of days and using consultation with government representatives, community groups and local vets we developed goals and indicators that lit each of our respective fires. But it became a paper exercise because we didn’t follow this trough to make sure we had the funds, time and clearly identified person responsible to make these indicators work for us. So we weren’t able to do good evaluations and hence learn and improve – we continued to stumble on in the dark.
  • NDWC Chennai 2013 - Evaluating ABC - Dr Elly Hiby and Lex Hiby

    1. 1. Evaluating ABCElly Hiby (elly.hiby@dogstrust.org.uk)Dogs Trust&Lex HibyConservation Research
    2. 2. Evaluating ABC• Why evaluate• What indicators can we measure• Methods for measuring indicators– Dog population survey• Novel tools for surveys(developed with funding from HSI)– Combining clinic database withsurvey data collected on a smartphone
    3. 3. • Are you involvedin running anABC programme?• Do you evaluateyour programme?• Does thisevaluation gobeyond reportingnumber of dogsthrough ABC?
    4. 4. What is evaluation?• Evaluation“the assessing and judging the value of a pieceof work, an organisation or a service. Its mainpurpose is to help an organisation reflect onwhat it is trying to achieve, assessing how far itis succeeding, and identify requiredimprovements”Evaluation Trust
    5. 5. What is evaluation?• Evaluation“the assessing and judging the value of a pieceof work, an organisation or a service. Its mainpurpose is to help an organisation reflect onwhat it is trying to achieve, assessing how far itis succeeding, and identify requiredimprovements”Evaluation Trust
    6. 6. Why evaluate?• Donors – reward their generosity and buildcommitment• Government and animal welfareorganisations – inspire them to replicate• Your organisation – pride and motivation• Your end beneficiary – the animals thatmatter to usLearn and improve
    7. 7. Indicators• Evaluation involves measuring a range ofindicators to expose important changes• When selecting indicators consider the aim ofABC programme and opinions of stakeholders– Government including health department– Local community both dog owning/caring and not– Donors– Your own organisation (encompassed in the aim)What do they want to see change?
    8. 8. Indicators• Animal welfare:– Body condition score– Skin condition– Lactating females(more visible andconsistent to scorethan litters of puppies)– Disease incidence(dog rabies cases)– Euthanasia rates– Dog population size• Communitysatisfaction– Nuisance complaints– Attitudes towards dogs• Human health– Dog bites– Human rabies cases
    9. 9. Methods of measurement• Official records• Community questionnaires/focus groups• Dog population surveys
    10. 10. Methods of measurement• Official records• Municipality records of nuisance complaints• Health centre records of dog bites and human rabies cases– Samples are sufficient, find a friendly source– Note locations (e.g. postcodes) if this is possible, canyou find data from within the ABC area and outsidefor comparison over time (incl pre-ABC) and betweentreatment and control– Note human population increase – e.g. don’t quotejust dog bite data, quote bites per 100,000 people– Right to Information Act (RTIA)
    11. 11. Colombo dog rabies casesDog rabies cases in CMC against elimination, vaccination andsterilisation from 1990 to end 201002000400060008000100001200014000199019911992199319941995199619971998199920002001200220032004200520062007200820092010YearNumberofdogseliminated,vaccinatedorsterilised010203040506070NumberofdograbiescasesEliminationVaccination CMCand BPTNumber ofsterilisationsDog RabiesStart of project
    12. 12. Methods of measurement• Community questionnaires/focus groups– WHO (2004) Guidelines for conductingcommunity surveys on injuries and violence– Questionnaires can reveal changes inquantitative data such as dogs bites (e.g. hasanyone in your household been bitten in thelast 12 months)– Focus groups can reveal attitudes andpractices (e.g. do you think stray dogs aredangerous or safe? What do you do if you arebitten by a dog?)
    13. 13. Colombo focus groupsa**
    14. 14. Methods of measurement• Dog population surveys– Index of abundance (aka ‘indicator survey’)• Quicker/cheaper• But only indicates change in density and not totalpopulation size– Estimate of total population size• Requires more time investment ahead of ABC, soonly invest in this if necessary• Can be relatively simple once ABC had begun
    15. 15. Jaipur Pink City counts
    16. 16. Example – surveying for a newABC project• Baseline1. Dog population estimate based on exhaustivesearch in small number of sample areasextrapolated to whole area (assuming you need atotal dog population estimate)• Refine after approximately 1 year of ABC2. Mark-resight of dogs notched through ABC (noveltool)
    17. 17. 1) Baseline population estimate• Establish smallest authority area for which informationon covariates is available (e.g. human population, roadlength)• Select small sample of these areas (as many as can berealistically managed, suggest minimum 5%)• Complete count of all dogs visible on public areas atpeak roaming time (usually early morning)• Extrapolate to entire city:– Using sampling fraction (multiply number of dogs counted bytotal number of areas/number of samples areas)or– Find reliable covariate (e.g. dog:human ratio) and extrapolate toentire area
    18. 18. Colombo, Sri Lanka
    19. 19. Colombo, Sri Lanka
    20. 20. Colombo, Sri Lanka
    21. 21. 1) Baseline population estimate• Extrapolate to entire city:– Using sampling fraction (multiply number of dogscounted by total number of areas/number of samplesareas)• E.g. counted 462 dogs in 7 wards out of a total of 47Estimated total roaming dog population = 463 X (47/7) = 3,102or– Find reliable covariate (e.g. dog:human ratio) andextrapolate to entire area• E.g. counted 462 dogs in 7 wards in which live 16,000 people= 1 dog:35 people. 100,000 people live in the city henceestimated total roaming dog population = 100,000/35 = 2,857
    22. 22. 1) Baseline population estimate• This is a ‘rough’ minimum roaming dogpopulation estimate:– Small sample size– Dogs roaming and visible in the early morning – nottotal roaming dog population as some will be missed– But can be a basis for lobbying and budgeting withthese clear limitations– Be clear this will be refined and is not the final figure
    23. 23. 2) Mark-resight of notched dogs• Dogs are notched during the ABC programmeproviding a sample of ‘marked’ dogs in the city.• Taking into account a survival estimate (someABC dogs will have died since the operation) thenumber of notched dogs in the city can becalculated• Combining that with proportion of notched dogscan be used to measure the population size
    24. 24. 2) Mark-resight
    25. 25. 2) Mark-resight of notched dogs• ABC example– ABC programme sterilises 3,500 dogs in first year, with anannual survival estimate of 0.7 the number of dogs notched andalive in the city is estimated to be 2,800– The percentage of notched dogs on the street is estimated to be46%– Total dog population estimate is 2,800 / 0.46 = 6,087Hiby et al (2011) A mark-resight survey method to estimate theroaming dog population in three cities in Rajasthan, India• Novel tool combining database for recordingclinic data with a smart phone app
    26. 26. 2) Mark-resight of notched dogs• Mark-resight estimates the total number ofroaming dogs rather than the number roaming atthe time of the initial baseline survey• Relies on a survival estimate– Can use the estimate in Reece et al (2008) Fecundityand longevity of roaming dogs in Jaipur, India; annualsurvival of 0.7– Can estimate a survival for your own area if usingindividual tattoos
    27. 27. 3) Index of abundance routes• Several standard routes of known length• Representative areas• Confounding factors– Time, person, weather, protocol…• Quick so can repeat– Seasonal breeding, politically important time…• Evaluate ‘like with like’
    28. 28. What to record• Information for each dog:– Age• Adult• Puppy – agree maximum age (e.g. 4 months)– Sex• Male• Female• Unknown– Reproductive status• Lactating• Sterilised (notched)– Welfare state• Body condition score• Skin condition
    29. 29. Welfare measures• Body condition score (5 point BCS)– Reduction % of dogs in score 1 BCS over time andbetween notched and unsterilised dogs• Skin condition (Presence/absence)– Reduction in % of dogs with skin conditions visible• TVTs?• Behaviour?1 – runs away immediately2 – keeps a distance but remains in area3 – approaches4 – in your pocket!
    30. 30. Using an Android phone to recorddata• An offline database plus somefree smartphone apps can helpyou to monitor your intervention
    31. 31. % Jaipur females pregnant051015202530Oct-08Nov-08Dec-08Jan-09Feb-09Mar-09Apr-09May-09Jun-09Jul-09Aug-09Sep-09Oct-09Nov-09Dec-09Jan-10Feb-10Mar-10Apr-10May-10Jun-10Jul-10Aug-10Sep-10Oct-10Nov-10Dec-10Jan-11Feb-11Mar-11Apr-11May-11Jun-11Jul-11Aug-11Sep-11Oct-11Nov-11
    32. 32. ADMIN:download andanalyze city dataFUNDING AGENCY:view by NGO and time periodNGO 1: uploadby month, viewNGO 2: uploadby month, viewNGO 3: uploadby month, view
    33. 33. Discussion point• Which indicator is best for dog abundance?• Total population for the city?– But cities are expanding in size with increasing humanpopulation in expanding areas = increasing carrying capacity =increasing total dog population– Average city dweller has no concept of total dog population size,but they do perceive the density and behaviour of dogs– Takes a lot of time/resources to establish a really reliable dogpopulation estimate ahead of ABC – although easy once ABChas started using phone/database tool• In comparison the number of dogs per km of street is:– Quick to estimate from the standard routes used in index ofabundance surveys– Relates more closely to the dog problem
    34. 34. Conclusion• Evaluation provides a measure of changein important indicators and an opportunityto improve performance
    35. 35. Conclusion• Animal welfare:– Body condition score– Skin condition– Lactating females(more visible andconsistent to scorethan litters of puppies)– Disease incidence(dog rabies cases)– Euthanasia rates– Dog population size• Communitysatisfaction– Nuisance complaints– Attitudes towards dogs• Human health– Dog bites– Human rabies cases
    36. 36. Conclusion• Methods of measurement– Official records– Community questionnaires/focus groups– Dog population surveys• Database and phone app
    37. 37. Whose responsibility?“ An important job had to be done and Everybodywas sure that Somebody would do it. Anybodycould have done it, but Nobody did it. Somebodygot angry about that because it was Everybody’sjob. Everybody thought that Anybody could do it,but Nobody realized that Everybody wouldn’t doit. It ended up that Everybody blamed Somebodywhen Nobody did what Anybody could havedone”Make someone responsible and provide timeand budget
    38. 38. Thank you forlisteningAny questions?

    ×