Over five years Khulisa and their partners conducted over 400 performance assessments for PEPFAR using Khulisa’s Performance Assessment Tool, that measured each PEPFAR programme's efficiency, effectiveness and impact.

1. Click to Edit Master Title Style
Rapid and Efficient Implementation
of a Large-Scale Performance
Evaluation Initiative
for the US Government in South Africa
In: Proper Use of Evaluation to Impact Funders and Society
AEA Annual Meeting, October 17, 2013
Susan J. Griffey, DrPH, BSN, FNP
Social & Scientific Systems, Inc.
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

2. Click to Edit Master Title Style
Outline
• A little background
• Key points
• What we did and what resulted
• Implications
• How you can apply this
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

3. Large-Scale Performance Evaluation
Initiative
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07
A Little Background

4. 27 Evaluators +
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

5. Using Standardized Performance Assessment
Click to Edit Master Title Style
(PA) Implementation
• Streamlined Performance Assessment (PA)
methodology
• Training
• Data collection
• Reporting for the PA and for the portfolio
• Operations and logistics
• Business and administrative processes
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

6. Assessed 60 Organizations
Click to Edit Master Title Style
• Not-for-profit, NGO
• For profit
• Academic/research institutes
• Partnerships and consortia
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

7. On 3 Programmatic Dimensions
Click to Edit Master Title Style
• Fiscal management (FM)
• Systems strengthening (SS)
• 1 or more technical program areas [TPA]), each
with several sub-areas
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

8. at 263 Locations across South Africa…
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

9. Over 10 Months
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

10. Produced Rapid Performance Results
Click to Edit Master Title Style
• Minimal burden on grantees and funders
• Met funder requirements for accountability and
compliance
• Able to give clients relevant results
– Rapidly
– In an immediately-useful way
• Value added from portfolio-level results
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

11. Standardize, Streamline, Automate for
Large-Scale Evaluation
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

12. Streamlined Performance Assessment (PA)
Click to Edit Master Title Style
Methodology
• Focused on critical items only
• Time-limited data collection at each
organization
– 1 day each at Headquarters and 2 service sites
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

13. Click to Edit Master Title Style
Streamlined evaluator training
and Peer Support
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

14. Data Collection
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07
Sample Data Collection Worksheet
Availability and awareness of
policies, plans, procedures,
and regulations
Human Capacity
Development & Systems
Strengthening
Organisational Administration
and Fiscal Management
Programme Management,
Implementation, and
Leadership
Service delivery
Referrals, Linkages,
Integration
Excellent
Strong
Some improvement
needed
Immediate attention
required

15. Report Templates
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

16. Operational and Logistical
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

17. Business and Administrative
Click to Edit Master Title Style
Planning/SOW
becomes invoice
Expense Claim =
Reimbursement
Request
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

18. Scheduling Software and UNavailability
Click to Edit Master Title Style
Calendar
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

19. K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07
Findings

20. Portfolio: Strong Overall in all 3 Programmatic
Click to Edit Master Title Style
Dimensions
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07
Technical Program
Areas
2.57/3.00
Fiscal
Management
2.69/3.00
Systems
Strengthening
2.63/3.00

21. % with STRONG Scores
Click to Edit Master Title Style
• Overall, FM much stronger
• SS and TPA were similar: 66% and 64%
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

22. We did more with less
Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

23. K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07
Implications

24. Implications
Click to Edit Master Title Style
“[The] report has provided [funder
program manager] with clear indications
as to what were some of the weaknesses
with the just-concluded grant and has
also provided them with clear guidance
and the way forward with the new grant
that just started in 2013.”
Funder
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

25. Click to Edit Master Title Style
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

26. Standardize, Streamline, Automate for Large-
Click to Edit Master Title Style
Scale Evaluation
Managers with multiple programs of any budget
level – large or small
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07

27. Contact Us!
Click to Edit Master Title Style
Sue Griffey, DrPH, BSN
Vice President and Director, Evaluation Center
Social & Scientific Systems, Inc.
– sgriffey@s-3.com @SueMentors
– www.s-3.com
Mary Pat Selvaggio, MPH, BS, Director for Research and Health
Edna Berhane, MPH, MIA, Associate Director
Khulisa Management Services
– mpselvaggio@khulisa.com, eberhane@khulisa.com
– www.khulisa.com
K h u l i s a
Management
Services (Pty)Ltd
Reg No.: 99 09520/07