Handout (.ppt format, 2479.5 kb)


Published on

Published in: Health & Medicine, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The Assessment Instruments – There are four different performance assessment instruments. These are: A State level assessment instrument, which addresses the “state public health system” or all entities that contribute to public health at the state level. A Local-level instrument to assess local public health system capacity and performance, An assessment that can be used by local governing bodies, such as boards of health or county councils; and Lastly, an international instrument has been developed collaboratively by CDC and the World Health Organization to assess national capacity and performance. So far this has been used in 42 countries – primarily in Latin America and the Caribbean. This presentation focuses on the national program, which includes the first three instruments. Also, there has been some work and interest around developing instruments that focus on specific categorical public health areas, such as laboratory services and environmental health.
  • To truly understand the value and uniqueness of the NPHPS, it is important to realize that there are four concepts that underpin the instruments and standards: 1. The instruments are based on and built around the Ten Essential Public Health Services- The Essential Public Health Services provide the fundamental framework for the NPHPS instruments. Using the Essential Services assures that the standards fully cover the gamut of public health activities needed at state and community levels. When you look at the 3 different NPHPSP instruments, you will see that each includes a chapter or section devoted to each Essential Service. 2. The standards focus on the overall public health system, as opposed to a single organization or entity - This assures that the contributions of all entities are recognized in assessing the provision of public health services. Clearly, the governmental public health agency – either at the state or local level – is a major player in the public health system, but agencies cannot provide the full spectrum of essential public health services alone. Hospitals, public safety, voluntary health organizations, mental health centers, schools, civic groups, faith institutions and others all contribute to accomplishing the actions necessary to achieve public health. 3. The standards describe an optimal level of performance ("the gold standard") rather than provide minimum expectations. Minimum standards often do not stimulate organizations to strive for higher levels of achievement. Optimal standards will stimulate organizations for a higher level of achievement and will provide a level to which all public health systems can aspire to achieve. The standards are intended to support and stimulate a process of quality improvement - The major outcome of responding to the performance standards instrument should be the identification of strengths and weaknesses within the state or local public health system. This information should be used to determine areas and methods for improvement. Ideally, this information should feed into a health improvement process and should result in planning and action around identified priority areas.
  • This is the format of the assessment instrument. The 10 Essential Services provide the framework for each instrument, so there are 10 sections or “chapters” – one for each Essential Service. Each Essential Service section is further divided into several indicators , which represent major components, activities, or practice areas of the Essential Service. Associated with each indicator are model standards (written in paragraph and bullet format) that describe aspects of optimal performance. Each model standard is followed by a series of assessment questions that serve as measures of performance. Using four response options (yes, high partial, low partial, and no), the measures elicit information on how well the model standard is being met. If a state or local public health system or a governing entity responds “yes” to all questions under any one standard, the responding entity should look similar to and function consistently with the model standard. However, the model standards are designed to represent optimum performance and there will likely be few model standards that are fully met. The model standards should stimulate continuous quality improvement that will help to improve state and local public health practice over time.
  • Lastly, the state and local instruments include two summary questions at the end of each indicator section (see Appendix D for an example). Respondents are asked to think about the model standard as a whole and use a four-point scale to assess the percentage of the model standard that: is achieved by the public health system collectively; and is the direct contribution of the public health agency. The four responses are: 0-25%; 26-50%; 51-75%; and 76-100%.
  • So…how do you get started? Here are some key steps to consider: To use any of the assessment instruments, begin by reviewing all of the materials and thinking about how you’ll want to approach the process . It can be a lengthy process, and careful preparation will assure that the time is spent well. We have resources, such as a User Guide and Frequently-Asked-Questions, that can help you to think about the entire process. 2. Identify and recruit participants - You’ll first need to begin by bringing together the necessary partners. Keep in mind that the state and local instruments require more extensive participation than the governance instrument. The Users Guide has a special section devoted to the identification and recruitment of partners. You can also use the Essential Services as a framework for identifying organizations or individuals which should be included. We’ll go over those in a few moments. 3) Orienting Participants – Before jumping into the performance assessment discussions, you might want to provide a brief overview of the NPHPSP, the Essential Public Health Services and the purpose of completing the assessment instrument. This can ensure that participants have a good understanding of the process and the outcomes and benefits that should result. 4) The next step is to discuss and complete the assessment instrument . This is a crucial step so I’ll go more into depth on this piece in a moment. Keep in mind that it will most likely take several sessions to work through the entire tool. Submit your responses to CDC and within a few days you will receive an electronic summary of your results, which includes your scores for all Essential Services, indicators and first-tier questions, as well as charts and graphs displaying the data. Discuss the results with the participants and determine challenges and opportunities that need to be addressed. The qualitative comments from earlier discussions can shed light on potential solutions or gaps. It is recommended , but not required to conduct the assessments through a coordinated statewide process. By having the local and state assessments conducted at the same time, or by having all local jurisdictions complete the local assessment, much can be learned from looking at the aggregate data and developing cross-cutting improvement plans.
  • Participation is crucial so it is necessary to fully think through the participants in this process. Because the state and local tools focus on the “system”, respondents should have broad representation from throughout the system in developing their responses to the instrument. The following questions may help in identifying potential participants: Who plays a role in the public health system? Who has a hand in providing the Essential Public Health Services? What broad, cross-sectional participation is needed? What community participants or consumer can we included? Who else do we need to include who might have expertise in certain areas? How many people should participate? And lastly, if you hold a series of meetings or use small groups to conduct the assessment, assure there is consistency or a core group among respondents. This is crucial for minimizing different approaches to the assessment and the impact it might have on scores. If, for example, one group scores themselves very critically while another group is more positive, scores will not be even and comparable throughout the instrument.
  • The second concept addresses the fact that the NPHPS focuses on the overall “public health system.” This assures that the contributions of all entities are recognized in assessing the provision of public health services. Clearly, the governmental public health agency – either at the state or local level – is a major player in the public health system, but agencies cannot provide the full spectrum of essential public health services alone. Hospitals, public safety, voluntary health organizations, mental health centers, schools, civic groups, faith institutions and others all contribute to accomplishing the actions necessary to achieve public health. When we talk about public health systems , a definition we commonly use is: “ all public, private, and voluntary entities that contribute to the delivery of essential public health services within a jurisdiction.” Another way of looking at it is to say that a public health system is a network of entities with differing roles, relationships, and interactions. All of these entities contribute to the health and well-being of the community.
  • Several different approaches can be used to complete the instrument. Here are three commonly used approaches. I will further elaborate on each, in the following slides. The one approach that should NOT be used is completing the instrument through individual surveys (e.g., sending out the instrument and then compiling the responses). This does not allow for development of consensus responses, there is no opportunity for qualitative feedback on why responses were selected, and there is no cross-learning among participants.
  • First, many sites conduct a 1-2 day retreat of system partners. An orientation is held at the beginning of the day, then participants work through the instrument (either in small groups or as one large group). The end of the day should be devoted to the identification of major themes. This helps to get the work done in a short timeframe. Also, the group gets through the process and sees the results at the end of the session. However, it r equires a significant time commitment and also can be overwhelming for participants (as well as those preparing for and leading the effort).
  • Second, many sites have split up the instrument so that responses are developed by small groups or subcommittees (e.g., each is responsible for three ES). If this is done, some attention should be given to maximizing cross-learning and consistency in responses. For example, consider having the same facilitators and recorders or a “core group” that participates in all subcommittees.
  • A third option is to conduct a series of meetings (e.g., 5 meetings at which 2 ES are discussed at each). This allows for a core group to participate, as well as experts to be brought in, as needed. The work is also done in small pieces, which can be seem more manageable. However, conveners need to explore how to minimize the drawbacks: attendance at meetings will change with participant availability, and the process can also seem to “drag on”
  • Regardless of how participation is structured, participants will need to agree on a process for discussing the assessment instrument and identifying responses. Consensus responses should be developed through dialogue among system partner organizations. There are several possible approaches that can be considered, as you design your process: Walk through the instrument and questions one by one – this will allow for a very methodical and thorough process. However, it can seem tedious to participants. Discuss the model standards with facilitator/recorder judgment on responses – Provide participants with only the model standards. Participants can discuss each model standard for a set period of time (e.g., 10 minutes) during which the facilitator ensures that the discussion hits all of the key points addressed in the model standard. The facilitator and recorders make judgments on the responses to the questions (asking follow-up questions as needed) based on the discussion. This keeps the interest of participants’ high, since the discussion is focused more on content than the process of identifying a response. As a drawback, the final responses can be greatly impacted by the facilitator’s and recorder’s perceptions of activities. Discuss the model standards with follow-up voting – Provide participants with the full instruments. Participants discuss each model standard for a set period of time (e.g., 10 minutes) similar to what is described above. After the model standard has been fully discussed, participants vote (using color-coded cards or raised hands) on the response to each question. Further discussion can occur where there is disparity in responses.
  • State and local public health systems should consider implementing the performance assessment instruments through a coordinated statewide approach. Using the instruments during the same time period can create opportunities to coordinate orientation activities, technical assistance, and improvement planning at the state and local levels. In fact, some states are using funding from the CDC Bioterrorism Grant Program to support their statewide efforts.
  • Green states are those that were active in pilot testing. Please note that Ohio only tested the local instrument in a sampling of local jurisdictions and Massachusetts also only tested the governance in a sampling of local boards of health. Red states are those that are currently using or about to use one or more of the instruments statewide (states active since OMB clearance): New Jersey is requiring statewide use of the local instrument (in 22 regional areas) and the governance instrument (by over 500 boards of health) and will be using the state instrument as well. Connecticut , through the state association of local health officers, is supporting their local agencies and their system partners to use the local instrument. Oklahoma, Colorado, New Mexico – has used both the state and local instrument Oregon and Arkansas – has used the local instrument Montana and North Dakota – has used the state instrument and there is current activity to implement the local instrument in many local jurisdictions. Kentucky – used the local instrument in a cohort of jurisdictions and is supporting voluntary use of NPHPSP and MAPP in the other jurisdictions. States with a blue star indicate those that have probable or definite plans for use in the future and/or are using the instrument in a cohort of sites: California – using MAPP (and NPHPSP as part of this) in a small cohort of local sites. Illinois recently passed legislation which adds a requirement for a state public health improvement plan to be developed every four years, using HP 2010 and the NPHPS as its frameworks.  South Carolina is planning for support of NPHPSP (ideally through the use of MAPP) in all of its districts. Ohio is establishing a standards process which sets minimum agency standards for local health agencies and also requires all local areas to periodically complete the NPHPSP Local Instrument.  States active as part of CDC BT cooperative agreement activities: OK, NM, CO, AR, NJ, MT (and KY is mandating MAPP as part of their BT cooperative agreement).
  • State Assessment: Divide 10 EPHS into three sequential groups; each met for 4 hours: Data-related: EPHS #1,2,10 Health services: EPHS#7,8,9 Community mobilization: EPHS#3,4,5,6 Partners were invited to one, two, or three groups depending on their expertise. Participants scored the instrument before the meeting and discussed their opinions, and then voted. Consensus was used for summary questions at the end of each indicator. Also discussed relevance of each model standard to practice in their state.
  • State assessment : CDC results report was sent out to participants, along with an importance ranking tool for each indicator. Importance ranks were graphed with assessment results and that was the beginning for action planning for performance improvement. Statewide planning process began with one 4-hour session, and the partners chose to continue with monthly meetings to complete.
  • There are numerous resources available to help. Many of these are available on the NPHPSP webpage at www.phppo.cdc.gov/nphpsp/, on partner organizations’ websites, or by calling the partner organizations
  • During the session, Chris Kinabrew will tell specific stories of TA provided through this call series…
  • Handout (.ppt format, 2479.5 kb)

    1. 1. Public Health Performance Improvement: How National Performance Standards Make the Difference (Part II) Got Skills? How To Build Your Public Health System Performance Assessment APHA 132 nd Annual Meeting Session 3247 Washington, DC
    2. 2. Recap of Session I Session II Objectives <ul><ul><li>identifying the benefits of using the performance standards </li></ul></ul><ul><ul><li>learning different methods for structuring the assessment process </li></ul></ul><ul><ul><li>gaining knowledge on using assessment results for performance improvement </li></ul></ul>
    3. 3. Structuring a Performance Standards Process <ul><li>Liza Corso, MPA </li></ul><ul><li>Centers for Disease Control and Prevention </li></ul>
    4. 4. NPHPSP <ul><li>Partners </li></ul><ul><ul><li>CDC, APHA, ASTHO, NACCHO, NALBOH, NNPHI, PHF </li></ul></ul><ul><li>Assessment Instruments </li></ul><ul><ul><li>State public health system </li></ul></ul><ul><ul><li>Local public health system </li></ul></ul><ul><ul><li>Local governance </li></ul></ul><ul><ul><li>International </li></ul></ul>
    5. 5. Four Concepts Applied in NPHPSP <ul><li>1. Based on the ten Essential Public Health Services </li></ul><ul><li>2. Focus on the overall public health system </li></ul><ul><li>3. Describe the “gold standard” for performance </li></ul><ul><li>4. Results drive performance improvement </li></ul>
    6. 7. Indicator Model Standard Measures Essential Service Instrument Format
    7. 8. Measures Summary Questions
    8. 9. Key Steps in the Process <ul><li>Identify, recruit and orient participants </li></ul><ul><li>Complete the instrument </li></ul><ul><ul><li>Use facilitators and recorders </li></ul></ul><ul><ul><li>Gain consensus responses </li></ul></ul><ul><li>Submit responses to CDC </li></ul><ul><li>Discuss CDC report with participants </li></ul><ul><li>Determine challenges and opportunities </li></ul><ul><li>Develop and implement improvement plans </li></ul>
    9. 10. Identifying Participants <ul><li>Key questions: </li></ul><ul><ul><li>Who plays a role in the public health system? </li></ul></ul><ul><ul><li>Who provides the Essential Services? </li></ul></ul><ul><ul><li>What broad, cross-sector participation is needed? </li></ul></ul><ul><ul><li>What consumers can be included? </li></ul></ul><ul><ul><li>Who else needs to be included (e.g., to assure expertise in certain areas) </li></ul></ul><ul><ul><li>How many people should participate? </li></ul></ul><ul><li>Ensure consistency or a core group among respondents </li></ul>
    10. 11. The “Public Health System” <ul><li>More than just the public health agency </li></ul><ul><li>“ Public health system” </li></ul><ul><ul><li>All public, private, and voluntary entities that contribute to public health in a given area. </li></ul></ul><ul><ul><li>A network of entities with differing roles, relationships, and interactions. </li></ul></ul><ul><ul><li>All entities contribute to the health and well-being of the community. </li></ul></ul>
    11. 12. Structuring Participation: Different Approaches <ul><li>Retreat (1 – 2 days) </li></ul><ul><li>Small Group Activities </li></ul><ul><li>Series of Meetings </li></ul>
    12. 13. Retreat <ul><li>Generally 1-2 days </li></ul><ul><li>Advantages: </li></ul><ul><ul><li>Short timeframe </li></ul></ul><ul><ul><li>Maintains momentum </li></ul></ul><ul><li>Drawbacks: </li></ul><ul><ul><li>Requires time commitment </li></ul></ul><ul><ul><li>Can be overwhelming </li></ul></ul>
    13. 14. Small Group Activities <ul><li>Advantages: </li></ul><ul><ul><li>Allows for expertise, as needed </li></ul></ul><ul><ul><li>Less overwhelming </li></ul></ul><ul><li>Drawbacks: </li></ul><ul><ul><li>Less cross-learning </li></ul></ul><ul><ul><li>Less consistency in response development </li></ul></ul>
    14. 15. Series of Meetings <ul><li>Advantages: </li></ul><ul><ul><li>Allows for expertise, as needed </li></ul></ul><ul><ul><li>Gets the work done in small pieces </li></ul></ul><ul><li>Drawbacks: </li></ul><ul><ul><li>Participation changes with meeting attendance </li></ul></ul><ul><ul><li>Process may seem to “drag on” </li></ul></ul>
    15. 16. Approaches in Determining Responses <ul><li>Walk through questions one by one </li></ul><ul><li>Discuss the model standard with facilitator/recorder judgment on responses </li></ul><ul><li>Discuss model standard with follow-up voting </li></ul>
    16. 17. Coordinated Statewide Process <ul><li>Use of multiple NPHPSP instruments in a coordinated approach </li></ul><ul><li>Benefits: </li></ul><ul><ul><li>Coordinated orientation, training, and technical assistance opportunities </li></ul></ul><ul><ul><li>Aggregate reports of assessment results </li></ul></ul><ul><ul><li>Coordinated improvement planning </li></ul></ul>
    17. 18. Statewide Activities Pilot / Field Test States Currently using or planning for use statewide
    18. 19. Lessons Learned through the Use of the NPHPSP Performance Assessment Evelyn A. Knight F. Douglas Scutchfield Ann V. Kelly Michelyn W. Bhandari Miriam Fordham University of Kentucky College of Public Health
    19. 20. Method <ul><li>First step in three phase project </li></ul><ul><li>1-1.5 hour interviews with state and local users selected to represent broad cross-section of users </li></ul><ul><ul><li>10 state assessment users </li></ul></ul><ul><ul><li>12 state coordinators of local assessments </li></ul></ul><ul><ul><li>5 local assessment users </li></ul></ul><ul><li>Incorporated results of earlier study of 9 local public health systems in Kentucky </li></ul>
    20. 21. Interviews with users <ul><li>Description of public health system in state or local area </li></ul><ul><li>How performance assessment carried out </li></ul><ul><li>Benefits/barriers of carrying out the performance assessment </li></ul>
    21. 22. Results <ul><li>Why complete the assessment? </li></ul><ul><li>Planning the assessment </li></ul><ul><li>Planning for facilitation </li></ul><ul><li>Recruiting system partners </li></ul><ul><li>Planning assessment sessions </li></ul><ul><li>Planning for performance improvement </li></ul><ul><li>Benefits of assessment </li></ul>
    22. 23. Why complete the assessment? <ul><li>State and Local Assessment </li></ul><ul><ul><li>Initiated coordinated quality improvement process for state and local levels </li></ul></ul><ul><li>Local Assessment Only </li></ul><ul><ul><li>Part of MAPP process; extension of Turning Point activities </li></ul></ul><ul><ul><li>Statewide effort to assess level of public health performance across all jurisdictions; justify need for full time staff in all jurisdictions </li></ul></ul><ul><li>State Assessment </li></ul><ul><ul><li>Initiate coordinated planning across the state </li></ul></ul><ul><ul><li>To educate state health department staff </li></ul></ul>
    23. 24. Planning the Assessment <ul><li>Allow for steep learning curve inside the sponsoring organization </li></ul><ul><li>Consider carrying out a trial assessment inside the sponsoring PH agency </li></ul><ul><li>Take advantage of training and technical assistance –Partners websites </li></ul><ul><li>Plan for training and orientation of assessment leaders and facilitators </li></ul><ul><li>Keep performance improvement in mind </li></ul>
    24. 25. Planning for facilitation <ul><li>Professional facilitation can help, but not necessary if training available </li></ul><ul><li>Have a plan for coming to consensus </li></ul><ul><li>Capture key issues for later discussion </li></ul><ul><li>Facilitators and recorders MUST be neutral </li></ul><ul><li>Pacing is imperative, which can be difficult for less experienced facilitators </li></ul>
    25. 26. Recruiting System Partners <ul><li>Build on existing relationships—establish a steering committee a la MAPP </li></ul><ul><li>If existing relationships are thin, allow more time and effort for recruitment </li></ul><ul><li>Help potential new partners see their public health role </li></ul><ul><li>Be selective </li></ul>
    26. 27. Planning assessment sessions <ul><li>Carefully orchestrate assessment retreats </li></ul><ul><ul><li>Single or multiple </li></ul></ul><ul><ul><li>If multiple sessions are used, do not extend out over too long a time period </li></ul></ul><ul><li>Divide the group or not….both seem to work </li></ul><ul><ul><li>If divided, assure good representation across EPHS groups </li></ul></ul><ul><li>Provide some advance orientation as well as at the assessment itself </li></ul>
    27. 28. Planning for performance improvement <ul><li>Consider how results will be used </li></ul><ul><ul><li>How to share with partners </li></ul></ul><ul><ul><li>How to prioritize EPHS – develop importance ratings </li></ul></ul><ul><ul><li>How to gather supporting information </li></ul></ul><ul><li>Revisit the model standards and the instrument </li></ul><ul><li>Know who will be responsible </li></ul><ul><li>PR </li></ul>
    28. 29. Benefits of Assessment for PH System Performance Improvement <ul><li>Building knowledge and greater understanding of public health across the community/system </li></ul><ul><ul><li>The conversation and system learning </li></ul></ul><ul><li>Identifying system strengths and areas to be changed </li></ul><ul><li>Establishing new networks </li></ul><ul><li>Building system identity </li></ul>
    29. 30. Summary <ul><li>Plan the assessment process to: </li></ul><ul><ul><li>meet your system’s needs </li></ul></ul><ul><ul><li>transition into a performance improvement process </li></ul></ul><ul><li>Greatest benefits include </li></ul><ul><ul><li>Education about public health </li></ul></ul><ul><ul><li>Sharing and networking </li></ul></ul><ul><ul><li>Building the public health system </li></ul></ul>
    30. 31. The National Public Health Performance Standards Resources to Assist Users Chris Kinabrew National Network of Public Health Institutes
    31. 32. Useful Products from NPHPSP Partners <ul><ul><li>Assessment Instruments (online, binder, standards only) </li></ul></ul><ul><ul><li>User Guide </li></ul></ul><ul><ul><li>Frequently Asked Questions </li></ul></ul><ul><ul><li>Reports with data </li></ul></ul><ul><ul><ul><li>For each respondent </li></ul></ul></ul><ul><ul><ul><li>Aggregate statewide data </li></ul></ul></ul><ul><ul><li>Tri-fold brochures </li></ul></ul><ul><ul><li>Posters </li></ul></ul><ul><ul><li>Performance Improvement Resources available at http://www.phf.org/Tools-Resources.htm#NPHPSP </li></ul></ul>
    32. 33. NPHPSP Activities <ul><li>Training Workshops </li></ul><ul><li>Statewide Kickoffs </li></ul><ul><li>User Teleconferences </li></ul><ul><li>Ongoing technical assistance </li></ul>
    33. 34. Training Workshops <ul><li>Annual event </li></ul><ul><li>Topics covered include: </li></ul><ul><ul><li>Orientation to 10 Essential Services </li></ul></ul><ul><ul><li>Using the Instruments </li></ul></ul><ul><ul><li>Coordination of Statewide Approach </li></ul></ul><ul><ul><li>Facilitation and Consensus Building </li></ul></ul><ul><ul><li>Data Submission and Reports </li></ul></ul><ul><ul><li>Developing and Implementing Improvement Plans </li></ul></ul><ul><li>CD-ROM from April, 2004 workshop available </li></ul><ul><li>Sign up now for info re Spring, 2005 training workshop </li></ul>
    34. 35. Statewide Kickoffs <ul><li>Colorado </li></ul><ul><li>New Mexico </li></ul><ul><li>Florida </li></ul>
    35. 36. NPHPS User Network <ul><li>Monthly conference calls with: </li></ul><ul><ul><li>Jurisdictions that have completed assessments </li></ul></ul><ul><ul><li>Interested Parties </li></ul></ul><ul><ul><li>Partner organizations </li></ul></ul><ul><li>Sharing information about: </li></ul><ul><ul><li>Implementation Strategies </li></ul></ul><ul><ul><li>Performance improvement processes </li></ul></ul><ul><ul><li>Tools </li></ul></ul><ul><ul><li>Resources </li></ul></ul>
    36. 37. Questions to Consider <ul><li>How does our state/local public health structure affect our planning? </li></ul><ul><li>Which tools do we want to use? </li></ul><ul><li>What leadership can sponsor the process and create buy-in? </li></ul><ul><li>What approaches will be most successful? </li></ul><ul><li>Should this connect with other current or past activities? </li></ul><ul><li>What resources exist to support the effort? </li></ul>
    37. 38. Exercise “The Local Public Health System and the Essential Services” Scott Fisher, MPH National Association of County and City Health Officials
    38. 39. Questions? Please stay tuned for additional info re 3 rd Annual Training Workshop in Spring 2005