PFFEvaluationReport-Year2

Year 2
Evaluation Report
for the
Partnerships for Families
Initiative
August 2006
Submitted by: The PFF Evaluation Team
Todd Franke (PI), Center for Healthier Children, Families and Communities,
University of California, Los Angeles
Devon Brooks (Co-PI), University of Southern California
Christina Christie (Co-PI), Claremont Graduate University
Stephen Budde (Co-PI), Juvenile Protective Association
Jan Nissly (Project Director)
Susan Kim (Agency Liaison)
Jane Yoo (Child Welfare Evaluation Coordinator)
Jorja Leap (Implementation Monitoring Coordinator)
Jaymie Lorthridge (MSW Student Intern)
Danny Hong (Doctoral Student Research Associate)
Alice Kim (Research Associate)
Funded by

This page was intentionally left blank.

1
August 2006
ACKNOWLEDGEMENTS
This report describes the evaluation activities and products of Year Two of the Part-
nerships for Families (PFF) Initiative. This past year marked a broadening in the partners
involved in PFF. We are grateful to have worked with so many dedicated social workers
and staff in the PFF Lead Agencies and their collaborating agencies (see the complete list of
agencies following the acknowledgments). Thank you for your passion, thoughtful feed-
back, and especially your patience!
We also owe gratitude and thanks to the Los Angeles County Department of Chil-
dren and Family Services (DCFS). Our work has benefited immensely from the leadership
of Dr. David Sanders and the perseverance and efforts of Angela Carter and Teri Gillams, as
well as countless other DCFS staff.
To our outstanding evaluation liaison, Antoinette Andrews, and other First 5 LA
staff, including Evelyn Martinez, Armando Jimenez, Dawn Kurtz, Roberto Roque, and
Karen Blakeney; thank you for your support and commitment to rigorous evaluation and
research. To the First 5 LA Commission, thank you for your commitment to children and
families, and for your appreciation and support of evaluation of innovative initiatives such
as PFF.
Last, but certainly not least, we thank the personnel who worked behind the scenes
to administer our contracts, namely, Carolyn McLaurine, Lauren Beastall, Selma Walker
and William Rypcinski.
Those mentioned above, and countless others, in one way or another, were integral
members of the Partnerships for Families Initiative Evaluation Team. For your commitment
to serving families and preventing child maltreatment, and for supporting us in the evalua-
tion of PFF, we offer you our deepest gratitude and respect. We look forward to continuing
this important partnership with you in the years to come.

2
Year 2 Evaluation Report for the Partnerships for Families Initiative
LIST OF AGENCIES
SPA I: CHILDREN’S BUREAU OF SOUTHERN CALIFORNIA
SPA II: THE HELP GROUP CHILDREN AND FAMILY CENTER
• ANTELOPE VALLEY HOSPITAL, HEALTHY HOMES • PENNY LANE
• ANTELOPE VALLEY DOMESTIC VIOLENCE
COUNCIL
• SPA 1 COUNCIL/CPC
• CHILD CARE RESOURCE CENTER • TARZANA TREATMENT CENTER
• CHILDREN’S CENTER OF THE ANTELOPE
VALLEY
• VALLEY CHILD GUIDANCE
• LANCASTER DCFS OFFICE
• YES2KIDS – ANTELOPE VALLEY CHILD
ABUSE PREVENTION COUNCIL
• LOS ANGELES COUNTY OFFICE OF EDUCATION
• CENTER FOR THE IMPROVEMENT OF CHILD
CARING
• THE HELP GROUP
• CHILD CARE RESOURCE CENTER • JEWISH FAMILY SERVICE
• EL PROYECTO DEL BARRIO • MISSION CITY COMMUNITY NETWORK
• FRIENDS OF THE FAMILY
• NORTH HOLLYWOOD AND SANTA CLARITA
DCFS OFFICES
• HAVEN HILLS, INC.
• SAN FERNANDO COMMUNITY MENTAL HEALTH
CENTER, INC

3
August 2006
SPA III: SPIRITT FAMILY SERVICES
SPA IV: PARA LOS NIÑOS
• ARROW CONTINUATION CAL-SAFE • MOUNT VIEW SCHOOL DISTRICT
• CHILD DEVELOPMENT MEDIA, INC. • NATIVITY CATHOLIC CHURCH
• CHRISTIAN FOOD CENTER • NORTH PARK CAL-SAFE PROGRAM
• CITRUS VALLEY HEALTH PARTNERS • NUEVA VISTA GENESIS PROGRAM
• CITY OF EL MONTE POLICE DEPARTMENT
• OPTIONS-A CHILD CARE & HUMAN SERVICES
AGENCY
• COMMUNITY ASSESSMET AND SERVICE CEN-
TER/PROTOTYPES
• OUR LADY OF GUADALUPE CATHOLIC CHURCH
• EL MONTE HIGH SCHOOL • PARENT’S PLACE FAMILY RESOURCE CENTER
• ENKI YOUTH & FAMILY SERVICES • PHFE-WIC PROGRAM
• FAMILY OUTREACH & COMMUNITY INTER-
VENTION SERVICES (FOCIS)
• ROWLAND UNIFIED SCHOOL DISTRICT
• GLENDORA AND EL MONTE DCFS OFFICES
• SAN GABRIEL VALLEY POMONA REGIONAL
CENTER
• HACIENDA/ LA PUENTE WORKMAN HIGH
CAL-SAFE
• SANTANA ALTERNATIVE-ACCEPT
• INFANT, CHILD AND FAMILY PROJECT • SPA 3 COUNCIL
• INTERNATIONAL INSTITUTE OF LOS ANGELES • WESTERN JUSTICE FOUNDATION
• LOS ANGELES COUNTY DEPARTMENT OF
HEALTH SERVICES
• WORKSOURCE CALIFORNIA
• LOS ANGELES COUNTY DEPARTMENT OF
MENTAL HEALTH
• YWCA SAN GABRIEL VALLEY-WINGS
• LOS ANGELES COUNTY PROBATION DEPART-
MENT
• ACORN • METRO NORTH DCFS OFFICE
• HOLLYGROVE CHILDREN AND FAMILY SERVICES • PLAZA COMMUNITY CENTER, INC
• KOREATOWN YOUTH AND COMMUNITY CENTER • ST. ANNE’S

4
SPA V: ST. JOHN’S CHILD AND FAMILY DEVELOPMENT CENTER
SPA VI: SHIELDS FOR FAMILIES, INC.
• ANOTHER WAY • ST. JOSEPH’S
• CLARE FOUNDATION • VENICE FAMILY CLINIC
• CONNECTIONS FOR CHILDREN
• VISTA DEL MAR – SCHOOL READINESS
PROJECT
• DEPARTMENT OF MENTAL HEALTH, WLA
OFFICE
• WEST LA DCFS OFFICE
• LEGAL AID FOUNDATION • WESTSIDE CHILD TRAUMA COUNCIL
• LES KELLEY FAMILY HEALTH CENTER • WESTSIDE CHILDREN’S CENTER
• OCEAN PARK COMMUNITY CENTER • WESTSIDE DOMESTIC VIOLENCE NETWORK
• SANTA MONICA/MALIBU UNIFIED SCHOOL
DISTRICT – INFANT & FAMILY SUPPORT
• WESTSIDE FAMILY HEALTH CENTER
• SOJOURN SERVICES FOR BATTERED WOMEN
• A COMMUNITY OF FRIENDS • KING/DREW MEDICAL CENTER
• CALIFORNIA STATE UNIVERSITY
DOMINGUEZ HILLS
• LOS ANGELES METRO CHURCHES
• COMMUNITY COALITION • LOS ANGELES UNIFIED SCHOOL DISTRICT
• CHILD WELFARE LEAGUE OF AMERICA • MARY HENRY HEALTH CLINIC
• DREW CHILD DEVELOPMENT CENTERS • PARA LOS NIÑOS
• EISNER PEDIATRIC & FAMILY MEDICAL
CENTER
• PROTOTYPES
• INSTITUTE FOR MAXIMUM HUMAN
POTENTIAL
• WATERIDGE DCFS OFFICE
• JENESSE CENTER

5
August 2006
SPA VII: BIENVENIDOS CHILDREN CENTER, INC.
SPA VIII: THE NATIONAL COUNCIL ON ALCOHOLISM AND DRUG DEPENDENCE OF
SOUTH BAY (NCADD/SOUTH BAY)
• ABC LEARNING CENTER
• HUMAN SERVICES ASSOCIATION
(SUB-CONTRACTOR)
• BELVEDERE DCFS OFFICE • LOS ANGELES UNIFIED SCHOOL DISTRICT
• BHS/BOYLE HEIGHTS RECOVERY CENTER • MELA
• EAST LOS ANGELES WOMEN’S CENTER • MEXICAN AMERICAN OPPORTUNITY FOUNDATION
• EASTMONT COMMUNITY CENTER • VIOLENCE INTERVENTION PROGRAM (VIP)
• ENKI HEALTH & RESEARCH SYSTEMS, INC. • WIC
• HEALTHY START – GARFIELD CLUSTER
• CITY OF CARSON PARKS AND RECREATION • PROJECT TOUCH
• CITY OF INGLEWOOD PARS, RECREATION AND
COMMUNITY SERVICES
• RICHSTONE FAMILY CENTER
• CITY OF LONG BEACH CENTERS FOR FAMILY
& YOUTH
• ROSE CITY RESEARCH
• CONNECTIONS FOR CHILDREN • SAN PEDRO COMMUNITY LEGAL SERVICES
• INGLEWOOD COALITION FOR DRUG &
VIOLENCE PREVENTION
• SOUTH BAY YOUTH PROJECT
• INSTITUTE FOR BLACK PARENTING • SOUTH BAY CENTER FOR COUNSELING
• LENNOX SCHOOL READINESS • SOUTHERN CALIFORNIA INDIAN CENTER
• LONG BEACH YMCA • TORRANCE DCFS OFFICE
• PACIFIC ASIAN COUNSELING SERVICES

6
TABLE OF CONTENTS
Acknowledgements 1
List of Agencies 2
Table of Contents 6
Section I: Introduction 8
The Partnerships for Families Initiative 8
Year 1 Evaluation Activities 8
Initial Year 2 Activities 8
Section II: Planning and Coordination 10
Meetings with First 5 LA and DCFS 10
Evaluation-Related Input Regarding PFF Lead Agency Applicants 11
Interview Protocol 12
Initial Meetings with PFF Lead Agencies 12
Follow-up Meetings with Lead Agencies 13
Collaborative Meetings 13
Grantee Meetings 14
Evaluation Workgroup Meetings 14
First 5 LA Data Sharing Meeting 15
Meetings Related to the Selection of Social Learning and Training Facilitator 15
Development of the PFF Initiative Evaluation Website 15
Evaluation Team E-Mail Address 16
Section III: Toward a Comprehensive Evaluation Plan 17
Meetings with First LA Research & Evaluation Commissioner Liaisons:
A New Focus for the Initiative Evaluation 18
Development of an Implementation Monitoring Plan 19
Evaluation Capacity Assessment of PFF Agencies 20
Benchmarks & Performance Measures 32
Constructing Agency and Collaborative Level Assessment 32
Agency Level Assessments 32
Collaborative Level Assessments 33
Wilder Collaboration Factors Inventory 33
PFF Network Inventory 34
Examining PFF Lead Agencies’ Data Collection Tools 36
Tracking/Documenting Collaborative Composition Listings in
RFP Responses 36
Analysis of Demographic Information from Current Intake Forms 36
Development of Family- and Child-Level Outcomes 37
Research on the FAF and the NCFAS 38
Construction of PFF Evaluation Manual 39

7
August 2006
Section IV: Technical Assistance to Inform Program Development 40
Technical Assistance to Applicant Agencies: Applicant
Information Workshop 40
Technical Assistance to First 5 LA on the Development of an
Evaluation Database 40
Section V: Reflections on Years 1 & 2 42
Re-Examination of Year 1 Suggestions 42
Overarching Theme One: Overcoming Barriers 42
Overarching Theme Two: Importance of a Clear and Consistent Vision 43
Overarching Theme Three: Relationships and Communication among
Key Partners in the Initiative 44
Overarching Theme Four: Mixed Feelings about the Current
Implementation Plan 44
Initiative Evaluation Strengths and Successes 45
Challenges to the Initiative Evaluation 46
Conclusion – Next Steps 47
References 50
Appendices 51

8
SECTION I: INTRODUCTION
The Partnerships for Families Initiative
Funded by First 5 LA, the Partnerships for Families Initiative (PFF) is a secondary
child maltreatment prevention initiative designed to strengthen families as well as the com-
munities in which they live. Specifically, the five-year initiative aims to create and coordi-
nate a web of local level partnerships between new and existing service agencies and groups
that are coordinated, accessible and responsive to the unique needs of children and families
(http://www.first5la.org/ourprojects/PartnershipsforFamiliesInitiative.php4). The popula-
tions targeted by PFF include families with children five years old or younger who are
identified as being at risk for maltreatment by the Department of Children and Family
Services and high-risk pregnant women (Please see the PFF RFP for more details on the
target populations).
Year 1 Evaluation Activities
Although an implementation plan for the PFF initiative was finalized and approved
by First 5 LA at the end of Year 1, the Evaluation Team was largely involved with pre-
implementation planning efforts of PFF stakeholders. Specifically, the Evaluation Team
completed a process study that qualitatively described various stakeholders’ perspectives on
what occurred as year one planning moved forward. Additionally, the Evaluation Team
provided technical assistance to First 5 LA staff, developed several iterations of an
evaluation plan in response to the initiative’s evolving nature, and conducted a preliminary
review of data sources and assessment tools currently used by First 5 LA, child and family
service agencies and the Department of Children and Family Services.
Initial Year 2 Activities
Early in year two, a key deliverable for the Evaluation Team was a summary report
for the first year of the evaluation. Accordingly, we prepared a comprehensive report that
was designed to facilitate refinement of the PFF Initiative, to assist First 5 LA in ongoing
organizational development activities, and to summarize the details of the Evaluation

9
August 2006
Team’s activities during the first year of the evaluation.
We submitted a draft of the report to First 5 LA in September 2005. By design, sub-
mission of a draft of the report provided us with an opportunity to ensure accuracy of our
findings, to solicit alternative interpretations of the findings, and to solicit general feedback
about the report.
After the revised, final version of the report was submitted in November 2005, the
Evaluation Team held discussions with First 5 LA PFF staff and Commission Board mem-
bers in order to more thoroughly describe findings and implications from the (pre-
implementation) process study, to offer some concrete suggestions, and to respond to any
questions about the report.
Prior to the second year of the evaluation, numerous meetings and conference calls
were held between the Evaluation Team and the First 5 LA PFF Evaluation Liaison in order
to develop the year two scope of work. Ultimately, the agreed-upon scope of work (see Ap-
pendix A) included the following six major activity areas: planning and coordination; tech-
nical assistance to applicant agencies; technical assistance to PFF Lead Agencies; bench-
marks and performance measures; monitoring program fidelity; and a comprehensive
evaluation plan. We describe our efforts in each of these major areas in the sections that fol-
low.

10
SECTION II: PLANNING AND COORDINATION
Just as collaboration is pivotal to the programmatic success of PFF and to other
large-scale, innovative child maltreatment prevention efforts, it is also vital to a relevant and
rigorous evaluation of PFF. To this end, the Evaluation Team has engaged in planning and
coordination activities to develop and/or strengthen relationships and ensure effective com-
munication among stakeholders. We have attempted to create and sustain productive work-
ing relationships with PFF evaluation stakeholders and to approach all tasks with a collabo-
rative spirit. In that vein, the Team has led or participated in a variety of collaborative ef-
forts during the past year, including those involving First 5 LA, the Department of Children
and Family Services (DCFS), the Partnerships for Families (PFF) Collaborative agencies,
and other key stakeholders. These efforts, which are described in the remainder of this sec-
tion, were instrumental in the successful planning and coordination of evaluation activities
during the second year. They have also set the stage for the more comprehensive evaluation
activities that will be carried out in the years to come.
Meetings with First 5 LA and DCFS
Beginning in October 2005, the Evaluation Team held regular, formal evaluation
planning meetings with First 5 LA and DCFS. These meetings generally were intended to
ensure that our planning efforts were consistent with the goals of other partners central to
the success of the PFF Initiative. The meetings were also designed to ensure that our efforts
were informed by and coordinated with other initiatives within First 5 LA, DCFS, and the
County.
Issues pertaining to various conceptual and logistical aspects of the Initiative, and
particularly issues relating to the design and implementation of the evaluation were ad-
dressed in our meetings with First 5 LA and DCFS. Chief among these issues were those
having to do with clarification of the following: the PFF Initiative conceptual model; target
population(s); referral procedures; expected short- and long- term outcomes; mechanisms
by which the outcomes are expected to be achieved (i.e., the theory of change model); per-
formance measures; data collection protocols; data sharing; and roles and responsibilities of
those involved in implementing and evaluating the Initiative.

11
August 2006
Meetings with First 5 LA and DCFS were held on at least a monthly basis through
February 2006, when it was determined collaboratively by First 5 LA, DCFS and the
Evaluation Team that subgroup meetings on specific areas of focus would be more effec-
tive.
Evaluation-Related Input Regarding PFF Lead Agency Applicants
After reviewing applications from RFP respondents, First 5 LA conducted site/audit
visits to speak with potential PFF lead agencies and to learn more about the agencies’ pro-
posed approaches. The Evaluation Team participated in each of the on-site “audit” visits
(conducted during December 2005) by the First 5 LA review committee. Inclusion of the
Evaluation Team in the site visits served two primary purposes. First, it allowed the
Evaluation Team to provide direct feedback to First 5 LA about the applicants’ evaluation
capacity as potential lead agencies. In this regard, applicants were assessed on various com-
ponents of evaluation, including training, experience, leadership support, data management,
and technical infrastructure. Secondly, site visits provided an opportunity for the Evalua-
tion Team to better understand any local variations around general evaluative activity.
Because of the differences in funding resources and service provision among the
PFF Collaboratives, the Evaluation Team examined the behavioral and perceptual differ-
ences around data collection, evaluation implementation, and evaluation reporting. Such an
examination allowed us to begin developing strategies to address any perceived barriers to
the initiative evaluation by grantees (such as delayed evaluation reporting), while reinforc-
ing existing strengths (such as the desire to share data more effectively).
Each site visit (except for the finance audit) consisted of a four-hour meeting that
involved both structured and unstructured interview questions. From the lead agency, the
Executive Director/CEO, Director of Finance/CFO, Director of Case Management, and Pro-
gram Evaluator (internal or external) were asked to participate in the site visit. In addition
to lead agency staff, a number of partner agency representatives were also present to address
questions about the proposed Collaboratives.
In order to comprehensively assess awareness of evaluation activity, most questions
posed during the audits were directed initially at non-evaluation personnel and then by the
more “formal” evaluators. During the structured interview process, we posed the questions

12
in the interview protocol presented below.
Interview Protocol
1. Please tell us a little about your planned communication structure (to share in-
formation).
2. How do you intend to share information with collaborative partners on a regu-
lar basis?
3. Tell us about your plans for quality control.
4. What will this process look like?
5. What sort of feedback loop do you propose to ensure successful implementa-
tion?
6. How will problems in your proposed system be addressed?
7. How do you plan to include clients in this process?
8. Describe your plans for data collection.
9. How will the lead agency work with the collaborative partners to obtain data?
10. How will the lead agency ensure cooperation of collaborative partners?
11. Please describe your IT capacity with regards to data entry and database ad-
ministration.
12. How will your existing infrastructure and staffing assist in regard to start-up
of the program?
13. Describe your plans for monitoring provision of services.
Initial Meetings with PFF Lead Agencies
After the lead agencies were approved by the First 5 LA Commission, the Evalua-
tion Team’s Agency Liaison facilitated meetings of the Evaluation Team and lead agency
staff. The objectives of the meetings were to:
• Provide information about the PFF Evaluation Team and to explain Team
members’ roles;
• Provide a brief overview of the evaluation plan to date;
• Gauge evaluation interest and potential areas for trainings;
• Assess evaluation capacity so that we can plan trainings and other support
accordingly; and
• Ascertain what agencies want to learn from the PFF evaluation.
Underlying these objectives was a desire to forge a foundation for a longer-term col-

13
August 2006
laborative relationship with the agencies. We believe that the initial meetings with lead
agencies were indeed an important step towards ensuring productive conversations as we
proceeded with the evaluation. This was evidenced during the meetings as many agencies
indicated that they appreciated that the Evaluation Team was initiating conversations with
the lead agencies and involving them in the planning and decision-making before program
implementation began.
In the initial meetings with PFF lead agencies, the Evaluation Team expressed its
interest in working with the lead agencies and their collaborating partners in order to maxi-
mize relevance and utility of the evaluation. We, the Evaluation Team, highlighted our de-
sire to involve agencies in the development of the evaluation plan. At the same time, we
emphasized the importance of trying to balance the [sometimes competing] desires of multi-
ple stakeholders.
As a means to concretely demonstrate the Evaluation Team’s commitment to col-
laboration and open communication, feedback from the initial meeting with the PFF agen-
cies was summarized and reported back to the agencies at the first Grantee Meeting in
March 2006.
Follow-up Meetings with Lead Agencies
As with many evaluations of complex initiatives and agencies experiencing organ-
izational change, we have found that the dynamic environment of the PFF Initiative calls for
continuous communication and coordination with key stakeholders in order to minimize
confusion and anxiety. Recently, as a follow-up to the initial meeting and part of on-going
dialogue with the lead agencies, the Evaluation Team’s Principal Investigator and Agency
Liaison have begun holding regular meetings with each of the lead agencies. The objectives
of the follow-up meeting are to further strengthen our working relationship, to address any
outstanding issues, and to respond to any questions the agencies might have.
Collaborative Meetings
Each Collaborative of PFF agencies regularly convenes meetings to review the pro-
gram and discuss potential strategic action. While their composition varies, the membership

14
of Collaborative meetings consists of stakeholders such as service providers, County depart-
ment representatives, and community members. In the initial meetings with lead agencies,
the Agency Liaison requested permission to participate in the Collaborative meetings.
For purposes of the evaluation, Collaborative meetings provide our Team with a
greater understanding of how the PFF Initiative is being conceptualized and implemented
by the Collaboratives. It also provides some context and insight into how Collaboratives are
operating. Finally, attending these meetings gives us with an opportunity to provide evalua-
tion-related updates, to clarify possible misperceptions, to respond to questions, and to col-
lect insights into the evaluation that should be shared with the Evaluation Team.
Grantee Meetings
During the planning period in particular, First 5 LA organized a series of meetings
of PFF agencies to provide support as they move into implementation. The Agency Liaison
attended Grantee Meetings hosted by First 5 LA. Consequently, the Evaluation Team has
gained a greater sense of programmatic issues such as the DCFS referral process. The
Evaluation Team also worked with First 5 LA staff to design a segment of the Grantee Ori-
entation Meeting. During this segment, we introduced individual members of the Team,
shared our approach to evaluation of the PFF Initiative (Appendix B), and addressed ques-
tions from grantees.
Evaluation Workgroup Meetings
In the spring of 2006, First 5 LA convened an Evaluation Workgroup, comprised of
programmatic and evaluation staff from the Lead Agencies and the First 5 LA PFF Initia-
tive, as well as members of the Evaluation Team. Members of the Evaluation Team, par-
ticularly the Agency Liaison, have participated in each of the Evaluation Workgroup meet-
ings throughout the year. At the Workgroup’s second meeting, the Evaluation Team pre-
sented a draft of the evaluation plan for consideration by the agencies. We also facilitated
discussions on the strengths and weaknesses of data collection tools that would also be used
for case planning purposes (Please also see section on Research on NCFAS and FAF, be-
low.)

15
August 2006
First 5 LA Data Sharing Meeting
The First 5 LA Research & Evaluation staff convened a meeting to facilitate sharing
data among a wide array of agencies and organizations throughout Los Angeles County. At
the request of First 5 LA, the Evaluation Team participated in the February 2006 First 5 LA
Data Sharing Meeting. At the meeting, the Evaluation Team provided information to other
attendees about plans for the initiative evaluation and the types of data that might ultimately
be available to others.
Meetings Related to the Selection of Social Learning and Training Facilitator
Per the original RFP, the Social Learning & Training Facilitator is a First 5 LA con-
sultant charged with facilitating collective learning to promote the accumulation of knowl-
edge among PFF agencies, First 5 LA and the Evaluation Team. The Evaluation Team was
invited to be a member of the selection committee for the Social Learning and Training Fa-
cilitator. Responsive to the timeframe, the Agency Liaison reviewed the submitted propos-
als and participated in the interviews. After much thoughtful conversation, the selection
committee reached consensus and selected a Social Learning and Training Facilitator for the
PFF Initiative.
Development of the PFF Initiative Evaluation Website
The PFF Evaluation Team wants to provide information about the initiative evalua-
tion and relevant information to PFF stakeholders and other persons involved in child wel-
fare and/or evaluation. Since information, particularly in this new initiative, is dynamic, the
Team determined that one way to disseminate information in a regular and timely fashion
would be through an evaluation website. First 5 LA was consulted about the idea and
agreed that a website would be a beneficial source of information and communication—
which would presumably result in enhanced collaboration and coordination among the
Evaluation Team, First 5 LA, collaborative agencies, and other key stakeholders.
The Evaluation Team has conceptualized the website as a user friendly and inclusive
portal that would feature information about the PFF Initiative and about best practices in
child welfare and evaluation. Various websites relating to child welfare and the practice of

16
evaluation were examined in order to determine the type of information to include on the
website.
Simultaneous to the website design research, vendors were sought, and Urban In-
sight was selected as the vendor of choice. As the web design was being developed by the
vendor, our Team continued to examine, gather, and develop content for the website. Cur-
rently the website features information about various aspects of the Partnerships for Fami-
lies Initiative such as information about the Evaluation Team, PFF Lead Agencies, Service
Planning Areas and the latest developments with the evaluation plan.
Since the initial conversation with First 5 LA, certain developments have impacted
the website. In June 2005, for instance, an RFQ was released for the position of Social
Learning and Training Facilitator for the PFF Initiative. The chosen facilitator (ICA - The
Institute of Cultural Affairs) was subsequently tasked with developing a website for the PFF
Collaborative agencies. Though the PFF Evaluation Team has launched its website, the
Team is collaborating with ICA so that the two PFF-related websites are not redundant.
While our website is focused more on the evaluation aspects of the initiative and related is-
sues, the website ICA is launching will focus more on the programmatic components. At
this time the PFF Initiative Evaluation website (http://www.pffevaluation.org) continues to
be refined.
Evaluation Team E-Mail Address
Aware that stakeholders may not know which member of the Evaluation Team to
contact with respect to particular questions or comments, the Team sought to provide an
easy method by which others could communicate with us. Hence, a pffeval@usc.edu e-mail
address was created. Once the Team receives an e-mail, the question is forwarded to the
appropriate Evaluation Team member to reply in a timely manner. As the Agency Liaison
has developed a strong relationship with the PFF agencies, many of the PFF agencies’ ques-
tions have been addressed by her through email, phone calls or in-person. We have estab-
lished this team e-mail infrastructure as we anticipate that in the future, other interested
stakeholders will submit questions using this email address.

17
August 2006
SECTION III: TOWARD A COMPREHENSIVE EVALUATION PLAN
In order to develop the most relevant and rigorous evaluation possible, the Evalua-
tion Team has solicited input from an array of PFF stakeholders, including First 5 LA,
DCFS, and PFF Collaborative agencies. To provide ample opportunities for stakeholders to
raise questions and voice opinions about the evaluation, the Team has held numerous meet-
ings and attended those regularly held by other stakeholders (e.g., Executive Team and
Point of Engagement outcomes meetings at DCFS; monthly Collaborative meetings).
Through the evaluation website, we also provide opportunities for visitors to send com-
ments or suggestions regarding the evaluation.
In response to significant changes in the evaluation plan that were requested by the
First 5 LA Commissioners, and with input from various stakeholders, the Evaluation Team
has prepared a third, comprehensive evaluation plan for the PFF Initiative.
A draft of the revised evaluation plan (Appendix C) was submitted to First 5 LA on
April 17, 2006. The Evaluation Team initially introduced the draft evaluation plan at an
April 21, 2006 Grantee meeting. The Team responded to many of the questions in-person
but there was not sufficient time to respond to all the questions. Therefore, PFF agency
staff submitted questions and subsequent to the meeting, the Evaluation Team created a
document with written responses (Appendix D). Many of the questions pertained to meth-
odology, availability of data to agencies, data collection effort and other general questions.
The draft was also shared with members of the Evaluation Workgroup on May 10, 2006.
The draft evaluation plan detailed the following evaluation components, starting
with five primary questions:
1. Was child safety enhanced as a result of participation in the PFF Initiative?
2. Did improved family well-being influence child safety?
3. What characteristics of the PFF Initiative were related to improved family
well-being (i.e., parenting and family functioning)?
4. At each the (a) agency level, (b) community level, and (c) county level, was
child maltreatment prevented as a result of participation in the PFF Initia-
tive?

18
5. At each the (a) agency level, (b) community level, and (c) county level, was
child permanency enhanced as a result of participation in the PFF Initia-
tive?
To address these questions, the following design elements were identified:
• Comparison group
• Cross sectional data
• Longitudinal data (pretest, posttest and follow-up)
• Qualitative and quantitative surveys and/or interviews
• Data retrieval from DCFS administrative data and agency administrative data
(including First 5 LA Grants Management data)
The evaluation plan proposed three potential comparison groups for measuring im-
pact of the PFF Initiative and included agency-level samples (the lead agency, partner agen-
cies, and collaboratives) for examining other aspects of the Initiative. Furthermore, meas-
ures and indicators were identified for the primary domains of the evaluation: child safety,
family well-being, engagement, service utilization and quality, collaboration, organizational
learning, and demographics for family, parent, and child.
To summarize the evaluation plan, the draft identified key data sources, and de-
scribed the procedures for answering the evaluation questions. In addition to forming hy-
potheses, the summary detailed how the proposed designs, samples, and measures come to-
gether to answer the primary evaluation questions for the PFF Initiative.
Meetings with First LA Research & Evaluation Commissioner Liaisons: A New Focus
for the Initiative Evaluation
The draft of the evaluation plan described above was submitted for review by the
First 5 LA Commissioner Liaisons. Shortly after the May Evaluation Workgroup meeting
mentioned above, the Evaluation Team met with the Commissioner Liaisons who chose a
new direction for the evaluation. In response and to maximize productivity of the next
meeting with the Commissioner Liaisons, the Evaluation Team modified the draft evalua-

19
August 2006
tion plan (May 2006, see Appendix C). In June 2006, the Evaluation Team met with the
First 5 LA Research & Evaluation Commissioner Liaisons to discuss specific details of the
new direction and then had conversations with First 5 LA staff to finalize the scope of work.
By mid-June 2006, the Evaluation Team had somewhat greater clarity about the new direc-
tives for the initiative evaluation for the coming year. At the request of the First 5 LA Com-
missioners Liaisons, the Evaluation Team will focus its efforts on two primary areas: large-
scale case studies of each of the PFF Collaboratives and a detailed values inquiry of PFF
stakeholders.
Development of an Implementation Monitoring Plan
In order to begin monitoring implementation of the PFF Initiative, the Evaluation
Team developed an Implementation Monitoring Protocol. This process began with several
efforts. First, extensive depth interviews conducted with First 5 LA Staff were carefully
reviewed to determine key areas of interest and potential research domains. Second, there
was continued field observation at First 5 LA Commission meetings and First 5 LA work
group meetings were used to inform instrument development. Third, there was a compre-
hensive research literature review of all implementation monitoring protocols utilized in
similar evaluation efforts. This review was presented as a set of “best practices” implemen-
tation monitoring instruments. Three major deliverables emerged from this work; they in-
clude a list of major research domains and two primary instruments that were designed to be
administered as part of the implementation monitoring process.
The details of the Implementation Monitoring Plan as developed are detailed below.
However, it should be noted that with numerous changes in the direction of the initiative
evaluation toward the end of evaluation year 2, the plan may not be executed as described.
Yet, some aspects of the plan described will be assessed in the current carnation of the
evaluation plan using a case study approach.
As a result of the literature review and Team meetings and discussion, it was deter-
mined that we will monitor and assess the following components of implementation:
• Strategies for engagement of clients,
• Collaboration among agencies in service provision,

20
• Service Characteristics & Delivery,
• Service Quality, and
• Capacity Building, both external and internal.
The first of two major instruments developed is a quarterly self-report form
(Appendix E) that will track and monitor implementation facilities. After meetings with
First 5 LA Grants Management, it was determined that much of this data would be collected
as a natural part of their work. Therefore, the Evaluation Team will rely on this secondary
data, rather than require that agencies collect and report on duplicative data. This quarterly
self-report form can be completed by agencies within the Collaborative. We estimate that
the questionnaire will be distributed to a sample of approximately 60-80 agency staff (the
lead staff person for each of the PFF agencies). This form allows staff in PFF agencies to
reflect on and document activities related to the implementation of the PFF.
The second instrument is an open-ended qualitative questionnaire (Appendix F).
The interview protocol consists of thematically oriented open-ended questions. In-person
interviews will be conducted with a sample of approximately 80 subjects that will include
those completing the self-report form (see paragraph above) as well as other staff including
the Executive Director, Program Director, Project Manager, Clinical Director, Case Manag-
ers, In-Home Outreach workers and data staff as well as the advisory committee members
of the eight Collaborative Agencies
Together, these two instruments have been designed to collect detailed qualitative
data that will offer a multi-dimensional picture of the entire implementation process. To
prepare for this work, an application to the UCLA Internal Review Board (IRB) was sub-
mitted and approved. (Please see IRB Approval Notice in Appendix G.)
Evaluation Capacity Assessment of PFF Agencies
Prior to the initial Team meetings with the Lead Agencies, the PFF Evaluation Team
administered an online evaluation capacity assessment to various staff members from each
site. In most cases, respondents consisted of the executive director and at least one local
evaluation/research expert. Questions from the survey were designed to assess evaluation
capacity with regard to three specific domains: technical capacity/data management, barriers

21
August 2006
to implementation, and staff training/awareness (see Appendix H).
Once data was collected, information was used primarily to help facilitate discus-
sions during the site visits around the agency’s assessed strengths and weaknesses around
evaluation. The aggregated data, which are presented below, are currently being used to
inform the initiative evaluation planning. Analysis of these data will continue into year three
of the evaluation.

22
Evaluation Capacity Assessment
PFF Lead Agencies
Total N = 17
(7 of 8 PFF Lead Agencies represented)
Exhibit A. Technical Capacity
Please estimate what percentage of your agency's staff has access to a work compute
Please estimate what percentage of your agency's staff (with computer access) are net-
worked together by one or more shared servers?
Response Response Total
0-25% 0%
26-50% 0%
51-75% 0%
75-100% 93.8%
Do not know 6.2%
0-25% 6.2%
26-50% 18.8%
51-75% 0%
75-100% 50%
Do not know 25%

23
August 2006
Please estimate what percentage of your agency's staff (with computer access) commu-
nicate with each other via email?
Does your agency have internet access?
Does your agency have high-speed internet access (cable-modem, DSL, T1 or faster)?
Does your agency have its own website?
0-25% 18.8%
26-50% 18.8%
51-75% 6.2%
75-100% 43.8%
Do not know 12.5%
Yes 100%
No 0%
Yes 100%
No 0%
Do not know 0%
Yes 100%
No 0%
Do not know 0%

24
How are data collected and stored at your agency? (check all that apply)
What percentage of your agency staff is responsible for some type of data collection,
storage, or analysis?
Paper/pencil assessments, forms, charts, records 100%
Internet/web-based database 62.5%
Computerized database managed locally 87.5%
Do not know 0%
0-25% 6.2%
26-50% 18.8%
51-75% 31.2%
75-100% 18.8%
Do not know 25%

25
August 2006
Exhibit B. Program Evaluation (Barriers)
Do you have a budget line item for evaluation for any of your funded programs?
Who performs evaluation activities (e.g. developing evaluation plans, writing reports,
administering pre- and post-tests, etc.) for your program(s)? Check all that apply.
What barriers or challenges, if any, have you or your program staff experienced in
meeting evaluation activities in the past 12 months? Check all that apply.
Yes 73.3%
No 6.7%
Do not know 20%
Evaluation consultant outside of agency 57.1%
Staff within agency and a part of program staff 100%
Staff within agency and NOT a part of program staff 28.6%
Limited staff capabilities 78.6%
Lack of resources or supplies (e.g., computers, statistical software) 42.9%
Lack of evaluation training 78.6%
Management perceptions regarding the value of evaluation 35.7%
Unclear program objectives 14.3%
Poor program evaluation plan 42.9%
Unrealistic program objectives 21.4%
Client recruitment 7.1%
Client retention 14.3%

26
Exhibit C. Evaluation Training (SELF)
How do you rate your ability to distinguish between process evaluation and outcome
evaluation?
Poor 0%
Fair 38.5%
Good 23.1%
Excellent 38.5%

27
August 2006
Please rate your skill level for each of the following evaluation activities.
Evaluation Activities
No
Skill
Some
Skill
Moderate
Skill
Extensive
Skill
A. Learning basic evaluation methods
(e.g., process and outcome evaluation)
0% 31% 46% 3%
B. Developing an evaluation plan 15% 23% 31% 31%
C. Designing surveys 23% 23% 31% 23%
D. Conducting focus groups 8% 38% 31% 23%
E. Collecting data (e.g., administering surveys) 8% 15% 54% 23%
F. Using evaluation information in program
design and implementation
8% 38% 23% 31%
G. Developing computerized tracking database 38% 23% 31% 8%
H. Entering data on computerized spreadsheet 23% 23% 31% 23%
I. Managing data on computerized spreadsheet 23% 31% 23% 23%
J. Analyzing data 8% 38% 31% 23%
K. Using statistical software to analyze data 54% 23% 15% 8%
L. Interpreting results from qualitative data
analysis
38% 15% 23% 23%
M. Interpreting results from quantitative data
analysis
31% 23% 23% 23%
N. Completing monthly report forms for funding
agencies
8% 23% 38% 31%
O. Writing evaluation reports 15% 15% 38% 31%
P. Using findings to improve programs 0% 15% 38% 46%
Q. Presenting program evaluation results 8% 8% 38% 46%

28
Table 1. Areas of Training Interest for RESPONDENTS
by Percentage of Responses
(From the list above, please select your top three areas of interest.)
% Item
54% Using evaluation information in program design and implementation
54% Using findings to improve programs
38% Developing an evaluation plan
23% Analyzing data
15% Learning basic evaluation methods (e.g., process and outcome evaluation)
15% Developing computerized tracking database
15% Managing data on computerized spreadsheet
15% Using statistical software to analyze data
15% Interpreting results from qualitative data analysis
15% Interpreting results from quantitative data analysis
8% Designing surveys
8% Collecting data (e.g., administering surveys)
8% Entering data on computerized spreadsheet
8% Writing evaluation reports
8% Presenting program evaluation results
0% Conducting focus groups
0% Completing monthly report forms for funding agencies

29
August 2006
Exhibit D. Evaluation Training (STAFF)
How do you rate your staff's ability to distinguish between process evaluation and out-
come evaluation?
Poor 8.3%
Fair 41.7%
Good 41.7%
Excellent 8.3%

30
Please rate your staff's skill level for each of the following evaluation activities.
Evaluation Activities
No
Skill
Some
Skill
Moderate
Skill
Extensive
Skill
A. Learning basic evaluation methods
(e.g., process and outcome evaluation)
8% 50% 42% 0%
B. Developing an evaluation plan 25% 50% 17% 8%
C. Designing surveys 17% 67% 17% 0%
D. Conducting focus groups 8% 67% 25% 0%
E. Collecting data (e.g., administering surveys) 0% 33% 67% 0%
F. Using evaluation information in program
design and implementation
17% 42% 42% 0%
G. Developing computerized tracking database 25% 58% 8% 8%
H. Entering data on computerized spreadsheet 8% 50% 25% 17%
I. Managing data on computerized spreadsheet 8% 58% 25% 8%
J. Analyzing data 25% 50% 25% 0%
K. Using statistical software to analyze data 33% 42% 25% 0%
L. Interpreting results from qualitative data
analysis
33% 50% 17% 0%
M. Interpreting results from quantitative data
analysis
33% 58% 8% 0%
N. Completing monthly report forms for
funding agencies
0% 33% 33% 33%
O. Writing evaluation reports 25% 42% 17% 17%
P. Using findings to improve programs 8% 42% 42% 8%
Q. Presenting program evaluation results 17% 42% 33% 8%

31
August 2006
Table 2. Areas of Training Interest for Agency’s STAFF
by Percentage of Responses
(From the list above, please rate your top three areas of interest for your staff.)
% Item
33% Collecting data (e.g., administering surveys)
33% Using findings to improve programs
25% Developing computerized tracking database
25% Analyzing data
17% Learning basic evaluation methods (e.g., process and outcome evaluation)
17% Developing an evaluation plan
17% Using evaluation information in program design and implementation
17% Entering data on computerized spreadsheet
17% Interpreting results from qualitative data analysis
17% Interpreting results from quantitative data analysis
17% Presenting program evaluation results
8% Designing surveys
8% Conducting focus groups
8% Managing data on computerized spreadsheet
8% Using statistical software to analyze data
8% Completing monthly report forms for funding agencies
8% Writing evaluation reports

32
Benchmarks & Performance Measures
Benchmarks and performance measures were developed to guide agency implemen-
tation. The Evaluation Team identified benchmarks and performances measures based on
evaluations of similar initiatives nationwide, other First 5 LA initiatives, and DCFS
programs. Several documents were produced for this purpose, outlining benchmarks and
performance measures to be aligned with the overarching implementation goals as well as
Initiative outcomes (see Appendices I, J, and K).
The benchmarks and performance measures covered two key areas: (1) service
delivery and service quality, with an emphasis on client engagement, and (2) collaboration
within networks, including supporting infrastructure for effective collaboration across
agencies.
The Evaluation Team worked directly with First 5 LA to apply the benchmarks and
performance measures for implementation planning and monitoring. In addition to meeting
with First 5 LA Grants Management, the PFF Evaluation Team members attended DCFS
evaluation meetings for Point of Engagement, and consulted with First 5 LA Research
Analysts to develop benchmarks and performances measures in preparation for PFF roll-out
and implementation monitoring.
Constructing Agency and Collaborative Level Assessment
Agency Level Assessments
Identification of the agency level measures focused primarily on organizational
scales commonly used in the child welfare and social service domains. Measures at this
level are focused internally and emphasize the work environment, relationships, or
situations as perceived by the agency’s staff members. The rationale in determining which
organizational assessments to include was twofold: its utility as an interagency comparison
measure and its external validity in relation to social service outcomes. Given the
collaborative nature of the PFF initiative, data at this level can provide an important link in
determining which organizational factors are associated with more effective collaborative
behavior and, in turn, which of these factors directly relate to improvements in child/family
service delivery.

33
August 2006
Agency level scales include the following:
• Organizational Climate Scale
• Overall Job Satisfaction Rating
• Employee Engagement Scale
• Anticipated Turnover
• Readiness for Organizational Learning and Evaluation (ROLE)
• Employee Demographics
Collaborative Level Assessments
As a supplement to the qualitative data analysis, the Evaluation Team also
constructed several instruments around interagency collaboration. It is important to note
that evaluation of collaborative networks in the child/family service sectors has traditionally
been limited due to difficulties in defining network membership and identification of
comparable networks. In many ways, the current PFF Initiative provides a unique opportu-
nity to overcome these barriers by establishing eight predetermined networks, each sharing
similar service goals and funding resources. Interagency collaboration will be examined via
two models: the Wilder Collaboration Factors Inventory and the PFF Network Inventory,
each of which is described below.
Wilder Collaboration Factors Inventory.
This instrument utilizes a 5-point Likert scale designed to assess 20 key factors
around each of the networks and their collaborative activity. Data will be collected from
representatives of each participating agency (both lead and partners). Below is a brief
description of the 20 factors.
• History of collaboration/cooperation in the community
• Collaborative group is seen as a legitimate leader in the community
• Favorable political and social climate
• Mutual respect, understanding, and trust

34
• Appropriate cross section of members
• Members see collaboration as in their self interest
• Ability to compromise
• Members share a stake in both process and outcome
• Multiple layers of participation
• Flexibility
• Development of clear roles and policy guidelines
• Adaptability
• Appropriate pace of development
• Open and frequent communication
• Established informal relationships and communication links
• Concrete, attainable goals and objectives
• Shared vision
• Unique purpose
• Sufficient funds, staff, materials, and time
• Skilled leadership
PFF Network Inventory.
The PFF Network Inventory is an adaptation of several instruments used in prior
collaborative network studies (CPPC). This assessment capitalizes on the establishment of
predetermined membership. By allowing each participant to rate the other collaborative
members on the following 6 dimensions:
1. Familiarity
How well do you know the work and the activities of this agency? The focus
is on activity directed to children and families in your Service Planning Area.

35
August 2006
2. Shared Activity on Specific Cases
Origin of referrals: Your organization receives service referrals from this
other organization, or, clients come to your agency because they were
informed by persons in this organization.
Tracking service referrals: Your agency refers or recommends clients to this
organization for services or supports.
Joint case planning or consultation: Your agency staffs cooperate on indi-
vidual client cases through case planning (family team meetings, integrated
service teams, etc.), or you share specific case information and service strate-
gies with staff of this other organization.
3. General Service Coordination
Participation in meetings on service coordination: Staff persons from both
organizations attend the same planning meetings or have contact about issues
of service strategy, agency coordination, service integration, and/or change
of the service system.
4. Change in Contact
Has your amount of interaction with this agency changed in the last 2 years?
If your own experience is shorter than two years, respond for the time period
that you know.
5. Your own Contact (with staff in other agencies in the last 3 months)
Number of your professional or work contacts. Estimate the number of your
own professional/work related contacts (calls or in-person) with staff from
each agency.
Number of your personal or social contacts. Estimate the number of your
own personal or social contacts (calls or in-person) with staff from each
agency.

36
6. Assess Quality
How would you rate the quality of the services provided by this agency?
Select the response that best summarizes your own assessment of the overall
quality of services and supports that each agency provides for children and
families.
Examining PFF Lead Agencies’ Data Collection Tools
Agencies typically have multiple funding streams that require different data collec-
tion tools. Often many data points are similar but different enough that they require differ-
ent forms. In an effort to reduce duplicative data collection efforts, agencies provided their
current data collection instruments in the proposals submitted to First 5 LA.
This past year we examined the tools used by the lead agencies. A chart entitled
“Data Collected by PFF Lead Agencies” was created to summarize the tools and domains
assessed by the tools (Appendix L). Please note that categories were imposed on the tools
and consequently, they may not be completely accurate in their placement, particularly
since the categories are necessarily mutually exclusive. Nonetheless, this chart provides a
sense of what tools are being used and by which agency.
Tracking/Documenting Collaborative Composition Listings in RFP Responses
The Evaluation Team reviewed all the applications from agencies responding to the
RFP. We created a document (available from First 5 LA) that tracks all the lead agencies
that responded to the RFP, for which Service Planning Area they applied, who the respec-
tive Collaborative partners are, who were funded partners and what services or component
they would have addressed. This information may be helpful in understanding the context
of the Collaboratives and how they evolve over time.
Analysis of Demographic Information from Current Intake Forms
In preparation for a conversation with PFF lead agencies around a base level of
commonality in demographic information, a chart was created that identifies fields in the
agencies’ current intake form(s) (Appendix M). For many of the fields, the chart also

37
August 2006
indicates how many categories are available. For example, while some agencies have a
blank field for race/ethnicity (i.e. allowing the respondent to reply with a category of their
choice), other agencies have categories that range from 5 to 20 ethnicities. Note that while
there needs to be some level of commonality, agencies may have more than the base-level
categories as long as they can be collapsed into the agreed upon categories.
The Evaluation Team plans to meet with PFF agencies and First 5 LA staff about
demographic information so as to explicitly determine the demographic information col-
lected. However, it should be noted that the PFF agencies have decided to use the Family
Assessment Form (FAF) which includes a section of demographic data. Moreover, we plan,
to some extent, to align to Department of Children and Family Services’ data.
Development of Family- and Child-Level Outcomes
A major part of the PFF Evaluation Team’s work in the second year of the Initiative
was to further develop the family- and child-level outcomes component of the evaluation
plan. According to First 5 LA’s Research and Evaluation RFQ, PFF Implementation Plan
and the Lead Agency RFP, family-level outcomes were to be included and examined in the
evaluation plan. The family level outcomes centered on the concept of family well-being.
In order to appropriately conceptualize and develop a plan addressing family well-being, the
Team gathered information on families such as the definitions of family, family types, fam-
ily processes, the relationship between child and family well-being, family well-being
outcomes, and the measurement of families-related processes and outcomes.
The process of providing a foundation for the concept of family well-being began
with the review of two documents: the Partnerships for Families Research and Evaluation
Project Proposal prepared by the Evaluation Team, and the Research and Evaluation Pro-
ject: Partnerships for Families RFQ prepared by First 5 LA. These documents provided de-
tails about how First 5 LA wanted family well-being to be measured, how the Team initially
intended to measure family well-being and current measurements being used to investigate
family well-being (see Appendix N).
Next, a review of large scale national and international child welfare projects, such
as Longscan and Families and Children Study (UK), was conducted. Concurrent with the

38
review of child welfare projects was a survey of literature from the social and health science
fields for information related to family well-being and the measurement of families.
The research guided the Team in its conceptualization of family well-being and
delineation of how family well-being could be measured (see Appendix O).
During the review of child welfare projects and applicable literature, a plethora of
measures was identified (see Appendix P). Pertinent measures were added to a database of
child welfare instruments. It became apparent that two measurements, the North Carolina
Family Assessment Scale (NCFAS) (Kirk & Reed-Ashcraft, n.d.) and the Family Assess-
ment Form (FAF) (Children’s Bureau of Southern California, 1997), were well suited for
use in the PFF Initiative. An extensive comparison of the two instruments was carried out
(see below).
Research on the FAF and the NCFAS
In developing the evaluation plan, the Evaluation Team has been committed to mini-
mizing the burden on both PFF agency staff and target families while at the same time
ensuring that the data used to assess both processes and outcomes is reliable and valid.
Further, guiding the development of the evaluation plan is a commitment to balancing the
programmatic needs of agencies with those of research. Consequently, when determining
which measures to use for collecting data on family and child processes and outcomes, our
goal was to rely on measures that could be used by the agencies as part of their routine prac-
tice and case planning.
A comprehensive review of the empirical and practice literature resulted in the iden-
tification of numerous measures that could potentially be used to satisfy both research and
case planning purposes. Among those measures were the Family Assessment Form (FAF)
and the North Carolina Family Assessment Scale (NCFAS).. . a As indicated previously, this
past year the Evaluation Team also reviewed the measures and forms currently used by the
agencies. Whenever possible, we planned to build on the practices and forms already in
place within the agencies. We found that several agencies were already using or familiar
with the FAF and/or the NCFAS. A review of their psychometrics and suitability for use
with the population of families who would be targeted by PFF indicated that either measure
would be appropriate for our evaluation. Thus, the final decision about which measure

39
August 2006
would be used for the evaluation was left to the PFF agencies.
The Evaluation Team encouraged the agencies to select a measure that could most
effectively be used in their case planning. First 5 LA convened a meeting of PFF grantees
to discuss the measures being considered for assessment of family well-being (see
Appendices Q, R, and S). At this meeting, presenters spoke to the use of the measures and
the PFF agency staff asked questions regarding the use and utility of each. At the conclu-
sion of the meeting, the PFF agencies that attended the meeting reached a consensus and
selected the FAF.
Construction of PFF Evaluation Manual
The Evaluation Team assisted First 5 LA in the development of the PFF Evaluation
Manual. The primary purpose for the manual is to help build on the capacity that already
exists within specific agencies and collaboratives. Grantees are encouraged to use their PFF
evaluation process to learn how their program theories—the stories that explain how pro-
gram activities will impact clients—play out in practice. Therefore, if a theory is not con-
firmed in practice, evaluation provides the organization with an opportunity to learn if the
reason why a program isn’t working as planned is in the implementation or the theory itself.
If the evaluation shows that the program theory is confirmed in practice, it may suggest the
need for program expansion. In either scenario, the organization is empowered with infor-
mation, which it can use to strengthen service delivery and increase organization and staff
capacity.
The PFF Evaluation manual is primarily a compilation of four (4) main sources:
First 5 LA Program Evaluation Kit (2002), U.S. Department of Health & Human Services
Administration on Children, Youth, and Families (ACYF) (2006), the Center for Disease
Control (CDC) Evaluation Working Group (1999), and The Results Accountability Imple-
mentation Guide (RAIG) (2005). ACYF, CDC, and RAIG are referenced in the Evaluation
Resources section of the manual along with additional evaluation resources. A draft of the
manual is contained in Appendix T.

40
SECTION IV: TECHNICAL ASSISTANCE TO INFORM PROGRAM
DEVELOPMENT
Understanding the importance of integrating and coordinating efforts, the Evaluation
Team has participated in various forums to support the initiative. This has included support
to potential lead agency applicants and First 5 LA. Through our participation, we believe
we increase the likelihood that the needs of the evaluation work well with other pieces of
the initiative.
Technical Assistance to Applicant Agencies: Applicant Information Workshop
The September 30, 2005 applicant information workshop provided an overview of
the RFP and allowed potential applicants to ask questions. Accordingly, the Principal In-
vestigator and Agency Liaison attended the Applicant Workshop to introduce the Evalua-
tion Team and to respond to evaluation-related questions. There were few questions at the
time relating to evaluation.
Technical Assistance to First 5 LA on the Development of an Evaluation Database
Early in the year, First 5 LA PFF staff requested the Evaluation Team’s participation
in the discussion with possible contractors who would develop a database for the Initiative.
In the past, First 5 LA has contracted with JMPT Consulting to design data systems for the
Family Literacy, Healthy Births and School Readiness Initiatives. Given this existing rela-
tionship and JMPT Consulting’s demonstrated ability to develop quality systems that meet
the needs of First 5 LA and grantees, First 5 LA was planning to continue working with
JMPT Consulting to develop a data system for PFF. However, before taking this step, First
5 LA wished to research another vendor who currently collects data from several of the PFF
lead agencies, in an effort to potentially reduce the level of duplicative data. First 5 LA
staff and the Evaluation Team met with the potential contractors. The Evaluation Team
provided feedback to First 5 LA regarding each developer’s capacity to supply the type of
data system that would likely be needed for the initiative evaluation. First 5 LA decided

41
August 2006
that JMPT has the technological capacity (e.g. data sharing that was demonstrated through
JMPT’s work in the Healthy Births Initiative) that best fits the needs of PFF and ultimately
selected JMPT Consulting to be the database developer for PFF. In November 2005, the
Evaluation Team began attending database-related meetings with First 5 LA, DCFS, and
JMPT staff in order to develop a database system that meets the needs of the evaluation and
of First 5 LA.

42
SECTION V: REFLECTIONS ON YEARS 1 & 2
Consistent with the concept of PFF as a learning initiative, the Evaluation Team
thought that it would be beneficial to reflect on the past two years of the Initiative and its
evaluation. We began by re-examining the issues that we recommended be considered by
First 5 LA staff and commissioners in our Year 1 report. We also spent some time
reflecting on Year 2 with regard to what we did well or not so well, and what we learned in
the process. This section details our conclusions about what aspects of the initiative evalua-
tion have worked well, the obstacles that have been encountered, and some future actions
we plan to take in order to maximize strengths and limit liabilities.
Re-Examination of Year 1 Suggestions
In last year’s initiative evaluation report, we identified four overarching themes
(Overcoming barriers, Importance of a clear and consistent vision, Relationships and com-
munication among key partners in the Initiative, and Mixed feelings about the current im-
plementation plan) and provided suggestions for First 5 LA to consider in further develop-
ing PFF. As detailed below, we have noted significant attention and related progress in sev-
eral areas, while others have been addressed only partially or even minimally.
Overarching Theme One: Overcoming Barriers
Five issues to consider were provided with regard to barriers to successful imple-
mentation of PFF:
• Fostering and maintaining employee commitment
• Succession planning
• Continuous feedback and reflection
• Appreciative inquiry
• Organizational capacity building
During evaluation year two, the Evaluation Team observed significant attention
being provided to PFF in several of these areas, including sustained high levels of employee

43
August 2006
commitment, continuous feedback and reflection, and organizational capacity building.
Now that the PFF agencies have become involved in the Initiative, succession planning (i.e.,
planning for expected levels of staff turnover) among program staff is becoming even more
important. Additionally, we suggest that First 5 LA continue to be mindful of providing
continuous feedback and reflection, ongoing attention to which is integral to any large-scale
change. We commend First 5 LA on contracting a Continuous Quality Improvement (CQI)
consultant to provide training to all the PFF agencies on what CQI is and how to incorporate
continuous learning into their program model.
We also recommend continued attention to appreciative inquiry and organizational
capacity building, given their relevance to organizational readiness and ability to change in
the face of an evolving initiative. One specific aspect of organizational capacity building
might include training regarding development of specific goals as they relate to the PFF the-
ory of change and outcomes, and a description of how the goals will be evaluated. In gen-
eral, we suspect that ongoing issues in this area exist primarily because they are both com-
mon and expected for an initiative in this early stage of development.
Overarching Theme Two: Importance of a Clear and Consistent Vision
Last year, when noting the importance of developing and maintaining a clear and
consistent vision of PFF, we raised the following issues:
• Further define the components and parameters of the PFF intervention
• Articulate the relationship of each intervention component to desired short
and long term outcomes
• Develop a plan for utilizing technical assistance
• Clarify how the PFF initiative involves change and innovation
• Establish the degree to which PFF provides guiding principles or highly-
defined models
• Understand the tension between breadth vs. intensity of PFF interventions
• Focus on engagement outcomes

44
• Examine terms of Commission Board leadership and Orientation to Initiative
Again, First 5 LA PFF staff members have focused attention in several of these areas
during the past year (e.g., hiring a Social Learning and Training Facilitator and holding nu-
merous related planning meetings with PFF agency personnel). However, for a variety of
reasons, many of these areas continue to need to be addressed. We stress the importance of
a clear and consistent vision particularly throughout the early stages of development of the
Initiative, including the coming year.
Overarching Theme Three: Relationships and Communication among Key
Partners in the Initiative
Regarding the relationships and communication patterns that existed among Initia-
tive partners during the first evaluation year, we suggested attention to the following:
• Define relationships amongst stakeholders
• Clarify and enhance patterns of communication amongst collaborators
• Capitalize on the benefits of conflict
During the past year, we note that First 5 LA has made great strides in this area, par-
ticularly given the increasing complexity brought on by the addition of new partners and the
departure of old ones. As expected with any changes in relationships, the various roles and
communication patterns between the numerous new PFF agencies, First 5 LA, DCFS, the
Evaluation Team, ICA, JMPT and other involved parties must be negotiated. First 5 LA
staff members have tried to address the roles and communication patterns by meeting more
regularly and communicating via email. Perhaps the single biggest remaining issue in this
area pertains to the [lack of] communication, either directly or indirectly, between the First
5 LA Commissioners and PFF stakeholders, particularly around the Commission's vision for
the Initiative and its evaluation.
Overarching Theme Four: Mixed Feelings about the Current Implementation
Plan
Previously, the Evaluation Team offered the following as means through which to
deal with mixed feelings that existed regarding the approved PFF Implementation Plan:

45
August 2006
• Work with DCFS and Collaboratives to identify specific means of innovation
• Promote PFF as a learning initiative
• Address sustainability
The Evaluation Team has not observed or noted lingering concerns about plans for
implementation of PFF. Quite the contrary, in fact; involved parties from all groups (First 5
LA, DCFS, and PFF agency staff) have consistently demonstrated dedication, optimism,
and excitement in relation to planning for PFF implementation. We note that First 5 LA
staff members have done an excellent job of promoting PFF as a learning initiative; this is
demonstrated in the way they discuss the Initiative and in their decision to contract with a
Social Learning and Training Facilitator. Unfortunately, though, the continued lack of clar-
ity regarding certain elements of the initiative may seriously hamper further efforts to pro-
mote PFF as a learning initiative.
Additionally, few efforts appear to have been directed at sustainability at the present
time, and we maintain that a focus on sustainability issues is critical, even at this stage of
the Initiative's development. It should be noted that an Evaluation Team activity identified
in our Year 3 scope of work will be to identify and examine organizational factors related to
program maintenance; this task will begin to address issues of sustainability in a limited
way.
Initiative Evaluation Strengths and Successes
As we reviewed year two of the evaluation, we noted the following key strengths of
the initiative evaluation and of the Evaluation Team itself:
First 5 LA has a strong interest in ensuring that PFF is a learning initiative. As
with any evaluation, learning is a key outcome, but it is not always as explicit as First 5 LA
has affirmed it to be. First 5 LA is endeavoring to “do business differently,” which is not an
easy road to navigate. First 5 LA should be commended for including a planning period and
for phasing-in implementation of PFF. These periods have provided agencies with greater
ability and room to learn, which ultimately strengthens programs and communities. First 5
LA is also providing consulting support to agencies so they can design processes that allow

46
data to inform their programs and case planning.
The Evaluation Team is flexible and adapts well to change. Because of substantial
revisions to directives regarding the initiative evaluation, the Evaluation Team has revised
the initiative evaluation plan numerous times. With each revision, we have assumed the task
with the same degree of commitment to designing and implementing a rigorous evaluation
and to collaborating with key stakeholders in the evaluation.
We have taken great effort to further strengthen our relationship with First 5 LA.
This year, there has been more frequent contact between the Evaluation Team (especially
the Team’s Project Director) and the First 5 LA PFF Evaluation Liaison (Research Analyst),
which we greatly appreciate. Consistent, regular contact has proven helpful in allowing the
Evaluation Team to move forward, even during the complex pre-implementation planning
period.
We have built a strong working relationship with PFF agencies. Believing that
direct and early involvement with the Collaborative agencies would result in the best possi-
ble initiative evaluation, the Agency Liaison has communicated and liaised with PFF
agency staff on all aspects of the evaluation. As a result, we have enjoyed frequent and
open communication, which has been extremely beneficial for developing productive work-
ing relationships, gaining input regarding the initiative evaluation, and increasing clarity
with regard to the purpose, tasks, and roles of the Initiative Evaluation Team.
Challenges to the Initiative Evaluation
Following are some of the barriers the Evaluation has observed and experienced.
Suggestions for how to address them are discussed in the Conclusion—Next Steps section.
Continued significant changes in the evaluation direction. For the Evaluation Team,
one of the hallmarks of a quality evaluation is ensuring it meets the needs of its users. To
this end, the Evaluation Team will continue to communicate and work with stakeholders to
ensure the evaluation’s relevance and utility. While the Evaluation Team is dedicated to
being responsive to the changing landscape and multiple interests of stakeholders, a con-
tinuing shift in the macro-level direction of the evaluation has negative repercussions on the
evaluation plan as well as on stakeholders. It should be noted that the PFF agencies have
been incredibly patient and understanding about the evolving nature of the evaluation and

47
August 2006
have earnestly participated in the initiative evaluation; however, we do not wish to unduly
test this patience nor burden them with tasks that soon become irrelevant.
The Team is dedicated to modifying the evaluation plan and to making mid-course
corrections in order to ensure that the plan continues to meet needs of the Initiative and of
First 5 LA. However, we find it problematic for the general direction of the evaluation to
change frequently and significantly. This is not to imply that we should collectively move
forward with an evaluation plan simply for the sake of moving forward; but rather, that we
arrive at a clear and consistent direction for the evaluation. This would not only benefit the
evaluation, but ensure that First 5 LA resources are used effectively and efficiently.
Inability to be proactive. This shifting sense of direction results in the Evaluation Team
and other stakeholders discarding the work that was previously carried out. As mentioned
above, the Evaluation Team has demonstrated agility in being responsive to changes in di-
rection. However, with each significant change, all parties are less clear on the scope of the
evaluation and, accordingly, their respective roles. While the Evaluation Team has been up
to the challenge, this situation leaves us in the position of being reactive and consequently,
potentially compromising the Team’s ability to create a quality evaluation. While we have
been able to respond, this is not an ideal condition for any of the stakeholders.
Unclear roles and responsibilities. Collaboration and collective action are fundamental
to PFF. The Evaluation Team agrees that all major stakeholders should be involved in the
development of the Initiative and its evaluation, and that responsibility for the success of
both should be held jointly. However, with some diffusion of responsibility and multiple
entities, each with demanding schedules, it becomes difficult to move the evaluation for-
ward at a steady pace. As outlined in our Year One report, it is imperative that roles and
responsibilities be more clearly defined.
Conclusion – Next Steps
In summary, while many issues identified in last year's report have been addressed
to some degree, some outstanding issues continue to challenge the success of the Initiative.
Those issues relate primarily to the lack of clarity about the Initiative itself and the roles/
responsibilities of key stakeholders—specifically of the First 5 LA Commissioners. This
lack of clarity has implications for the work carried out by the PFF staff and agencies, the

48
support of the Initiative by PFF agency staff, and the development and implementation of an
evaluation. In order to address this issue, as well as the other challenges identified above,
we offer for consideration the following:
• We suggest that it is of paramount importance to the Initiative to develop as
much clarity as possible with regard to the PFF program model and the roles
of all involved parties. In particular, the Initiative would benefit greatly from
attention to the following, especially now that implementation is underway:
revision of the PFF concept paper and theory of change; development of a
PFF vision statement by the Commission; and explicit, written description of
all key partners, their respective roles/responsibilities, and patterns of com-
munication.
• The Evaluation Team believes it would be beneficial to meet with First 5 LA
Commissioners on a more regular basis. While our previous meetings were
more ad hoc, the Evaluation Team would like to meet with the First 5 LA
Commissioner liaisons on a quarterly basis to ensure that they continue to be
informed and engaged in the process, and that they assume some degree of
responsibility for the success of the evaluation. Of course, the evaluation
would continue to be informed by the macro-level direction provided by the
First 5 LA Commission. Similarly, we think the Collaborative agencies
should also have an opportunity to meet or communicate more regularly with
liaisons of the Commission.
• As the evaluation moves forward, the Evaluation Team would greatly appre-
ciate receiving clearer and more consistent feedback from the First 5 LA
Commission, particularly with regard to the PFF initiative model and First 5
LA vision for the Initiative. Alternatively, we would appreciate more auton-
omy in determining the evaluation approach (consistent with the parameters
set forth by the Commission).
• Finally, we ask that the First 5 LA Commissioners and PFF staff review our
evaluation report from Year One, as numerous issues identified therein (as

49
August 2006
outlined above) continued to plague the Initiative and Evaluation this past
year.

50
REFERENCES
Children’s Bureau of Southern California (1997). Family Assessment Form: A Practice-
Based Approach to Assessing Family Functioning. Washington, DC: CWLA Press.
Kirk, R. S. & Reed-Ashcraft, K. (n.d.). User’s Guide for the North Carolina Family Assess-
ment Scale (NCFAS) Version 2.0. Chapel Hill, NC: University of North Carolina at
Chapel Hill School of Social Work.

51
August 2006
APPENDICES
Appendix A: Year 2 Scope of Work
Appendix B: Evaluation-Related Documents Shared at Grantee Orientation
Appendix C: First 5 LA Partnerships for Families Initiative Evaluation Plan
Appendix D: Partnership for Families Initiative Evaluation: Aims and Questions
Appendix E: PFF Quarterly Reporting Form
Appendix F: PFF Qualitative Interview Schedule
Appendix G: Notice of IRB Approval
Appendix H: Building Evaluation Capacity Assessment Tool
Appendix I: PFF Benchmarks & Performance Measures: Structure of Indicators
Appendix J: PFF Implementation Research and Evaluation Questions
Appendix K: PFF Indicators of Readiness: Planning Phase
Appendix L: Data Collection Tools Table
Appendix M: Agency Intake Data Table
Appendix N: Expected PFF Family Level Outcomes
Appendix O: Family Domains
Appendix P: Measurement of Outcomes/ Research Questions
Appendix Q: Synopsis of FAF Psychometric Properties
Appendix R: Synopsis of Family Domains (FAF & NCFAS)
Appendix S: Synopsis of NCFAS 1.4 Psychometric Properties
Appendix T: PFF Evaluation Manual

PFFEvaluationReport-Year2

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (11)

Similar to PFFEvaluationReport-Year2

Similar to PFFEvaluationReport-Year2 (20)

PFFEvaluationReport-Year2