

Legislative Program Review
& Investigations Committee
Department of Children and Families
Monitoring and Evaluation
December 2007
DEPARTMENT OF CHILDREN AND FAMILIES MONITORING AND EVALUATION
Executive Summary i
Introduction 1
Purpose 1
Study Approach 2
Methodology 6
Report Organization 8
I. Background: Overview of DCF. 9
Mission, Guiding Principles, and Goals 9
Major Duties and Responsibilities 13
State Mandate Areas and Programs 15
Organization and Budget 22
II. DCF Internal Monitoring and Evaluation Activities 29
Bureau of Continuous Quality Improvement 29
Other Internal Monitoring and Evaluation Efforts 41
Contracted Monitoring and Evaluation 43
DCF Information Systems 48
III. DCF External Monitoring and Evaluation Activities 51
U.S. Department of Health and Human Services 51
Federally-Required Child Abuse Prevention and Treatment Plan 65
DCF Federal Grant Funding 68
Judicial Oversight 70
Legislative Oversight 83
Accrediting Body Oversight 88
Other Regulatory Monitoring and Evaluation Reporting 94
IV. Outside Investigations and Reviews 97
Office of the Child Advocate 97
Child Fatality Review Panel 102
State Attorney General 105
V. Advisory Group Monitoring and Evaluation Activities 107
Overview 107
Agencywide and Area Advisory Groups 109
DCF Facility Advisory Groups 109
Federally Mandated Advisory Groups 111
Issue-Specific Advisory Groups 111
Inactive Advisory Groups 113
Advisory Groups Requiring DCF Participation 113
VI. Results from Monitoring and Evaluation Efforts 117
Overview 117
Detailed Monitoring and Evaluation Results Within Child Protective Services 119
Detailed Monitoring and Evaluation Results Within Behavioral Health Services 132
Detailed Monitoring and Evaluation Results Within Juvenile Services 138
Detailed Monitoring and Evaluation Results Within Prevention Services 139
Detailed Agencywide Results 141
Summary of Results from Key Monitoring and Evaluation Efforts 142
VII. Findings and Recommendations 145
Agency Goals 146
Overall System Assessment 150
Internal Efforts 159
External Efforts 171
Outside Investigations and Reviews 178
Advisory Group Efforts 181
Executive Summary
Department of Children and Families Monitoring and Evaluation
Formed in 1974 as a consolidated children's agency, the Connecticut Department of Children and Families (DCF) has broad authority and primary responsibility for state mandates concerning child protection, children's behavioral health, juvenile delinquency, and prevention services related to children and families. The department has been studied, audited, reviewed, and subject to legal action almost continuously since it was created due to ongoing concerns about its ability to carry out its challenging mission.
Numerous internal quality improvement efforts, as well as oversight by multiple outside entities including federal and other state agencies, various advisory groups, the courts, and the legislature, have focused on how to achieve better outcomes for the children and families DCF serves. The Legislative Program Review and Investigations Committee (PRI) alone had completed seven different reports on the department prior to undertaking a review of the overall DCF accountability system in April 2007. Unlike previous PRI reviews, this study evaluated a critical function -- monitoring and evaluation of agency results -- rather than a particular mandate.
An effective results-based monitoring and evaluating system is important for three main reasons: 1) it provides an agency with productive feedback on actual outcomes and progress toward goals; 2) it allows agency staff, policymakers, and stakeholders to know where the agency is successful, where it is not, and how to make improvements; and 3) ultimately, it helps the agency provide services that meet clients' needs and make cost-effective use of taxpayer resources. The purpose of the 2007 program review committee study was to determine areas of strength and weakness, as well as gaps and redundancies, in the existing DCF accountability system, and to identify needed improvements.
Study approach and methods. The committee study employed two primary research methods: interviews with key stakeholders; and analysis of monitoring and evaluation reports and other documents produced through DCF quality assurance, performance evaluation, and oversight efforts. There were four main sources of efforts:
1) internal monitoring and evaluation efforts such as: provider licensing, performance-based contracting, ombudsman activities, and various department self-reviews and contracted evaluation studies;
2) external oversight efforts by federal agencies, federal and state courts, legislative committees, and independent entities like national accreditation organizations;
3) outside investigations and reviews, such as those carried out by the state Office of the Child Advocate (OCA) and the state attorney general; and
4) monitoring and evaluation activities by advisory groups established under federal or state law.
To put into context all the information gathered about the process, sources, and results of DCF monitoring and evaluation, the core components of the current system were also compared with a national model for child welfare agency quality improvement.
The program review committee's final report contains an assessment of the overall DCF monitoring and evaluation system, details the system's positive features as well as deficiencies, and recommends nearly 40 administrative and legislative changes to improve its effectiveness. The report also summarizes data on agency accomplishments that were compiled by program review staff from more than 100 different monitoring and evaluation documents analyzed during the study.
Main findings. The program review committee found little attention has been given to examining DCF as a whole or assessing how well the agency is achieving its broad goals of safety, permanency, and well-being for all children and families. Further, while the department is responsible for carrying out four major mandates, monitoring and evaluation is focused primarily on the child protective services mandate, due largely to the ongoing impact of the federal Juan F. lawsuit consent decree and requirements of federal agencies.
The PRI study showed there is greater emphasis on tracking how services for children and families are delivered rather than on assessing their end results. While high quality service delivery is important, the crucial indicator of effectiveness is whether programs are making a difference and achieving stated goals. In general, more attention to outcome information is needed throughout the DCF accountability system.
The committee's review also identified pockets of strength within the system, such as the Juan F. exit plan process and related DCF area office quality improvement processes, the department's licensing procedures, the agency's recently revised special review process, and the activities of on-site facility monitors.
Some major weaknesses were revealed as well. In particular, the agency's contracting process provides little accountability, consequences for poor performance are rare, and working relationships with private providers need improvement. The committee also found ineffective use of some important sources of feedback on services and programs, such as child fatality reviews, OCA investigations, and even the department's own program review reports and contracted evaluations.
In part, these deficiencies are due to both fragmentation of quality improvement efforts within the agency and the fact that results data are not regularly integrated and analyzed. Both problems are related to the department's information systems, which are themselves fragmented and in some cases inadequate. Another challenge is a lack of department staff with the analytic skills and research experience needed to use results data and information. Further, there is no centralized place – like an agencywide strategic plan – where all DCF goals and information about service delivery and outcomes are brought together.
Duplication of external monitoring efforts also was revealed by the program review committee's examination of statutorily required DCF plans and reports. The committee determined several mandates could be eliminated without a loss of accountability, as certain documents have become obsolete or been replaced by newer sources of similar information. In addition, reducing the number and clarifying the purpose of reporting mandates could improve the quality of information on department results available to the legislature and the public.
Committee recommendations. Ultimately, the point of all monitoring and evaluation efforts, whether internal, external, investigatory, or advisory, is to ensure programs and services are having desired results. Taken together, the PRI committee recommendations listed below are aimed at making the current DCF accountability system more effective by:
● making agency goals explicit;
● integrating quality improvement activities and incorporating best practices throughout the agency;
● improving the quality and quantity of available data; and
● promoting the use of results information to better meet the needs of children and families.
Recommendations
1. The current statutory provision for a Department of Children and Families biennial five-year master plan shall be repealed and replaced with a mandate for ongoing strategic planning. Specifically:
a) Beginning July 1, 2008, the department shall start the process of developing a vision, mission, and strategic goals with the advice and assistance of representatives of the children and families served by the agency, public and private providers, advocates, and other stakeholders.
b) The department should dedicate staff, under the direction of the commissioner or a deputy commissioner, to: 1) prepare a strategic planning document that includes action steps and time frame for implementation to fulfill the vision, mission, and goals developed with stakeholders; 2) track and report on progress in achieving the plan's goals at least annually; and regularly review, revise, and update the department's strategic plan as needed.
c) The first plan shall be completed and submitted to the legislature and the governor by July 1, 2009.
d) The department's strategic plan shall be submitted to the agency's State Advisory Council for Children and Families for review and comment prior to submission to the legislature and governor. Progress in carrying out the plan shall be reported to the council by the DCF commissioner at least quarterly and to the legislature and governor annually.
2. The department should reinforce and expand the role of the Service Evaluation and Enhancement Committee (SEEC) in integrating monitoring and evaluation efforts across the agency and initiating proactive intervention on agencywide issues.
3. DCF performance-based contracts should specify the data required from providers. Performance standards or expected outcomes should be stated in the contract. DCF should monitor data submissions for accuracy.
4. DCF should review currently required data elements from providers and determine whether they are necessary or analyzed in any way. Data elements that are unnecessary should be eliminated and additional data elements that pertain to outcomes should be added to performance-based contract requirements.
5. DCF shall compile necessary required data elements to compare actual and expected outcomes based on the performance-based contract. Failure to meet contract expectations should result in discussion and joint plans for progress in meeting expectations.
Until automated systems are deemed reliable, DCF should monitor contract expectations manually. Summary reports should be shared with providers so that they may monitor their performance against the aggregated data. Reports should be distributed to providers and DCF staff made more aware of the existence of these reports.
6. A central repository should be created by DCF of contracted research and evaluation reports and internally produced research and evaluation reports. This repository should be accessible and searchable by all DCF staff and should include the Office of Policy and Management (OPM) feedback form as applicable.
7. The DCF licensing unit should expand internal self-monitoring by inspecting High Meadows and Connecticut Children's Place, the two DCF facilities not currently under external licensure or accreditation. The licensing unit should follow the child care facilities regulations standards used to inspect external residential treatment facilities similar to High Meadows and Connecticut Children's Place.
8. The department should establish an internal written policy for responding to recommendations from the internal special reviews of child fatalities and other critical incidents. The policy should require a corrective action plan be developed, implementation of accepted recommendations be monitored, and a status report be prepared for the commissioner every 90 days. A forum to discuss results and lessons learned should be scheduled with managers and key staff from all relevant areas of the department within 45 days of release of the report.
9. DCF should be permitted to establish a long-term research partnership with the Child Health and Development Institute and its affiliate, the Connecticut Center for Effective Practice, through a multi-year, sole source contract to carry out a broadly defined research and evaluation agenda related to the agency's mission.
10. DCF should reexamine the role of its program lead position and consider the allocation of time necessary for this responsibility. DCF should also develop a team approach for working with contracted providers that will ensure contract obligations are being met, provide assistance when necessary so that programs do not reach a crisis point, and support and assist programs with quality improvement.
11. Considering contractor monitoring best practices, DCF should examine the roles of staff within the Grants Development and Contracts Division to determine whether some of the 19 positions could be reallocated from the financial/accounting function of contract management to program development and implementation support activities.
12. DCF should maintain a centralized and complete electronic grants and contracts library on the department's intranet. Grants and contracts missing should be scanned into the library. Previous years' contracts should be maintained for future reference.
13. The department should require the Grants Development and Contracts Division to receive and review feedback from area office and program lead staff on the performance of a provider before deciding to renew a contract. If concerns are raised about a provider, then discussions with the appropriate parties should occur and a performance improvement plan developed.
14. A workgroup should be convened by the department and the Offices of Policy and Management and the Attorney General to clarify the guidelines regarding contract bidding and related programming suggestions.
15. DCF should develop a protocol for providers to submit suggested programs or program enhancements. A form for submitting the idea should be developed and timelines for response from DCF publicized.
16. DCF should work with the Department of Administrative Services (DAS) to develop: 1) an appropriate job classification for staff positions within the agency responsible primarily for research and analysis; and 2) recruitment strategies for obtaining personnel with the necessary qualifications to fill them.
Furthermore, the department should increase its internal analytic capacity. The size and scope of the Risk Management Unit staff should be expanded to include the following duties in addition to compiling information to support the SEEC function: interpreting data produced by the state's behavioral health Administrative Service Organization (ASO); compiling contracted evaluation results; maintaining the research
repository recommended earlier; supporting agency strategic planning activities; and sharing outcome, best practices, and results information agencywide.
17. For programs exceeding $20 million in funding, DCF should require an external evaluation be conducted to assess the outcomes of the program.
18. DCF should develop and issue guidelines for staff and consultants regarding the format for final evaluation reports.
19. The Office of the Child Advocate should undertake an investigation to assess adequacy and integrity of the internal process for reviewing and responding to allegations of staff child abuse and neglect. It should also examine compliance with C.G.S. §17a-103c.
20. Wilderness School staff should work with the Department of Public Health to develop a more appropriate licensure as a wilderness school rather than as a camp.
21. Replace the following statutory reports:
a) DCF biennial five-year master plan (C.G.S. §17a-3);
b) DCF annual report on the Connecticut Juvenile Training School (CJTS) (C.G.S. §17a-6b and C.G.S. §17a-6c); and
Repeal the statutory reports listed below:
c) Children's Behavioral Health Advisory Committee (CBHAC) annual local systems of care status report (C.G.S. §17a-4a(e));
d) CBHAC biennial recommendations on behavioral health services (C.G.S. §17a-4a(f));
e) Quarterly hospital reports to DCF on psychiatric care (C.G.S. §17a-21);
f) KidCare Community Collaborative annual self-evaluations (C.G.S. §17a-22b);
g) DCF/DSS five-year independent longitudinal evaluation of KidCare (C.G.S. §17a-22c(c));
h) DCF monthly report to legislature on children in subacute care in psychiatric or general hospitals who cannot be discharged (C.G.S. §17a-91a);
i) Cost-benefit evaluation of juvenile offender programs (C.G.S. §46b-121m);
j) Licensed child care facilities annual reports (C.G.S. §17a-145);
k) DCF annual evaluation reports on Unified District #2 to the education commissioner (C.G.S. §17a-37(d));
l) DCF to conduct studies to evaluate effectiveness (C.G.S. §17a-3(a)(6)); and
m) Adoption Advisory Committee report (C.G.S. §17a-116b(g)(3)).
22. All DCF facilities shall be required to produce an annual report for their respective advisory groups. The report shall contain at a minimum the following:
1. aggregate profiles of the residents;
2. description and update on major initiatives;
3. key outcome indicators;
4. costs associated with operating the facility; and
5. description of education programs and outcomes.
23. Research and evaluation reports produced through federal grant requirements should be included in the report repository recommended earlier concerning contracted evaluation reports and internally produced research products.
24. DCF should adopt a written policy requiring that formal results from research and evaluation reports produced from federal grants be reviewed and considered when agency managers make decisions concerning future funding and/or continuation of programs developed with federal grants.
25. DCF should convene a workgroup including program leads, a representative from the Juan F. court monitor's office, and DSS to develop a treatment plan and review process that satisfies both the internal DCF and federal (e.g., Private Non-Medical Institution Initiative or PNMI) requirements.
26. A pilot program should be created to assess the feasibility of conducting one treatment plan conference to be held at court that combines: the Specific Steps identified during the initial case status conference at court and the corresponding DCF treatment plan conference currently held in the area office.
27. The statutes concerning the Office of the Child Advocate and the Child Fatality Review Panel (CFRP) shall be amended to require the Department of Children and Families, and other state entities subject to OCA and CFRP investigative activities, to provide a written response to formal recommendations made by the child advocate and the panel for improving state services provided to children.
The agency response should: include proposed corrective actions to address identified problems and a time frame for implementation of improvements; and be provided to OCA or CFRP within 45 days of receipt of the recommendations. Copies of the agency response also should be submitted to the legislative committees of cognizance and the appropriations committee.
28. The resources necessary to improve the OCA data management system should be provided during the next fiscal year, either by the Department of Information Technology (DOIT) making this project a priority or through funding for a consultant to design and implement an upgraded system for the child advocate's office.
29. DCF should establish a policy for area office advising bodies to adopt a model whereby advising body members attend DCF area office quality improvement meetings, and DCF area office representatives attend advising body meetings, furthering promotion of a partnership.
30. DCF facility advisory boards shall be required by statute and it shall be mandated that all boards respond to their facility's annual report and that they add recommendations deemed necessary.
31. The role of the State Advisory Council for Children and Families (SAC) should be strengthened to include monitoring the agency's progress in achieving its goals as well as offering assistance and an outside perspective. The board's statute shall be written to clarify this role and DCF's participation with the board concerning strategic planning as recommended above. The council's meetings should be held at locations that facilitate participation by members of the public, such as the Legislative Office Building, and its agendas and minutes should be posted on the DCF website. The department should provide the council with funding for administrative support services and to ensure members representing families from across the state can serve on the council.
32. DCF should establish an electronic mechanism, for example a blog, where members of area office advising bodies can share information with each other, the SAC, and vice versa. Additionally, minutes and agendas from all meetings should be posted on the DCF website.
33. DCF should fund all three required Citizen Review Panels equally.
34. The Children's Behavioral Health Advisory Committee (C.G.S. §17a - 4a) should be incorporated into the State Advisory Council as opposed to remaining a separate entity.
35. Repeal the statutory requirement for the Adoption Advisory Council (C.G.S. §17a-116b).
36. Repeal the statutory requirement for the Connecticut Juvenile Training School Public Safety Committee (C.G.S. §17a-27f).
37. DCF shall hire an external consultant to:
a) perform a gap analysis1 and workflow analysis with the focus on integrating the functions of the department with technology modeled to support the service model;
b) develop a project plan; and
c) develop a request for proposals to procure the team needed to integrate the data systems and replace the LINK System.
Introduction
Department of Children and Families Monitoring and Evaluation
The Connecticut Department of Children and Families (DCF) has broad authority and primary responsibility for the state's main child welfare mandates -- protecting children from abuse and neglect, providing children's behavioral health and juvenile justice services, and carrying out prevention efforts for children and families at risk of abuse, neglect, mental illness, substance abuse, and juvenile delinquency. To carry out its mission, the department has an annual budget exceeding $800 million and a full-time staff of nearly 3,500 employees.
Since its formation as a consolidated children's agency in 1974, the department has been studied, audited, reviewed, and subject to legal action almost continuously. Multiple entities outside the agency, such as the state Office of the Child Advocate (OCA), legislative committees, the Judicial Branch, national accrediting bodies, federal agencies that provide funding for children's services, and the federal courts, track aspects of DCF performance and provide oversight of various program outcomes. Concerns continue to be raised about the efficiency, effectiveness, and advocacy capabilities of DCF through these many oversight efforts.
At the same time, there has been a number of initiatives to improve department operations and services in recent years. Many changes have been made in response to the ongoing federal Juan F. consent decree concerning Connecticut's child welfare system.
Reports from the federal court monitor show caseloads for the agency's social workers comply with national standards, community-based treatment and support services have been expanded, and there is more collaboration with other agencies involved with children and families such as the Departments of Social Services, Mental Health and Addiction Services, Developmental Services (DSS, DMHAS, DDS) and the Court Support Services Division (CSSD) of the Judicial Branch. The Department of Children and Families also has instituted various internal monitoring and evaluation efforts, such as those carried out by its continuous quality improvement bureau, as ways to strengthen management and policy decision making.
An effective process for tracking and assessing results is the cornerstone of accountability and improved performance of state agencies. In April 2007, the Legislative Program Review and Investigations Committee (PRI) voted to undertake a comprehensive assessment of efforts to monitor and evaluate DCF.
Purpose
The purpose of the program review committee study of DCF monitoring and evaluation was to determine areas of strength and weakness, as well as any gaps and redundancies, within the existing agency accountability system. The main goal was to identify improvements to internal and external oversight efforts that would lead to better agency performance and, ultimately, to better outcomes for children and families.
Specifically, the study centered on: 1) describing how goals set for and by the agency are measured and tracked; 2) evaluating the department's progress in attaining its goals; 3) examining the extent to which the results of monitoring and evaluation efforts are used by DCF to improve the services it provides to children and families; and 4) identifying ways to increase the overall effectiveness of the DCF accountability system. Unlike all previous PRI studies of the Department of Children and Families, the focus was on a critical function -- monitoring and evaluation -- rather than a particular agency mandate.
Previous committee DCF studies. It is not uncommon for the program review committee to conduct multiple studies of a state agency over time, especially a complex one like the Department of Children and Families. Six prior PRI studies evaluated how the agency carried out its various mandates and identified ways to make specific programs more efficient and effective.2 The committee's last report on DCF, issued in December 1999, examined implementation of its overall consolidated children's services mission. PRI proposed a significant restructuring of department duties that was intended to achieve higher quality services, better coordination and integration, and stronger leadership and oversight.
While there has not been a major reorganization of child protection, behavioral health, and juvenile justice programs in Connecticut since the 1999 PRI study, a number of changes have occurred within DCF and in the general environment of children's services. Since 1999, DCF's capacity for self-evaluation and corrective action has increased and there are more external mechanisms for providing productive feedback and accountability for results. Major developments in effective oversight within and outside DCF are highlighted in Chapter I, and described more fully in Appendix C.
In reviewing these developments, it appeared to the program review committee that better monitoring and evaluation efforts might attain the improvements in agency performance sought previously through restructuring proposals. This study, therefore, centered on the effectiveness of the system for tracking, assessing, and using information on DCF outcomes to reach agency goals for children and families.
Study Approach
The committee study scope was limited to monitoring and evaluation of DCF that has occurred both within and outside of the agency over the past three to five years. For the purposes of the study, the following definitions were used:
● The term “monitoring” refers to the effort to systematically track program delivery.3 It can answer such questions as: has a program been delivered as planned and to the group for which it was intended? Did particular activities occur within a given time frame? Did the program serve the number of children it was expected to serve?
● The term “evaluation” means efforts to determine the extent to which programs are effective, which can answer such questions as: what impact has the program had on the people it served? Did the expected program outcomes occur? Is anyone better off?4 What is the program's cost in relation to its benefits?
If the DCF monitoring and evaluation system is working well, the quality of agency programs and services should continually improve, benefiting the clients, and justifying the public's investment. When information on actual results is produced, and then used by the agency to guide decisions on policies, operations, and resources, more efficient and effective services for children and families should result. The key research question is: do existing efforts to track agency accomplishments and assess client outcomes result in better services for children and families?
The committee's overall approach to the study, illustrated in Figure 1, had five main components. These components, described in detail in Appendix B, were:
1. Capture and categorize, by source, efforts to monitor and evaluate DCF accomplishments and identify the goals the agency is trying to achieve;
2. Assess how well the various efforts to measure agency goals and progress made are working;
3. Summarize the outcome information produced and reported (e.g., results achieved, deficiencies noted and recommended improvements);
4. Describe the impact of the feedback information on DCF decisions about policies, resources, and services; and
5. Recommend ways to make the current monitoring and evaluation system more effective, thereby improving the quality of DCF programs and services for children and families.
As Figure 1 shows, four main sources of DCF monitoring and evaluation were identified and analyzed: internal efforts; external efforts; outside investigations and reviews; and advisory groups established under federal or state law. Each source is summarized in Table 1.



![]()
Table 1. Examples of DCF Monitoring and Evaluation Efforts by Source |
|
Internal Monitoring & Evaluation: ● DCF Bureau of Continuous Quality Improvement (BCQI) ● DCF performance-based contracting activities ● Internal child fatality and critical incident reviews ● Office of the Ombudsman activities |
External Monitoring & Evaluation: ● Federal Oversight ○ U.S. Department of Health and Human Services (DHHS) n Children's Bureau of Administration for Children and Families (ACF) n Substance Abuse and Mental Health Services Administration (SAMHSA) ○ U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention (OJJDP) ● Judicial Branch/Federal Court Monitoring (e.g., Juan F. Court Monitor) ● Independent Accreditation Groups ○ The Joint Commission (hospitals) ○ Council on Accreditation (child welfare agencies) ○ Commission on Accreditation for Corrections (correctional facilities) ● Legislative Oversight ○ Committees of Cognizance ○ Statutory reporting requirements ○ Results-Based Accountability |
Outside Investigations and Reviews: ● Office of the Child Advocate (OCA) ● Child Fatality Review Panel (CFRP) ● Office of the Attorney General (OAG) |
|
Advisory Groups (established under state or federal law): ● State & Area Advisory Councils to DCF ● DCF Facility Advisory Groups ● Connecticut Citizen Review Panel(s) (required by federal law) ● Connecticut Behavioral Health Partnership Oversight Council (BHPOC) |
Source: PRI staff analysis. |
Methodology
The committee, through its staff, employed two main research methods to study the DCF monitoring and evaluation system. The methods were: key stakeholder interviews; and analysis of monitoring and evaluation reports and related documents produced through all four sources of efforts -- internal, external, investigative, and advisory.
Stakeholder interviews. Committee staff conducted approximately 100 interviews with: division and unit personnel within the DCF bureaus; court monitor staff; Office of the Child Advocate staff; advisory group chairs; federal agency officials; external evaluators; and representatives of providers and advocacy groups. Given the time and resource constraints of the study, not every area office or DCF facility could be visited; however, efforts were made to visit or interview staff from a full range of locations and types of operations.
Document analysis. Committee staff reviewed 126 reports and materials pertaining to monitoring and evaluation of DCF. The study focused on reports and other materials documenting monitoring and evaluation efforts that occurred within the past three to five years (through September 2007). In order to assess the efforts, PRI staff evaluated each document using an internally developed standardized rating system to answer the questions shown in Table 2. Ratings required agreement between two PRI staff who had independently reviewed the documents and then met to discuss their ratings.
Study limitations. Several limitations were encountered during the committee study. First, PRI staff was unable, within the study time frame, to completely assess every effort to monitor and evaluate the Department of Children and Families. For example, while work force development and employee performance evaluation procedures have an important role in supporting quality improvement, the DCF human resources division, the department's Training Academy, and the agency's use of the Performance Assessment and Recognition System (PARS) were not examined.
Additionally, a key department program, foster care, was undergoing a major restructuring at the time of the committee review. Nearly all aspects of foster care monitoring and evaluation were being revamped, so little about efforts in that area could be assessed as it was too soon to know the impact of the new procedures.
Further, not every activity or product within a given type of monitoring and evaluation effort (e.g., licensing visit report, quality improvement plan, advisory group meeting, evaluation report, etc.) could be examined. Due in part to the fragmentation of the DCF monitoring and evaluation system, it is likely that other documents that could have been included in the analysis were not identified. However, the committee believes that the statistical reports, studies, and other documented information about major programs within the department reviewed by PRI staff are a representative sample of monitoring and evaluation currently underway.
There were also situations where multiple monitoring and evaluation efforts were overlapping or occurring simultaneously. This made it difficult to discern which effort led to changes in programs or facilities, particularly when recommendations were similar. Another challenge was attributing outcomes to particular programs, especially when children and families were receiving a variety of services and supports at the same time.
Table 2. Areas Assessed in Each Monitoring and Evaluation Document | |
Question Areas* |
Description |
Is the focus of the monitoring and evaluation on DCF agencywide goals, mandate areas, or specific programs? |
PRI staff identified the purpose of each monitoring and evaluation effort. |
What is tracked: process (service delivery), outcome (end result) or both? |
PRI staff classified monitoring and evaluation as: 1. “process only” for those efforts that pertained to the way in which service is delivered; 2. “outcome only” for those efforts that addressed expected end results or outcomes of the service; or 3. “both process and outcome” for efforts that addressed both aspects of monitoring and evaluation. |
Are the goals and issues studied S.M.A.R.T.? |
The stated goal(s) or issue(s) subject to the specific monitoring and evaluation effort was rated on whether it was: specific, measurable, attainable, relevant, and trackable (“S.M.A.R.T.”). On a scale from 1 to 5, where 5=very positive. |
How well do the measures used match up with the goals? |
The measures selected to assess progress on reaching the goal were rated on their logical relationship to the goal (versus chosen for convenience/ready availability). On a scale from 1 to 5, where 5=very well. |
How good a job was done in collecting information to answer the question or ascertain progress in attaining the goal? |
Examining the mechanical efforts to obtain information to address the goal or issue, PRI staff rated monitoring and evaluation efforts on the extent to which measures were consistent, with good quality data and little or no missing information. On a scale from 1 to 5, where 5=the goal is measured consistently, with good quality data and little or no missing information, or the issue or question is clearly addressed, with good quality data or sources of information. |
Were the monitoring and evaluation findings used to make changes? |
Step 1: PRI staff assessed whether the findings addressed organizational barriers, communication barriers and/or resource barriers. Step 2: Through interviews, PRI staff determined if the monitoring and evaluation findings were used by DCF to make changes to policy, training, or services, or if legislative changes were made. On a scale from 1 to 7, where 1=no findings were used, to 7= findings were fully used to identify organizational and resource barriers and to make changes to policies, training, and services, and used to seek legislative changes. |
Were the recommendations stated clearly, did they flow logically from the findings, and did they contain actions? |
Based on the reviewed documents recommendations were rated on a scale from 1 to 5, where 5=very positive. |
Were the recommendations adopted? |
Based on interviews and available documents, PRI staff estimated the percent of recommendations that were adopted. On a scale from 1 to 5, where 1=not at all, to 5=completely. |
* In addition to these questions, PRI staff looked at how the data were collected and then analyzed. Source: PRI staff. | |
Lastly, it is important to note the rating system employed for the study does not capture the magnitude of a particular monitoring and evaluation activity. The committee staff did not try to assess the significance of each effort or any resulting recommendation. Staff calculated only the percent of recommendations implemented, and did not attempt to rate their impact or importance.
Report Organization
This report consists of seven chapters and a number of appendices. Chapter I provides background information on the Department of Children and Families including descriptions of its mission, goals, major mandates and activities, current organization, and operating budget. The agency's internal efforts to monitor and evaluate its programs and services through various quality assurance and improvement functions such as licensing, case reviews, performance-based contracting, program reviews, risk management analysis, and contracted evaluations are described in Chapter II.
External monitoring and evaluation of DCF activities that are carried out by federal agencies, the courts, the legislature, other state agencies, and national accrediting organizations are described in Chapter III. Outside investigations and reviews, which are conducted by the Office of the Child Advocate, the Child Fatality Review Panel, and the state attorney general's office are discussed in Chapter IV. The many advisory groups responsible under state or federal law for assisting the department in carrying out its mission and assessing its performance are highlighted in Chapter V.
Information about the results of DCF programs and services that was compiled from the monitoring and evaluation reports reviewed for this study is provided in Chapter VI. The program review committee's findings concerning the strengths and deficiencies of current internal, external, investigatory, and advisory monitoring and evaluation efforts are presented in Chapter VII, along with recommendations for improving the DCF accountability system through administrative and legislative changes.
Agency response. It is the policy of the Legislative Program Review and Investigations Committee to provide agencies subject to a study with an opportunity to review and comment formally on committee recommendations prior to publication of the final report. A written response to this study was submitted by the Department of Children and Families and is presented in Appendix A.
Chapter I
Background: Overview of DCF
Connecticut established its consolidated children's agency, the Department of Children and Families, in the 1970s. The legislature combined the state's primary child welfare programs in one organization with the intent of achieving a comprehensive, coordinated statewide system of services for children and families who are at risk because of abuse or neglect, delinquency, mental illness, emotional disturbance, or substance abuse problems.
Since its formation, the department has undergone numerous internal reorganizations, shifts in policy and practice, and almost continuous critical review as it seeks to carry out its complex mission. Background information on DCF is presented in this chapter and includes: an overview of the agency's mission and operating principles; descriptions of its major mandates and associated programs and activities; and a summary of the department's current organization and budget. A brief history of the agency and children's services in Connecticut is provided in Appendix D.
Mission, Guiding Principles and Goals
The purpose and goals of the Department of Children and Families are implied in many of its legislative mandates, although there is no single statutory policy statement about the agency's role. Over time, the department has adopted various mission statements that reflect its broad scope as well as the general evolution of child welfare policy and practice. The current mission of DCF, as stated on the agency's website, is: to protect children, improve child and family well-being and support and preserve families.
DCF management officially adopted six guiding principles for all agency activities based on its mission statement. They include the following overarching principle encompassing the core agency mission and five specific principles intended to guide department practice:
● Overarching Principle: Safety, Permanency, and Well-Being
● Principle One: Families as Allies
● Principle Two: Cultural Competence
● Principle Three: Partnerships
● Principle Four: Organizational Commitment
● Principle Five: Work Force Development
Descriptions of each principle were developed by the department and are provided to all employees and contracted providers, and made available to the general public. A copy of the agency's mission and guiding principles document is presented in Appendix E.
Many goals have been established internally and externally for the Department of Children and Families. At this time, the department does not have a single document containing all goals for the overall agency, its mandate areas, or its specific programs. The information about DCF goals presented below was compiled from a variety of sources, including state
statutes, agency plans and budget documents, mission and goal statements included on the agency's webpage, and interviews with agency staff. A summary is in Appendix F.
Agencywide goals. As its overarching principle indicates, the Department of Children and Families has three main goals for children: 1) safety; 2) permanency; and 3) well-being. These goals, like the agency mission statement and its guiding principles, are not specified in state statute. However, they are implied in many of the laws directing DCF operations.
Safety and permanency as goals for children in the department's care and custody do have a statutory basis. Since 1998, DCF is required by law to prepare a written plan for each child and youth under agency supervision that includes, but is not limited to: “… a goal for permanent placement … which may include reunification with the parent, long-term foster care, independent living, transfer of guardianship or adoption. The child's or youth's health and safety shall be the paramount concern in formulating the plan.” Under another state statute, it is the policy of Connecticut to protect children from abuse, strengthen the family and make homes safe for children, and provide a temporary or permanent nurturing and safe environment for children when necessary.
Other agencywide goals are the department's Positive Outcomes for Children. These 22 positive outcomes mirror the exit plan outcome measure established under the federal Juan F. child welfare consent decree, which are described in detail in Chapter III and summarized in Appendix F. All of the positive outcomes/exit plan outcome measures are focused on safety, permanency, and the well-being of children and families. The agency mission, guiding principles, and positive outcomes are posted throughout the agency and the department has developed and revised an action plan for meeting the performance goals set under the Juan F. consent decree exit plan.
Child protection mandate goals. The department's goals related to its children's protective services mandate are based on state statutory policy directives to protect children from abuse and neglect, plan for permanent placement, and provide comprehensive services to meet the needs of at risk children and their families. They parallel the child welfare goals set for states under federal legislation. Like its agencywide goals, DCF's main child protection goals are: safety; permanency; and well-being.
Specific child protection goals include the 22 outcome measures for the Juan F. exit plan and the closely-related federal outcomes standards for state child welfare agencies. These standards are summarized, with all other major agency goals, in Appendix F.
As noted above, the department has an action plan, with specific strategies and time frames, for achieving compliance with the Juan F. consent decree goals. Progress in implementing the plan is regularly assessed by department management as well as the court monitor. The Juan F. action plan also is incorporated in DCF's Child and Family Services Plan, developed in accordance with federal requirements to outline the agency's child welfare goals and strategies for achieving them. Another document containing department child protection goals is its Performance Improvement Plan that must be prepared and implemented in response to federal Child and Family Services Review (CFSR) findings. (CFSR and other federal requirements are discussed in more detail in Chapter III.)
Behavioral health mandate goals. The goals of the DCF's behavioral health mandate, as defined in the agency's FY2008-2009 biennium governor's budget document, are:
● to address children's behavioral health needs, serve children in their homes and communities to the greatest extent possible, and use the most effective, evidence-based practices in all behavioral health services.
Goals for the department's overall behavioral health system are not clearly set out in statute. However, expected outcomes for the state's major behavioral health reform initiative, the Connecticut Behavioral Health Partnership, and for KidCare, the children's services component overseen by DCF, are described in state law. The statutory goals for KidCare are included in Appendix F.
DCF participates in the statewide mental health planning process the Department of Mental Health and Addiction Services carries out to meet federal mental health block grant funding requirements. DCF prepares the section of the federal plan on children's services, which must describe how the state will implement an organized, community-based system for improving mental health services for children with serious emotional disturbances.
In addition to describing the current state service system, the federal mental health plan must: identify and analyze system strengths, needs, and priorities; and discuss performance goals and action plans for improvement. Although goals and measures are outlined in the children's services section, the document does not appear to be used by DCF or its behavioral health bureau as a strategic guide for providing services.
A two-year strategic plan that sets goals for Riverview, the children's psychiatric hospital operated by DCF, was developed by facility staff with the help of the DCF Bureau of Continuous Quality Improvement in the spring of 2007. A multidisciplinary hospital staff workgroup is responsible for implementation, and progress is reviewed quarterly by facility management, a BCQI representative, and an on-site monitor from the Office of the Child Advocate.
Juvenile justice mandate goals. DCF's juvenile justice goals, as outlined on the agency's Juvenile Services Bureau website, are:
● to serve children in the juvenile justice system and their families; protect public safety; collaborate with the courts, communities, and partners; and provide a continuum of effective prevention, treatment, and transitional services children need to succeed in their families and communities.
Further, there are specific statutory goals for the state juvenile justice system, which apply to the courts as well as DCF. These are also listed in Appendix F and are generally reflected in the juvenile services bureau goal statement.
A statewide juvenile justice strategic plan was prepared by the DCF Juvenile Services Bureau and the Court Support Services Division of the Judicial Branch with input from many public and private stakeholders.5 Issued in August 2006, it sets a vision, mission, 10 guiding principles, and 12 broad system goals in four areas (resource development; coordination, collaboration, and information sharing; data analysis; and work force development).
A workgroup of staff from the DCF Juvenile Services Bureau and the Court Support Services Division, advocates, and parents, with the help of a consultant, operationalized the statewide plan into a results-based accountability format. In addition, DCF and CSSD have jointly developed a plan that both carries out the goals and meets the required service outcomes under the final settlement agreement for the Emily J. juvenile justice class action lawsuit.
Staff at the DCF Connecticut Juvenile Training School (CJTS) developed a strategic action plan for that secure juvenile justice facility in the summer of 2005. In addition to setting six main goals for improving programming and accountability, the plan: defined objectives and outcomes for each goal; included specific action steps for each one; and outlined implementation time frames and responsibilities. Progress was monitored and strategies were revised as needed on a monthly basis until the end of 2006. Strategic planning for CJTS has been put on hold pending a final decision about the facility's future.
Prevention mandate goals. State statute specifically includes prevention services as a DCF responsibility in providing comprehensive services to children and families at risk for abuse, neglect, delinquency, and behavioral health problems. The department's goals for its prevention mandate are set out in detail on the agency webpage and budget document. In brief, they are to:
● promote a range of services that enable children and their families to thrive independently in their communities; and
● apply evidence-based or best practice prevention approaches to ensure successful transition from DCF involvement, or to prevent DCF involvement at all, by children and their families.
The DCF prevention office also has adopted seven guiding principles that reflect and expand on the agencywide guiding principles (see Appendix E). Further, the department developed a five-year child welfare prevention plan in 2006 that outlines four goals related to primary prevention and early intervention efforts carried out by the agency. Progress is monitored by the prevention office director, who provides status reports as needed or on request to agency top management.
Major programs. Goals of each of the major department programs within each of the four mandate areas are also listed in Appendix F. The main source for program-specific goals is the agency's budget document. All of the more than 70 specific budgeted programs reviewed have stated goals, although they do vary in specificity, measurability, and relevance.
Many of the program goals are related to outcomes for children and families, usually in very broad terms (e.g., “foster positive youth development”), but a significant number primarily relate to how services are to be delivered (e.g., “receive appropriate services in the least restrictive setting”). Few of the program goals identified by PRI staff incorporate the agency's guiding principles concerning family-centered practice, partnerships, and cultural competence. For the most part, however, they are consistent with the agency's overall and mandate area goals.
Major Duties and Responsibilities
The Department of Children and Families has broad authority and responsibility for protecting and supporting children and families by carrying out state and federal child welfare, juvenile justice, and children's mental health and substance abuse programs. Current state statutes require the department to:
● “…plan, create, develop, operate or arrange for, administer and evaluate a comprehensive and integrated state-wide program of services including preventive services for children and youths…” who are abused, neglected or uncared for, mentally ill or emotionally disturbed, substance abusers, delinquent, or whose behavior does not conform to the law or acceptable community standards;6
● provide a “flexible, innovative, and effective program for placement, care, and treatment” of committed, transferred, and voluntarily admitted children and youth, as well as provide appropriate services as needed to the families of children and youth in its care;
● work in cooperation with other agencies and organizations to provide or arrange for preventive programs, including but not limited to teenage pregnancy and youth suicide prevention;
● establish or contract for services for the “identification, evaluation, discipline, rehabilitation, aftercare, treatment, and care of children and youth served by the agency….”; and
● “… undertake or contract for or otherwise stimulate research concerning children and youth….”
At present, the agency contracts with nearly 200 different private providers for more than 100 types of services for its clients. The Department of Children and Families, as specified in state statute, also operates the state's only public psychiatric hospital for children and youth, two residential treatment facilities, and a secure correctional facility for delinquent boys. The department runs a therapeutic program for troubled youth through its Wilderness School, another facility named in statute. Table I-1 provides a brief description of each DCF facility.
Monitoring and evaluation. The agency has a number of specific statutory charges to monitor, assess, and evaluate its activities. It is required to:
● collect, interpret, and publish statistics related to children and youth in the department;
● conduct studies of any program, service, or facility developed, operated, contracted for, or supported by the department to evaluate its effectiveness; and
● prepare and submit biennially to the General Assembly a five-year master plan that includes but is not limited to:
the department's long-range goals and their current level of attainment; and
an overall assessment of the adequacy of children's services in Connecticut.
Table I-1. Facilities Operated by DCF | |
Name/Location |
Scope |
Riverview Hospital for Children and Youth Middletown |
98-bed psychiatric hospital for children and adolescents ages 5 through 17. Patients admitted when intensive 24-hour care and treatment is necessary in a protected environment. |
High Meadows Hamden |
42-bed residential treatment facility for severely emotionally disturbed adolescents (ages 12 to 17) who require intensive and comprehensive 24-hour services but not a closed setting. |
Connecticut Children's Place (CCP) East Windsor |
Formerly the State Receiving Home, now a 54-bed residential diagnostic center for children and youth ages 10 to 18, who are in need of protection due to abuse, neglect, abandonment, unmanageable behavior, or sudden disruption in their current placement or residence. Diagnostic and evaluation services and brief treatment are available while permanent placement is pending. |
Connecticut Juvenile Training School (CJTS) Middletown |
Secure facility for approximately 100 boys who are committed delinquents; intended to prepare residents for successful community re-entry through educational, treatment, and rehabilitative services. (Opened in 2001 to replace Long Lane School) |
The Wilderness School East Hartland |
Therapeutic camp/outdoor expedition program for troubled youth age 13 and over intended to foster positive development; 20-day, 5-day, 1-day and alumni follow-up programs are provided. |
|
Source: Connecticut General Statutes and DCF agency website. | |
DCF is also required by law to award funding to community service programs in proportion to their effectiveness. Furthermore, it must: evaluate the programs based on analysis of their outcomes and an assessment of service needs; and collect, maintain, and analyze data used for evaluation on an ongoing basis. As noted below in the discussion of the current agency organization, a grants development and contracts division within the Bureau of Finance has responsibility for the DCF performance-based contracting process. The agency's contract monitoring procedures, including how contractor performance information is used for decision making, is described in Chapter II.
Under state statute, DCF must report each year to the governor and legislature on the status of all children committed to the department. It also must establish and maintain a central registry of all children with permanency plans that recommend adoption and, under legislation enacted in 1999, have a system in place to monitor progress in implementing such plans. Information on the status of the various reports, plans, and reviews the department is required by state or federal law to produce, or to receive from service providers and advisory groups, is provided in more detail in Chapter III.
Legislation enacted in 2005 requires the department to seek accreditation from the national accrediting body for public child welfare agencies, the Council on Accreditation (COA). The COA accreditation process and standards and DCF efforts to comply with this requirement are also discussed in Chapter III.
Federal mandates. DCF is the state agency responsible for carrying out a number of federal mandates in areas of child welfare, children's behavioral health, and juvenile delinquency. Currently, the department is subject to oversight by: the U.S. Department of Health and Human Services, Administration for Children and Families; the Substance Abuse and Mental Health Services Administration of HHS; and the U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention. It must prepare any required state plans, grant applications, and reports for these federal agencies.
Federal monitoring and evaluation activities related to DCF, such as the Child and Family Services Reviews carried out for all state child welfare agencies, were examined in depth by committee staff. Major federal oversight activities on DCF services and programs for children and families are described in Chapter III.
Advisory groups. More than a dozen councils, committees, commissions, and boards established in accordance with state and federal law have responsibility for advising and assisting DCF or generally providing input to the governor and/or legislature on matters within the department's purview. These groups include:
● general agency advisory groups, such as the state and area advisory councils and the advisory groups for DCF facilities; and
● program or issue-specific advisory groups, such as the Behavioral Health Partnership Oversight Council and the Youth Suicide Advisory Board.
Program review staff reviewed the roles of these advisory groups in tracking program outcomes, assessing performance, and making recommendations to DCF for service improvements. Descriptive information on the advisory groups is presented in Chapter V.
State Mandate Areas and Programs
The department's many programs and activities are generally organized by its four main statutory mandate areas: child protective services; children and youth behavioral health services; juvenile justice services for adjudicated delinquents; and prevention services. DCF also categorizes its treatment services within each area on a continuum ranging from community-based and in-home services to increasingly intensive out-of-home placements. Like federal and other state children's agencies, providing appropriate care in the least restrictive, most family-like environment possible is the underlying goal of most of the department's efforts.
Each DCF mandate area and the main programs and activities it includes are described briefly below. Figure I-1 summarizes, by area, the many types of services carried out or funded by the department at the time of the committee's review. (The most recent annual data available for DCF activities were for FY 06 while funding information reflects FY 07 appropriations).
Child protection. Efforts to protect children from abuse or injury are the core work of DCF in its role as the state's primary child welfare agency. If children cannot remain safely at home, the department must arrange temporary placements with relatives, in foster homes, or in other residential settings. When reunification with their families is not possible, DCF is required to seek permanent homes for children through other means, such as adoption and subsidized relative care.
Services in the child protection area usually start with the Child Abuse and Neglect Hotline, which is the state's single point of contact for reporting suspected child abuse and neglect. It is operated 24 hours a day, seven days a week by DCF. Reports accepted for investigation are forwarded to trained professional social work staff in the department's area offices. If abuse or neglect is substantiated, the case is assigned to an area office treatment social worker for ongoing services to help ensure the child is safe and the family is supported. DCF received 43,500 abuse and neglect reports, investigated 28,790, and substantiated 7,568 during FY 06.
The treatment social worker is responsible for providing appropriate services to the child and family. If the child's safety can be assured without removal, services may include in-home supports, such as a parent aide or substance abuse screening. If removal is required, out-of-home care is provided. In accordance with federal and state requirements, DCF must develop an initial written treatment plan for every child under its supervision within a specific time frame and treatment plans must be reviewed every six months.
In most cases, children who are removed from their homes are placed in foster homes, all of which are licensed by the department. On average during FY 06, about 3,200 children were living in foster care. If the department determines reunification with the child's own family is not possible, the social worker will try to achieve permanency through other options such as adoption, a subsidized guardianship with a relative, or sometimes, in the case of older children, independent living arrangements. In FY 06, over 1,200 children were living with licensed relative caregivers and over 700 youths were in independent living situations. Also that year, the department finalized 498 adoptions and granted 308 subsidized guardianships.
Behavioral health. DCF is responsible for addressing the behavioral health needs of Connecticut's children by planning, developing, and providing appropriate mental health and substance abuse assessment, treatment, and aftercare services. The agency provides behavioral health services to: children committed to DCF because of abuse and/or neglect; delinquents committed to its custody; and to children and youth with behavioral health needs and no involvement with DCF. State law allows families to apply on a voluntary basis to the department for state funded mental health and substance abuse services for children under 18.
FIGURE I-1. DCF Agency Services and Programs by Mandate Area (as of FY 07) | |||
Child Protection (CP) |
Behavioral Health (BH) |
Juvenile Justice (JJ) |
Prevention |
CP Community-Based Services n Hotline n Social Work (Area Offices) n In-Home (family preservation, parent aide, substance abuse screening) |
BH Community-Based Services n KidCare n Emergency mobile psych n Care coordination n Parent advocacy n Child guidance clinics n Extended day treatment n Substance abuse treatment including family-focused and supportive housing programs n Flexible Funding n Intensive in-home treatment ● MST (multi-systemic therapy) ● MDFT (multi-dimensional family therapy) ● FFT (functional family therapy) ● IICAPS (intensive in-home child and adolescent psychiatric services) ● FST (family support team) |
JJ Community-Based Services n Parole Services n Aftercare for Delinquent Youth ● MST(multi-systemic therapy) ● Outreach, Tracking and Reunification and Choice ● | |