Executive Summary
Department of Children and Families Monitoring and Evaluation
Formed in 1974 as a consolidated children's agency, the Connecticut Department of Children and Families (DCF) has broad authority and primary responsibility for state mandates concerning child protection, children's behavioral health, juvenile delinquency, and prevention services related to children and families. The department has been studied, audited, reviewed, and subject to legal action almost continuously since it was created due to ongoing concerns about its ability to carry out its challenging mission.
Numerous internal quality improvement efforts, as well as oversight by multiple outside entities including federal and other state agencies, various advisory groups, the courts, and the legislature, have focused on how to achieve better outcomes for the children and families DCF serves. The Legislative Program Review and Investigations Committee (PRI) alone had completed seven different reports on the department prior to undertaking a review of the overall DCF accountability system in April 2007. Unlike previous PRI reviews, this study evaluated a critical function -- monitoring and evaluation of agency results -- rather than a particular mandate.
An effective results-based monitoring and evaluating system is important for three main reasons: 1) it provides an agency with productive feedback on actual outcomes and progress toward goals; 2) it allows agency staff, policymakers, and stakeholders to know where the agency is successful, where it is not, and how to make improvements; and 3) ultimately, it helps the agency provide services that meet clients' needs and make cost-effective use of taxpayer resources. The purpose of the 2007 program review committee study was to determine areas of strength and weakness, as well as gaps and redundancies, in the existing DCF accountability system, and to identify needed improvements.
Study approach and methods. The committee study employed two primary research methods: interviews with key stakeholders; and analysis of monitoring and evaluation reports and other documents produced through DCF quality assurance, performance evaluation, and oversight efforts. There were four main sources of efforts:
1) internal monitoring and evaluation efforts such as: provider licensing, performance-based contracting, ombudsman activities, and various department self-reviews and contracted evaluation studies;
2) external oversight efforts by federal agencies, federal and state courts, legislative committees, and independent entities like national accreditation organizations;
3) outside investigations and reviews, such as those carried out by the state Office of the Child Advocate (OCA) and the state attorney general; and
4) monitoring and evaluation activities by advisory groups established under federal or state law.
To put into context all the information gathered about the process, sources, and results of DCF monitoring and evaluation, the core components of the current system were also compared with a national model for child welfare agency quality improvement.
The program review committee's final report contains an assessment of the overall DCF monitoring and evaluation system, details the system's positive features as well as deficiencies, and recommends nearly 40 administrative and legislative changes to improve its effectiveness. The report also summarizes data on agency accomplishments that were compiled by program review staff from more than 100 different monitoring and evaluation documents analyzed during the study.
Main findings. The program review committee found little attention has been given to examining DCF as a whole or assessing how well the agency is achieving its broad goals of safety, permanency, and well-being for all children and families. Further, while the department is responsible for carrying out four major mandates, monitoring and evaluation is focused primarily on the child protective services mandate, due largely to the ongoing impact of the federal Juan F. lawsuit consent decree and requirements of federal agencies.
The PRI study showed there is greater emphasis on tracking how services for children and families are delivered rather than on assessing their end results. While high quality service delivery is important, the crucial indicator of effectiveness is whether programs are making a difference and achieving stated goals. In general, more attention to outcome information is needed throughout the DCF accountability system.
The committee's review also identified pockets of strength within the system, such as the Juan F. exit plan process and related DCF area office quality improvement processes, the department's licensing procedures, the agency's recently revised special review process, and the activities of on-site facility monitors.
Some major weaknesses were revealed as well. In particular, the agency's contracting process provides little accountability, consequences for poor performance are rare, and working relationships with private providers need improvement. The committee also found ineffective use of some important sources of feedback on services and programs, such as child fatality reviews, OCA investigations, and even the department's own program review reports and contracted evaluations.
In part, these deficiencies are due to both fragmentation of quality improvement efforts within the agency and the fact that results data are not regularly integrated and analyzed. Both problems are related to the department's information systems, which are themselves fragmented and in some cases inadequate. Another challenge is a lack of department staff with the analytic skills and research experience needed to use results data and information. Further, there is no centralized place – like an agencywide strategic plan – where all DCF goals and information about service delivery and outcomes are brought together.
Duplication of external monitoring efforts also was revealed by the program review committee's examination of statutorily required DCF plans and reports. The committee determined several mandates could be eliminated without a loss of accountability, as certain documents have become obsolete or been replaced by newer sources of similar information. In addition, reducing the number and clarifying the purpose of reporting mandates could improve the quality of information on department results available to the legislature and the public.
Committee recommendations. Ultimately, the point of all monitoring and evaluation efforts, whether internal, external, investigatory, or advisory, is to ensure programs and services are having desired results. Taken together, the PRI committee recommendations listed below are aimed at making the current DCF accountability system more effective by:
● making agency goals explicit;
● integrating quality improvement activities and incorporating best practices throughout the agency;
● improving the quality and quantity of available data; and
● promoting the use of results information to better meet the needs of children and families.
Recommendations
1. The current statutory provision for a Department of Children and Families biennial five-year master plan shall be repealed and replaced with a mandate for ongoing strategic planning. Specifically:
a) Beginning July 1, 2008, the department shall start the process of developing a vision, mission, and strategic goals with the advice and assistance of representatives of the children and families served by the agency, public and private providers, advocates, and other stakeholders.
b) The department should dedicate staff, under the direction of the commissioner or a deputy commissioner, to: 1) prepare a strategic planning document that includes action steps and time frame for implementation to fulfill the vision, mission, and goals developed with stakeholders; 2) track and report on progress in achieving the plan's goals at least annually; and regularly review, revise, and update the department's strategic plan as needed.
c) The first plan shall be completed and submitted to the legislature and the governor by July 1, 2009.
d) The department's strategic plan shall be submitted to the agency's State Advisory Council for Children and Families for review and comment prior to submission to the legislature and governor. Progress in carrying out the plan shall be reported to the council by the DCF commissioner at least quarterly and to the legislature and governor annually.
2. The department should reinforce and expand the role of the Service Evaluation and Enhancement Committee (SEEC) in integrating monitoring and evaluation efforts across the agency and initiating proactive intervention on agencywide issues.
3. DCF performance-based contracts should specify the data required from providers. Performance standards or expected outcomes should be stated in the contract. DCF should monitor data submissions for accuracy.
4. DCF should review currently required data elements from providers and determine whether they are necessary or analyzed in any way. Data elements that are unnecessary should be eliminated and additional data elements that pertain to outcomes should be added to performance-based contract requirements.
5. DCF shall compile necessary required data elements to compare actual and expected outcomes based on the performance-based contract. Failure to meet contract expectations should result in discussion and joint plans for progress in meeting expectations.
Until automated systems are deemed reliable, DCF should monitor contract expectations manually. Summary reports should be shared with providers so that they may monitor their performance against the aggregated data. Reports should be distributed to providers and DCF staff made more aware of the existence of these reports.
6. A central repository should be created by DCF of contracted research and evaluation reports and internally produced research and evaluation reports. This repository should be accessible and searchable by all DCF staff and should include the Office of Policy and Management (OPM) feedback form as applicable.
7. The DCF licensing unit should expand internal self-monitoring by inspecting High Meadows and Connecticut Children's Place, the two DCF facilities not currently under external licensure or accreditation. The licensing unit should follow the child care facilities regulations standards used to inspect external residential treatment facilities similar to High Meadows and Connecticut Children's Place.
8. The department should establish an internal written policy for responding to recommendations from the internal special reviews of child fatalities and other critical incidents. The policy should require a corrective action plan be developed, implementation of accepted recommendations be monitored, and a status report be prepared for the commissioner every 90 days. A forum to discuss results and lessons learned should be scheduled with managers and key staff from all relevant areas of the department within 45 days of release of the report.
9. DCF should be permitted to establish a long-term research partnership with the Child Health and Development Institute and its affiliate, the Connecticut Center for Effective Practice, through a multi-year, sole source contract to carry out a broadly defined research and evaluation agenda related to the agency's mission.
10. DCF should reexamine the role of its program lead position and consider the allocation of time necessary for this responsibility. DCF should also develop a team approach for working with contracted providers that will ensure contract obligations are being met, provide assistance when necessary so that programs do not reach a crisis point, and support and assist programs with quality improvement.
11. Considering contractor monitoring best practices, DCF should examine the roles of staff within the Grants Development and Contracts Division to determine whether some of the 19 positions could be reallocated from the financial/accounting function of contract management to program development and implementation support activities.
12. DCF should maintain a centralized and complete electronic grants and contracts library on the department's intranet. Grants and contracts missing should be scanned into the library. Previous years' contracts should be maintained for future reference.
13. The department should require the Grants Development and Contracts Division to receive and review feedback from area office and program lead staff on the performance of a provider before deciding to renew a contract. If concerns are raised about a provider, then discussions with the appropriate parties should occur and a performance improvement plan developed.
14. A workgroup should be convened by the department and the Offices of Policy and Management and the Attorney General to clarify the guidelines regarding contract bidding and related programming suggestions.
15. DCF should develop a protocol for providers to submit suggested programs or program enhancements. A form for submitting the idea should be developed and timelines for response from DCF publicized.
16. DCF should work with the Department of Administrative Services (DAS) to develop: 1) an appropriate job classification for staff positions within the agency responsible primarily for research and analysis; and 2) recruitment strategies for obtaining personnel with the necessary qualifications to fill them.
Furthermore, the department should increase its internal analytic capacity. The size and scope of the Risk Management Unit staff should be expanded to include the following duties in addition to compiling information to support the SEEC function: interpreting data produced by the state's behavioral health Administrative Service Organization (ASO); compiling contracted evaluation results; maintaining the research
repository recommended earlier; supporting agency strategic planning activities; and sharing outcome, best practices, and results information agencywide.
17. For programs exceeding $20 million in funding, DCF should require an external evaluation be conducted to assess the outcomes of the program.
18. DCF should develop and issue guidelines for staff and consultants regarding the format for final evaluation reports.
19. The Office of the Child Advocate should undertake an investigation to assess adequacy and integrity of the internal process for reviewing and responding to allegations of staff child abuse and neglect. It should also examine compliance with C.G.S. §17a-103c.
20. Wilderness School staff should work with the Department of Public Health to develop a more appropriate licensure as a wilderness school rather than as a camp.
21. Replace the following statutory reports:
a) DCF biennial five-year master plan (C.G.S. §17a-3);
b) DCF annual report on the Connecticut Juvenile Training School (CJTS) (C.G.S. §17a-6b and C.G.S. §17a-6c); and
Repeal the statutory reports listed below:
c) Children's Behavioral Health Advisory Committee (CBHAC) annual local systems of care status report (C.G.S. §17a-4a(e));
d) CBHAC biennial recommendations on behavioral health services (C.G.S. §17a-4a(f));
e) Quarterly hospital reports to DCF on psychiatric care (C.G.S. §17a-21);
f) KidCare Community Collaborative annual self-evaluations (C.G.S. §17a-22b);
g) DCF/DSS five-year independent longitudinal evaluation of KidCare (C.G.S. §17a-22c(c));
h) DCF monthly report to legislature on children in subacute care in psychiatric or general hospitals who cannot be discharged (C.G.S. §17a-91a);
i) Cost-benefit evaluation of juvenile offender programs (C.G.S. §46b-121m);
j) Licensed child care facilities annual reports (C.G.S. §17a-145);
k) DCF annual evaluation reports on Unified District #2 to the education commissioner (C.G.S. §17a-37(d));
l) DCF to conduct studies to evaluate effectiveness (C.G.S. §17a-3(a)(6)); and
m) Adoption Advisory Committee report (C.G.S. §17a-116b(g)(3)).
22. All DCF facilities shall be required to produce an annual report for their respective advisory groups. The report shall contain at a minimum the following:
1. aggregate profiles of the residents;
2. description and update on major initiatives;
3. key outcome indicators;
4. costs associated with operating the facility; and
5. description of education programs and outcomes.
23. Research and evaluation reports produced through federal grant requirements should be included in the report repository recommended earlier concerning contracted evaluation reports and internally produced research products.
24. DCF should adopt a written policy requiring that formal results from research and evaluation reports produced from federal grants be reviewed and considered when agency managers make decisions concerning future funding and/or continuation of programs developed with federal grants.
25. DCF should convene a workgroup including program leads, a representative from the Juan F. court monitor's office, and DSS to develop a treatment plan and review process that satisfies both the internal DCF and federal (e.g., Private Non-Medical Institution Initiative or PNMI) requirements.
26. A pilot program should be created to assess the feasibility of conducting one treatment plan conference to be held at court that combines: the Specific Steps identified during the initial case status conference at court and the corresponding DCF treatment plan conference currently held in the area office.
27. The statutes concerning the Office of the Child Advocate and the Child Fatality Review Panel (CFRP) shall be amended to require the Department of Children and Families, and other state entities subject to OCA and CFRP investigative activities, to provide a written response to formal recommendations made by the child advocate and the panel for improving state services provided to children.
The agency response should: include proposed corrective actions to address identified problems and a time frame for implementation of improvements; and be provided to OCA or CFRP within 45 days of receipt of the recommendations. Copies of the agency response also should be submitted to the legislative committees of cognizance and the appropriations committee.
28. The resources necessary to improve the OCA data management system should be provided during the next fiscal year, either by the Department of Information Technology (DOIT) making this project a priority or through funding for a consultant to design and implement an upgraded system for the child advocate's office.
29. DCF should establish a policy for area office advising bodies to adopt a model whereby advising body members attend DCF area office quality improvement meetings, and DCF area office representatives attend advising body meetings, furthering promotion of a partnership.
30. DCF facility advisory boards shall be required by statute and it shall be mandated that all boards respond to their facility's annual report and that they add recommendations deemed necessary.
31. The role of the State Advisory Council for Children and Families (SAC) should be strengthened to include monitoring the agency's progress in achieving its goals as well as offering assistance and an outside perspective. The board's statute shall be written to clarify this role and DCF's participation with the board concerning strategic planning as recommended above. The council's meetings should be held at locations that facilitate participation by members of the public, such as the Legislative Office Building, and its agendas and minutes should be posted on the DCF website. The department should provide the council with funding for administrative support services and to ensure members representing families from across the state can serve on the council.
32. DCF should establish an electronic mechanism, for example a blog, where members of area office advising bodies can share information with each other, the SAC, and vice versa. Additionally, minutes and agendas from all meetings should be posted on the DCF website.
33. DCF should fund all three required Citizen Review Panels equally.
34. The Children's Behavioral Health Advisory Committee (C.G.S. §17a - 4a) should be incorporated into the State Advisory Council as opposed to remaining a separate entity.
35. Repeal the statutory requirement for the Adoption Advisory Council (C.G.S. §17a-116b).
36. Repeal the statutory requirement for the Connecticut Juvenile Training School Public Safety Committee (C.G.S. §17a-27f).
37. DCF shall hire an external consultant to:
a) perform a gap analysis1 and workflow analysis with the focus on integrating the functions of the department with technology modeled to support the service model;
b) develop a project plan; and
c) develop a request for proposals to procure the team needed to integrate the data systems and replace the LINK System.
1 Refers to identifying technology requirements and assessing existing capabilities to determine where needs are not being met.