12. Evaluation Use

The California Department of Social Services created the California Evidence-Based Clearinghouse for Child Welfare, which allows child welfare providers and professionals to “identify, select, and implement evidence-based child welfare practices that will improve child safety, increase permanency, increase family and community stability, and promote child and family well-being.” This tool helps identify evidence-based practices and provides guidance and support for program implementation. The clearinghouse’s numerical rating scale categorizes programs into six tiers of evidence and uses a relevance scale as a complement to the scientific rating scale and to demonstrate applicability for client populations.

Issue Areas: Child Welfare

The Colorado Governor’s Office of State Planning and Budgeting (OSPB) and the Colorado state legislature have aligned evidence definitions for a five-step evidence continuum that informs budget and spending decisions. OSPB requires agencies to identify the level of evidence and the results of any evaluations of programs as a part of the budget proposal process. OSPB also requires agencies to report on cost-benefit analyses performed on agency programs as a part of the annual budget process.

The Department of Higher Education’s Colorado Opportunity Scholarship Initiative (COSI) conducts an annual quasi-experimental design evaluation using propensity score matching to evaluate the success of programs that have received grant funding.  Using COSI’s innovative model and the annual reporting requirements, COSI staff can monitor progress of grantees in achieving shared goals.

The Colorado Department of Human Services has been conducting cost-benefit analyses by evaluating at-risk Colorado youth based on their performance under Pay For Success contracts. If outcomes meet criteria of achievement based on the contract, success payments made are detailed in an annual report to the Joint Budget Committee.

Issue Areas: Child Welfare, Criminal Justice, Education, Health

A 2015 Connecticut law defines three tiers of evidence for programs operated by the Connecticut Departments of Correction, Children and Families, and Mental Health and Addiction Services, and the Court Support Services Division of the Judicial Branch: evidence-based, research-based, and promising. The law requires these agencies to categorize their programs by the evidence tiers in even-numbered fiscal years. Additionally, the law charges the Institute for Municipal and Regional Policy at Central Connecticut State University with submitting an annual report containing a cost-benefit analysis of the programs. In 2020, the report entitled Benefit-Cost Analyses of Evidence-Based Programs showed that 108 programs and services administered by the Judicial Branch’s Court Support Services Division and the Department of Correction were identified as evidence-based.

Issue Areas: Child Welfare, Criminal Justice, Health

The Florida Department of Juvenile Justice requires all residential commitment prevention contractors to implement at least one evidence-based model from the agency’s Sourcebook of Delinquency Interventions. The sourcebook lists all juvenile justice programs according to three levels of evidence: evidence-based practices, promising practices, and practices with demonstrated effectiveness, as defined in the book. The Department also introduced a Standardized Program Evaluation Protocol, a monitoring tool to ensure providers implement programs with fidelity.

Issue Areas: Criminal Justice

The North Carolina Office of State Budget and Management released program inventories in the areas of child and family health and juvenile justice. The state also publishes periodic Results First progress reports and has created evidence definitions to categorize programs based on their levels of evidence.

Issue Areas: Child Welfare, Criminal Justice, Health

The FY21 New Jersey budget and FY22 budget each appropriated $1 million in funding for the New Jersey Policy Lab, led by Rutgers University, that will contribute high-quality, nonpartisan research and analysis of policy-based solutions facing New Jersey. The initiative, granted by the New Jersey Office of the Secretary of Higher Education, will also build coalitions across governments and community organizations to support evidence-based policy initiatives, assist stakeholders in troubleshooting implementation issues, provide data modeling for policy recommendations, and evaluate the impacts of proposed statewide public policy measures.

Issue Areas: Education

New Mexico has published a series of inventory and cost-benefit reports in the areas of children’s behavioral health, adult behavioral health, early education, child welfare, criminal justice, healthcare, and education. In 2019, the Legislative Finance Committee’s evaluation unit published analyses on substance abuse treatment outcomes and Medicaid spending and managed care administration. In FY21, the evaluation unit is partnering with Abdul Latif Jameel Poverty Action Lab (J-PAL) North America to evaluate the effectiveness of the state’s early college high schools. All reports from the evaluation unit can be found here.

Issue Areas: Criminal Justice, Child Welfare, Economic Mobility, Education, Health

The Ohio Department of Education’s evidence-based clearinghouse provides a common evidence framework by aligning evidence standards from various clearinghouses. The framework assists educators in identifying evidence-based interventions according to content focus area, subject, grade bands, urban/rural settings, student demographics, and the four levels of evidence in the federal Every Student Succeeds Act.

Issue Areas: Education

As part of the implementation of the 2003 Oregon law on evidence-based health, youth, and corrections programs, Oregon Health Authority maintains an ongoing inventory of approved evidence-based practices in behavioral health, including culturally specific services.

The Oregon Health Authority (OHA) has established the Health Evidence Review Commission to continually review and update an ongoing inventory of evidence-based practices at work within OHA. The commission vets medical evidence to identify new and valuable practices and maintains a publicly available list of practices in a wide range of clinical and treatment areas. In 2020, Oregon Youth Authority provided an updated report identifying the cost-effectiveness of their programs and services.

Issue Areas: Child Welfare, Criminal Justice, Education, Health

In 2020, the Pennsylvania Workforce Development Board unanimously voted to adopt a four-tiered evidence framework for the state’s workforce system. The definitions for evidence of effectiveness include: strong evidence, moderate evidence, preliminary evidence, and pre-preliminary evidence.

In 2020, the Pennsylvania Department of Education (PDE) launched the PDE Evidence Resource Center to help Pennsylvania schools thoughtfully apply high-quality, relevant research to their local settings. By listing evidence-based interventions in the areas of instruction and curriculum, student support and professional development, and student wellness, this tool helps school districts identify and select strategies to address every aspect of school improvement. This tool is also being used for the PDE’s research agenda that allows the department to identify and measure its progress in meeting its research and learning priorities.

Issue Areas: Economic Mobility, Education, Workforce

Tennessee’s Office of Evidence and Impact (OEI) defined five evidence steps. The Office also completed inventories of state- and federally-funded programs in the areas of corrections, mental health, and substance abuse services as a part of the state’s evidence-based budgeting initiative. In 2021, OEI published program inventories for higher education, children’s services, and human services.

Tennessee’s Department of Education, Department of Higher Education, and Department of Health have conducted program evaluations independently or in partnership with universities that have directly impacted policy, budget, and programmatic decisions. For example, state evaluations on voluntary pre-K, financial aid for nontraditional postsecondary students, and the smoking cessation program for pregnant women have informed timing of program expansions or affirmed the continuation of programs with high confidence in results.

Issue Areas: Child Welfare, Criminal Justice, Education, Health

In 2020, the Texas Workforce Commission, in partnership with the Office of the Governor, Texas Workforce Investment Council, and Rural Capital Area Workforce Board, collaborated to develop a new evidence framework to better measure the impact of publicly funded workforce development programs. In July 2020, the Commission approved $1.8 million in Workforce Innovation and Opportunity Act (WIOA) statewide funding for the Building Construction Trades project, which will utilize the evidence framework to better measure the success of the grant program.

Issue Areas: Workforce

Project KIDS is a special performance audit of the Office of the State Auditor that integrates financial, operational, and performance data to improve data-driven decision-making in Utah K-12 public education. Project KIDS follows resources down to the individual student level to understand how K-12 public education dollars are spent. It then analyzes course enrollment and demographic data to identify which students participate in particular classes and programs. Thus, individual resource profiles are created for all Utah students based on their unique course-taking patterns and demographics. Additionally, each student’s spending information is tied to their academic performance. District and charter school administrators are issued more detailed dashboards with privileged access to student-level data for their specific local education agency. By integrating student-level spending and performance, this methodology allows for an exploration of what resource allocations drive high student achievement. It also facilitates bottom-up, not top-down, student spending averages. Project KIDS recognizes that a top-down spending average fails to differentiate between various student needs and consequent spending. Thus, the individual student resource profiles are utilized to build up spending averages for each classroom, school, and local education agency. This framework enables stakeholders and the public to ask deeper questions about equitable and effective spending practices. They can use the school spending map and other analytic resources in weighing spending decisions.

Issue Areas: Education, Equity

In 2018, the Vermont Agency of Education adopted four tiers of evidence, based on the federal Every Student Succeeds Act (ESSA) for use in distributing education funds, including the bulk of federal- and state-funded grants (such as BEST/Act 230 and Flexible Pathways). The guidance states: “applying the same requirement to both federal and state grants will allow for greater consistency in the review and approval of grant applications.”

Issue Areas: Education

Since 2012, at the direction of the legislature, the Washington State Institute for Public Policy (WSIPP) has produced a series of inventories of evidence-based, research-based, and promising programs. These inventories are developed with the aim of informing state agencies about effective and cost-beneficial options for service provision. In addition, a 2013 state law directed the Department of Corrections, in consultation with WSIPP, to: (1) compile an inventory of existing programs; (2) determine whether its programs were evidence-based; (3) assess the effectiveness of its programs, including conducting a cost-benefit analysis; and (4) phase out ineffective programs and implement evidence-based programs.

As a result of laws like these that require evidence reviews and/or direct outcome evaluations, WSIPP has published hundreds of cost-benefit analyses in a wide variety of issue areas over the past 10 years. The WSIPP cost-benefit framework allows the translation of program effects into dollars and cents and has been emulated by several state governments across the country. These inventories are developed with the aim of informing state agencies about effective and cost-beneficial options for service provision. Recent inventories published by WSIPP include: children’s mental health, child welfare, and juvenile justice (2020) and K-12 educational programs for struggling students (2020), which is also featured on the state superintendent’s Menus of Best Practices and Strategies website.

The 2021 legislature directed WSIPP to publish benefit-cost analyses of wilderness therapy programs for youth mental health as well as the use of domestic steel in transportation and capital budget contracts. WSIPP also has ongoing projects examining the long-term costs and benefits of legalizing recreational cannabis, investigating the cost-effectiveness of integrating involuntary treatment for substance use disorders and mental health disorders, and exploring the costs and benefits of the state quality rating system for early childhood education providers.

Issue Areas: Child Welfare, Criminal Justice, Economic Mobility, Education, Health, Workforce