WIOA State Plan Common Elements

Description of Overall Approach to Assessment Activities

Evaluation and assessment activities in California deploy a variety of qualitative and quantitative research methodologies to assess program efficacy using both in-house staff and contract researchers.  Preferred methods include survey research, focus groups, communities of practice, literature reviews, analyses of descriptive statistics, and applied inferential statistics. Some of these activities happen during the course of regular operations, for example, through routine structured interactions with organizations receiving workforce grants, and through the implementation of the Cross-System Analytics and Assessment for Learning and Skills Attainment (CAAL-Skills) workforce data initiative. Other evaluation and assessment activities are formalized and implemented through the use of third party subject matter experts hired to provide specialized research.

Typically the state’s evaluation and assessment activities aim to investigate how specific operational practices and policy interventions affect the success of workforce programs and their participants. Relevant research questions include the following:

• What types of services are most likely to improve the labor market outcomes of program participants, especially over the medium to long term?
• How can operational practices be better structured to facilitate improved operations and better outcomes for workforce program participants?
• What program supports ensure program sustainability, continued program participation for participants, and limit premature program exit for participants while ensuring innovation and sustainability?
• What practices facilitate partnership between service providers?
• What practices facilitate regional partnerships and industry engagement?
• How can services be made more customer-centered?

Answering these types of questions requires taking a long and broad view on program operations and outcomes. The sections that follow provide an overview of assessment and evaluation activity that has taken place since July 1, 2018.

Overview of Assessment Activities July 1, 2018 through June 30, 2020

The following is a list of core and other one stop partner program evaluation and assessment activities initiated, developed, and/or completed in the two year window between July 1, 2018 and June 30, 2020.

• Evaluations of Slingshot and Workforce Accelerator “Innovation” Grant Programs (Initiated May 2017, Completed December 2018)
• Evaluation of High Road Training Partnerships (Initiated January 2017, Anticipated Completion Spring 2021)
• Evaluations of Regional Plan Implementation Grants 2.0 and 3.0 (Initiated February 2019, Anticipated Completion December 2021)
• Evaluation of AB 1111 State Funded Grant Program (Initiated July 2019, Anticipated Completion Fall 2021)
• Evaluation of America’s Job Centers of California (Initiated Summer 2019, Anticipated Completion December 2021)
• Development of the CAAL-Skills Data Repository (initiated August 2016, continual ongoing development through the present)
• Development of the CAAL-Skills Workforce Metrics Dashboard 2020 Report (Initiated in Spring 2017, Anticipated Completion Fall 2020)
• Development of the CAAL-Skills Workforce Metrics Dashboard 2022 Report (Initiated Summer 2019, Anticipated Completion Fall 2022)
• California Policy Lab (UC Regents) analysis of CAAL-Skills workforce program data (Initiated Summer 2018, Anticipated Completion August 2021)

These assessments and findings are described in the paragraphs that follow, highlighting methodologies employed, timelines adopted, and key findings and recommendations where appropriate. The manner in which findings and recommendations have impacted state policy and strategy is summarized in narrative that follows the assessment summaries.

Descriptive Summaries of Contract Evaluation Activities

Evaluation of Slingshot and Workforce Accelerator

Governor’s Discretionary Fund “Innovation” Grant Programs

(Initiated May 2017, Completed December 2018)

Background

Launched in 2014, SlingShot was California’s regional prosperity initiative. The initiative was focused on bringing together leaders from industry, education and government – including both the economic and workforce development systems – to identify and address ‘demand-side’ workforce issues as well as expanding opportunities for economic mobility. The state’s 12 SlingShot regions represented the broad geographic, demographic and economic diversity that makes up California.  Work done under the SlingShot initiative set the stage for regional work that continues today in California’s 15 Regional Planning Units (RPUs).

The Workforce Accelerator Fund (Accelerator), launched in 2014, is a grant program that continues to provide resources to innovative projects with the goal of helping individuals with barriers to employment find jobs and advance in careers at wage levels that support their economic well-being. Through the Accelerator initiative, the State Board has awarded over a hundred grants testing innovative ideas, forging new partnerships, and creating new models and modes of service delivery to improve participant outcomes.

Evaluation Overview

In June 2017, the California Workforce Development Board awarded the Corporation for a Skilled Workforce (CSW), Social Policy Research Associates (SPRA), and BW Research Partnership (BW) contracts to analyze the Slingshot and Workforce Accelerator Grant Programs and to make recommendations to improve the programs. CSW served as the evaluation coordinator for the research being conducted while BW served as the principal researcher for SlingShot and SPRA served as the principal researcher for Accelerator.

Researchers used a variety of methodologies to study both grant programs, including site visits, interviews, and data analysis. The evaluation contracts were extended during the course of 2018 through to December 2018 to further facilitate dissemination of findings.

The SlingShot Evaluation

BW Research Partnership evaluated the statewide SlingShot initiative and sought to do the following:

• Understand what regions learned from SlingShot
• Understand how the initiative evolved over time 
• Understand how regional engagement efforts could be built and sustained 
• Understand how to scale efforts for greater impact

The evaluators identified the following “challenge themes” from participants in the SlingShot Grant program:

• Industry Engagement: Despite many initial successes in the SlingShot initiative, it was still difficult engaging industry stakeholders in a consistent and significant way. All regions identified ongoing industry engagement as a major obstacle to progress, though many were encouraged by examples where individual companies actively engaged.
• Sustainability: Successfully innovating and creating system change typically takes time as well as trust and credibility with industry, education and community partners. Regions consistently identified the challenges associated with sustainability: growing trust and credibility with partners; developing early-stage innovation, planning and implementing to scale and expanding but with uncertainty about the timing of additional resources. All systems resist change and tend to remain inert without the necessary set of factors to help them move to a new condition. There is also great elasticity built into all systems, and they will return to their previous state unless there is deliberate and sustained pressure to move to a new model.

The evaluators identified the following “success themes” from participants in the SlingShot Grant program:

• Empowering Regional Collaboration:  While the Slingshot initiative emphasized new and innovative approaches, the fact is that the ability to move quickly, engage employers and other partners, create new programs and pathways, vet industry-recognized credentials, and braid existing resources was often built upon prior relationships among partners who had developed trust by working together and collaborating on earlier projects or plans.
• Key Players as Catalysts: Looking across the profiles of all regions, the impact of SlingShot’s role of introducing “catalysts” into the system becomes clear. In the success stories of each region, there were always key players, be they individuals or organizations that acted as catalysts, without whom critical connections would not have been made, foundational programs and processes would not have been developed, and important stakeholders would not have been engaged and brought into the fold. In the absence of such key players acting as catalysts, supported by the SlingShot model, it is difficult to imagine these types of innovations coming to fruition so consistently and frequently across regions.
• Flexibility to Experiment and Pivot: In a state the size of California, with its regional economic and demographic diversity, it is clear that the problems, opportunities and solutions will vary immensely from region to region. A “one-size-fits-all” approach to workforce development and innovation would clearly not be viable. Regions were offered flexibility in embracing the SlingShot process and tailoring it to the unique needs of each region, along with the latitude and time to try new strategies and program designs – and then to change direction based on early learnings. This aspect of the initiative was vital to the overall success of SlingShot.

Additionally the evaluators identified the following key recommendations to the State:

• There is a need to support efforts that grow stronger regional networks and deeper collaboration. The state can support these efforts by first understanding the level of engagement necessary to build collaboration and sustainable system change. Activities the state can use to support these efforts include making opportunities available for regions to bring together regional partners, either through state-funded convenings or by creating viable networks for regions to share planning and implementation activities.
• There is a need to continue supporting regional “pivoting” due to changing labor markets or partner dynamics. Regions consistently voiced that the ability to pivot in response to new information and incorporate evolving input from regional partners was an essential characteristic of successful SlingShot initiatives. The state is instrumental in a region’s ability to be nimble and responsive to regional partners and labor market needs. Being open to the emergent nature of employers-voiced skill needs, and how regional education and training partners can adapt to those needs is vital.
• There is continued need to elevate regional communication and information sharing. To build trust and flexibility, regions committed themselves to improving communication and information sharing, both internally and externally. Nevertheless, an important state role is identifying, describing and categorizing best practices, consistent protocols and other standardized practices for regional engagement.
• There is a need to support formal decision-making processes and shared administrative responsibilities at the regional level. Decision making structures and administrative responsibilities imply power determinations. SlingShot was about shared power among many partners. The state can support emerging decision-making processes by ensuring that the flexibility inherent in SlingShot continues into future initiatives, particularly as it relates to formal decision-making and administrative responsibilities.
• There is a need to measure what matters. No matter what the stage of development or the final objectives of the project, it is invaluable for regions to begin the process of deciding what to measure, and then starting to measure what is important, to the point where their efforts become based on targets set and modified as additional information and learning dictates.

The manner in which these findings have shaped state strategy are addressed in the narrative that follows the descriptions of assessments.

The Accelerator Evaluation

Social Policy Research Associates (SPR) evaluated the first five grant cycles of the Workforce Accelerator Fund (Accelerator) to learn about how the Accelerator initiative worked operationally on the ground as well as the way the initiative helped the workforce system build capacity to innovate.  SPRA’s findings include the following:

• Direct service approaches were more common among grantees than were indirect ones, but projects combining both direct and indirect approaches were most common.
• Just over half of projects were new endeavors to Accelerator grantees. Slightly more projects were new approaches to grantees as compared to projects that strengthened or enhanced projects previously piloted by grantees.
• Most grantees were well-connected to regional planning efforts. Over 75 percent of grantees reported being part of a regional planning effort including Adult Education Block Grants, the California Community College Strong Workforce Initiative, the State Board’s SlingShot Initiative, or RPU.
• Around two-thirds of the Accelerator projects and funds were awarded to grantee organizations that identified as non-profit, education, or labor agencies.
• Grantees identified gaps in serving high barrier populations in their area. They responded to the goals of the Accelerator by engaging customers and partners to gain an understanding of issues facing their communities.
• Grantees designed new and innovative approaches to address the issues they identified, employing multiple wide-ranging strategies to improve employment and earnings for participants. Also, many developed a capacity for innovation that will foster on-going innovative practices within their organizations.
• Grantees defined success outside the bounds of WIOA performance measures. Grantees used the opportunity presented by Accelerator to test out new ideas with goals that differed from WIOA-defined outcomes. In some cases, they arrived at innovations that were ready to be sustained or scaled.
• Grantees scaled their projects in size, population, scope, or reach. These approaches included: increasing the number of job seekers or employers served; serving additional types of job seekers or employers; expanding the type or nature of services provided; or expanding to new geographic areas.
• For some projects, successful scaling was critical to sustainability and occurred simultaneously rather than after a project was sustained. This occurred primarily when projects needed to expand to reduce costs or operated in a way in which they did not need to continue service delivery in an area once initially delivered.
• Accelerator facilitated systems change with a small “s” by helping grantees institutionalize changes within their organizations. These small “s” changes included: using data more effectively, developing partnerships with new types of organizations, serving employers and job seekers with barriers, and developing a capacity to innovate.
• Accelerator facilitated systems change with a large “S” by shifting California’s public workforce system as a whole. These large “S” changes included: diversifying the types of organizations in the system, encouraging traditional workforce system agencies to work with new partners, helping the system improve services to barriered job seekers and employers, and helping the system to coordinate and integrate programs and services.

The evaluators made the following program recommendations:

• Grantees, and the workforce system as a whole, would benefit from longer initial Accelerator grant periods and new grant types focused on sustaining and expanding as well as scaling to new geographic areas.
• Additional research and evaluation would support sustaining and scaling. Many Accelerator projects show promising results, but these results typically have not been rigorously tested. Furthermore, grantees worked with relatively small numbers of job seekers and employers so any evidence of effectiveness is still preliminary. Additional evaluation support—through technical assistance or the hiring of evaluators—would help grantees refine outcome measures, develop stronger data collection methods, and improve their evaluation designs.
• Certain policy levers may support sustaining and scaling Accelerator projects. Some Workforce Development Boards (WDBs) are unwilling to use Workforce Innovation and Opportunity Act (WIOA) Title IB formula funds to serve barriered individuals due to concerns about how these individuals might affect performance measures, even despite the promising results from many Accelerator projects that aid these populations.

The manner in which these findings have shaped state strategy are addressed in the narrative that follows the descriptions of assessments.

Evaluation of High Road Training Partnerships

Governor’s Discretionary Fund Grant Program

(Initiated January 2017, Anticipated Completion Spring 2021)

The High Road Training Partnerships (HRTP) initiative is a sector-based approach to addressing critical workforce issues related to equity, job quality, and environmental sustainability. HRTPs are industry-based, worker-focused training partnerships that build skills for California’s high road employers—firms that compete based on quality of product and service achieved through innovation and investment in human capital. HRTPs generate family-supporting jobs and have the following characteristics:

• Industry-Led Problem Solving: Foundational is that the industry – including leaders representing both employers and workers – lead the problem solving for the workforce demands unique to that industry.
• Partnerships as a Priority: It is key to have strong and durable industry partnership as a goal in and of itself. Here it means a formal relationship that is neither loose nor ad hoc, but forms the basis of sustained problem-solving.
• Worker Wisdom: Existing HRTPs in California have developed innovative ways to explicitly bring worker voice into their strategies and tactics as a core value undergirding the partnership.
• Industry-Driven Education & Training Solutions: Partnerships can tap into training that already exists, develop and deliver their own programs, or use a hybrid approach specific to their particular workforce needs.

In December 2017, the California Workforce Development Board executed a contract with the UCLA Labor Center to develop a process evaluation for the High Road Training Partnerships Initiative, documenting the experience of HRTP project partners and monitoring their successes and challenges as they work to address economic inequality, climate change, and job quality utilizing a regional workforce skills infrastructure and labor management partnerships. The contract was subsequently extended in April 2020.

The UCLA evaluation is qualitative in nature and primarily utilizes interviews and stakeholder engagement to develop rich contextual case study descriptions pertaining to the manner in which High Road Training Partnerships operate.  To date, the evaluator has drafted eight case studies covering workforce partnership activities in the following industry sectors: Logistics, Healthcare, Hospitality, Janitorial, Ports, Transit, Public Service, and Water Utilities. A review of these case studies suggests the following:

• Intermediaries are key to the development of Industry partnerships
• Skills training must be informed by the subject matter expertise of industry leaders
• Disadvantaged communities can access good jobs through the building of a regional skills infrastructure that provides the proper program supports to those receiving training

Work for this evaluation is ongoing and is expected to be complete in Spring 2021 though information gleaned from the case studies already informs the state’s strategies for program development. This is addressed in the narrative that follows the descriptions of assessments.

Evaluation of Regional Plan Implementation

Governor’s Discretionary Fund Grant Programs 2.0 and 3.0

(Initiated February 2019, Anticipated Completion December 2021)

Since the first WIOA State Plan, California has emphasized the need for workforce programs to collaborate regionally to align programs and services with the needs of regional industry sectors driving economic development and job growth. The CWDB has organized the state’s various Local Workforce Development Boards into Regional Planning Units (RPUs) and has required LWDBs in RPUs to develop regional plans explaining how local services and operations are articulated with regional labor market dynamics. Regional efforts under WIOA are an outgrowth of regional initiatives undertaken as part of the SlingShot Initiative.  The CWDB has supported these efforts by investing discretionary funds in local efforts to implement regional plans.

In May of 2019, the California Workforce Development Board executed a contract with the Corporation for a Skilled Workforce and the Council for Adult and Experiential Learning (CAEL) to provide an assessment and evaluation of State efforts to build regional partnerships and support the implementation of WIOA Regional Plans using discretionary funds. This contract was updated in May of 2020 to ensure that activities for two separate RPU grant periods were included in the scope of work.

The RPI evaluation is designed to take a formative approach, informed by elements of developmental evaluation, to the implementation of regional planning across the fourteen workforce development regions. The goal is to understand the ways each part of the state is approaching regional work and how their efforts to date align with CWDB’s overall vision. By understanding the implementation process of each region, CSW hopes to identify promising practices being used in various regions as well as gaps or challenges in implementing regional visions.

Qualitative research methods are being used to identify common elements of regional work across the fourteen diverse regions. Data collection activities involve the participation of lead project staff, decision makers and regional partners. The objective is to develop a clear picture of the state’s vision for regions by doing all of the following:

• Understanding the policy rationale for regionalism, including the statutory framework and the state plan vision
• Conducting interviews and focus groups with local workforce board directors, regional leadership, regional organizers (ROs) and training coordinators (RTCs), technical assistance providers and relevant project stakeholders and staff to develop a detailed and contextually rich examination of the regions, focusing on successful initiatives and lessons learned
• Conducting site visits to leverage engagement with regional partnership stakeholders, including (potentially) employers and other community partners
• Developing a synthesis and analysis of the qualitative data surfaced to develop recommendations on how to support regions in the future, ensuring connection between regional strategic thinking and local service delivery

Work for this evaluation is ongoing and is expected to be complete at the end of 2021 though preliminary findings from the evaluators suggest the following:

• Tracking the success of regional efforts requires thoughtful approaches to the way regional plan implementation is measured
• Current process measures for tracking the progress of regional efforts developed (in response to the SlingShot evaluation) need to be simplified, rethought, and supplemented with quantitative outcome measures more clearly linked to program enrollments, services, and participant outcomes

The implications of these findings for California’s program strategy are discussed in the final section of this narrative.

Evaluation of AB 1111 State Funded Grant Program

(Initiated July 2019, Anticipated Completion Fall 2021)

The AB 1111 Grant program is a state funded grant program designed to expand the reach of the state’s workforce system by building partnerships between Local Workforce Development Boards (LWDBs) and Community Based Organizations (CBOs) that provide services to individuals with barriers to employment. The design of the AB 1111 program has been informed by the experiences of LWDBs and CBOs who have participated in the Accelerator grant program.  Under the grant program, LWBD-CBO partnerships are intended to harness the subject matter expertise of CBOs who have roots in relevant communities or who have specialized knowledge pertaining to service delivery for target populations so as to utilize CBO expertise and facilitate greater access to the workforce system. The program also allows LWDBs and CBOs to initiate services using state money before enrolling participants into WIOA, addressing performance related disincentives to providing services to California’s hardest to serve.

In November 2019, the California Workforce Development Board announced it had awarded the Corporation for a Skilled Workforce and Social Policy Research Associates a contract to provide an analysis of the outcomes of the individuals served by the initiative, an analysis of the effectiveness of grantee programs’ service delivery and design, and an analysis of the initiative’s impact on the workforce system as a whole. Research questions that are the focus of this evaluation include the following:

• To what extent do Initiative participants (overall, by target group, and by grantee) complete their programming funded under the grant?
• To what extent do Initiative participants (overall, by target group, and by grantee) transition into or become integrated into the broader workforce and education system as evidenced by enrollment in these programs (WIOA Titles I & II; Adult Education Block Grant; community colleges; vocational rehabilitation)?
• To what extent do grantees achieve their performance goals for serving specified numbers of target group members? For transitioning those participants into the broader workforce and education system? For supporting those participants so that they complete their services and attain success on program performance metrics? If grantees do not achieve these goals, why not?
• What are the key service delivery and design elements (whether proven or newly developed) of the most successful grantee programs (as evidenced by grantees’ success in achieving their own performance goals and by the outcomes of their participants)? How do these vary by target group? By geographic context?
• To what extent do grantee programs supplement and align workforce system services with services provided by community-based organizations (CBOs) to build the capacity of the workforce system to better serve individuals with barriers?
• To what extent do grantee programs successfully integrate individuals from target populations into Regional Planning Unit (RPU) career pathways programs in target industry sectors as specified in Regional Plans?

Quantitative and qualitative research methods employed for this analysis include the following: background analysis of grant application documents and quarterly grantee narrative reports; review of related local and regional workforce plans; interviews and surveys of Local Workforce Development Board staff; interviews and surveys of CBO staff; quantitative analysis of related administrative and performance data. The contract is pending final execution and work is expected to be completed by Fall 2021.  There are no results to report at this time.

Evaluation of America’s Job Centers of California

(Initiated Summer 2019, Anticipated Completion December 2021)

In February 2020, the California Workforce Development Board announced it had awarded the Corporation for a Skilled Workforce and the Ray Marshall Center at the LBJ School of Public Affairs (at the University of Texas, Austin) a contract to analyze how different operational practices in the America’s Job Centers of California (AJCCs) affect the distribution of services provided to WIOA Title I program participants. The objective of this evaluation is to determine whether local decisions about how to operate AJCCs systematically affects access to the different types of services available to program participants.

• Are some operational and service delivery models associated with a higher propensity to provide some services rather than others? 
• Does the “mix” of services clients receive have any systematic relationship to the labor market outcomes of program participants? 

This evaluation is designed to utilize surveys, site visits, interviews, focus groups, and a quantitative analysis of services delivered during the period July 2015 to December 2018. The CWDB is currently working with EDD to identify the specifications for the data file to be used for quantitative analyses and the time horizon for which wage and employment data will be used to measure outcomes, but outcome data will likely run through December 2019, allowing the measurement of employment and wage outcomes four quarters post-exit. Data on participants’ pre-program earnings is expected to also be available, and will provide appropriate benchmarks for gauging wage gains. The contract is pending final execution and work is expected to be completed by December 2021.  There are no results to report at this time.

Overview of the CAAL-Skills Data Repository and Related Evaluation Activities

As part of its efforts to facilitate comprehensive program evaluation, the CWDB launched the Cross-System Analytics and Assessment for Learning and Skills Attainment (CAAL-Skills) workforce data initiative in August of 2016. Development and implementation of the initiative is continuously ongoing and aims to fulfil the following objectives:

• Compliance with the California Workforce Innovation and Opportunity Act which requires that the state’s workforce system be data-driven and evidence-based
• Compliance with Assembly Bill No. 2148 (K. Mullin, Chapter 385, Statutes of 2014) and Assembly Bill No. 1336 (K. Mullin, Chapter 211, Statutes of 2017) which requires:
  • the development and display of a workforce metrics dashboard
  • submission of periodic legislative reports on workforce program participant labor market outcomes
  • periodic completion of statistically rigorous evaluations of the labor market impacts of workforce programs
• Compliance with WIOA SEC. 116 (e) program requirements pertaining to statistically rigorous evaluation and assessment of WIOA programs
• Development of customized data reports and analyses based on participating workforce program research needs

CAAL-Skills is an ongoing workforce data-initiative that merges administrative data from participating data-sharing partners in a data repository housed at the CWDB. Participating programs are party to data-sharing agreements that identify how participating program data may be used. Programs currently participating in CAAL-Skills include the following:

• WIOA Title I Adult  
• WIOA Title I Dislocated Worker
• WIOA Title I Youth  
• WIOA Title II
• WIOA Title III Wagner-Peyser
• WIOA Title IV
• CalWORKS (California’s TANF program)
• CalFresh (California’s SNAP E&T program)
• Trade Adjustment Assistance
• Incumbent Worker Training accessed through the California’s Employment Training Panel program
• State Certified Apprenticeship
• Career Technical Education programs at the California Community Colleges
• Rehabilitative programs at the California Department of Corrections and Rehabilitation and the Prison Industry Authority

CAAL-Skills Evaluation and Assessment Activities

July 1, 2018 to June 30, 2020

The following is an itemized overview of CAAL-Skills evaluation and assessment activities that occurred during the two year window between July 1, 2018 and June 30, 2020.

CAAL-Skills Workforce Metrics Dashboard 2020 Report

(Initiated Spring 2017, Anticipated Completion Fall 2020)

The 2020 Workforce Metrics Dashboard Report summarizes and graphically displays credential attainment, employment, and wage data for WIOA Core Programs, CalWORKS (California’s TANF program), the Trade Adjustment Assistance program, the Employment Training Panel’s State-Funded Incumbent Worker Training program, State Certified Apprenticeships, and the California Community College system’s Career Technical Education programs.

Once complete the dashboard report will provide descriptive statistics on aggregate outcomes for individuals who exited participating programs in State Fiscal Years 2014-15 and 2015-16 (which run from July 1 to June 30 each year).

For most programs the data will be displayed using a standard set of tables and graphs that organizes credential attainment and labor market outcome data by race, ethnicity, gender, age, veteran status, regional planning unit of service providers, type of services received, type of training received, training completion status, and type of credential received.  Standard table and graphs also present information on post-exit industry sector of employment, and greater detail on the dispersion of wages for program exiters using box-and-whisker plots.

Draft Tables and accompanying narrative for the 2020 Workforce Metrics Dashboard Report are not yet complete though a summary of some preliminary findings relevant to current program strategies is briefly discussed at the end of this narrative.

CAAL-Skills Workforce Metrics Dashboard 2022 Report

(Initiated Summer 2019, Anticipated Completion Fall 2022)

The 2022 Workforce Metrics Dashboard Report will summarize and graphically display credential attainment, employment, and wage data for WIOA Core Programs, CalWORKS (California’s TANF program), CalFresh (California’s SNAP E&T program), CalWORKS (California’s TANF program), the Trade Adjustment Assistance program, the Employment Training Panel’s State-Funded Incumbent Worker Training program, State Certified Apprenticeships, the California Community College system’s Career Technical Education programs, as well as Rehabilitative programs operated by the California Department of Corrections and Rehabilitation, and the Prison Industry Authority.

Once complete the dashboard report will provide descriptive statistics on aggregate outcomes for individuals who exited participating programs in State Fiscal Years 2016-17 and 2017-18 (which run from July 1 to June 30 each year).

For most programs the data will be displayed using a standard set of tables and graphs that organizes credential attainment and labor market outcome data by race, ethnicity, gender, age, veteran status, regional planning unit, type of services received, type of training received, training completion status, and type of credential received.  Standard tables and graphs also will present information on post-exit industry sector of employment, and greater detail on the dispersion of wages for program exiters using histograms and box-and-whisker plots.

Draft Tables and accompanying narrative for the 2022 Workforce Metrics Dashboard Report have yet to be developed as the CAAL-Skills technical team is still receiving, merging, cleaning, and organizing the 2016-17 and 2017-18 administrative data of participating programs.

UC California Policy Lab analysis of CAAL-Skills Data

(Initiated Summer 2018, Anticipated Completion August 2021)

In March 2019, the California Workforce Development Board awarded the California Policy Lab (U.C. Regents) a contract to analyze CAAL-Skills de-identified program participant labor market outcome data for individuals who exited participating workforce programs in State Fiscal Years 2014-15 and 2015-16.

The California Policy Lab partners with California’s state and local governments to generate scientific evidence that solves California’s most urgent problems, including homelessness, poverty, crime, and education inequality. CPL builds close working partnerships between policymakers and researchers at UCLA and UC Berkeley to help evaluate and improve public programs through empirical research and technical assistance.

CPL’s lead researchers are Til von Wachter and Jesse Rothstein.  Till von Wachter is a Professor of Economics at the University of California Los Angeles, Faculty Director of the California Policy Lab, Director of the Federal Statistical Research Data Center, and Associate Dean for Research for the Social Science Division. Prof. von Wachter has been an expert witness in numerous testimonies before committees of U.S. Congress, and has provided expert assistance to the City and County of Los Angeles, the U.S. Department of Labor, the Canadian Labor Ministry, the OECD, the United Nations, and the IMF. Jesse Rothstein is Professor in the Goldman School of Public Policy and the Department of Economics, and Director of the Institute for Research on Labor and Employment at UC Berkeley. He served as Chief Economist at the U.S. Department of Labor and as Senior Economist with the Council of Economic Advisers, Executive Office of the President under the Obama Administration.

Under the research contract CPL’s scope of work includes the following:

• data assessment and written recommendations regarding data sources and/or data documentation
• development of a written report of recommended statistical analyses 
• work with CWDB to develop consensus among program partners for the proposed analyses
• conducting the agreed upon analyses 
• develop draft report
• review of findings with programs and conducting follow up analyses 
• finalization of findings

The CAAL-Skills data file to be analyzed was delivered to the UC Policy Lab in February of 2020 and includes information on all the variables used to produce CAAL-Skills Workforce Metrics Dashboard Reports. Importantly, the data file has been augmented to include ten years of labor market data, running from 2009 to 2019, for workforce program participants included in the study, allowing the researchers to construct appropriate control groups and utilize longer time horizons when examining the labor market impacts of participating workforce programs.

Programs included in the study are as follows: the WIOA Core Programs, CalWORKS (California’s TANF program), CalFresh (California’s SNAP E&T), the Trade Adjustment Assistance program, the Employment Training Panel’s State-Funded Incumbent Worker Training program, State Certified Apprenticeships, and the California Community College system’s Career Technical Education programs.

Data analysis is ongoing and as of yet, there are no preliminary or final research results to report.

Description of Substantive Findings and Policy Strategy

Many of the results of the evaluations and assessments listed above are incomplete or preliminary as most of the evaluation activities detailed above are ongoing, meaning that final work products have not been developed or vetted for distribution. 

Additionally, it is important to note that the State is hesitant to draw concrete policy conclusions from the outcome data available in the draft performance tables available through the CAAL-Skills workforce data initiative and the CAAL Skills Workforce Dashboard 2020 Report.

Proper program evaluation requires the use of sophisticated statistical methods to control for factors other than participant services when explaining individual labor market outcomes. This is the reason California has entered into a partnership with the California Policy Lab (UC Regents) to conduct sophisticated statistical analyses of the labor market impacts of workforce programs on participants post program exit.

Keeping these caveats in mind, the CWDB is aware of preliminary data findings suggesting better wage and employment outcomes for those receiving training services and earning a credential than for those who don’t receive training and who don’t receive a credential. Preliminary CAAL-Skills outcomes Tables suggest that:

• RPUs with a higher propensity to train WIOA Title I participants have higher post-exit wages and employment rates for program participants, even in more rural areas of the state that are less economically dynamic
• Individuals that complete WIOA Title I training programs earn more and are more likely to be employed than participants that do not complete training programs
• More intensive training programs and those that directly take into account industry needs, result in higher levels of employment and greater wages for WIOA Title I program participants
• Credential attainment is linked to improved labor market outcomes for program participants
• The type of credential a participant receives matters in terms of likelihood of employment and earnings, with more intensive training and education programs leading to better outcomes

These findings are consistent with an ever-expanding body of literature that links social stratification to levels of human capital development and educational attainment as well as with other CAAL-Skills program data that show linkages between levels of program participant skills attainment, higher wages, and better employment outcomes, especially for those programs that build skills through OJT and Apprenticeship.

In addition, some preliminary CAAL Skills data suggests that supportive services lead to higher completion rates and credential attainment for TANF recipients enrolled in career education programs at the California Community Colleges. Given the starting disadvantages of TANF recipients in relation to the more general population of CTE participants, this finding seems to strongly support the value of supportive services in facilitating program completion.

In sum, the CAAL-Skills data suggests that training is a good means to improve employment outcomes for workforce program participants. This suggests that investment in appropriate training programs (directly related to regional labor market dynamics) and the provision of supportive services to ensure training program participants complete their training and earn an industry recognized credential are important means for fighting poverty.

Consistent with the foregoing, the State will continue to invest in and emphasize policies, programs, and grants that foster human capital development by upskilling and reskilling program participants in a manner consistent with industry needs.  Such policies and investments include sector focused job training and continued investment in High Road Training Partnerships. Similarly, the State will continue to invest in regional initiatives that build regional skills infrastructure while also encouraging broad and inclusive local and regional partnerships focused on customer-centered service delivery.

Here it is important to also point out that findings and recommendations from the SlingShot, Accelerator, HRTP, and Regional Plan evaluations continue to inform the way California approaches policy strategy and operations as the State works to build a regional skills infrastructure that links WIOA program participants to good quality jobs through various local service delivery structures, the AJCC system,  and perhaps most importantly, regional training providers who meet the needs of quality employers in industry sectors driving regional economic growth.

Based on the findings and recommendations from the aforementioned assessments, California will do the following:

• Emphasize partnerships and alignment across programs and funding streams, including partnerships between CBOs and programs operating through the AJCCs
• Emphasize industry engagement in the development of HRTPs, OJT programming, apprenticeship, and WIOA Regional Plans
• Develop and fund intermediary activities to bring local and regional partnerships together, giving special emphasis to activities that link local service delivery structures with regional skills infrastructures designed to meet regional workforce needs
• Continue partnership activities between State departments who are part of the workforce system so as to ensure collaboration and collective responsibility for shared participant outcomes
• Invest in regional planning, regional partnership, regional organizing activities and TA support to the regions by state merit staff and other subject matter experts
• Utilize performance metrics to track participant outcomes for grants that fund direct services and subject direct service grant programs to rigorous assessment and evaluation activities (this has already started with the AB 1111 grant program)
• Develop metrics that track skills building investments at the regional level by setting goals and collecting data through the regional planning process and by developing new data reporting requirements for regional planning units awarded regional planning implementation grants
• Evaluate programs and program outcomes over longer time horizons as well as increasing grant periods for services, so as to encourage more attention to the medium and longer term upward mobility of program participants post-exit