Performance Measurement
Staff Briefing
October 20, 1999

Performance Measurement

Section I - Background

Purpose of the Study

The purpose of the study is to identify ways to strengthen and systematize the availability and use of performance measurement information in the General Assembly. It will examine how a performance measurement system should be structured and identify the requirements necessary to ensure the system put in place meets the needs of the General Assembly. This briefing paper provides background information on the concept of performance measurement, national trends, and Connecticut's current efforts and needs.

Definition of Performance Measurement

Performance measurement as used in this study means the systematic measuring of an agency's or program's activities, outputs, and outcomes and their relationship to the agency's or program's stated objectives.

Uses of Performance Measurement Information

Performance measurement data serve many different purposes for a variety of users. They can be useful to agency managers and executive and legislative branch policy-makers for:

The structure of performance measurement systems vary based on whether their primary use is managing agency operations, developing policy, budgeting, or a combination of the three.

Figure I-1 shows the relationship between the scope of the data collected under a performance measurement system and uses of the information. The diagram is adapted from one produced by the Florida legislature's Office of Program Policy Analysis and Government Accountability (OPPAGA) and depicts a comprehensive system in which performance measures are reduced in number and broadened in scope as uses of the data change from decisions affecting agency operations to those dealing with the state budget.

Figure I-1 illustrates that at the highest level of a comprehensive performance measurement system, budget allocation, the data used to make decisions involves a few broad measures of efficiency (e.g., cost per unit of output) and effectiveness (e.g., cost per outcome). At the policy level, the focus is on a limited number of key output (e.g., unemployed individuals receiving job training, miles of road paved, criminals incarcerated, etc.) and outcome (e.g., percent of job training graduates employed in field of training, highway roughness index rating, crime rate, etc.) measures. At the agency level, represented by the bottom tier of the pyramid, managers analyze a wide array of input, output, and outcome indicators to fine-tune operations and reallocate resources.

Basic Elements of a Performance Measurement System

There is no single structure for a performance measurement system that meets every organization's needs. However, there are basic elements that should be present in any good system. These include:

Figure I-2 shows a model performance measurement system in terms of a process in which the elements identified above are connected in a sequence of steps. The process begins with a plan, moves through a data collection and analysis phase, and ends with decisions being made based on an assessment of quantitative data.

Performance Measurement Models

There are two basic performance measurement models. In this study, one is labeled the special review model. Under this approach data on government operations are gathered from all available sources and analyzed by a staff reporting to a central authority. The central authority, typically a special commission, is empowered to make recommendations. The other model is commonly referred to as performance based budgeting. It involves the systematic identification and collection of performance data, which are then incorporated into the regular budget process.

It should be noted neither model is rigidly structured. Both offer opportunities for variation, particularly the performance based budget model.

Other States

The special review model has been in existence for many years. It is frequently used in periods of budgetary crisis. Indeed, as will be discussed in the next section, Connecticut has used this model on several occasions. Information on the model’s use in other states is limited, however, due to its highly localized focus and temporary nature. At least one state, Texas, did attempt to institutionalize this approach.

Texas Performance Review. In 1991, Texas enacted legislation requiring the examination of the organization, management, and programs of the entire state government. The process was called the Texas Performance Review (TPR). The authority to carry out the review was given to the nonpartisan Legislative Budget Board. The board designated the state comptroller as the project leader.

During the initial round, the comptroller was supported by a staff of 100 auditors, research analysts, and other specialists from state agencies and the private sector. The methodology followed included traditional staff work such as research documents, interviews, and surveys. It also included a variety of other means such as soliciting information from state employees and the general public using toll-free hotlines, public hearings, and press releases.

According to the comptroller, the initial project produced over 200 recommendations and a projected savings of $4 billion. After legislative debate, about two-thirds of the recommendations, accounting for $2.4 billion in savings, were adopted. At the completion of the initial project, the legislature authorized the comptroller to continue TPR on a permanent basis.

In subsequent years the comptroller continued to issue reports with hundreds of recommendations and billions of dollars in projected savings or revenue increases. The comptroller reported between 85 and 90 percent of the recommendations and associated savings were adopted.

Recently, after the sitting comptroller left office, the TPR project was modified. The comptroller’s review of all state government operations has been replaced with a focus on auditing education districts. A citizens council has been formed to address other aspects of the government in a manner similar to the way the Thomas Commission reviewed the Connecticut's state government.

Performance based budgeting. As Table I-1 shows, many states are moving toward a the performance based budget model. The table identifies 31 states that in 1996 indicated an intention to implement some form of a performance measurement system. The table names 28 states that included such performance data in their budgets, 12 of which are already using the data to a some degree as a policy tool. Connecticut was one of only five states reporting it did not have performance measures.

As Table I-1 shows, most of the states involved in the performance budgeting process are early in the implementation phase. Presented below is more detailed information on three states recognized as leaders in performance based budgeting.

Texas In addition to undertaking activities relevant to the special review model, Texas also has pursued the performance based budget model. Texas requires each agency to have a strategic plan that includes goals, objectives, and performance measures. State agencies are required to report performance data to the nonpartisan Legislative Budget Board on a quarterly basis. The state auditor reviews the data for validity and reliability. The board’s staff analyzes each agency’s performance based on the data. The staff's analysis is added to the information used in the budget decision-making process.

Florida. In Florida, performance budgeting is being phased in over seven years, which began in 1994. As summarized at the National Legislative Program Evaluation Section's 1999 annual meeting, the Florida process consists of four steps:

1. agencies propose programs and measures;
2. governor's office includes this information in the budget recommendation;
3. legislature incorporates the information into the budget recommendation; and
4. after one year, the legislature's Office of Policy Analysis and Governmental Accountability conducts a study to evaluate performance.

Minnesota. In Minnesota, state agencies biennially submit performance reports showing progress on specific goals. The measures contained in the reports are developed with the help of the state's budget and administrative departments. The measures are reviewed for reliability, validity, and usefulness by the Office of the Legislative Auditors (OLA). The auditors are required by law to comment on the reports. The performance reports are not directly linked to the budget process, but are intended to provide the legislature with information to facilitate policy and budget discussions.

Section II - Performance Measurement in Connecticut

Statutory Requirements for Performance Measurement

Over time the Connecticut General Assembly has introduced, modified, and attempted to strengthen various aspects of financial and programmatic oversight. Much of this attention has focused on the Auditors of Public Accounts and the Program Review and Investigations Committee. Each office plays a key role in aiding the legislature to meet its oversight responsibility. In recent years, the focus has shifted away from modifying these two special function offices toward developing a system to ensure performance data are integrated into the legislature’s budget and policy decision-making processes.

The legislature’s attention has been on creating a process wherein performance measurement data will be identified and flow to decision-makers. As a result, Connecticut has statutes requiring such things as: the development of a long-range vision for the state; agency strategic plans; agency performance measures; and progress reports.

This section outlines what these mandates have been, what entities have been charged with carrying out the mandates, and what the results have been.

State Entities Responsible for Performance Measurement

Table II-1 identifies entities having a role in the operation of the state's performance measurement system. Each entity's responsibilities with respect to the process outlined in Figure I-2 (i.e., step 1 -- planning, step 2 -- identification of measures, step 3 -- collection of data, and step 4 -- analyzing data) are classified in the column labeled "Roles". Excluded from the table are step 5 -- decision-making operations not referenced by the state statutes in a performance measurement context. Following Table II-1 is a slightly more detailed description of the entities named in the table.

Connecticut Progress Council. The council was created by PA 93-387 to develop a long-range vision for the state and to define benchmarks to measure the state's progress. The council is composed of 28 members including legislators and representatives of the executive branch and the private sector. It is authorized to draw personnel and other resources from state agencies and the nonpartisan offices of the legislature. In its 1995 report - - the only one issued to date - - the council identified 300 benchmarks to be used to measure progress. Examples of the measures include incidents of child abuse and neglect per 1,000 population, violent crimes per 100,000 population, percentage of students at or above goal on Connecticut Mastery Test, etc.

Office of Policy and Management. In addition to preparing the governor’s budget proposal and implementing and monitoring the execution of the budget, OPM has a statutory mandate to determine the effectiveness of the policies, management, organization, operating procedures, and services of state agencies and to recommend improvements (C.S.G. 4-66). Primary responsibility for this charge resides with OPM's Strategic Management Division, which serves as a management consultant to state agencies. The division consists of two groups totaling 19 members. The Energy Group has a staff of 12. The Performance Evaluation Group has 7 staff, two of which are assigned to agencies develop strategic business plans, develop performance measures, and evaluate programs. The division indicates it conducts about 10 performance reviews annually.

Auditors of Public Accounts. The primary responsibility of the auditor's office is to determine whether state agencies are in compliance with state and federal financial requirements. The statutes also give the state auditors authority to examine an agency's performance to determine its effectiveness in achieving an expressed legislative purpose (C.S.G. 2-90(c)). Using this power the state auditors have included a performance evaluation component in selected audits and established a Performance Audit Team, which can be assigned to review specific state programs. Such assignments typically are an outgrowth of a financial audit. The performance audit team has a staff of five and completes about two or three reviews per year. Its reports are issued separately from financial audits. In conducting the reviews, the team assesses the objective of the program, determines program results, identifies factors inhibiting performance, assesses compliance with laws and regulations, evaluates management oversight, and recommends program improvements.

Legislative Program Review and Investigations Committee. The committee is a bipartisan, statutory committee of the Connecticut General Assembly. It was established in 1972 to evaluate the efficiency, effectiveness, and statutory compliance of selected state agencies and programs and recommend remedies where needed. The committee is served by a staff of 12 including a staff director, 10 analysts, and an administrator. It completes six to eight reviews per year.

Office of Fiscal Analysis. The Office of Fiscal Analysis is a nonpartisan professional office of the Connecticut General Assembly. Its primary function is to provide technical support to the Committee on Appropriations and the Committee on Finance, Revenue and Bonding, as well as theother committees and members of the legislature. The office is served by a staff of 25 including a director and four administrators. Among the duties of the office relevant to performance measurement are: analyzing budget requests; assisting in the development of means by which budgeted programs can be periodically reviewed; preparing short analyses of the costs and long-range projections of executive programs and proposed agency regulations; analyzing and preparing critiques of the governor's proposed budget; studying selected executive programs during the interim; and preparing fiscal notes upon favorably reported bills that require the expenditure of state or municipal funds or affect state or municipal revenue.

Office of the State Comptroller. The Office of the State Comptroller (OSC) maintains the state's computer-based accounting system and is the primary producer of comprehensive financial information for the state. OSC's functions include providing accounting and financial services, developing accounting policy, preparing financial reports, and analyzing and compiling agency and state-level management information. Among relevant performance measurement activities carried out by the comptroller's office are reviews of reports of the Auditors of Public Accounts aimed at ensuring agency compliance with the auditor's recommendations and conducting independent audits under OSC's performance review program.

 Performance Study Commissions

In the past, when the state has experienced severe and widespread financial difficulties, it has created special commissions to study the management and structure of the state's government and recommend changes. There have been four such commission's created over the past 30 years. The first of these was the 1971 Governor's Commission on Services and Expenditures (Etherington). This was followed in order by the 1976 Committee on the Structure of State Government (Filer), the 1991 Commission to Study the Management of State Government (Thomas), and the 1992 Commission to Effect Government Reorganization (Harper-Hull).

Each commission was set up as a temporary body charged with reviewing government operations and identifying ways to: reduce costs; improve management; and make the delivery of state services more effective and efficient. All of the commissions recommended reductions in the number of state agencies and elimination of those state services deemed unnecessary or duplicative. Of note, every commission made recommendations of one type or another aimed at improving the availability of information to executive and legislative decision-makers for analyzing and managing state government operations.

Combined, the four commissions made over 2,000 recommendations. Although systematic follow-up studies are absent, there is anecdotal information that many of the recommendations were implemented either through executive directive or legislative action. However, as the Connecticut Public Expenditures Council discussed in a paper reviewing the work of the first three commissions, the positions taken on a recommendations by special interest groups generally had more influence on the fate of proposals than evidence of a cost reduction or system improvement.

Compliance

In a July 1994 report to the General Assembly, OPM outlined efforts undertaken by the executive branch to comply with the statutory requirements for agency performance measures (P.A. 92-8, May Special Session). The report indicated it would take several years to develop a meaningful performance measurement system, which according to OPM meant the measures would be used in making decisions about program activities, policies, and budget priorities. The report included preliminary measures for 21 state agencies.

In a March 1996 memo to the chairs of the General Assembly’s Appropriations and Planning and Development Committees, OPM indicated it was working with state agencies to institute a business planning concept as a means of linking the state's benchmark project with "specific state agency goals, objectives and their related performance measures." OPM noted the Department of Revenue Services and the Department of Children and Families were already involved in a pilot test of this approach. (The memo was in response to P.A. 95-232, which required OPM to submit a plan for the use of benchmarks in the development of the budget.)

In September 1998, OPM issued a set of guidelines for developing an agency strategic plan and performance indicators. In the preface to the guidelines, it was noted eight agencies had already developed their first business plan. The OPM web site has links to guidelines used in selected other states to help their agencies develop and implement performance measurement systems.

Overall, OPM has taken steps to provide state agencies with information and guidance on the steps necessary to develop a performance measurement system. Unfortunately, the documentation committee staff obtained so far indicates much of the effort has been aimed at starting and restarting the process at the agency level. The statutory mandates, as spelled out in sections 4-67m, 4-67r and 4-73 and discussed above, have not yet been met. Complicating matters, the progress council failed to update its benchmarks as required by C.S.G. Sec.4-67r and, for all practical purposes, has stopped functioning.

In an effort to respond to legislative inquiries, OPM is in the process of surveying state agencies to determine the status of their performance measurement systems. A detailed analysis of the responses is expected to be completed in fall of 1999.

Section III - Assessment of Connecticut's System and Future Directions

Recent Assessments

There have been two reviews of Connecticut's existing performance measurement system. One was done for the State Comptroller's Office by a private consulting firm. The other study was undertaken by a special legislative task force. Each study is summarized below as is a preliminary analysis of the state's performance measurement system by the program review committee staff.

Consultant's report. In January 1998, Deloitte & Touche LLP submitted a report to the State Comptroller's Office containing recommendations for improving the way the state collects management information and makes the data available for analysis. The report indicated Connecticut's performance measurement system contributed little of value to:

Deloitte & Touche LLP attributed this in part to a budget process that allocates and accounts for funds on a line-item basis. The report indicated this process contributes to a commonly held view among state agency managers that the collection and reporting of performance information is a "bother," and the results are widely ignored by executive and legislative branch budget analysts.

The report notes most of the performance measures developed in response to state statutory requirements are simple indicators of workload and output. The consultants expressed an opinion such measures are of little value in assessing efficiency and effectiveness.

All four of the consultant's key recommendations addressed ways to improve the state's management information, and one dealt specifically with performance measurement. This recommendation calls for Connecticut to:

Legislative Task Force. As an outgrowth of legislative interest in the connection between oversight and performance based budgeting, a legislative task force studied the issue between the 1997 and 1998 sessions of the General Assembly. It took testimony from the heads of relevant Connecticut agencies and staff from key performance review agencies in Texas and Florida. In addition, the task force examined documents describing the oversight and monitoring efforts being made in Connecticut and other states. In February 1998 the task force issued its report.

Task Force Findings:

Task Force Recommendations:

Preliminary staff analysis. The program review committee staff conducted its own analysis of the state's performance measurement system. This effort focused on comparing the status of the state's activities with respect to the requirements of a good performance measurement system referenced in the steps shown in Figure I-2. The results of the comparison are outlined in Table III-1.

The table above reflects the staff’s opinion that existing state statutes meet the basic requirements of a good performance measurement system -- agency planning and the identification of performance measures. The table highlights the areas where the staff believes the state’s performance measurement system is not working well. It also shows the importance the staff attaches to the need for the executive and legislative branches to commit to performance measurement, if the approach is to be successful.

External Factors Influencing Connecticut

It should be noted there are factors pushing Connecticut to implement a performance measurement system. In particular, two events external to state government are likely to put pressure on the state to implement a performance measurement system. One involves new requirements that may be tied to taking federal money. The other concerns probable changes in national accounting standards.

Discussion Options

A number of different methods could be used to address the needs identified in Table III-1. Based on an assessment of current and previous efforts in Connecticut and activities in other states, the committee staff believes if the state wants to systematize its approach to evaluating performance it should move in one of two directions.

One path leads toward creating a mechanism external to the executive and legislative branches to review the performance of governmental operations. The other route moves toward establishing a performance measurement process within one or both branches of government. Regardless of the direction chosen, there are several options available under each approach. Below, are outlines of a few of the basic alternatives available for consideration under either of the major directions selected.

External approaches. The first set of options, which follow, deal with assigning responsibility for performance measurement to an entity outside of the direct control of either the executive and legislative branches.

Internal approaches. The second group of options involve making changes within the existing governmental structure. Specifically, these options differ from each other based on two components: the legislative entity given lead responsibility; and the related authority for assuring the process is carried out. (See Figure I-2) Underlying this set of options is the assumption a commitment to performance measurement on the part of both executive and legislative leaders exists. Without such a commitment OPM will do no better fulfilling its obligations than it currently does, and the legislative resources identified in the options will be subjected to being diverted to other tasks.

The first and second of these options are based on a performance measurement system designed to address policy matters as well as budget concerns. The third option is offered to deal with an approach where the legislature's primary emphasis from the perspective of the legislative branch would be on performance budgeting.

1. Create a new legislative staff office with responsibility to:

2. Expand the mandates and resources of the state auditors or the program review committee and assign responsibility to:

3. Expand the mandate and resources of the legislature’s Office of Fiscal Analysis and reorganize the budget process to reflect a performance-based budgeting system.

Summary. As noted above, all five of the options presented are for discussion purposes and in a preliminary outline format. Details such as how often any component of the government should be reviewed, the recipients of the recommendations, and requirements for action on the recommendations have to be worked out. Of course, the five options do not exhaust all of the possible alternatives, and it is expected additional approaches will surface based on the staff-continuing review, public hearing testimony, and discussion among the committee members. 

Return to Year 1999 Studies