Measuring The Performance Of Government Technology Programs: Lessons From Manufacturing Extension
December 1997
Working Paper Number:
CES-97-18
Abstract
Managers of government technology programs are under increasing pressure to demonstrate the effectiveness of their programs. In this paper we examine the issues involved in credibly evaluating such programs in the context of recent efforts to evaluate manufacturing extension programs in the U.S. We provide a stylized model of the dynamic competitive environment in which the plants and firms targeted by these programs operate and discuss its implications for evaluation. We compare and contrast the various methodologies and data sets used to evaluate manufacturing extension. We conclude that the best currently available method for measuring the overall effectiveness of programs such as manufacturing extension is to combine program administrative data with existing panel data sets.
Document Tags and Keywords
Keywords
Keywords are automatically generated using KeyBERT, a powerful and innovative
keyword extraction tool that utilizes BERT embeddings to ensure high-quality and contextually relevant
keywords.
By analyzing the content of working papers, KeyBERT identifies terms and phrases that capture the essence of the
text, highlighting the most significant topics and trends. This approach not only enhances searchability but
provides connections that go beyond potentially domain-specific author-defined keywords.
:
estimation,
econometric,
manufacturing,
industrial,
agency,
manufacturer,
organizational,
incorporated,
innovation,
competitiveness,
expenditure,
valuation,
policymakers,
econometrician,
policy
Tags
Tags are automatically generated using a pretrained language model from spaCy, which excels at
several tasks, including entity tagging.
The model is able to label words and phrases by part-of-speech,
including "organizations." By filtering for frequent words and phrases labeled as "organizations", papers are
identified to contain references to specific institutions, datasets, and other organizations.
:
Longitudinal Research Database,
Census Bureau Longitudinal Business Database,
Medical Expenditure Panel Survey,
National Research Council,
General Accounting Office
Similar Working Papers
Similarity between working papers are determined by an unsupervised neural
network model
know as Doc2Vec.
Doc2Vec is a model that represents entire documents as fixed-length vectors, allowing for the
capture of semantic meaning in a way that relates to the context of words within the document. The model learns to
associate a unique vector with each document while simultaneously learning word vectors, enabling tasks such as
document classification, clustering, and similarity detection by preserving the order and structure of words. The
document vectors are compared using cosine similarity/distance to determine the most similar working papers.
Papers identified with 🔥 are in the top 20% of similarity.
The 10 most similar working papers to the working paper 'Measuring The Performance Of Government Technology Programs: Lessons From Manufacturing Extension' are listed below in order of similarity.
-
Working PaperUsing Matched Client And Census Data To Evaluate The Performance Of The Manufacturing Extension Partnership🔥
April 1995
Working Paper Number:
CES-95-07
This paper proposes a framework for evaluating the Manufacturing Extension Partnership (MEP). The MEP is administered by the National Institute of Standards and Technology (NIST) as part of its effort to improve the global competitiveness of U.S. manufacturing industries. As the name implies, the MEP is modelled after agricultural extension. Rather than farmers the MEP's target population is small and medium sized manufacturers, generally those with less than 500 employees. The MEP currently supports 44 manufacturing extension centers around the country. These centers provide technical and business assistance for manufacturers much as county extension agents do for farmers. The goal of evaluation is to see if MEP engagements lead to positive outcomes from the view of important MEP stakeholders (e.g., MEP clients, MEP centers, NIST, state and local governments and Congress). These outcomes are discussed in McGuckin and Redman (1995) and include: Process Outcomes (e.g., adoption of a new technology by a client); Intermediate Outcomes (e.g., reduction in the clients defect rate); Business Outcomes (e.g., survival and profits) and Policy Outcomes (increases in employment,wages and/or exports). The evaluation framework described in this paper has two components. The first component is an evaluation dataset which contains measures of many of the program outcomes listed above for both MEP clients and a representative control group of non- clients. This dataset will be constructed by linking MEP client records with plant level Census data housed at the Center for Economic Studies of the Census Bureau. The Census data provides measures of several outcome and control variables which are comparable across both plants and time. The Census data include observations for all manufacturing plants in the U.S. from which representative control groups can be constructed. The MEP client records provide data on the type and intensity of extension engagements. Linking these rich sources of information yields a comprehensive and powerful dataset for MEP evaluation. The second component is an evaluation methodology which exploits this rich dataset to make statistical inferences about the impact of MEP services, while carefully controlling for other influences. By using this methodology, we can address many of the shortcomings which plagued previous attempts to evaluate extension services. In addition to evaluation, the dataset described in this paper may be used to profile the characteristics of MEP clients and compare them to non-clients. The Census data contain the complete universe of manufacturing establishments in the U.S.View Full Paper PDF
-
Working PaperManufacturing Extension And Productivity Dynamics🔥
June 1998
Working Paper Number:
CES-98-08
This paper presents results from an investigation of the effects of manufacturing extension on the productivity dynamics of client plants. Previous econometric studies of manufacturing extension had very little time series information. This limited what researchers could say about the relative timing of extension services and performance improvements. In turn, this makes it difficult to attribute performance improvements to the receipt of extension services. In this paper, I use a panel of client and nonclient plants to more carefully analyze the dynamics of extension and productivity. The results suggest that the timing of observed productivity improvements at client plants is consistent with a positive impact of manufacturing extension. Estimated program impacts are within the range of those found in previous studies.View Full Paper PDF
-
Working PaperGOVERNMENT TECHNICAL ASSISTANCE PROGRAMS* AND PLANT SURVIVAL: THE ROLE OF PLANT OWNERSHIP TYPE🔥
February 1999
Working Paper Number:
CES-99-02
This paper compares the survival rates of plants participating in manufacturing extension programs to nonparticipating plants. Participating plants receive technical and business assistance from one of a nationwide network of extension centers intended to assist smaller manufacturers. Results suggest that plant survival is related to plant size, age, productivity, capital intensity and ownership type. Importantly, the impact of extension services differs across ownership types. Participating in extension increases the probability of survival for single unit plants, but not for multi units. This result is consistent with the notion that single unit plants have less access to information on new technologies and would, therefore, benefit more from technical assistance programs such as manufacturing extension.View Full Paper PDF
-
Working PaperMeasuring the Impact of the Manufacturing Extension Partnership
September 1996
Working Paper Number:
CES-96-08
In this paper, I measure the impact of the Manufacturing Extension Partnership (MEP) on productivity and sales growth at manufacturing plants. To do this, I match MEP client data to the Census Bureau's Longitudinal Research Database (LRD). The LRD contains data for all manufacturing establishments in the U.S. and provides a number of measures of plant performance and characteristics that are measured consistently across plants and time. This facilitates valid comparisons between both client and non-client plants and among clients served by different MEP centers. The National Institute of Standards and Technology (NIST) administers the MEP as part of their effort to improve the competitiveness of U.S. manufacturing. The program provides business and technical assistance to small and medium sized manufacturers much as agricultural extension does for farmers. The goal of the paper is to see if measures of plant performance (e.g., productivity and sales growth) are systematically related to participation in the MEP, while controlling for other factors that are known or thought to influence performance. Selection bias is often a problem in evaluation studies so I specify an econometric model that controls for selection. I estimate the model with data from 8 manufacturing extension centers in 2 states. The control group includes all plants from each state in the LRD. Preliminary results indicate that MEP participation is systematically related to productivity growth but not to sales growth.View Full Paper PDF
-
Working PaperEvaluating the Impact of MEP Services on Establishment Performance: A Preliminary Empirical Investigation
July 2012
Working Paper Number:
CES-12-15
This work examines the impact of manufacturing extension services on establishment productivity. It builds on an earlier study conducted by Jarmin in the 1990s, by matching the Census of Manufacturers (CMF) with the Manufacturing Extension Partnership (MEP) customer and activity datasets to generate treatment and comparison groups for analysis. The scope of the study is the period 1997 to 2002, which was a period of economic downturn in the manufacturing sector and budgetary challenges for the MEP. The paper presents some preliminary findings from this analysis. Both lagged dependent variable (LDV) and difference in difference (DiD) models are employed to estimate the relationship between manufacturing extension and labor productivity. The results presented are inconclusive and paint a mixed picture as they demonstrate the benefits and limitations of using Census microdata in program evaluation. They also point to the need to conduct analyses that could help to better understand the dynamic impact of MEP services.View Full Paper PDF
-
Working PaperEvaluating the Long-Term Effect of NIST MEP Services on Establishment Performance
March 2015
Working Paper Number:
CES-15-09
This work examines the effects of receipt of business assistance services from the Manufacturing Extension Partnership (MEP) on manufacturing establishment performance. Several measures of performance are considered: (1) change in value-added per employee (a measure of productivity); (2) change in sales per worker; (3) change in employment; and (4) establishment survival. To analyze these relationships, we merged program records from the MEP's client and project information files with administrative records from the Census of Manufacturers and other Census databases over the periods 1997'2002 and 2002'2007 to compare the outcomes and performance of 'served' and 'unserved' manufacturing establishments. The approach builds on, updates, and expands upon earlier studies comparing matched MEP client and non-client performance over time periods ending in 1992 and 2002. Our results generally indicate that MEP services had positive and significant impacts on establishment productivity and sales per worker for the 2002'2007 period with some exceptions based on employment size, industry, and type of service provided. MEP services also increased the probability of establishment survival for the 1997'2007 period. Regardless of econometric model specification, MEP clients with 1'19 employees have statistically significant and higher levels of labor productivity growth. We also observed significant productivity differences associated with MEP services by broad sector, with higher impacts over the 2002'2007 time period in the durable goods manufacturing sector. The study further finds that establishments receiving MEP assistance are more likely to survive than those that do not receive MEP assistance. Detailed findings of the study, as well as caveats and limitations, are discussed in the paper.View Full Paper PDF
-
Working PaperStatistics on the International Trade Administration's Global Markets Program
September 2015
Working Paper Number:
CES-15-17
Recent mandates for evidence-based policy choices from both the Executive and Legislative branches of the federal government underscore the importance of understanding the relationship between program participation and business outcomes. In this paper, we examine the correlations between participation in an export-promotion program and business outcomes. We use this experience to provide more general lessons learned about combining program data on treatments with Census Bureau micro data that can be used as a control. Note this paper does not evaluate a program, but instead provides critical information about a program. The mission of the Commercial Service/Global Markets program is to help companies either start or increase their exports of goods and services. It pursues this mission through advocacy, events, and counseling. This study looks at a very small part of the overall program. While we cannot rule-out several sources of bias in our results, we do observe several consistent patterns across our models. In particular, program participation is positively correlated with export growth and change and, for small businesses, also with positive employment growth. However, overall, and for large firms in particular, there is a negative correlation with employment growth and counseling. The paper concludes with a 'Lessons Learned' section that highlights areas where measurement can be improved.View Full Paper PDF
-
Working PaperRelative Effectiveness of Energy Efficiency Programs versus Market Based Climate Policies in the Chemical Industry
April 2018
Working Paper Number:
CES-18-16
This paper addresses the relative effectiveness of market vs program based climate policies. We compute the carbon price resulting in an equivalent reduction in energy from programs that eliminate the efficiency gap. A reduced-form stochastic frontier energy demand analysis of plant level electricity and fuel data, from energy-intensive chemical sectors, jointly estimates the distribution of energy efficiency and underlying price elasticities. The analysis controls for plant level price endogeneity and heterogeneity to obtain a decomposition of efficiency into persistent (PE) and time-varying (TVE) components. Total inefficiency is relatively small and price elasticities are relatively high. If all plants performed at the 90th percentile of their efficiency distribution, the reduction in energy is between 4% and 13%. A modest carbon price of between $9.48/ton and $14.01/ton CO2 would achieve reductions in energy use equivalent to all manufacturing plants making improvements to close the efficiency gap.View Full Paper PDF
-
Working PaperEVIDENCE OF AN 'ENERGY-MANAGEMENT GAP' IN U.S. MANUFACTURING: SPILLOVERS FROM FIRM MANAGEMENT PRACTICES TO ENERGY EFFICIENCY
April 2013
Working Paper Number:
CES-13-25
In this paper we merge a well-cited survey of firm management practices into confidential U.S. Census microdata to examine whether generic, i.e. non-energy specific, firm management practices, 'spillover' to enhance energy efficiency in the United States. We find the relationship in U.S. plants to be more nuanced than past research on UK plants has suggested. Most management techniques have beneficial spillovers to energy efficiency, but an emphasis on generic targets, conditional on other management practices, results in spillovers that increase energy intensity. Our specification controls for industry specific effects at a detailed 6-digit NAICS level and shows that this result is stronger for firms in energy intensive industries. We interpret the empirical result that generic management practices do not necessarily spillover to improved energy performance as evidence of an 'energy management gap.'View Full Paper PDF
-
Working PaperThe Direct and Indirect Costs of Food Safety Regulation
September 2008
Working Paper Number:
CES-08-31
The cost of compliance with the Pathogen Reduction Hazard Analysis Critical Control Program (PR/HACCP) rule of 1996 has been controversial since it was first proposed. Surveys have provided some cost information but examined plant size and other indirect effects with limited data and did not make cost estimates of direct cost components, such as mandated tasks. This paper addresses those deficiencies with data from a national survey of meat and poultry plants on PR/HACCP costs. Results indicate that (1) mandated tasks are the most costly component of the PR/HACCP rule, (2) regulation favors large plants over small ones, and (3) private actions are nearly as costly as direct regulation.View Full Paper PDF
