Lessons for Targeted Program Evaluation: A Personal and Professional History of the Survey of Program Dynamics
August 2007
Working Paper Number:
CES-07-24
Abstract
The Survey of Program Dynamics (SPD) was created by the 1996 welfare reform legislation to facilitate its evaluation. This paper describes the evolution of that survey, discusses its implementation, and draws lessons for future evaluation. Large-scale surveys can be an important part of a portfolio of evaluation methods, but sufficient time must be given to data collection agencies if a high-quality longitudinal survey is expected. Such a survey must have both internal (agency) and external (policy analyst) buy-in. Investments in data analysis by agency staff, downplayed in favor of larger sample sizes given a fixed budget, could have contributed to more external acceptance. More attention up-front to reducing the potentially deleterious effects of attrition in longitudinal surveys, such as through the use of monetary incentives, might have been worthwhile. Given the problems encountered by the Census Bureau in producing the SPD, I argue that ongoing multi-purpose longitudinal surveys like the Survey of Income and Program Participation are potentially more valuable than episodic special-purpose surveys.
Document Tags and Keywords
Keywords
Keywords are automatically generated using KeyBERT, a powerful and innovative
keyword extraction tool that utilizes BERT embeddings to ensure high-quality and contextually relevant
keywords.
By analyzing the content of working papers, KeyBERT identifies terms and phrases that capture the essence of the
text, highlighting the most significant topics and trends. This approach not only enhances searchability but
provides connections that go beyond potentially domain-specific author-defined keywords.
:
analysis,
economist,
survey data,
survey,
agency,
respondent,
longitudinal,
valuation,
fiscal,
survey income,
assessing
Tags
Tags are automatically generated using a pretrained language model from spaCy, which excels at
several tasks, including entity tagging.
The model is able to label words and phrases by part-of-speech,
including "organizations." By filtering for frequent words and phrases labeled as "organizations", papers are
identified to contain references to specific institutions, datasets, and other organizations.
:
Social Security Administration,
Center for Economic Studies,
Office of Management and Budget,
Urban Institute,
Current Population Survey,
1940 Census,
Survey of Income and Program Participation,
General Accounting Office,
Social Security,
Department of Health and Human Services,
National Academy of Sciences
Similar Working Papers
Similarity between working papers are determined by an unsupervised neural
network model
know as Doc2Vec.
Doc2Vec is a model that represents entire documents as fixed-length vectors, allowing for the
capture of semantic meaning in a way that relates to the context of words within the document. The model learns to
associate a unique vector with each document while simultaneously learning word vectors, enabling tasks such as
document classification, clustering, and similarity detection by preserving the order and structure of words. The
document vectors are compared using cosine similarity/distance to determine the most similar working papers.
Papers identified with 🔥 are in the top 20% of similarity.
The 10 most similar working papers to the working paper 'Lessons for Targeted Program Evaluation: A Personal and Professional History of the Survey of Program Dynamics' are listed below in order of similarity.
-
Working PaperMeasuring Poverty in the United States: History and Current Issues🔥
April 2006
Working Paper Number:
CES-06-11
Formal measurement of poverty in the United States is now about 40 years old. This paper first briefly describes the origins and basis of the official poverty thresholds adopted by the federal government in the late 1960s. Then, it discusses in some detail some of the more current issues that observers suggest must be addressed if changes are to be made. The final sections discuss recent efforts to propose alternates to the current official approach.View Full Paper PDF
-
Working PaperManagement Challenges of the 2010 U.S. Census🔥
August 2011
Working Paper Number:
CES-11-22
This paper gives an insider's perspective on the management approaches used to manage the 2010 Census during its operational phase. The approaches used, the challenges faced (in particular, difficulties faced in automating data collection), and the solutions applied to meet those challenges are described. Finally, six management lessons learned are presented.View Full Paper PDF
-
Working PaperAlternative Measures of Income Poverty and the Anti-Poverty Effects of Taxes and Transfers
June 2005
Working Paper Number:
CES-05-08
The Census Bureau prepared a number of alternative income-based measures of poverty to illustrate the distributional impacts of several alternatives to the official measure. The paper examines five income variants for two different units of analysis (families and households) for two different assumptions about inflation (the historical Consumer Price Index and a 'Research Series' alternative that uses current methods) for two different sets of thresholds (official and a formula-based alternative base on three parameters). The poverty rate effects are analyzed for the total population, the distributional effects are analyzed using poverty shares, and the anti-poverty effects of taxes and transfers are analyzed using a percentage reduction in poverty rates. Suggestions for future research are included.View Full Paper PDF
-
Working PaperThe Management and Organizational Practices Survey (MOPS): Cognitive Testing*
January 2016
Working Paper Number:
CES-16-53
All Census Bureau surveys must meet quality standards before they can be sent to the public for data collection. This paper outlines the pretesting process that was used to ensure that the Management and Organizational Practices Survey (MOPS) met those standards. The MOPS is the first large survey of management practices at U.S. manufacturing establishments. The first wave of the MOPS, issued for reference year 2010, was subject to internal expert review and two rounds of cognitive interviews. The results of this pretesting were used to make significant changes to the MOPS instrument and ensure that quality data was collected. The second wave of the MOPS, featuring new questions on data in decision making (DDD) and uncertainty and issued for reference year 2015, was subject to two rounds of cognitive interviews and a round of usability testing. This paper illustrates the effort undertaken by the Census Bureau to ensure that all surveys released into the field are of high quality and provides insight into how respondents interpret the MOPS questionnaire for those looking to utilize the MOPS data.View Full Paper PDF
-
Working PaperEffects of a Government-Academic Partnership: Has the NSF-Census Bureau Research Network Helped Improve the U.S. Statistical System?
January 2017
Working Paper Number:
CES-17-59R
The National Science Foundation-Census Bureau Research Network (NCRN) was established in 2011 to create interdisciplinary research nodes on methodological questions of interest and significance to the broader research community and to the Federal Statistical System (FSS), particularly the Census Bureau. The activities to date have covered both fundamental and applied statistical research and have focused at least in part on the training of current and future generations of researchers in skills of relevance to surveys and alternative measurement of economic units, households, and persons. This paper discusses some of the key research findings of the eight nodes, organized into six topics: (1) Improving census and survey data collection methods; (2) Using alternative sources of data; (3) Protecting privacy and confidentiality by improving disclosure avoidance; (4) Using spatial and spatio-temporal statistical modeling to improve estimates; (5) Assessing data cost and quality tradeoffs; and (6) Combining information from multiple sources. It also reports on collaborations across nodes and with federal agencies, new software developed, and educational activities and outcomes. The paper concludes with an evaluation of the ability of the FSS to apply the NCRN's research outcomes and suggests some next steps, as well as the implications of this research-network model for future federal government renewal initiatives.View Full Paper PDF
-
Working PaperThe Impact of Household Surveys on 2020 Census Self-Response
July 2022
Working Paper Number:
CES-22-24
Households who were sampled in 2019 for the American Community Survey (ACS) had lower self-response rates to the 2020 Census. The magnitude varied from -1.5 percentage point for household sampled in January 2019 to -15.1 percent point for households sampled in December 2019. Similar effects are found for the Current Population Survey (CPS) as well.View Full Paper PDF
-
Working PaperThe EITC over the business cycle: Who benefits?
December 2014
Working Paper Number:
carra-2014-15
In this paper, I examine the impact of the Great Recession on Earned Income Tax Credit (EITC) eligibility. Because the EITC is structurally tied to earnings, the direction of this impact is not immediately obvious. Families who experience complete job loss for an entire tax year lose eligibility, while those experiencing underemployment (part-year employment, a reduction in hours, or spousal unemployment in married households) may become eligible. Determining the direction and magnitude of the impact is important for a number of reasons. The EITC has become the largest cash-transfer program in the U.S., and many low-earning families rely on it as a means of support in tough times. The program has largely been viewed as a replacement for welfare, enticing former welfare recipients into the labor force. However, the effectiveness of the EITC during a period of very high unemployment has not been assessed. To answer these questions, I first use the Current Population Survey (CPS) matched to Internal Revenue Service data from tax years 2005 to 2010 to assess patterns of employment and eligibility over the Great Recession for different labor-force groups. Results indicate that overall, EITC eligibility increased over the recession, but only among groups that were cushioned from total household earnings loss by marriage. I also use the 2006 CPS matched to tax data from 2005 through 2011 to examine changes in eligibility experienced by individuals over time. In assessing three competing causes of eligibility loss, I find that less-educated, unmarried women experienced a greater hazard of eligibility loss due a yearlong lack of earnings compared with other labor-market groups. I discuss the implications of these findings on the view of the EITC as a safety-net program.View Full Paper PDF
-
Working PaperIncome Packaging and Economic Disconnection: Do Sources of Support Differ from Other Low-Income Women?
December 2013
Working Paper Number:
CES-13-61
Income packaging, or piecing together cash and non-cash resources from a variety of sources, is a common financial survival strategy among low-income women. This strategy is particularly important for economically disconnected women, who lack both employment income and public cash assistance receipt. Using data from the confidential Census Bureau versions of the Survey of Income and Program Participation, this study compares the use of public and private supports between disconnected and connected low-income women, controlling for differences in state welfare rules and county unemployment rates. Findings from bivariate comparisons and multilevel logistic regressions indicate that disconnected women utilize public non-cash supports at similar rates to connected women, but rely more heavily on private sources. Conclusions focus on the policy implications for outreach and program development.View Full Paper PDF
-
Working PaperMeasuring the Impact of COVID-19 on Businesses and People: Lessons from the Census Bureau's Experience
January 2021
Working Paper Number:
CES-21-02
We provide an overview of Census Bureau activities to enhance the consistency, timeliness, and relevance of our data products in response to the COVID-19 pandemic. We highlight new data products designed to provide timely and granular information on the pandemic's impact: the Small Business Pulse Survey, weekly Business Formation Statistics, the Household Pulse Survey, and Community Resilience Estimates. We describe pandemic-related content introduced to existing surveys such as the Annual Business Survey and the Current Population Survey. We discuss adaptations to ensure the continuity and consistency of existing data products such as principal economic indicators and the American Community Survey.View Full Paper PDF
-
Working PaperWhen and Why Does Nonresponse Occur? Comparing the Determinants of Initial Unit Nonresponse and Panel Attrition
September 2023
Working Paper Number:
CES-23-44
Though unit nonresponse threatens data quality in both cross-sectional and panel surveys, little is understood about how initial nonresponse and later panel attrition may be theoretically or empirically distinct phenomena. This study advances current knowledge of the determinants of both unit nonresponse and panel attrition within the context of the U.S. Census Bureau's Survey of Income and Program Participation (SIPP) panel survey, which I link with high-quality federal administrative records, paradata, and geographic data. By exploiting the SIPP's interpenetrated sampling design and relying on cross-classified random effects modeling, this study quantifies the relative effects of sample household, interviewer, and place characteristics on baseline nonresponse and later attrition, addressing a critical gap in the literature. Given the reliance on successful record linkages between survey sample households and federal administrative data in the nonresponse research, this study also undertakes an explicitly spatial analysis of the place-based characteristics associated with successful record linkages in the U.S.View Full Paper PDF
