-
Nonresponse and Coverage Bias in the Household Pulse Survey: Evidence from Administrative Data
October 2024
Working Paper Number:
CES-24-60
The Household Pulse Survey (HPS) conducted by the U.S. Census Bureau is a unique survey that provided timely data on the effects of the COVID-19 Pandemic on American households and continues to provide data on other emergent social and economic issues. Because the survey has a response rate in the single digits and only has an online response mode, there are concerns about nonresponse and coverage bias. In this paper, we match administrative data from government agencies and third-party data to HPS respondents to examine how representative they are of the U.S. population. For comparison, we create a benchmark of American Community Survey (ACS) respondents and nonrespondents and include the ACS respondents as another point of reference. Overall, we find that the HPS is less representative of the U.S. population than the ACS. However, performance varies across administrative variables, and the existing weighting adjustments appear to greatly improve the representativeness of the HPS. Additionally, we look at household characteristics by their email domain to examine the effects on coverage from limiting email messages in 2023 to addresses from the contact frame with at least 90% deliverability rates, finding no clear change in the representativeness of the HPS afterwards.
View Full
Paper PDF
-
Citizenship Question Effects on Household Survey Response
June 2024
Working Paper Number:
CES-24-31
Several small-sample studies have predicted that a citizenship question in the 2020 Census would cause a large drop in self-response rates. In contrast, minimal effects were found in Poehler et al.'s (2020) analysis of the 2019 Census Test randomized controlled trial (RCT). We reconcile these findings by analyzing associations between characteristics about the addresses in the 2019 Census Test and their response behavior by linking to independently constructed administrative data. We find significant heterogeneity in sensitivity to the citizenship question among households containing Hispanics, naturalized citizens, and noncitizens. Response drops the most for households containing noncitizens ineligible for a Social Security number (SSN). It falls more for households with Latin American-born immigrants than those with immigrants from other countries. Response drops less for households with U.S.-born Hispanics than households with noncitizens from Latin America. Reductions in responsiveness occur not only through lower unit self-response rates, but also by increased household roster omissions and internet break-offs. The inclusion of a citizenship question increases the undercount of households with noncitizens. Households with noncitizens also have much higher citizenship question item nonresponse rates than those only containing citizens. The use of tract-level characteristics and significant heterogeneity among Hispanics, the foreign-born, and noncitizens help explain why the effects found by Poehler et al. were so small. Linking administrative microdata with the RCT data expands what we can learn from the RCT.
View Full
Paper PDF
-
The 2010 Census Confidentiality Protections Failed, Here's How and Why
December 2023
Authors:
Lars Vilhuber,
John M. Abowd,
Ethan Lewis,
Nathan Goldschlag,
Robert Ashmead,
Daniel Kifer,
Philip Leclerc,
Rolando A. Rodr�guez,
Tamara Adams,
David Darais,
Sourya Dey,
Simson L. Garfinkel,
Scott Moore,
Ramy N. Tadros
Working Paper Number:
CES-23-63
Using only 34 published tables, we reconstruct five variables (census block, sex, age, race, and ethnicity) in the confidential 2010 Census person records. Using the 38-bin age variable tabulated at the census block level, at most 20.1% of reconstructed records can differ from their confidential source on even a single value for these five variables. Using only published data, an attacker can verify that all records in 70% of all census blocks (97 million people) are perfectly reconstructed. The tabular publications in Summary File 1 thus have prohibited disclosure risk similar to the unreleased confidential microdata. Reidentification studies confirm that an attacker can, within blocks with perfect reconstruction accuracy, correctly infer the actual census response on race and ethnicity for 3.4 million vulnerable population uniques (persons with nonmodal characteristics) with 95% accuracy, the same precision as the confidential data achieve and far greater than statistical baselines. The flaw in the 2010 Census framework was the assumption that aggregation prevented accurate microdata reconstruction, justifying weaker disclosure limitation methods than were applied to 2010 Census public microdata. The framework used for 2020 Census publications defends against attacks that are based on reconstruction, as we also demonstrate here. Finally, we show that alternatives to the 2020 Census Disclosure Avoidance System with similar accuracy (enhanced swapping) also fail to protect confidentiality, and those that partially defend against reconstruction attacks (incomplete suppression implementations) destroy the primary statutory use case: data for redistricting all legislatures in the country in compliance with the 1965 Voting Rights Act.
View Full
Paper PDF
-
Producing U.S. Population Statistics Using Multiple Administrative Sources
November 2023
Working Paper Number:
CES-23-58
We identify several challenges encountered when constructing U.S. administrative record-based (AR-based) population estimates for 2020. Though the AR estimates are higher than the 2020 Census at the national level, they are over 15 percent lower in 5 percent of counties, suggesting that locational accuracy can be improved. Other challenges include how to achieve comprehensive coverage, maintain consistent coverage across time, filter out nonresidents and people not alive on the reference date, uncover missing links across person and address records, and predict demographic characteristics when multiple ones are reported or when they are missing. We discuss several ways of addressing these issues, e.g., building in redundancy with more sources, linking children to their parents' addresses, and conducting additional record linkage for people without Social Security Numbers and for addresses not initially linked to the Census Bureau's Master Address File. We discuss modeling to predict lower levels of geography for people lacking those geocodes, the probability that a person is a U.S. resident on the reference date, the probability that an address is the person's residence on the reference date, and the probability a person is in each demographic characteristic category. Regression results illustrate how many of these challenges and solutions affect the AR county population estimates.
View Full
Paper PDF
-
Noncitizen Coverage and Its Effects on U.S. Population Statistics
August 2023
Working Paper Number:
CES-23-42
We produce population estimates with the same reference date, April 1, 2020, as the 2020 Census of Population and Housing by combining 31 types of administrative record (AR) and third-party sources, including several new to the Census Bureau with a focus on noncitizens. Our AR census national population estimate is higher than other Census Bureau official estimates: 1.8% greater than the 2020 Demographic Analysis high estimate, 3.0% more than the 2020 Census count, and 3.6% higher than the vintage-2020 Population Estimates Program estimate. Our analysis suggests that inclusion of more noncitizens, especially those with unknown legal status, explains the higher AR census estimate. About 19.8% of AR census noncitizens have addresses that cannot be linked to an address in the 2020 Census collection universe, compared to 5.7% of citizens, raising the possibility that the 2020 Census did not collect data for a significant fraction of noncitizens residing in the United States under the residency criteria used for the census. We show differences in estimates by age, sex, Hispanic origin, geography, and socioeconomic characteristics symptomatic of the differences in noncitizen coverage.
View Full
Paper PDF
-
Estimating the U.S. Citizen Voting-Age Population (CVAP) Using Blended Survey Data, Administrative Record Data, and Modeling: Technical Report
April 2023
Authors:
J. David Brown,
Danielle H. Sandler,
Lawrence Warren,
Moises Yi,
Misty L. Heggeness,
Joseph L. Schafer,
Matthew Spence,
Marta Murray-Close,
Carl Lieberman,
Genevieve Denoeux,
Lauren Medina
Working Paper Number:
CES-23-21
This report develops a method using administrative records (AR) to fill in responses for nonresponding American Community Survey (ACS) housing units rather than adjusting survey weights to account for selection of a subset of nonresponding housing units for follow-up interviews and for nonresponse bias. The method also inserts AR and modeling in place of edits and imputations for ACS survey citizenship item nonresponses. We produce Citizen Voting-Age Population (CVAP) tabulations using this enhanced CVAP method and compare them to published estimates. The enhanced CVAP method produces a 0.74 percentage point lower citizen share, and it is 3.05 percentage points lower for voting-age Hispanics. The latter result can be partly explained by omissions of voting-age Hispanic noncitizens with unknown legal status from ACS household responses. Weight adjustments may be less effective at addressing nonresponse bias under those conditions.
View Full
Paper PDF
-
Determination of the 2020 U.S. Citizen Voting Age Population (CVAP) Using Administrative Records and Statistical Methodology Technical Report
October 2020
Authors:
John M. Abowd,
J. David Brown,
Lawrence Warren,
Moises Yi,
Misty L. Heggeness,
William R. Bell,
Michael B. Hawes,
Andrew Keller,
Vincent T. Mule Jr.,
Joseph L. Schafer,
Matthew Spence
Working Paper Number:
CES-20-33
This report documents the efforts of the Census Bureau's Citizen Voting-Age Population (CVAP) Internal Expert Panel (IEP) and Technical Working Group (TWG) toward the use of multiple data sources to produce block-level statistics on the citizen voting-age population for use in enforcing the Voting Rights Act. It describes the administrative, survey, and census data sources used, and the four approaches developed for combining these data to produce CVAP estimates. It also discusses other aspects of the estimation process, including how records were linked across the multiple data sources, and the measures taken to protect the confidentiality of the data.
View Full
Paper PDF
-
Nonemployer Statistics by Demographics (NES-D):
Exploring Longitudinal Consistency and Sub-national Estimates
December 2019
Working Paper Number:
CES-19-34
Until recently, the quinquennial Survey of Business Owners (SBO) was the only source of information for U.S. employer and nonemployer businesses by owner demographic characteristics such as race, ethnicity, sex and veteran status. Now, however, the Nonemployer Statistics by Demographics series (NES-D) will replace the SBO's nonemployer component with reliable, and more frequent (annual) business demographic estimates with no additional respondent burden, and at lower imputation rates and costs. NES-D is not a survey; rather, it exploits existing administrative and census records to assign demographic characteristics to the universe of approximately 25 million (as of 2016) nonemployer businesses.
Although only in the second year of its research phase, NES-D is rapidly moving towards production, with a planned prototype or experimental version release of 2017 nonemployer data in 2020, followed by annual releases of the series. After the first year of research, we released a working paper (Luque et al., 2019) that assessed the viability of estimating nonemployer demographics exclusively with administrative records (AR) and census data. That paper used one year of data (2015) to produce preliminary tabulations of business counts at the national level. This year we expand that research in multiple ways by: i) examining the longitudinal consistency of administrative and census records coverage, and of our AR-based demographics estimates, ii) evaluating further coverage from additional data sources, iii) exploring estimates at the sub-national level, iv) exploring estimates by industrial sector, v) examining demographics estimates of business receipts as well as of counts, and vi) implementing imputation of missing demographic values.
Our current results are consistent with the main findings in Luque et al. (2019), and show that high coverage and demographic assignment rates are not the exception, but the norm. Specifically, we find that AR coverage rates are high and stable over time for each of the three years we examine, 2014-2016. We are able to identify owners for approximately 99 percent of nonemployer businesses (excluding C-corporations), 92 to 93 percent of identified nonemployer owners have no missing demographics, and only about 1 percent are missing three or more demographic characteristics in each of the three years. We also find that our demographics estimates are stable over time, with expected small annual changes that are consistent with underlying population trends in the U.S.. Due to data limitations, these results do not include C-corporations, which represent only 2 percent of nonemployer businesses and 4 percent of receipts.
Without added respondent burden and at lower imputation rates and costs, NES-D will provide high-quality business demographics estimates at a higher frequency (annual vs. every 5 years) than the SBO.
View Full
Paper PDF
-
Predicting the Effect of Adding a Citizenship Question to the 2020 Census
June 2019
Working Paper Number:
CES-19-18
The addition of a citizenship question to the 2020 census could affect the self-response rate, a key driver of the cost and quality of a census. We find that citizenship question response patterns in the American Community Survey (ACS) suggest that it is a sensitive question when asked about administrative record noncitizens but not when asked about administrative record citizens. ACS respondents who were administrative record noncitizens in 2017 frequently choose to skip the question or answer that the person is a citizen. We predict the effect on self-response to the entire survey by comparing mail response rates in the 2010 ACS, which included a citizenship question, with those of the 2010 census, which did not have a citizenship question, among households in both surveys. We compare the actual ACS-census difference in response rates for households that may contain noncitizens (more sensitive to the question) with the difference for households containing only U.S. citizens. We estimate that the addition of a citizenship question will have an 8.0 percentage point larger effect on self-response rates in households that may have noncitizens relative to those with only U.S. citizens. Assuming that the citizenship question does not affect unit self-response in all-citizen households and applying the 8.0 percentage point drop to the 28.1 % of housing units potentially having at least one noncitizen would predict an overall 2.2 percentage point drop in self-response in the 2020 census, increasing costs and reducing the quality of the population count.
View Full
Paper PDF
-
Nonemployer Statistics by Demographics (NES-D): Using Administrative and Census Records Data in Business Statistics
January 2019
Working Paper Number:
CES-19-01
The quinquennial Survey of Business Owners or SBO provided the only comprehensive source of information in the United States on employer and nonemployer businesses by the sex, race, ethnicity and veteran status of the business owners. The annual Nonemployer Statistics series (NES) provides establishment counts and receipts for nonemployers but contains no demographic information on the business owners. With the transition of the employer component of the SBO to the Annual Business Survey, the Nonemployer Statistics by Demographics series or NES-D represents the continuation of demographics estimates for nonemployer businesses. NES-D will leverage existing administrative and census records to assign demographic characteristics to the universe of approximately 24 million nonemployer businesses (as of 2015). Demographic characteristics include key demographics measured by the SBO (sex, race, Hispanic origin and veteran status) as well as other demographics (age, place of birth and citizenship status) collected but not imputed by the SBO if missing. A spectrum of administrative and census data sources will provide the nonemployer universe and demographics information. Specifically, the nonemployer universe originates in the Business Register; the Census Numident will provide sex, age, place of birth and citizenship status; race and Hispanic origin information will be obtained from multiple years of the decennial census and the American Community Survey; and the Department of Veteran Affairs will provide administrative records data on veteran status.
The use of blended data in this manner will make possible the production of NES-D, an annual series that will become the only source of detailed and comprehensive statistics on the scope, nature and activities of U.S. businesses with no paid employment by the demographic characteristics of the business owner. Using the 2015 vintage of nonemployers, initial results indicate that demographic information is available for the overwhelming majority of the universe of nonemployers. For instance, information on sex, age, place of birth and citizenship status is available for over 95 percent of the 24 million nonemployers while race and Hispanic origin are available for about 90 percent of them. These results exclude owners of C-corporations, which represent only 2 percent of nonemployer firms. Among other things, future work will entail imputation of missing demographics information (including that of C-corporations), testing the longitudinal consistency of the estimates, and expanding the set of characteristics beyond the demographics mentioned above. Without added respondent burden and at lower imputation rates and costs, NES-D will meet the needs of stakeholders as well as the economy as a whole by providing reliable estimates at a higher frequency (annual vs. every 5 years) and with a more timely dissemination schedule than the SBO.
View Full
Paper PDF
