Rationale for making comparisons using non-overlapping periods

The general ACS handbook does say that "these comparisons can be made with caution" (footnote 19 on p. 17 of https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_general_handbook_2020.pdf). I vaguely recall someone mentioning once that what you're basically getting is the difference between the last year of the more recent dataset and the first year of the earlier dataset (so 2014 vs 2019, in your example).

As Todd G noted, the usual significance tests don't work, but the Census Bureau used to provide a modification, which is multiplying the standard error of the difference by a factor that depends on the proportion of years that overlap. (Specifically, you subtract that proportion -- 4/5 = 0.8 in your example -- from 1 and take the square root.) I don't know if the Census Bureau is still publishing that guidance, but it does appear in the 2013-vintage ACS documentation (https://www2.census.gov/programs-surveys/acs/tech_docs/statistical_testing/2013StatisticalTesting3and5.pdf).

So it's doable, but I don't know whether it's desirable. You could easily imagine this being used as a "magic" way to make single-year comparisons for geographies without one-year estimates, but keep in mind:

• That modification to the usual significance tests is an approximation of an approximation (as I understand it). So there's more room for things to go awry.
• On average, the ACS samples about 2% of households each year. In a census tract with 2,000 total households, that's only ~40 households. So: even if a change over time were statistically significant, I would hesitate to draw any firm conclusions about change between two years, especially if I'm looking at subgroups (for example, renter households, or households with Black householders).

I'm looking forward to seeing other responses as well!

--Matt

The general ACS handbook does say that "these comparisons can be made with caution" (footnote 19 on p. 17 of https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_general_handbook_2020.pdf). I vaguely recall someone mentioning once that what you're basically getting is the difference between the last year of the more recent dataset and the first year of the earlier dataset (so 2014 vs 2019, in your example).

As Todd G noted, the usual significance tests don't work, but the Census Bureau used to provide a modification, which is multiplying the standard error of the difference by a factor that depends on the proportion of years that overlap. (Specifically, you subtract that proportion -- 4/5 = 0.8 in your example -- from 1 and take the square root.) I don't know if the Census Bureau is still publishing that guidance, but it does appear in the 2013-vintage ACS documentation (https://www2.census.gov/programs-surveys/acs/tech_docs/statistical_testing/2013StatisticalTesting3and5.pdf).

So it's doable, but I don't know whether it's desirable. You could easily imagine this being used as a "magic" way to make single-year comparisons for geographies without one-year estimates, but keep in mind:

• That modification to the usual significance tests is an approximation of an approximation (as I understand it). So there's more room for things to go awry.
• On average, the ACS samples about 2% of households each year. In a census tract with 2,000 total households, that's only ~40 households. So: even if a change over time were statistically significant, I would hesitate to draw any firm conclusions about change between two years, especially if I'm looking at subgroups (for example, renter households, or households with Black householders).

I'm looking forward to seeing other responses as well!

--Matt