Rationale for making comparisons using non-overlapping periods

Hi ACS friends,

I'm trying to explain to some brilliant but non-ACS-knowledgeable coworkers and others why the recommended approach is to use *non-overlapping* periods when making comparisons over time.  I've looked at the Census doc (https://www.census.gov/programs-surveys/acs/guidance/comparing-acs-data.html) and all I can find is the guidance that says Do use non-overlapping datasets, Do not use overlapping datasets.  I'm having a hard time finding the why.  All I can come up with is this:

4-5ths of the sample is the same in adjacent datasets, e.g., 2014-2018 vs. 2015-2019. You can think of Census removing respondents from 2014, and adding respondents from 2019, but the respondents from 2015, 2016, 2017, and 2018 are the same. We should compare two completely different datasets, which means two non-overlapping periods.

I'm curious how others answer this question as well!  -Diana

Top Replies

Parents
• Diana and Todd Z--

If you were comparing two separate time-periods, two datapoints at different times, then you would want to account for the statistical inference "margin of error" around the two estimate points. There's a formula for this.

See the "Determining Statistical Significance" section (p 55) in the "Understanding and Using ACS Data" handbook
https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_general_handbook_2020.pdf
But if the two time-periods are overlapping, this is the statisticians' equivalent of "double-dipping your chip" in the serving bowl. Some people will say "no big deal."  Other people will condemn it as unmannered barbarism.  Not so much for germophobe reasons, but because the "double-dip" (the same survey cases being included in both the 2010-14 and 2014-18 estimates) makes the standard significance test invalid.
Bottom line: Do you want to be able to say something about statistical significance of the differences over time? (And does your audience include statisticians?)
--TG