Rationale for making comparisons using non-overlapping periods

Diana Lavery over 3 years ago

Hi ACS friends,

I'm trying to explain to some brilliant but non-ACS-knowledgeable coworkers and others why the recommended approach is to use *non-overlapping* periods when making comparisons over time. I've looked at the Census doc (https://www.census.gov/programs-surveys/acs/guidance/comparing-acs-data.html) and all I can find is the guidance that says Do use non-overlapping datasets, Do not use overlapping datasets. I'm having a hard time finding the why. All I can come up with is this:

4-5ths of the sample is the same in adjacent datasets, e.g., 2014-2018 vs. 2015-2019. You can think of Census removing respondents from 2014, and adding respondents from 2019, but the respondents from 2015, 2016, 2017, and 2018 are the same. We should compare two completely different datasets, which means two non-overlapping periods.

I'm curious how others answer this question as well! -Diana

Top Replies

Gretchen Gooding over 3 years ago +1

A colleague suggested looking at this paper (https://www.census.gov/library/working-papers/2012/adrm/rrs2012-03.html). There's a short discussion about non-overlapping periods in section 4.4 on page 9…

Parents

Todd Graham over 3 years ago

Diana and Todd Z--

If you were comparing two separate time-periods, two datapoints at different times, then you would want to account for the statistical inference "margin of error" around the two estimate points. There's a formula for this.

See the "Determining Statistical Significance" section (p 55) in the "Understanding and Using ACS Data" handbook
https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_general_handbook_2020.pdf

But if the two time-periods are overlapping, this is the statisticians' equivalent of "double-dipping your chip" in the serving bowl. Some people will say "no big deal." Other people will condemn it as unmannered barbarism. Not so much for germophobe reasons, but because the "double-dip" (the same survey cases being included in both the 2010-14 and 2014-18 estimates) makes the standard significance test invalid.

Bottom line: Do you want to be able to say something about statistical significance of the differences over time? (And does your audience include statisticians?)

--TG
Cancel
Up 0 Down

Reply

Cancel

Reply

Todd Graham over 3 years ago

Diana and Todd Z--

If you were comparing two separate time-periods, two datapoints at different times, then you would want to account for the statistical inference "margin of error" around the two estimate points. There's a formula for this.

See the "Determining Statistical Significance" section (p 55) in the "Understanding and Using ACS Data" handbook
https://www.census.gov/content/dam/Census/library/publications/2020/acs/acs_general_handbook_2020.pdf

But if the two time-periods are overlapping, this is the statisticians' equivalent of "double-dipping your chip" in the serving bowl. Some people will say "no big deal." Other people will condemn it as unmannered barbarism. Not so much for germophobe reasons, but because the "double-dip" (the same survey cases being included in both the 2010-14 and 2014-18 estimates) makes the standard significance test invalid.

Bottom line: Do you want to be able to say something about statistical significance of the differences over time? (And does your audience include statisticians?)

--TG
Cancel
Up 0 Down

Reply

Cancel

Children

No Data