Looking ahead to Thursday's ACS release, I'm trying to examine change over time for relatively small groups. Because margins of error tend to be too wide in the one-year data to say much about change over time, the natural temptation is to compare the 2016 one-year data to the 2011-2015 five-year data.
I've always heard cautions against comparing estimates of different period lengths, even if the periods don't overlap (e.g., here on slide 22, here on slide 27). But I don't know why. The closest I've come to finding a rationale is this paper, which explains that one-year estimates and multi-year estimates are "fundamentally different in their construction" due to an extra step in the multiyear weighting to reduce variance for small areas.
So:
(1) Why exactly is caution advised when comparing estimates of different period lengths, even if the periods don't overlap and there are no other comparability issues?
(2) Is this caution more like "Don't ever do this because the usual statistical tests will be meaningless," or "In theory this isn't sound, but in practice it's basically okay"?
Thanks!
--Matt