Comparing Non-Overlapping 5 year estimates from 2006-2010 and 2011-2015

Has anyone compared non-overlapping 5-year estimates from 2006 - 2010 and 2011 - 2015 at the tract level?  What has been your experience and how do you feel about the reliability of the results?

I looked at ‘Drove Alone to Work’ comparing ACS tables B08006.  I calculated the percent change between the two time periods.  Next, I applied the Census Bureau’s statistical testing tool (with the confidence interval parameter set at 90%) to the data to determine which census tracts are significantly different from each other for the two periods.  Tracts that are not significantly different are considered to have no change.  (This testing takes into account the margin of error associated with the survey estimates, based on responses from only a sample of the full population.)

8% of the tracts I am looking at tested significantly different and are considered to have changed.  For ‘Drove Alone to Work’ I think 8%, with significant change over 10 years seems reasonable.  Has anyone else done this test on a different ACS characteristic?  What do you think of your results?

I then calculated the Coefficient of Variation to get reliability estimates on the tracts that tested significantly different.  I found 83% (2006-2010) and 92% (2011- 2015) of the data to be highly reliable and eliminated just 5 tracts having low reliability.

 

The magnitude of change I am seeing in the tracts for ‘Drove Alone to Work’ is a bit disconcerting.  I am seeing only 1 of 116 tracts with an increase/decrease under 15%.  56 of 116 tracts with an increase/decrease of 15%-30%, 26 of 116 tracts with an increase/decrease of 31%- 45% and 18 tracts with increase/decrease over 46%.  Has anyone else had results where they have found very few tracts with slight change while the majority of tracts (tested statistically significant and reliable) show medium to large change?

 

What do you think about comparing non-overlapping 5-year estimates in general?  Do you find this method to be sound?  Would you take a different approach to comparing the data?

  • I just have three things to add. First, if you are not on it, the transportation people have a Census data list serve where you might want to post this. You can join by going to www.chrispy.net/.../ctpp-news. Second, unless I miss read your post you only had about 9 tracts with any significant changes. That to me suggests that all the tracks with those large changes are not significant. What is troubling is that I could see people using the non-overlapping 5 year estimates and not paying attention to the MOEs . Having been in the transportation planning game for over 35 years I can assure you that on the face of it, changes like you were seeing just do not happen in such a short period of time with a variable as dominant as 'drive alone to work'. Of course, you could be in some crazy unique area where some externality caused the changes. Lastly, I would be careful looking at this as a change over 10 years. It is really only a change between the 06-10 and 11-15 period estimates, not really 10 years.

    Next week the transportation folks are having a census data conference in KC. I am hoping that issues like the one above will be talked about. We have a lot to learn when it comes to using ACS productively. www.trb.org/conferences/census2017.aspx
  • To answer your specific questions: Comparing non-overlapping 5-year estimates is the recommended approach. But, in general, MOEs for most census tract characteristics are fairly large. So if you can, I would recommend aggregating tracts to a larger area before comparing.

    Note also: If I've understood your process (you're only looking at the change for those that tested as significant?), you are likely to find larger change among those tracts because you've already excluded most (perhaps all) tracts with small changes using the significant difference criteria. Because of the large MOEs at the tract level, only those with a large change will appear to have a significant difference over time. (But perhaps I've misunderstood the steps you're describing? Either way - my recommendation remains: Aggregate if possible.)
  • In reply to Ed Christopher:

    Hi Ed,
    Thank you for the link to the ctpp list serve, I will join.

    Second, I mistyped 18% as 8%. I tested 683 tracts with the statistical testing tool and found 121 of those tracts to be statistically different (18%). 116 of those 121 tracts have a CV under 30%. (I eliminated the 5 tracts with CV’s over 30% for low reliability.) I am considering the magnitude of change for the 116 tracts that tested significant for change and have CV’s under 30%.

    I agree with you that the magnitude of change seems unlikely for the period of time. At this point I am wondering if anyone has gotten results that they feel confident in at the tract level.
  • In reply to Beth Jarosz:

    Hi Beth,
    Thank you for your input. Yes, I am only looking at the change for those that tested as significant. I was wondering if the exclusion of tracts with small changes was due my significance testing criteria. I set the confidence parameter at 90%. I will think about lowering this a bit.