Statistically comparing estimates

Matthew BraultNick KobelMark Mather I appreciate your responses!  I also sent this question to the help desk at the Census Bureau.  If and when they get back to me, I will post here.  Thanks again!

Matthew Brault Nick Kobel Mark Mather Here is the response I got from the Census Bureau directly.  Just some food for thought!

"You are correct that the poverty rate for Cook county is not independent from Illinois.  Depending on what they are using the comparison for will determine if they should take the covariance into account in some manner.  It is reasonable to compare counties to the state estimate, even though they are not independent in order to examine how the poverty rates may vary at a finer geographic level.  

To get into the details, there are two ways that I can think of to adjust the MOE to take the covariance into account.  One method is to use the Variance Replicate Estimate (VRE) tables (view documentation).  The VRE tables are the same as published detailed tables with the added benefit that they include replicate estimates to calculate the variance (as well as margin of error, or MOE).  This is the method that the ACS uses to calculate the MOE published on data.census.gov.  The documentation provides examples on how to calculate the MOE using the replicate estimates for a ratio.  As an interesting aside, if you subtracted the Cook county estimate from the Illinois estimate (and did the same for the 80 replicate estimates to calculate the MOE), you would effectively be calculating the MOE for all counties in Illinois except for Cook county.  This is because the estimate for Illinois is made up of the sum of the weights for Cook county plus the sum of the weights for all other counties in Illinois.  So, subtracting out Cook county leaves you with the weighted estimate of all other counties.  The same logic applies to the replicate estimates, which are used to calculate the MOE.

Another method you could use to take the covariance into account would be to calculate it in a more classical sense.  It has been a while for me, but this method uses the correlation as well as the variance of the individual estimates.  For this approach, you would need the microdata.  While this is not available, there is the Public Use Microdata Sample (PUMS).  PUMS is a subsample of the full ACS microdata.  In addition, the only geography available below the state level on the PUMS files is the Public Use Microdata Area, or PUMA.  PUMAs are completely contained within a state and are designed to contain roughly 100,000 people.  Using Tigerweb, it appears that the boundaries for Cook county lines up with the boundaries of several PUMAs.  So, it appears that you could aggregate several PUMAs together to obtain an estimate for Cook county.  Then, you can calculate the covariance.

In practice, both methods would require some technical expertise as well as a decent amount of work.  From experience and anecdotal evidence, the approximate MOE that does not take the covariance into account between two estimates tends to be fairly close to the MOE that does incorporate the covariance.  I took a look at the 20015-2019 ACS 5-year poverty rates for Cook county and Illinois (as well as the US).  According to data.census.gov, Cook county has an MOE of +/-0.2, while Illinois has an MOE of +/-0.1.  If you approximate the MOE, you get MOE_approx = sqrt( 0.1^2 + 0.2^2) or about 0.22.  It should also be noted that for very small, but not zero, MOEs, the ACS rounds to the lowest accuracy.  So, if the MOE for Illinois were much small than 0.1, it would still be rounded to 0.1.  That is to say, the MOE for Illinois may be conservative to begin with.

So, if you want to calculate the covariance, it requires some additional work.  If you are producing this estimate for a research project (or some other reason) and need something more exact, then putting in the extra work might be worth it.  Otherwise, for a simple comparison, the amount of work to calculate the covariance vs. the fact that the covariance may not have any effect on the result would, from a practical standpoint, argue for not taking it into account.

As a final note, it is worth mentioning that if you are examining poverty rate by other demographics, the MOE can be much higher.  For example, in Cook county, for Native Hawaiian and Other Pacific Islanders alone (NHOPI), the poverty rate is 15.5% +/-8.2%.  The total NHOPI estimate (both in and not in poverty) in Cook county is about 1,900, and about 4,300 in Illinois total.  In this instance, due to the small sample size (as well as the large percent of NHOPI in Cook county compared to the rest of Illinois), the covariance could play a much larger role."

Matthew Brault Nick Kobel Mark Mather Here is the response I got from the Census Bureau directly.  Just some food for thought!

"You are correct that the poverty rate for Cook county is not independent from Illinois.  Depending on what they are using the comparison for will determine if they should take the covariance into account in some manner.  It is reasonable to compare counties to the state estimate, even though they are not independent in order to examine how the poverty rates may vary at a finer geographic level.  

To get into the details, there are two ways that I can think of to adjust the MOE to take the covariance into account.  One method is to use the Variance Replicate Estimate (VRE) tables (view documentation).  The VRE tables are the same as published detailed tables with the added benefit that they include replicate estimates to calculate the variance (as well as margin of error, or MOE).  This is the method that the ACS uses to calculate the MOE published on data.census.gov.  The documentation provides examples on how to calculate the MOE using the replicate estimates for a ratio.  As an interesting aside, if you subtracted the Cook county estimate from the Illinois estimate (and did the same for the 80 replicate estimates to calculate the MOE), you would effectively be calculating the MOE for all counties in Illinois except for Cook county.  This is because the estimate for Illinois is made up of the sum of the weights for Cook county plus the sum of the weights for all other counties in Illinois.  So, subtracting out Cook county leaves you with the weighted estimate of all other counties.  The same logic applies to the replicate estimates, which are used to calculate the MOE.

Another method you could use to take the covariance into account would be to calculate it in a more classical sense.  It has been a while for me, but this method uses the correlation as well as the variance of the individual estimates.  For this approach, you would need the microdata.  While this is not available, there is the Public Use Microdata Sample (PUMS).  PUMS is a subsample of the full ACS microdata.  In addition, the only geography available below the state level on the PUMS files is the Public Use Microdata Area, or PUMA.  PUMAs are completely contained within a state and are designed to contain roughly 100,000 people.  Using Tigerweb, it appears that the boundaries for Cook county lines up with the boundaries of several PUMAs.  So, it appears that you could aggregate several PUMAs together to obtain an estimate for Cook county.  Then, you can calculate the covariance.

In practice, both methods would require some technical expertise as well as a decent amount of work.  From experience and anecdotal evidence, the approximate MOE that does not take the covariance into account between two estimates tends to be fairly close to the MOE that does incorporate the covariance.  I took a look at the 20015-2019 ACS 5-year poverty rates for Cook county and Illinois (as well as the US).  According to data.census.gov, Cook county has an MOE of +/-0.2, while Illinois has an MOE of +/-0.1.  If you approximate the MOE, you get MOE_approx = sqrt( 0.1^2 + 0.2^2) or about 0.22.  It should also be noted that for very small, but not zero, MOEs, the ACS rounds to the lowest accuracy.  So, if the MOE for Illinois were much small than 0.1, it would still be rounded to 0.1.  That is to say, the MOE for Illinois may be conservative to begin with.

So, if you want to calculate the covariance, it requires some additional work.  If you are producing this estimate for a research project (or some other reason) and need something more exact, then putting in the extra work might be worth it.  Otherwise, for a simple comparison, the amount of work to calculate the covariance vs. the fact that the covariance may not have any effect on the result would, from a practical standpoint, argue for not taking it into account.

As a final note, it is worth mentioning that if you are examining poverty rate by other demographics, the MOE can be much higher.  For example, in Cook county, for Native Hawaiian and Other Pacific Islanders alone (NHOPI), the poverty rate is 15.5% +/-8.2%.  The total NHOPI estimate (both in and not in poverty) in Cook county is about 1,900, and about 4,300 in Illinois total.  In this instance, due to the small sample size (as well as the large percent of NHOPI in Cook county compared to the rest of Illinois), the covariance could play a much larger role."