Hi everyone!
Has anybody looked into this apparent statistically significant difference between the two surveys on educational attainment data?
2020 CPS counts a total of 113381 thousand 18+ civilian noninstitutionalized population with at least an associates degree, out of a total 252117, so ~45% in share.
Estimate for the same scope using 2020 5-years ACS PUMS is 97417 out of 253296, or ~38%.
The discrepancy is even more pronounced for only doctoral degree holders, where CPS estimates 4.7M vs ACS 3.2M, a nearly 50% higher number for the former.
Checking both survey's questionnaires, the question for educational attainment is essentially identical and reasonably unambiguous. What would be some factors leading to this rather stark discrepancy?
Really appreciate it if anyone could shed some light for me!
Sincerely,
Michael
Best,
Hey Tom, Bernie,
Thank you both for replying! Particular Tom, for your warm recommendations.
In regards to the PUMS age universe setting, I double checked on the parameters and it shouldn't have caused any issues. The statistically significant difference still exist if we change the scope to 25+ yr olds, where ACS PUMS suggest 41% vs CPS 48% of the population has at least an associate's degree.
The official subject definitions document is very helpful and points to an article by a Census Bureau staff member N. Scanniello covering the exact subject matter. The unfortunate thing is it is fairly dated (2004), and it appears the discrepancy back then between the two surveys were considerably smaller than contemporary estimates. The paper mentions an approximate 1.3% difference in share of 25 yrs+ population with an associate degree (2.0% in 2020), 0.9% difference in bachelor's degree (3.3% in 2020), and not-statistically-significant difference in advanced degree holders --- very different from current day situation where CPS ASEC has a nearly 50% higher estimate for doctoral holders (4.7M vs 3.2M). The explanations offered in the paper are fairly conventional surveying issues such as nonresponses or wording differences, and in my humble opinion is becoming inadequate for justifying the differences exhibiting today. It would be very helpful if the Bureau someday updates this report.
After some personal considerations, I have decided to settle on seeing this difference being simply due to unaccounted-for-errors likely in selection bias. If one were to blindly guess which of ACS or CPS has the "truer" value I would undoubtedly lean to ACS for its clearly stronger coverage rate and sampling size. However, in less academic environment such as citations in business presentations I would probably grudgingly use CPS figures. One reason being the apparent higher familiarity most people nowadays have with CPS data over ACS. Common newspaper headline figures such as non-farm payrolls and unemployment rates are all based on the more frequent CPS survey, allowing it to gain much greater public attention in general. It would be quite the pain to suggest an alternative dataset that has rather significant differences and try explaining the details. Another reason being people generally prefer to see more "favorable" estimates, for a variety of reasons. And insisting to use a "bad" number sometimes cause unnecessary suspicions questioning your motives, which can be annoying as well.
Due to these complications I will go on to suggest US having 4.7M doctoral degree on the report I'm working on despite being against best judgement. And I will seek solace from knowing the 1.5M phantom PhD holders out there wouldn't care less.