Obamacare critics are furious and even some of the law's supporters are dumbfounded about some news that broke on Tuesday, via a story by Robert Pear in the New York Times. The subject is the Current Population Survey, or CPS, which is one of the tools that the government uses to measure how many people have health insurance. The Census Bureau operates the survey and, as Pear reported, it is changing the wording of survey questions. That decision will make it difficult to compare future results to past results.
Conspiracy theorists will say that Obama is "cooking the books"—in this case, by trying to hide data on whether the law is actually helping the uninsured. Count me as extremely skeptical. The change may or may not make sense on the merits. But it shouldn't change what we know about the law's impact on the number of uninsured—or even when we know it.
Figuring out whether Obamacare is reducing the number of people without health insurance is obviously a very important question. And the preliminary data—from independent organizations like Gallup and the Rand Corporation—isn’t precise. Those surveys show that more people have insurance because of the law, but they don’t reliably say how many or what kind of insurance people have. That requires more comprehensive, more thorough surveys—the kind that the federal government has traditionally provided.
The CPS is one of those tools. It’s been around for 50 years, has a reasonably large sample, and asks detailed questions about health insurance status. But CPS also has flaws. One is that the main question on health insurance is confusing. The survey, which goes out into the field during late winter and early spring, asks respondents whether they were insured in the previous year. But some people answer the question with reference to their current status, rather than the previous year’s—and some don’t have precise recollections of what happened the previous year.
“The Current Population Survey (i.e. Census) has always been a terrible way to measure the uninsured,” says Larry Levitt, senior vice president at the Kaiser Family Foundation. “The survey is done in the February to April period every year, but asks about insurance coverage from the prior year. The measure of the uninsured is theoretically the number of people who were uninsured for all of the previous year, but researchers have known for quite some time that it doesn’t really seem to be measuring that, because of the way people answer the questions. It seems to be more like a measure of the number of uninsured at a point in time, but it’s of course always been ambiguous what point in time.”
Another problem with the CPS is that its great strength—the detailed questions about type of insurance, and so on—made the old questionaire somewhat obsolete. Going forward, for example, it’s important to know whether people got coverage through one of the new marketplaces. But to do that, the survey has to have a question about the marketplaces, which didn’t even exist until this year. For these reasons, the Census Bureau decided to overhaul the survey, designing new questions that would help clarify previous ambiguities and produce information that reflects how insurance works under Obamacare. Apparently they opted against including a small sample with the old questions, just to preserve the ability to make comparisons, figuring then both sets of data (new questions and old) would be less reliable as each one's sample size got smaller.
That decision will clearly have some negative consequences. It means that the data coming out in the future, starting this October, won't look like the data that came out in the past. As Bloomberg View columnist Megan McArdle points out, making that kind of change in the middle of such an important transition is not typically a smart thing to do. And it's not just conservatives or libertarians unhappy with the transition. Scholars hoping to study health care trends over the long term, going back many years, just lost a valuable data source. "Altering the questions right now so that we can’t measure what’s going on is terrible," writes Aaron Carroll, the University of Indiana pediatrician and researcher. "If they were so bad they needed altering, a few years ago would have been better. Or, a few years from now. But right now? It’s killing me."
But whether or not those downsides outweigh the upsides—I've been going back and forth on that all day—the change shouldn't get in the way of measuring how many people got insurance from Obamacare. For one thing, the government runs two other major surveys on the uninsured. One is called the National Health Interview Survey, or NHIS. Sponsored by the Centers for Disease Control but also conducted by the Census Bureau, this survey asks a set of detailed questions about health, including insurance status.
The Census Bureau also runs something called the American Community Survey, or ACS. It's newer than the CPS and it didn't start asking about health insurance until a few years ago. Its questions are also less detailed. But it asks respondents about their insurance status today, not last year, so it doesn't have the same ambiguity. It is also the largest of the surveys, by far. It has a sample size of 3 million households over the course of the year, according to the Census Bureau, which works out to 250,000 a month. The total CPS sample size is 100,000 households. (For more on the two, see this Robert Wood Johnson Foundation primer from a few years ago.)
In addition, it's not as if the change in CPS makes studying Obamacare's impact impossible. Remember, the survey asks people about their insurance status during the previous year. As a result, the new questions will produce a set of data about 2013, making possible comparisons to 2014. (Sarah Kliff at Vox.com explains this in greater detail.) Of course, those results might not be out until 2015, well after the midterm elections. That timing is sure to set off alarms. But—and this is the crucial point—definitive information about 2014 was never going to come out before people cast their ballots in November. The best we'll probably get is some preliminary data from NHIS and perhaps some sketchy numbers from CPS. Solid numbers won't be available until fuller releases of data during 2015. "The bottom line is that there are three major surveys, none of which were going to give us a reliable number before the election,” says Jonathan Gruber, the MIT economist and Obamacare architect.
Maybe the decision about CPS is the wrong one and maybe, somewhere along the chain of command, an official decided this shift would play well politically. I honestly don't know. But what was true before this change is true now. We'll find out whether the Affordable Care Act is reducing the number of people without insurance—and by roughly how much. It's just going to take a while.