The outrage over Mitt Romney’s extended off-the-record riff to wealthy donors about the fact that “47 percent of Americans pay no income tax” has shown no sign of dying down. As of now, this looks like the defining moment of his presidential campaign. In lumping together those who have no federal income tax liability with those “who believe the government has a responsibility to care for them,” the former Massachusetts governor gave new life to every crude caricature of conservatives as class warriors for the ultrarich.

But did off-the-record Romney have a point? Is it a problem that we have narrowed the federal income tax base, or is there a case that conservatives seeking to contain the growth of government should strive to make the income tax base even narrower?

In a 2001 interview with Nicholas Lemann of the New Yorker, Republican Senator Jim DeMint of South Carolina called the narrowing of the tax base “a major crisis in democracy.” Just months before the first Bush tax cut removed millions of households from the federal income tax rolls, DeMint warned that “the tax code will destroy democracy, by putting us in a position where most voters don’t pay for government.” DeMint’s dark premonition wasn’t enough to get President Bush to revamp his tax cut, but the idea has grown more popular among conservatives in the intervening years, hence Romney’s riff.

Critics of DeMintism point out that virtually all of the households that don’t pay federal income taxes pay other taxes, including payroll taxes and state and local taxes. Indeed, some pundits have declared that this is Romney’s saving grace: Hardly anyone in the GOP nominee’s 47 percent actually believes that he or she is part of the 47 percent.

The deeper problem isn’t that too few people are paying federal income taxes, as DeMint and Representative Michelle Bachmann and now Mitt Romney maintain. Rather, it is that most of the taxes we do pay, with the big exception of retail sales taxes, are quite stealthy. In the days before income tax withholding, taxpayers had to write checks to the federal government. Now, many taxpayers have a better sense of the size of their income tax refund than of what they’ve actually forked over to the federal government.