During the protracted health care debate of 2009 and 2010, reformers suffered from choosing an easy target rather than the one with all the power. Time after time, they would depict the insurance industry as the source of all the system’s ills. Certainly, this made some sense, considering that the changes on offer amounted to insurance reform: guaranteeing coverage to anyone who seeks it, limiting premium prices, and building state health care exchanges where consumers could shop for insurance options.

But the more entrenched opponents of reform were the health care providers—the hospitals that charge patients and their insurance companies. They didn’t want a public option—i.e. government-provided insurance—to compete with private insurance because they feared there would be pressure to reduce prices from a non-profit institution empowered with Medicare’s bargaining advantage. They didn’t want the government to set prices or really any constraint on their ability to make profits. And so those ideas died in favor of more modest reforms.

The same problem is playing out in the debate over higher education. The left is fairly united on making college affordable to anyone who wants to enroll, and ending the crippling burden of student debt. Hillary Clinton has even expanded her plan to make public colleges and universities tuition-free for anyone with a family income up to $125,000 a year.

So far, the identified villains in this debate include student loan companies (since most student loans are now federally issued, that means the private companies that collect payments on the loans and that routinely cheat borrowers). Another is for-profit colleges, the source of a disproportionate number of student debt defaults. Just as with the insurance industry, those are worthwhile and even necessary targets. There is no reason for student loan servicer middlemen to exist; the government surely knows how to take a payment, as it controls the world’s largest collections agency, the Internal Revenue Service. And deceptive for-profit colleges are certainly a scourge.

But nobody has really taken on the obvious opponent to higher education affordability reform: the colleges that would be forced into competition with the tuition-free ones.

Those colleges aren’t standing idly by. Private college presidents have publicly attacked the idea of creating more free colleges, saying that their institutions would struggle to survive if they had to challenge a free alternative. One of these presidents, the typically lauded Sheila Bair of Washington College, former president of the Federal Deposit Insurance Corporation, called the proposal unaffordable. She also said it would roll back the “richness of choice” available to students who can select public institutions, women’s or religiously affiliated colleges, and historically black colleges and universities.

Clinton’s plan offers some support to these groups as well, including a $25 billion fund to support minority-serving institutions. She proposes an executive moratorium on federal student loan payments, to sort borrowers into interest-rate refinancing and income-based repayment plans in which nobody would pay back over 10 percent of their income. And she would also continue the Obama administration’s efforts to forgive student loans for borrowers defrauded by colleges that lure them in based on misleading job placement statistics or other statements.

Guess who’s against that? Colleges! They claim that it would subject them to frivolous lawsuits from students. If a graduate doesn’t get the career they want, under the new rules they might have standing to sue, the colleges argue. The opponents include not just private colleges, but public universities—which would be tuition-free under Clinton’s plan, incidentally, so it’s unclear what a loan forgiveness effort would be able to recover.

It’s not hard to figure out why colleges resist changes to the way we finance higher education: They do incredibly well under the status quo. The cost of a four-year education, including tuition, fees, room, and board, has risen nearly three-fold, from $16,213 for a private non-profit in 1975 to $43,291 today (adjusted for inflation), according to the College Board. For public colleges, the price jumped from $7,833 in 1975 to $19,548 today.

At the public level, states have pulled back funding for higher education, causing some of the increase. But all colleges benefit from the current system of financing higher education with debt. Colleges can increase prices without students feeling immediate pressures on affordability. Only after they graduate do borrowers get the bill that haunts them the rest of their lives.

Where is all this money going? Many point to the rise of university administrator salaries and staffs. (You might know them as the college presidents complaining about all these changes Hillary Clinton and Barack Obama want to make to the higher education system.) Another possible culprit is bloated construction costs, which go toward building show palaces for…well, for the administrators to crow about, and attract more students to pay those high tuition rates.

Whatever explanation appeals to you, the failures of the current system point clearly toward supplying a public debt-free option as a way to drive down costs. This would provide an anchor against skyrocketing costs, and force the cleanup of administrative bloat and unnecessary construction spending. You can force public colleges to lower costs as a condition of accepting tuition reimbursement. And if a glut of student loans causes prices to rise, then a free public option would reverse the effect.

This goes back to the core issue: Incumbents prospering from a system don’t have much interest in seeing it change. And those wanting to reform the system must challenge those incumbents. It’s easier to single out the easy villains, the Sallie Maes and the Corinthian Colleges. But that just sidesteps the real opponent, and will lead to something far less than reform.

Most colleges are seen in a fairly benevolent light. Large higher-education institutions are often major employers in their communities. They drive innovation, and provide sanctuary to some of our best thinkers. And for many adults, they are wrapped in the warm and fuzzy gauze of nostalgia. It’s hard to get people to see them as propping up a crisis that is over-burdening students and even stunting the growth of our economy. But until we do, it’s going to be very difficult to see any change.