With the news that Attorney General Eric Holder is resigning, many journalists are rightfully touting his record on civil rights issues. His record on reining in Wall Street is less sparkling.

Holder took over the Justice Department in 2009. It was at a time of national crisis, with the economy cratering and millions of Americans out of work. His job was not, of course, to rebuild the economy. It was to hold accountable those who caused the crisis. On that, he has fallen woefully short: Criminal prosecutions of financial crimes have been rare under Holder’s DOJ, and it will have consequences for years to come.

To understand why, think back to before the Great Recession. In the early and mid-2000s, a housing bubble built up as borrowers took on more and more debt to take out mortgages on new homes. Lenders packaged these sub-prime loans into complicated financial products and sold them to investors, not caring about their quality. But even if the borrowers defaulted, investors reasoned, housing prices wouldn't fall across the entire country, thereby allowing investors to foreclose on the house and recoup their money. But borrowers everywhere defaulted, housing prices plummeted nationwide, and banks were left holding massive amounts of toxic assets that were worth pennies on the dollar. Eventually, Lehman Brothers collapsed and the federal government passed the Toxic Asset Relief Program to bail out the banks.

Why did borrowers take out loans they couldn’t pay back? Most of the blame lies with the banks and their executives. David Dayen has documented the massive fraud committed by banks before the crisis. As he explained one year ago:

The originators faked the appraisals to make bigger loans, and fudged the income statements of their borrowers. The banks sold bonds backed by these mortgages to investors without revealing the shoddy quality of the loans. They knew about the shoddy quality – they hired third parties to examine the mortgages – but when they found all the toxic stuff, they used that information to get a discount from the originators, rather than pass on the warnings to investors. When the investments started going bad, investment banks used fraudulent accounting as bad as Bernie Madoff, taking large segments of their toxic assets off the books, pretending risky assets were safe, hiding the truth from shareholders and regulators.

Under Holder’s leadership, the DOJ has brought and settled civil cases with many banks for these crimes. Last November, JP Morgan reached a $13 billion settlement with the DOJ. In July, Citigroup settled for $7 billion. Most recently, the department announced a record $17 billion settlement with Bank of America. Holder and the department have often touted these settlements as evidence that they are being tough on the banks. But often the top-line numbers on the settlements overstate what the banks will actually pay. Bank of America, for instance, may only pay $12 billion.

For many of these settlements, the DOJ didn’t even bring a civil case, as Dean Starkman explained recently in this magazine. “[B]y imposing a fine without documenting the underlying abuses,” he wrote, “the Justice Department has permitted the banks, for a price, to bury their sins.” Even worse, in the majority of these cases, the banks weren’t required to admit to criminal wrongdoing. That policy has finally begun to change, as shown by the BNP Paribas’s guilty plea in June. But it has taken more than five years to reach this point.

Meanwhile, Holder hasn’t even tried to file criminal cases against major banks. In February 2013, the DOJ settled with HSBC for $2 billion after the bank was caught laundering money for drug cartels and Saudi Arabian banks with ties to terrorist organizations. If there was ever a case that deserved criminal prosecution, this was it. But the DOJ took a pass. Faced with bipartisan criticism from Congress, Holder responded, “I am concerned that the size of some of these institutions becomes so large that it does become difficult for us to prosecute them when we are hit with indications that if you do prosecute, if you do bring a criminal charge, it will have a negative impact on the national economy, perhaps even the world economy.” That is, the bank was too big to jail. Holder immediately walked back his comments, but that mindset seems to have played a big role in the department’s decision not to prosecute any Wall Street banks for criminal wrongdoing in relation to the financial crisis.

“Holder came into office in the immediate aftermath of a devastating financial crisis caused by an epidemic of corporate crime and wrongdoing,” Robert Weismann, the president of the regulatory watchdog Public Citizen, said in a statement Thursday. “Five years later, he has failed utterly to hold the perpetrators of the crisis accountable.”

The DOJ’s prosecution of high-level executives has been even worse than its prosecution of banks. In fact, the department has prosecuted zero executives for those crimes. One way to convict Wall Street executives is to flip low-level bankers, as the DOJ is now doing for crimes manipulating foreign exchange markets. But that tactic has not been used to prosecute crimes after the financial crisis.

Prosecuting bankers would show the public that regulators are committed to finding and punishing financial fraud. More importantly, it would deter Wall Street from committing those same crimes again. The possibility of going to prison could deter bankers from doctoring loan forms or using fraudulent accounting practices. Instead, many top-level executives walked away with golden parachutes and are living nicely despite their crimes. What’s going to deter the newest Wall Street execs from making the same choices?

Prosecuting the banks with their well-funded legal teams for criminal crimes wouldn't have been easy. But the DOJ has a lot of legal firepower as well. Holder simply never tried to use it to hold Wall Street executives accountable. That is a major blemish on Holder’s record. Bankers sleep easier at night thanks to his decisions. And when the next financial crisis hits—and when we discover that financial fraud was a major cause of it—Holder will deserve blame as well.