Amid the latest wave of sexual harassment allegations, you might conclude, with some relief, that predators are finally being held accountable for long-repressed abuse. But while several men have lost their jobs, there’s been little accountability in a legal sense. Several cases reported in the media—from Harvey Weinstein to Kevin Spacey to Russell Simmons—would constitute crimes if proven, but thus far there haven’t been any indictments. In Weinstein’s case there have been reports of criminal investigations, but in general the #MeToo movement has played out in the press and on social media.

This makes many uncomfortable. Each story has its own particulars, but they all inspire demands for the same conclusion: effectively, banishment from the public square. Bill Maher warned against lumping in Al Franken’s alleged groping with Roy Moore’s alleged stalking of minors. The mantra to “believe women” bumps up against questionable accusations, such as assault allegations against Senator Richard Blumenthal that were made by what appears to be a Twitter bot. Social media doesn’t make allowances for legal concepts like the presumption of innocence, and it can justifiably lead to fears of mob rule or partisan exploitation.

But we should identify the real culprit for this state of affairs: the long, slow abandonment of the rule of law in America. The reason adjudicating sexual misconduct claims has been left to the media and the crowd is that people have no expectation that the legal system will adjudicate those claims fairly. How can anyone blame them? They have witnessed endless instances of powerful people, mostly wealthy men, getting away with criminality and deception, in every context imaginable. When you don’t have a working justice system, you get a kind of vigilantism as a result. The problem isn’t the vigilantism—it’s the broken framework that leads desperate people to take matters into their own hands.

That powerful people face little sanction for misbehavior is an old story, as true in gender as it is in class. But brazen impunity for the powerful is a hallmark of our era. The worst financial crisis in America in nearly a century led to practically no convictions for those whose actions facilitated the meltdown. The Catholic Church shuttled around sex-abusing priests for decades with little reckoning. Cops shoot black people and go back on the job.

On the opposite end of the spectrum we have a wildly punitive justice system for those unlucky enough to have no money or power. Black men are sentenced to more jail time for committing the exact same crimes as whites. Nonviolent drug offenders rot in jail. Mandatory minimum sentences and three strikes laws trap many in an unforgiving correctional system.

This inequity has scratched at the social fabric of America for practically forever. Racism, sexism, and classism create imbalances in a system intended to be blind and without prejudice. Today, any reasonable observer has to have lost some faith in traditional modes of justice to protect the weak and the powerless.

That’s doubly true in the case of sexual harassment, which women (and men in many cases) have endured for far too long. Women faced skepticism and outright disbelief when they spoke out. Despite the almost universal presumption that the absence of human resources departments facilitates abuse, human resources departments or structures set up in the workplace to report abuse often work in the interests of protecting the company, not protecting victims. Non-disclosure agreements and private arbitration have allowed corporations to silence women, with the force of law. And when alleged serial harassers like Weinstein and Bill O’Reilly settled disputes out of court, they did so while admitting no wrongdoing—a tactic familiar to Wall Street banks that agreed to pay financial crisis–related fines.

Before the Clarence Thomas hearings in 1991, there was barely even a language for sexual harassment in the workplace. And even now, despite mandatory training sessions featuring cheesy videos, a culture of silence has predominated. This culture gets reinforced by a culture of no accountability for people with power. Victims have come to believe that law enforcement doesn’t work in their interests when their abuser is prominent, or even if the abuser simply wields some degree of power over them. So they turn to our digital town halls and document their stories.

It’s true that social media is not equipped to handle these allegations with the care of the constitutional framework we built for settling disputes. As a journalist, I believe in the media’s ability to deal with these issues with fairness and respect for the truth, but it can’t replace a system of evidence and procedure and precise readings of the law.

More important, making Twitter and Facebook the primary safety valves for women with tales of abuse leaves behind all the less heralded women who endure the same abuse at their jobs but don’t have the same megaphone. Hotel housekeeping unions and other labor groups have actually done a far better job than the Screen Actors Guild in protecting their members on the job, but lower-wage women are simply at a disadvantage if internet outcry becomes the main accountability mechanism for sexual misconduct.

These inequities, however, are directly tied to the reason victims feel no recourse but to step outside the legal system. We should be deeply enraged that, for large slices of the population, no mechanism exists to punish misconduct swiftly and dispassionately. The burst of allegations since the Weinstein scandal broke—which bears parallels to the failure of similar allegations to stop Donald Trump from becoming president—springs from frustrations with the justice system. The crowd isn’t good at enforcing the law because it’s not their job. But what are people supposed to do when they have no other choice?

There’s a toxicity to a two-tiered justice system that goes well beyond the particular crimes being excused, whether they pertain to financial malfeasance or sexual harassment. It creates a rot at the heart of our society. And it creates rage, which shoots out in unpredictable ways. When the law loses legitimacy, as Jill Leovy details in Ghettoside, her remarkable account of south Los Angeles, citizens resort to street justice. Apply that to the rest of America and you understand how women—actually how all of us—regularly feel.

So those who are handwringing about treating people fairly in this moment need to look at the root cause. We haven’t been treating people fairly in America for a long time. Too many have suffered from a broken justice system. The best response to the #MeToo revolution is to restore the rule of law so women don’t have to use a hashtag to ensure their story of assault gets heard.