“It remains an enduring challenge to our nation’s education system to reconcile the pursuit of diversity with the constitutional promise of equal treatment and dignity,” Justice Anthony Kennedy wrote in a Supreme Court opinion two years ago. The University of Texas, he a majority of justices concluded, had met this challenge with its admissions policies. The 4-3 ruling in 2016’s Fisher v. University of Texas effectively meant that American universities could lawfully consider racial diversity when admitting new students as long as Kennedy, the court’s swing justice, remained on the court.

Now Kennedy is gone, and with him, a fifth vote on the Supreme Court to uphold the constitutionality of such admissions policies. Justice Brett Kavanaugh’s confirmation all but guarantees that the court will revisit the issue in the near-future. His presence may also give the court’s conservative wing the votes it would need to chip away at 40 years of precedents affirming that American higher education has a compelling interest in ensuring a diverse student body.

Harvard’s legal battle over its own admissions practices began before Kennedy retired. But his departure raises the stakes even higher as the Ivy League university’s own case goes to trial this week in Boston. Harvard is defending its policies for admitting new students from a lawsuit brought by a group of Asian-American applicants who say they were kept out by an informal quota system at the school. Though many aspects of the case are unique to Harvard’s quirky system for choosing new students, it could still eventually give the high court the opening it needs to make far-reaching changes.

For years, the face of the movement to curb affirmative action was a young white woman. Abigail Fisher kicked off a years-long legal battle with the University of Texas after it denied her application to UT-Austin’s 2008 freshmen class. Representing her in court was the Project for Fair Representation, a conservative legal organization that specializes in challenging the legislative victories of the civil-rights movement.

The organization’s founder, Edward Blum, is a not a lawyer. But he has an uncanny knack for bringing momentous cases before the Supreme Court. He orchestrated the successful legal campaign to gut the Voting Rights Act of 1965, which culminated in the court’s 2013 ruling in Shelby County v. Holder. The 5-4 decision struck down Congress’s formula for determining which states had to seek federal approval before changing their voting laws. Though Chief Justice John Roberts claimed the nation had moved on from the Jim Crow era, the ruling sparked an immediate surge in voter suppression in Republican-led states across the country.

In 2016, the Project for Fair Representation also urged the Supreme Court to rethink how it enforces the “one man, one vote” principle. The organization’s lawsuit in Evenwel v. Abbott tried to compel states to apportion their legislative districts by voting population, not by total population. Had it succeeded, the nation’s whiter and more rural regions would have seen a tremendous boost in legislative power at the expense of diverse urban areas with more diverse communities. Instead, the Supreme Court unanimously rejected the proposition.

That same year, the justices rejected Blum and Fisher’s bid to strike down the University of Texas’s method for accepting new students. But the case hinted at another way forward. In a dissenting opinion, Justice Samuel Alito suggested that the university’s methods increased black and Hispanic representation in the student body at the cost of Asian-American representation. “In UT’s view, apparently, Asian-Americans are not worth as much as Hispanics in promoting ‘cross-racial understanding,’ breaking down ‘racial stereotypes,’ and enabling students to ‘better understand persons of different races,’” he wrote, quoting from a brief filed by an Asian-American legal group.

The Harvard lawsuit was brought in 2014 by Students for Fair Admissions, a nonprofit organization also established by Blum. It represented a group of Asian-American applicants who claimed that the school’s admissions policy violated federal education laws by effectively setting a de facto racial quota through their evaluation of students’ personalities. At the time, Harvard and other Ivy League schools were under growing scrutiny for declining admission rates for students of Asian descent.

“Harvard and other academic institutions cannot and should not be trusted with the awesome and historically dangerous tool of racial classification,” the group argued in its complaint. “As in the past, they will use any leeway the Supreme Court grants them to use racial preferences in college admissions—under whatever rubric—to engage in racial stereotyping, discrimination against disfavored minorities, and quota-setting to advance their social-engineering agenda.”

Students for Fair Admissions’ lawsuit isn’t without its critics in the Asian-American community, some of whom have expressed concern that their situation is being used as a vehicle to undermine efforts to help other disadvantaged communities. “My stance on affirmative action is a general reminder to the rest of America—and especially to Edward Blum—that I, along with so many other Asian Americans, refuse to be tools of white supremacy, and that we stand in alliance with all communities of color,” Thang Q. Diep, a Harvard student, said at a rally on campus against the lawsuit on Sunday.

If the lawsuit or another one like it gives the Supreme Court an opportunity to strike down race-conscious admissions programs, colleges and universities may have to turn to alternative measures to ensure a diverse student body. One of the most controversial options would be to focus on socioeconomic factors instead of race. In 1996, California voters passed Proposition 209 to ban the state from considering race, gender, or ethnicity in higher education and certain other spheres. California’s state university system responded with sweeping changes to its admissions policies to offset the effects.

“Schools have reduced their reliance on standardized test scores for admissions, banned legacy preferences for the children of alumni, encouraged more community-college transfers to four-year institutions, and created new outreach programs to high-poverty high schools,” Richard Kahlenberg, a senior fellow at the Century Foundation, wrote in 2014. “In part because of these efforts, UCLA and UC–Berkeley are far more socioeconomically diverse than most selective colleges.” This, he argued, bolstered the higher-education system’s ability to encourage social mobility.

Another proposal is to rethink the admissions process entirely. The Atlantic’s Alia Wong wrote earlier this year that some experts are suggesting that elite schools could use a lottery system of sorts to fix structural issues in the current process. “To continue to promote diversity, the school could give extra weight to certain applicants depending on, say, their zip code, the kind of high school they attended, their income, and their race,” Wong explained. “Then admissions officers could use those criteria to whittle down their batch of 40,000 applicants to a much smaller pool of qualified contenders and from there select the final 2,000 or so through a lottery (not everyone who’s admitted attends).”

Neither of these alternatives is likely to work as well at achieving as racially diverse a student body as some of the current measures used by the nation’s top colleges and universities. The most effective tool may be race-conscious admissions policies in one form or another. But with the Supreme Court’s disapproval almost certainly imminent, more creative options may be needed to achieve the same goals.