Vox's Ezra Klein argued Tuesday that Steve Scalise, the House Majority Whip whose political career is in jeopardy over a speech he gave to white supremacist leaders in 2002, made compromises with a racist political culture to win elections in Louisiana. Klein is right that it’s those compromises, and not the speech itself, that ultimately put Scalise’s position at risk. But this only points to a broader problem that all politicians face: When can you compromise your views to get elected? And when is that unacceptable?

When news broke that Scalise spoke to the European-American Unity and Rights Organization (EURO), a group founded by former Klu Klux Klan leader David Duke, Red State’s Erick Erickson offered a representative reaction of the political world: How could he not have known that Duke was involved? But that prompts another question: If he did know, why would he ever have gone to the event? To me, those questions are equally unanswerable—it doesn’t make sense that he wouldn’t know, and it doesn’t make sense that he would give the speech if he did know. Given that Scalise doesn’t have a record of racist comments or a history of attending gatherings with white nationalists, I, like Klein, am willing to give him the benefit of the doubt that this was just an honest mistake. And under that belief, there’s no reason that Scalise cannot continue to serve as the House Majority Whip. He is no less qualified this week for that position than he was last week.

But Klein makes a strong argument that Scalise’s speech, even if it was an honest mistake, is representative of compromises he had to make to win elections in Louisiana. He also twice voted against making MLK day a state holiday, first in 1999 and second in 2004. The latter passed by a 90-6 vote. (Scalise’s office did not return a request for comment on whether he would still vote the same way today.)

Klein also points to a Roll Call article from 2002, which the company reposted on Monday, in which Scalise said he had similar “conservative” views as Duke but argued that Duke, who had previously served in Louisiana's House of Representatives and was considering another run for office, was unelectable. “The novelty of David Duke has worn off,” he said. “The voters in this district are smart enough to realize that they need to get behind someone who not only believes in the issues they care about, but also can get elected. Duke has proven that he can’t get elected, and that’s the first and most important thing.” Klein is right that these comments are actually more damming than Scalise’s speech. It’s worth quoting him at length:

The question was an opportunity for Scalise to go much further, to say something like, "David Duke is a racist who this state should be ashamed of. He would be an embarrassment to this district if he was elected to Congress." Instead, he implies sympathy with Duke's ends, but argues that Duke can't win. He argues, in effect, that even if you agree with Duke — and Scalise suggests that maybe he does, at least on some issues — voting for Duke is counterproductive.

If you gave Scalise truth serum and asked him about this answer today, my guess is he could give you a pretty good explanation: Duke was popular in Scalise's district, and attacking him as a racist would have backfired. The best way to doom Duke's campaign was to gently, insistently question his electability. Remember: you can't change people's minds about politics unless they think you already agree with them.

Scalise might well have ended up at the David Duke-backed European-American Unity and Rights Organization without knowing who they were or really bothering to find out. He might well have been trying to destroy Duke by questioning his electability rather than his views. But that's only because he was practiced at appealing to the kind of people who really did support David Duke and really were sympathetic to the European-American Unity and Rights Organization. And, now that Scalise has risen through Louisiana politics to become a nationally influential figure, that's the problem.

Scalise is not the only one who has faced this problem. In that 2002 Roll Call article, David Vitter, the current Louisiana senator and a budding politician at the time, also argued that Duke was unelectable. “I honestly think his 15 minutes of fame have come and gone,” he said. “When he’s competed in a field with real conservatives, real Republicans, Duke has not done well at all.” That’s an almost identical message to what Scalise said—and that makes sense. Vitter faced the same political culture that Scalise faced and both have succeeded in rising through the ranks. It’s not surprising that they made similar compromises along the way. Should that disqualify him from holding a leadership position in the Senate? Maybe so.

This points to a broader problem for political candidates: They make compromises all the time to get elected. When does one cross the line from acceptable to unacceptable? Taken to extremes, that’s an easy question. If a state still supported slavery, it would be unacceptable for a politician to adopt that position in order to get elected. That’s a morally repugnant position, no matter a state’s political culture. But the question is much more difficult in the case of Scalise and Vitter. They tacitly tolerated, and maybe even accepted, Louisiana’s racist culture. Most importantly, they didn’t push hard to change it.

The politics of this may be so bad for Scalise and the Republican Party that he will have no choice but to step down from his leadership position. Certainly, if more evidence surfaces that he associated with white supremacists or made racist comments, he may lose his congressional seat altogether. But without such evidence, and politics aside, it’s a tough call whether he should actually lose his position as majority whip.

This article has been updated to clarify that David Duke served in Louisiana's House of Representatives, not the U.S. House.