Four members of the House were worried about the precedent set by his removal—and the possibility it would be used against Black lawmakers.