Sidney Poitier in the 1967 adaptation of ‘In the Heat of the Night.’United Artists / Photofest

In September of 1950, Hughes Allison, a playwright and writer of pulp fiction, responded to a manuscript sent by a writer who requested that he reply via the Mystery Writers of America. The author, Polly MacManus, singled out Allison for a reason: He was black, she was white, and she wanted his advice on how to write from the point of view of a black detective.

Allison warned MacManus he would be “brutally frank” in his letter: “I urge you to abandon the attempt to write about Negroes.” He felt MacManus’s detective “neither talks nor acts nor thinks like any college-trained Negro I’ve ever met.” Another black character in the manuscript, a maid, “is the most unrealistic chauvinist I’ve ever encountered in fiction. And in real life, she would be regarded by the Negro detectives whom I know as the kind of stool pigeon only heaven could spawn.”

He then followed up with what he considered to be one of the major pitfalls white writers face when writing black characters:

When you begin handling Negroes as major characters in fiction you immediately enter into that big and enormous and important and most complex area of American life called the Negro Questionwhere no answer can be secured from any part of that question if conjecture is allowed to play even a small part. You can’t guess. You have to know. You have to know Negro life as Negroes live itand they live on numerous political, economic, social, and intellectual levels growing out of cause-and-effect patterns, the character of which is historical. The history of this matter is well documentedso well documented that those who are informed can tell at a glance who knows and who is guessing.

MacManus may have had good intentions, but those, Allison wrote, “are seldom consistent with the harsh facts of history.”

The best thing about John Ball’s 1965 detective novel In the Heat of the Night may be its setup: While visiting the South, a black homicide investigator from Pasadena reluctantly joins forces with the town’s racist white police chief to solve a murder, and both gain a grudging respect for the other in the process. Ball had said he’d conjured up the idea as far back as 1933, while making an itinerant living as a magician working the coasts, but claimed he had to wait for a more hospitable time in America to write it.

IN THE HEAT OF THE NIGHT by John BallPenguin Classics, 176 pp. $15

That story may have been true, or it may have been a clever marketing ploy to heighten interest for readers at the time. It doesn’t change that Ballan adventure-thriller writer whose prior careers included being a military and commercial airline pilot and a music criticwas an unlikely author to conjure up the source material for Detective Virgil Tibbs.

“They call me Mister Tibbs.” The iconic line uttered by Sidney Poitier in the 1967 film may be in the original book but lacks the vigor and pathos that makes it so indelible on screen. Also added for the film was the famous scene where a wealthy, white plantation owner slaps Tibbs in the face and the detective immediately slaps him back. Poitier’s roiling anger underneath the surface, smoldering for filmgoers to see? The actor’s invention. In Ball’s novel, Tibbs is calm and collected and can sometimes come off as the dreaded “magical Negro.” At the end of the book, police chief Gillespie thanks Tibbs for his service, even though the detective had been arrested and harassed by white cops while on the job. Tibbs responds, “You’re a man to be admired, Chief Gillespie.”

It isn’t odd to see In the Heat of the Night back in circulation in 2015. The book won the 1966 Edgar Award for best first novel, and the great film led to two less-than-great sequels, as well as a very watchable 1988 television series with Howard Rollins as Tibbs and Carroll O’Connor as Gillespie.

But it is odd to see the novel included as part of Penguin Classics, whose reissue in December marked the book’s fiftieth anniversary. The foreword by John Ridley, Oscar-winning screenwriter of 12 Years a Slave and a fine crime novelist in his own right, teems with ambivalence. Ridley calls Tibbs a “monumental” charactercertainly truebut mitigates the compliment, saying Tibbs is “a liberal version of a black man so utterly flawless as to be equally other than something human.” Tibbs is, according to Ridley, “a black man more carefully constructed than fully realized, an approximation of life designed to thwart a common enemy but to be of no threat to its originator.”

Ridley is being charitable. Ball’s version of Tibbs is a caricature, or at best a template for better, more authentic adaptations. Tibbs serves a similar function to John Shaft, another black detective created by a white author, Ernest Tidyman, who became iconic as a result of Richard Roundtree’s portrayal in the eponymous 1971 blaxploitation classic. Neither author lived in their characters’ skins, which, as Aya De Leon wrote in her 1993 essay “The Black Detective in the White Mind,” was a problem: “Creating a man and coloring him brown does not make him a black man; putting a badge or a private detective’s license into his hand does not make him a hero.”

Hughes Allison attended a 1947 Mystery Writers of America meeting on a mission: He was in the midst of a career change. A decade earlier, his Federal Theater Project-sponsored play The Trial of Dr. Beck ran on Broadway for a few weeks. It was a gripping courtroom drama of a light-skinned man charged with killing his wealthy, darker-skinned wife, who got rich selling hair-straightening products for black women. His next produced play, It’s Midnight Over Newark, was named for Allison’s adopted hometown in 1941 (he moved there with his family from Greenville, South Carolina, in 1919, at age eleven). The play, a mix of theater and didactic treatise on discrimination at Newark hospitalsthe first act began with a disembodied voice shouting “What is a Negro?” over the loudspeakerlikely led to reforms in hiring African American staffers in Newark emergency rooms. Theater, however, didn’t pay the bills. Allison was also a reporter for pulp magazines like True Story during the Depression, but publishers like Rinehart & Company rejected his novels.

Hughes Allison, author of the first black detective story in ‘Ellery Queen’s Mystery Magazine,’ pictured in 1950Hughes Allison Archive, Newark Public Library

The man Allison came to hear was Frederic Dannay, one half of the crime-writing duo Ellery Queen, and the founder and editor of Ellery Queen’s Mystery Magazine. In its first six years, the magazine published crime stories from literary writers such as William Faulkner, as well as crime writers like Dashiell Hammett (in reprint, anyway) and Agatha Christie. At the meeting, Dannay reportedly told the crowd, there was no subject matter considered taboo for publication in the magazine.

Allison approached Dannay in the hotel bar. According to an account reprinted in Paula Woods’s 1995 anthology of black crime fiction, Spooks, Spies, and Private Eyes, Allison “challenged [Dannay] to print a serious crime story, set in the Negro community, with a Negro as the main character.” The way Allison saw it, there hadn’t yet been a serious black detective character. Rudolph Fisher’s detective, Dr. John Archer from The Conjure-Man Dies (1932), was more of a comic character; Chester Himes was writing stories and novels, but his Harlem Detectives series were still several years away and would be more appreciated in France, where Himes settled in 1953, than stateside.

Dannay accepted the challenge, and over the summer and fall Allison got to work creating his black detective, a plainclothesman named Joe Hill, operating in a fictionalized Newark called Oldhaven. Hill, unafraid to show emotion or throw a punch, was based in part on Carlton B. Norris, a black detective on the Newark force. In the story, Hill spends seven and a half years in uniform before being promoted to plainclothes, and the murder he investigates turns out to be the work of a black man who, Hill relates in disgust, “knows his own peopleNegroes. He draws his cult membership from the most ignorant, the most superstitious, the most stupid of them. He knows they regard all public institutionsincluding law enforcement agencies and the courtsas hostile to their interests.”

Soon after his meeting with Dannay, Allison explained in a letter why the idea of a black detective was particularly timely. “Most large cities in the North (and a few in the South, I understand) use Negro plainclothesmen a great deal now [and] seek out good ones. Practically all prosecutors and district attorneys in Northern urban communities have Negro investigators and assistants on their staffs today.”

Allison elaborated further in a newspaper profile on why he wanted to dramatize the Newark police: “[They] have done a better job in race relations than almost any other public institution or agency. The average teacher, for instance, thinks he or she can take a short course on minorities at Columbia University and learn a shortcut in dealing with Negro children. The police go into these children’s homes, make friends with their families, are willing to talk matters over.” Allison made this point deliberately, since this sets up a pivotal scene in the Joe Hill story.

After he received the draft from Allison, Dannay reached out to three members of the Newark Police department, including Norris, for their “frank opinions.” The input seemed to help; the Newark deputy police chief deemed Allison’s tale “a darn good story” while a lieutenant insisted, “I’ve been somewhat of a character all my life, anyway.” Norris himself was more laconic: “How could I have stopped [Allison]? We here learn to take everything in our stride.”

Allison’s story, “Corollary,” was first submitted for the Ellery Queen’s annual contest; it netted an honorable mention and a $200 cash prize. Before “Corollary” appeared in the magazine’s July 1948 issue, the New Jersey Herald News quoted Allison saying he “intended to make Joe Hill ... a permanent fiction character like Sherlock Holmes.” The first black writer to appear in Ellery Queen’s and the first black member of Mystery Writers of America looked ahead to a promising future in detective fiction.

It didn’t quite work out that way.

In the Heat of the Night introduces Virgil Tibbs in a clipped, declarative style: “Sam sized him up at a glance, and knew at once he did not belong in Wells. He was fairly slender and dressed up in city clothes, including a white shirt and a tie. Sam guessed that he might be about 30, but it was always hard to tell about blacks. Instead of being stretched out on the bench, he was wide awake and sitting up straight as though he were expecting something to happen.” Sam Wood, the white beat cop who discovers the murder victim, considers Tibbs as a suspect, and the suspense of this chapter, as Wood and the blustering Gillespie realize their mistake, is palpable, yet also contrived.

Compare this description of Tibbs to the one Ball wrote of his fictional detective just three years later for the annual Mystery Writers of America meeting in 1968: “The detective personality with whom I have the privilege of being best acquainted is Mr. Virgil Tibbs, who is a member of the Pasadena, California, Police Department. Mr. Tibbs happens to be a Negro, and I mean that literally. A while ago a certain story suggested itself to me which absolutely required that the principal investigator be a Negro. It was my extraordinary good fortune that

Mr. Tibbs was present and was able to resolve matters in the heat of the night. … Mr. Tibbs’s racial heritage was not the result of casting about for the yet unsung, but the stern necessity of the situation at hand.”

The style difference is owed to Ball’s editor at Harper & Row, Joan Kahn. When she received the first draft of what would become In the Heat of the Night, she recognized the greatness of the concept and got to work fashioning a diamond out of rough coal. She and Ball worked on revision after revision, and the result—great sales, critical success, the Edgar Award, an even better film—portended a long and fruitful working relationship. Then Ball submitted his second Virgil Tibbs novel. Kahn’s reaction wasn’t preserved for posterity, but she was likely frustrated: Here is her author, who has produced a great series opener, one that is stunningly relevant and contemporary and fresh, and now he’s placed his greatest character, detective Virgil Tibbs, in a nudist colony?

Indeed, The Cool Cottontail (1966) opens with a dead body discovered at a nudist resort, and Tibbs has to disrobe to investigate. The seriousness of In the Heat of the Night, was immediately undermined by the premise of its sequel. Kahn required further revisions to the manuscript even after a total rewrite; the marked-up proofs in Ball’s archives, at Boston University, indicate the author’s frustration, even contempt; “no” scribbled with force on page after page to stet her changes. The most telling of Kahn’s edits emerges early on as she runs a line through an expository paragraph: “They know Tibbs,” she wrote of other characters in the scene. “Why should they react?”

Three years after publishing The Cool Cottontail, Ball left Harper & Row for Little, Brown, which published the next three Virgil Tibbs novels (there were seven in all). None of those subsequent books came close to the power of In the Heat of the Night, jettisoning sober realism for more fanciful plots in far-flung locations like Kathmandu; in Tibbs’s final appearance, Singapore (1986), published two years before Ball’s death, the detective is summoned to the Far East to solve a crime at the behest of an “old friend” who hardly figures in earlier books.

It would take another two and a half decades after In the Heat of the Night for the next iconic black detective character to emerge, Easy Rawlins, from Walter Mosley’s Devil in a Blue Dress (1990). Mosley didn’t just live in the skin of Rawlins, a post-World War II private eye, he breathed his entire experience of America, contemporary and the past, into the character, right from the novel’s opening line: “I was surprised to see a white man walk into Joppy’s bar.” And the end of that paragraph: “When he looked at me I felt a thrill of fear, but that went away quickly because I was used to white people by 1948.”

Allison never published another story with Ellery Queen’s Mystery Magazine. But soon after the publication of “Corollary,” he received a letter from Ben Burns, the white, Jewish, Chicago-born editor and co-founder of Ebony and its brother publication, Negro Digest. (Burns wrote about this improbable career move in his 1996 memoir Nitty Gritty: A White Editor in Black Journalism.) Burns liked “Corollary” and wanted to know if Allison had another Joe Hill story. Allison did. “Imposture” appeared in the March 1949 issue of Negro Digest, but the magazine folded a year or so later.

In a 1950 interview with the Newark Evening News, Allison explained: “It’s a battle to get a story about a Negro detective published in a national magazine, you know. I send the publisher a page-by-page explanation of what I’m doing in the story, and how I know what I’m talking about. He has to be ready in case some letters of protest arrive from Mississippi or Georgia.”

Allison struggled to write as he aged. A third Joe Hill story, “Parallax,” began well, but the existing manuscript, located in the University of South Carolina’s archives, lacks an ending. Allison also wrote 90 pages of an incomplete novel, Resurgence, with Joe Hill in a supporting role. Health problems plagued Allison until his death at age 66 in 1974, as did a protracted battle to keep his home in Newark from being sold to the city.

Literary erasure is not always deliberate, but literary championing must be. Rachel Howzell Hall wrote earlier this year about being one of the few black mystery writers at annual genre conferences in an essay titled “Colored and Invisible” for The Life Sentence, a web site for crime and noir writers: “It can be lonely in those grand ballrooms, in those lesser ballrooms, at that reception. And there have been times when I’ve retreated to my hotel room, emotionally exhausted from being visibly invisible all day.”

For a moment in the 1990s, after Walter Mosley and Devil in a Blue Dress, crime fiction made room for more black writers. But then writers like Eleanor Taylor Bland, Penny Mickelbury, Paula Woods, Charlotte Carter, and others perhaps fell away in the relentless turnover of the publishing industry: canceled contracts, merged companies, and shifting editorial priorities. In recent years, few black crime-genre writers have reached Mosley’s level of popularity. To date, there are no available statistics on how well, or how badly, they are represented in the industry.

Recognizing the problem and addressing it accordingly takes work, and time. Yet it is frustrating, even shameful, how few writers of color get through the mystery corridors with a fictional representation of their own experiences. The door opened, briefly, for Hughes Allison. Before editorial neglect slammed it shut, Allison showed, years before Mosley, Himes, or any black detective fiction writer, what it was to live in his character’s skin.