Color perception is an ancient and active philosophical problem. It’s an instance of the wider category of sensory perception, but since the color spectrum fits on a single line (unlike, say, touch and taste), it has always been of particular interest. In her new book Outside Color, University of Pittsburgh professor M. Chirimuuta gives a serendipitously timed history of the puzzle of color in philosophy. To read the book as a layman feels like being let in on a shocking secret: Neither scientists nor philosophers know for sure what color is.

“Of all the properties that objects appear to have,” Chirimuuta writes, “color hovers uneasily between the subjective world of sensation and the objective world of fact.” The early history on the color perception debate alternated between partisans of these two camps. The Scholastic or Aristotelian model is a simple realism: Objects have colors that observers perceive in them. Like a seal that leaves a stamp in hot wax, an object’s color leaves its imprint temporarily on our eye. Since Scholastic realism presents no conflict between what we see and what there is, it was a convincing and long-lasting explanation. Color is what it looks like.

In the seventeenth century, the scientific revolution turned this common-sense explanation on its head. Galileo, that philosopher of inversions, suspected the realists had it exactly wrong. Colors, he wrote in Il Saggiatore (The Assayer), “hold their residence solely in the sensitive body; so that if the animal were removed, every such quality would be abolished and annihilated.” In other words, an unseen tree falls in black and white. If color exists in objects, then the scientific-minded must ask where. Unable to define and measure color the same way they located primary or physical qualities (shape, size, and speed), these early philosopher-scientists figured it had to be all in our heads.

Whether color is in our brains or in the world, realists and anti-realists agreed that there are right and wrong answers. Both models are frictionless, with perceiving subjects interacting with a real and objective reality. Chirimuuta calls them “detection models,” but they assume too much. If we imagine that every existing physical referent for something like The Dress—the Internet-famous garment that in February bewildered online America with a debate about whether a Scottish woman’s outfit was white-and-gold or blue-and-black—had been burned in a tragic warehouse fire, neither perspective could give a consistent answer as to its color without also imagining a normative viewer.

Outside Color is most engaging when it’s interrogating fundamental questions like these. “If we step back a moment,” Chirimuuta writes, “we can appreciate how very weird it is to even expect there to be a connection between the manifest visual world, brought to us by our senses, and the rarefied scientific image of a world made up of physical particles, etc.” To believe in the atom—never mind quantum theory or more extravagant scientific models—is to accept that our unaided perceptive faculties have very limited access to physical reality. Our perception gives us useful information about the world that allows us to make decisions as actors within it, but that doesn’t necessarily mean it’s true or consistent.

Ibrahim Mohammad El-Salahi, “The Tree” © 2015 Artists Rights Society (ARS), New York / DACS, London Ibrahim El-salahi/Courtesy Of The Tate Modern

The scientific revolution established that colors don’t inhere in objects, but it wasn’t the end of the road. Once Galileo located color in the brain, philosophers and scientists were left to repeat the same question: Where? Modern biologists have given us a couple of models. Trichromacy, based on the three types of cone cells in the eye, suggested each was responsible for a cardinal color, and together they mix to create the normal spectrum in the brain. Opponent process theory said the cone information then travels through two channels for cardinal colors (red-green and blue-yellow) and one for brightness (black-white) that together create the spectrum through contrast. It’s a compelling scientific resolution to the color problem: Lighting conditions interact with objects and the eye sorts it all out and illustrates wavelength information using colors.

One of the reasons I think philosophy isn’t very popular in the United States is that the secular among us assume not only that there exists a scientific explanation for everything, but that someone in a laboratory or a library somewhere already knows it. Primary science education plays up this assumption, preferring testable information to ongoing mysteries—I am reminded of an eleventh-grade physics exam on opponent processing. But here’s what they don’t tell you in school: The neurological and physical evidence that supports this model is extremely inconsistent.

In a 2013 study published in the Journal of Neurophysiology, scientists measured the response from primary visual cortex neurons in macaque monkeys (one of the few animals that see like we do) as they prompted them with chromatic contrasts. They were trying to prove the existing biological model of color perception by testing neuron response after isolating the function of neural channels. If the researchers could see the way the brain mixes colors, they could once and for all find red. But it didn’t work. Instead, “analysis of neuronal contrast—response functions and signal-to-noise ratios yielded no evidence for a special set of ‘cardinal color directions,’ for which visual cortex neurons are particularly sensitive.” Not only couldn’t they find the red-green and blue-yellow channels, nearly half the color neurons also fired in response to brightness changes, suggesting the tripartite channel model is inaccurately simplistic.

Scientific attempts to reduce colors to wavelengths have been equally unsuccessful. If it were simple, then colors as we observe them should match up with what’s called the “surface spectral reflectance,” (SSR) which can be measured with a digital receptor. But if you try to replicate color vision this way, you get far too many colors, and objects lack internal constancy because of the effects of lighting conditions. Where we see shadows, a computer sees a different color. But even when they apply highly sophisticated algorithms for illumination in a 3-D space, researchers have been unable to replicate human levels of color constancy. NYU professor of psychology and neural science Laurence T. Maloney writes that the failure of these SSR models thus far suggests “there are cues present in real scenes that we do not understand.” People are able to use color to judge scenes better than computers and cameras can, and scientists aren’t sure how.

Take a popular optical illusion, designed in 1995 by Edward Adelson, a professor of vision science in the Department of Brain and Cognitive Sciences at MIT:

The human perception system sees a checkerboard with a cylinder, while a basic SSR measurement shows squares A and B read the same. “Illusion” implies that our system is fooled, but as far as useful information goes, the checkerboard interpretation is probably better. Try as they might, mathematicians can’t make the computers see the checkerboard. Rather than a demonstration of how easily fooled we are, optical illusions like this one are examples of the brain’s mysterious and irreplicable abilities. It interprets its environment with a sophistication that exceeds our ability to measure and reconstruct physical phenomena. The usual framing has it wrong:
Despite A and B having the same SSR, humans are still able to see the checkerboard.

The difference between philosophy and the history of philosophy is the positing of a new answer, a new problem, a new step on the road of investigation. Here is where Chirimuuta really excels, and Outside Color becomes truly exciting. She poses a new mode of thinking about chromatic perception: color adverbialism. Instead of a brown dog, Chirimuuta wants us to see the dog brown-ly. It sounds silly, but turning color into a process better fits some exceptional cases than the standard model. Color is a mode of interpreting information, and sometimes it tells us more than pigment. It can tell us about motion: a black-and-white wheel set spinning reveals the rainbow. It can tell us about depth: Long distances appear blue because higher wavelength red light scatters less. “Color is not an object of sight but a way of seeing things,” Chirimuuta writes.

In this model, color is like a heads-up display, one with a complexity that outpaces any attempts to replicate it. When searching for an appropriate metaphor, it occurred to me that we consciously put color to use in a similar way all the time. When we divide up sports teams with jerseys, for instance, we see Team A red-ly and Team B blue-ly. The SSRs vary as light flickers on polyester, but humans can still differentiate between the teams. Countries on a map take colors because it’s an easy way for us to see their boundaries, and we’re not fooled by the shadows a lamp casts on a globe. Color adverbialism is a challenging philosophical answer to what appears in our limited imagination like a solely scientific question.

From an adverbialist position, The Dress isn’t black and blue or white and gold. Some people see it one way, some people the other. But the conflict directed viewers to use color a little more, to use it to investigate their environment. Some of them quickly discovered that tilting their screens allowed them to see both colors, as a light filter engaged and disengaged. This is the kind of interpretation the best computer models can’t make, but people are built to engage our world with curiosity. If that’s not philosophical investigation of the highest and most human order, then I don’t know what is.