History of color science: predicting biology from behavior
An example of the (un)reasonable effectiveness of behavioral research in neuroscience
In Ted Chiang's Exhalation, a mechanical being notices that their species’ perception of time has changed. Armed with a hypothesis for why this might be, the scientist builds an intricate device that allows it to peer inside its own brain to observe the machinery that gives rise to its thoughts. In doing so, it discovers ‘the true source of life, and as a corollary, the means by which life will one day end’.
I love the image that Exhalation conjures of a person observing their own brain in action. But something else that struck me about this story was how the initial hunch came not from looking inside the brain, but by observing their own behavior and their environment. Centuries before we invented fMRI machines, stuck electrodes inside the brains, or engineered proteins to make the brain glow, scientists made unreasonably accurate predictions about how our brains work. They did this by making careful observations of the world around us, introspecting on their experiences and behavior, and through rigorous reasoning. This post is about one example of the boldness of predicting biology from behavior1: the science of color vision.
We know that most humans are 'trichromats'. Our eyes have three color receptors that are sensitive to different wavelengths of light and this gives rise to our ability to perceive color. But how exactly did scientists figure this out?
Studying the stimulus: What is light?
All physicists are simply future neuroscientists. Newton was no different.
To understand how looking at light gives rise to the perception of color, we need to first understand what light is. Newton did a series of elegant experiments to understand the fundamental components of light.
"As the Rays of light differ in degrees of Refrangibility, so they also differ in their disposition to exhibit this or that particular colour (...) Some Rays are disposed to exhibit a red colour and no other; some a yellow and no other, some a green and no other, and so of the rest"- Isaac Newton, 1671 [1]
He let sunlight in through a tiny hole in his dark room. He placed a prism in the path of the light and let the refracted light fall upon the screen. Light passed through the prism and formed a rainbow on his screen. He noted that different rays of light differ in how they were refracted, and that the degree of their refractiveness corresponded to the color in which they appeared.
If you're curious about exactly why this happens, I recommend checking out this video by 3blue1brown.
Then, Newton cut a slit on the screen to allow only a portion of the rainbow to pass through. He placed a second prism in the path of this ray. He observed that:
"The color of the light was never changed in the least. If any part of the red light was refracted, it remained totally of the same red color as before. No orange, no yellow, no green or blue, nor other new color was produced by that refraction."
Newton's prism had split light into its fundamental components. He then used lenses to converge the rays that passed through the first prism. He observed that he could get back the original light, suggesting that the decomposition of light was reversible.
But this does not address the relationship between the wavelength of a ray of light and why it appears a certain color to humans.
Studying behavior: Metamers
"There is no red in a 700 nm light, just as there is no pain in the hooves of a kicking horse." - Steven Shevell.
Color is in the eye of the beholder. Or, more accurately, in the brain of the beholder. This insight was crucial in uncovering the biological basis for color vision. It shifted the emphasis from looking outwards, only studying light itself, to looking inwards and studying the system that is doing the actual perceiving. It might seem obvious, but most things that are obvious in hindsight were nothing but when they were first discovered!2
People have spent a long time wondering what it means to experience color and whether my blue is the same as your blue. Newton and others lay the foundations for understanding the basics of how light worked, but for a long time, we still lacked the tools to study the physiology of the eye. How, then, to get any insight about what gives rise to the perception of color?
Newton's work showed us that sunlight is made up of spectral components of different wavelengths. Early theories suggested that the eye contained 'retinal elements' that vibrated in response to light stimulation, and these vibrations helped us see. But if light is made of infinitely many rays of different wavelengths, are there infinitely many retinal elements that each respond to a given wavelength?
Thomas Young (of double-slit fame), amongst other 'future neuroscientists' including Helmholtz, Maxwell, and Grassman, was one of the first scientists to propose that there were only three distinct receptor types in the retina. The basis of this assertion was a set of neat behavioral results in what we now call 'color-matching' experiments 3 [2] .
They presented a person of normal color vision with a 'test' light made up of some arbitrary spectral composition. Then, they asked the subject to 'match' the appearance of the test color by adjusting the intensities of a set of lights with fixed spectral distributions - called primary lights.
They found that having two primary lights was insufficient to match all colors, while having four was too many. These behavioral experiments proved that it was necessary and sufficient to have three appropriately chosen primary lights to match the appearance of any light of arbitrary spectral composition. This meant that there were infinitely many lights with different spectral distributions that all appeared identical to humans.
To illustrate this, look outside your window at the sunlight. Okay, now look at this photo of the sky:
In these two cases, your eyes were stimulated by light with very different spectral power distributions. Below, the left panel shows an approximation of the spectral power of the sun. On the right is the spectral distribution of the light coming out of a typical computer monitor which has been calibrated to produce an image that is identical to sunlight to most human observers. Such stimuli, that are physically different but perceptually indistinguishable, are called metamers.
Side note: most monitor displays are built with only three different types of phosphors that emit light of particular spectral distributions. By adjusting the relative intensities of the three lights, we can produce the percept of any color. But a picture of sunlight on a display monitor is only a metamer of the sunlight we see outside our window. Understanding the science of human color perception made the engineering of these devices possible - a wonderful example of the impact of basic science. (In fact, based on these behavioral studies, the International Commission on Illumination codified these standards in 1931, decades before we actually measured cells in the eye!)
The scientists of the 1800s didn't stop at identifying the number of distinct primary lights required to match human vision. They also made precise predictions of ‘color-matching functions’: the intensity of each chosen primary light needed to match a monochromatic light of a given wavelength. For example, to get a monochromatic light of wavelength 550nm, what we need is an intensity of 1 for R, intensity of 1 for G, and intensity of 0 for B.
The color matching functions are not unique: if we used two distinct sets of primary lights, we will arrive at different color-matching functions. But the two functions are related by a linear transformation. We can think of this linear transformation as adjusting the intensities of the first set of primary lights to match the intensities of the second set of primary lights.
Based on these findings, they predicted that the human eye must contain three types of receptors. Moreover, they predicted that the absorption spectra of these three receptors must explain the color-matching results, i.e. they will be related to the color-matching functions by a linear transformation.
This was all figured out by the 1850s. It took us more than a hundred years after that to actually test and verify these predictions by studying the properties of photoreceptors in the human eye!
Studying biology: Cones
To conclude the saga and discover the biological basis of human trichromatic vision, in the late 1900s, a team of experimentalists from Stanford University Baylor, Nunn, and Schnapf [4] painstakingly measured the responses of the different cone types in the retina, the sheet of tissue at the back of the eye where light falls. Our retina primarily contains two types of photo-receptors: Rods and Cones. When light falls on these cells, they convert this to electrical signals. Rods operate more in 'scotopic' vision, that is when it is relatively dark. Cones operate in photopic vision, when it is bright, and mediate 'color vision' because there are three types of cones that are sensitive to different wavelengths. The fact that there are three types of cones, was known by the 1980s. But their exact spectral sensitivities, that is the shape and amplitude of how much each wavelength of light excites a given cone, had not been measured. These measurements were key in testing whether the cone spectral sensitivities could explain the behavioral results from the color-matching experiments.
Baylor and colleagues isolated single cones, passed monochromatic lights of various wavelengths, and observed the sensitivity of different cones. They found three different cone types with spectral response functions that peaked at different wavelengths (approximately 430, 531, and 561 nm). Further, they showed that there is a linear transformation that converts the cone photocurrent measurements into the color-matching functions predicted from the behavioral experiments. This paper finally provided the biological basis to explain the results from the color-matching experiments.
I first learned about this from Eero Simoncelli when I attended the CSHL course on vision. Here's a quote from a talk of his where he does a much better job of conveying the essence of this scientific story and why it's worthy of praise. (In his own career, he sought to and succeeded in discovering a similar scientific story: Metamers and pattern vision [5] )
"It's amazing that you can have a theory and a set of behavioral experiments that make very precise and clear predictions that get verified and tested in a mechanistic sense almost a hundred years later."
Studying others: Predicting Behavior from Biology
So far, we've talked about a story of the triumph of behavioral science in predicting biology. But often, progress in neuroscience has come from an Inside-Out approach - studying biology to predict psychology. Here’s one such case that I first came across in Ed Yong's book ‘An Immense World’.
Once scientists established that normal human vision is trichromatic, they started to look for situations where this was not the case. Color-blindness is an obvious example that people were aware of for centuries (Maxwell, one of the protagonists of the color-matching experiments, wrote about its implication for color-blindness [6]). But what of those that can see more than I can?
Gabriele Jordan and colleagues were interested in whether there may be tetrachromats hiding amongst us - humans with four distinct cones who were able to perceive colors that are identical to most of us trichromats. Instead of hoping that the tetrachromats would self-identify and call up their nearest color researcher, these scientists put together a set of conditions to narrow the search down. The key to figuring out the right set of conditions lay in understanding the genetics behind cone cells.
The genes that determine the properties of your L and M cones are in your X chromosome. Females typically have two X chromosomes (XX), while males have one X and one Y chromosome (XY). Because females have a spare copy of the L/M cone genes, a defect in one of them can be compensated for by the other, which is part of the reason why it’s less likely for females to be color-blind compared to males. This is also why females are much more likely to be tetrachromats. It's possible for them to inherit both an X-chromosome with the typical M and L cone genes and the other X-chromosome with a wonky copy of, say, the M-cone gene such that this cone responds to a slightly different wavelength compared to typical human M-cones. But how to narrow down which females to study to find tetrachromacy? The key is to look for the mothers/daughters of males who exhibit a specific, milder type of color deficiency called "anomalous trichromacy". An anomalous trichromat has an M or L cone with slightly shifted wavelength preference. So they do have trichromatic vision, but experience some difficulty when discriminating certain colors. The mother/daughter of an anomalous trichromat is an 'obligate carrier' of this spectrally shifted hybrid cone, in addition to having the standard S, M, and L cones, making them potential candidates for tetrachromacy.
So these researchers put out TV ads to recruit boys who are anomalous trichromats and their mothers to participate in their study. They narrowed down their list to twenty four eligible mothers and had them do a behavioral experiment called the Rayleigh Discrimination Test4.
They presented three colored circles in quick succession. Two of the three circles were made of monochromatic light of wavelength 590nm. The other circle was a mixture of two lights: Red (670nm) and Green (546nm). The order of presentation of the three circles was randomized each time and the participant had to guess which of the three was the mixture circle. For participants with normal trichromatic vision, for certain ratios of R and G, the mixture circle will appear to be identical to the monochromatic circle. However, if a person had a fourth cone with an appropriate preferred wavelength, and they were able to use the signals from that cone for perception, they would have been able to tell apart colors that were identical to others5.
In this Journal of Vision paper from 2010 [7], Jordan and colleagues found that only one of the twenty four obligate carriers they studied was actually able to discriminate between lights that were totally identical to trichromats. When they plotted the behavioral performance of all the participants as a function of mixture ratio, they found that error rates increased at intermediate ratios for most participants. Except, one subject, only known to the world as cDa29, had close to zero errors. She had no trouble doing this task which stumped the rest of them!
So far, cDa29 is the only reported human tetrachromat in the literature. There are many other species that are tetrachromats (e.g. some birds have a fourth cone with sensitivity in the UV range). If trichromatic color perception lets us perceive colors that lie within a triangle (where each corner is one of the three primary colors), tetrachromatic vision might allow one to perceive colors that live within a pyramid. Imagine: a whole other dimension of colors!
In the case of the search for a human tetrachromat, our knowledge of biology — knowing that the properties of our cones constrain our perception and that a subset of females who are mothers of anomalous trichromats might be candidates for tetrachromacy — informed who to study and what to look for in the behavior. So we've come full circle, from behavior predicting biology to biology predicting behavior.
Most of what I’ve written about here has previously been explained by others. In particular, I recommend checking out:
Ed Yong's An Immense World (chapter on color: "Yurple, Rurple, Gurple")
Eero Simoncelli's lecture on probing sensory representations
Thanks to Rithika Sankar for patiently explaining basic genetics to me. Thanks to K.R. Adhithya for feedback.
Please let me know if you spot any errors. I'd like to be as accurate as possible, and I appreciate your feedback.
For more on the importance of studying behavior to understand the brain, I highly recommend checking out Yael Niv on ‘The primacy of behavioral research for understanding the brain’ and Krakauer et al.’s ‘Neuroscience Needs Behavior’.
For example: Ancient Greeks believed that we emit beams of light from our eyes which enables us to see.
Disclaimer: The color-matching demo is only to give an idea of what the experiment would’ve looked like. We cannot perform a true trichromatic color matching experiment on a computer monitor since they’re built with only three types of phosphors.
Disclaimer in previous footnote applies here as well.
Having four cones is a necessary but not sufficient condition for exhibiting behavioral tetrachromacy. Jordan and colleagues speculate that a few more conditions need to be met: the extra cone needs to be responsive to a very specific wavelength, it has to be abundant in the part of the retina that perceives information in the fovea, i.e. the central portion of our visual field, and the brain needs to be able to process its responses to actually use the information the fourth cone relays to give rise to perception (e.g. see this modelling study from Sejnowski’s lab). But, for a counter-point, see this 2007 study where they showed that mice (typically dichromatic) genetically engineered to have an extra cone end up making trichromatic color discriminations! Another fascinating study showed that humans with a condition called ‘achromatopsia’ who underwent gene augmentation therapy were able to somewhat perceive red objects better than before. (Footnote in this footnote: these therapies will hopefully one day make color-blindness curable, and if so, it will be thanks to the centuries of curiosity-driven basic science that advanced our understanding of color vision)
References:
[1] A new theory of light and colors. Isaac Newton (1671)
[2] On the theory of light and colors. Thomas Young (1801)
[3] N.P.L. Colour-matching Investigation: Final Report. Stiles and Burch (1958)
[4] Spectral sensitivity of cones of the monkey Macaca fascicularis. Baylor, Nunn, Schnapf (1987)
[5] Metamers of the ventral stream. Freeman and Simoncelli (2011)
[6] Experiments on colour, as perceived by the eye. James Maxwell (1855)
[7] The dimensionality of color vision in carriers of anomalous trichromacy. Jordan et al. (2010)