Tuesday, October 16, 2007

Seeing isn't believing

TAKE a moment to observe the world around you. Scan the horizon with your eyes. Tilt your head back and listen. You're probably getting the impression that your senses are doing a fine job of capturing everything that is going on. Yet that is all it is: an impression.

Despite the fact that your visual system seems to provide you with a continuous widescreen movie, most of the time it is only gathering information from a tiny patch of the visual field. The rest of the time it isn't even doing that. Somehow from this sporadic input it conjures up a seamless visual experience.

What is going on? Bang in the middle of your retina is a small patch of densely crowded photoreceptors called the fovea. This is the retina's sweet spot, the only part of the eye capable of seeing with the rich detail and full colour we take for granted. This tiny spot - which covers an area of our visual field no bigger than the moon in the sky - feeds your visual system almost all of its raw information.

To build up a big picture, your eyes constantly dart about, fixating for a fraction of a second and then moving on. These jerky movements between fixations are called saccades, and we make about three per second, each lasting between 20 and 200 microseconds.

The curious thing about saccades is that while they are happening we are effectively blind. The brain doesn't bother to process information picked up during a saccade because the eyes move too rapidly to capture anything useful. All in all, your visual system works like a man blundering around in the dark waving around a flickering torch with a very narrow beam.

Despite the fact that you don't normally notice saccades, you can catch them in action. Look at your eyes close-up in the mirror and flick your focus back and forth from one pupil to another. However hard you try you cannot see your eyes move - even though somebody watching you can. That's because the motion is a saccade, and your brain isn't paying attention. Now pick two spots in the corners of your visual field and flick your gaze from one to the other and back again. If you're lucky you'll notice, just barely, a brief flash of darkness. This is your visual cortex clocking off.

So how does your brain weave such fragmentary information into a seamless movie? This remains something of a mystery. The best explanation, according to Andrew Hollingworth of the University of Iowa in Iowa City, is that your short-term and long-term visual memories retain information from previous fixations and integrate them into a here-and-now visual experience (Visual Cognition, vol 14, p 781).

There is also some guesswork going on. You can get a feel for this from the frozen-time illusion - the sensation that you sometimes get when you look at a clock and the second hand appears to freeze momentarily before tick-tocking back into action.

This happens because of saccades. To compensate for the temporary shut-down of vision, your brain makes a guess at what it would have seen, but it does so retrospectively. So the 100 or so milliseconds of blindness gets back-filled with the image that appears after the saccade is over. If your eyes happen to alight on the clock just after the second hand has moved, your brain assumes that the hand was in that location for the duration of the saccade too. The "second" then lasts about 10 per cent longer than normal, which is enough for you to notice.

The weirdness isn't confined to vision. Your auditory system is also full of gaps and glitches that the brain cleans up so we can make sense of the world. This is especially true of speech.

In everyday life we encounter lots of situations that obscure or distort people's voices, yet most of the time we understand effortlessly. This is because our brain pastes in the missing sounds, a phenomenon called phonemic restoration. It is so effective that it is sometimes hard to tell that the missing sounds are not there.

A good demonstration of this effect was published last year by Makio Kashino of NTT Communication Science Laboratories in Atsugi, Japan. He recorded a voice saying "Do you understand what I'm trying to say?" then removed short chunks and replaced them with silence. This made the sentence virtually unintelligible. But when he filled the gaps with loud white noise, the sentence miraculously becomes understandable (Acoustic Science and Technology, vol 27, p 318).

"The sounds we hear are not copies of physical sounds," Kashino says. "The brain fills in the gaps, based on the information in the remaining speech signal." The effect is so powerful that you can even record a sentence, chop it into 50-millisecond slices, reverse every single slice and play it back - and it is perfectly intelligible. You can listen to Kashino's sound files at http://asj.gr.jp/2006/data/kashi/index.html.

Another demonstration of the brain's ability to extract meaning from distorted signals is a form of synthesised speech called sine-wave speech. When you first hear a sentence in sine-wave speech it sounds alien and unintelligible, somewhat reminiscent of whistling or birdsong. But if you listen to the same sentence in normal speech and then return to the sine-wave version, it suddenly snaps into auditory focus. Try as you might, you cannot "unhear" the words that you didn't even realise were words the first time you heard them (listen to demos at http://www.mrc-cbu.cam.ac.uk/~mattd/sine-wave-speech and http://www.lifesci.sussex.ac.uk/home/Chris_Darwin/SWS).

According to Matt Davis of the UK Medical Research Council's Cognition and Brain Sciences Unit in Cambridge, this happens because the brain has circuits that respond to speech, but doesn't switch them on unless it detects spoken language (Hearing Research, vol 229, p 132). Sine-wave speech isn't speech-like enough to trigger the circuits, but once you know it is speech they spring into action. "It's an example of top-down influence," says Davis. "What you know about what you're hearing changes the way you hear it."

Given the tricks that your visual and auditory systems play, it probably comes as no surprise that when they get together, fights can break out. A good demonstration of this is the McGurk effect, in which listening to a series of identical syllables such as "ba ba ba ba" while watching somebody mouth "ba da la va" makes you hear "ba da la va". Try it for yourself at www.faculty.ucr.edu/~rosenblu/lab-index.html.

Until recently, psychologists believed that the visual system always trumps the other senses, but in 2000 a team of psychologists at the California Institute of Technology in Pasadena proved that this isn't the case. They showed volunteers a single flash on a computer screen. If they accompanied the flash with two very short beeps, the volunteers saw two flashes - in other words, this time the auditory system wins (Nature, vol 408, p 788). See the illusion at www.cns.atr.jp/~kmtn/soundInducedIllusoryFlash2/index.html.

No comments: