InformationWeek 500: Using Sound, Video To Represent Data
"Data is getting too voluminous … to compare numbers in side by side columns. We are multi-sensory beasts. We coordinate voluminous amounts of data every day through our senses," said JoAnn Kuchera-Morin, a music composer who serves as director of the AlloSphere Research Facility at the University of California at Santa Barbara.
The AlloSphere is a walk in, spherical-shaped lab where visitors mount a bridge to be treated to a three-dimensional visual and audio representation of complex data. Speakers and images on screens embedded in the wall of the sphere surround the visitor and saturate the senses with sound and moving graphics.
Kuchera-Morin in an opening presentation to the 370 attendees showed a three-dimensional representation of an individual's brain, with sound growing in intensity as the fly-through camera approached a blood rich portion of the cortex. A random or chaotic set of numbers can be represented as noise, but numbers in a definite relationship to each other can be represented by pitch. A long piece of string will vibrate twice as fast if it is cut in half, and that relationship can be applied in the form of notes to the data, she noted.
She then showed a visualization of data that represented a mapping of the human brain. In the AlloSphere, the mapping was a kind of fly-through of peaks and valleys, accompanied by sound that grew stronger as one approached a blood dense area.
With three-dimensional images representing folds in the brain and sounds representing blood movement, Kuchera-Morin's presentation took on an air of gliding through a subterranean set of passageways, accompanied by alternately increasing, then fading sounds. The images were displayed on big screens behind the speaker.
Kuchera-Morin showed how the sole electron of a hydrogen atom can be visually represented as a wobbly disk that changes shape as it orbits its nucleus. As one hydrogen atom approaches another, the disk emits a photon as the electrons pass near the other. This data can be represented by sound; likewise, hydrogen atoms combining with zinc results in electrons becoming excited and moving into higher orbital states, a change that can also be represented in sound. "We're convinced as artists that we can actually start to use our senses to understand data, that we can parse data the way we see, hear and taste the physical world," she said. Patterns in the data that trigger sounds would make those patterns readily identifiable to a listener.
A scientist, who was skeptical that a visual and sonic representation of the data of a hydrogen atom would change his thinking on the nature of hydrogen, came out of the AlloSphere with a different frame of mind. After seeing what a more tangible representation of the data, he said, "The sound signatures of an atom can create a new vocabulary that will let us push on to the next level."
Kuchera-Morin said soldiers can be taught how to react when under attack or what to do at the sound of gunfire. But throw in the smell of gun powder, and they react on a more instinctive level. Equations and large data sets likewise can elicit a different reaction in the analyst when the data is transformed into sound and images, she said.
There are future uses of such data representations in the business world, she assured her listeners. But it will take a combination of artists, engineers and business analysts to assess what's inside the data, determine what patterns the business user is looking for and how to represent it in more tangible form when it's found. Data explorations, like those executed in the AlloSphere, will require a collaboration of parties that have not previously joined forces to see how far new, sensory representations of data can go and what they may reveal.
