What follows is a response to questions about important applications of sonification. The questions were posed in a message over the ICAD (International Community for Auditory Display) mailing list from David H. Jameson, Manager, Computer Music Center, IBM Research Division. I must admit that my first reaction to his message was one of irritation because I tend to chafe at the way some people use mailing lists to get subscribers to do their thinking for them. But the question of "why the world should care about sonification" kept invading my thoughts for days so I gave in to the irritant and organized my thoughts on the matter.

Why the world should care about sonification.

To: David Jameson <dhj@watson.ibm.com>, icad@santafe.edu
From: Ron Pellegrino <ronpell@microweb.com>?
Subject: RE: What's a compelling example of sonification?
Date: 9/15/98

At the root level the "killer" apps for sonification are learning and playing, what most intelligent people would consider fundamental reasons for being. Certain segments of the education and entertainment industries target the learning and playing processes, so for those particular segments sonification must be considered a critical component of their work. Any person or institution involved in emerging sonification technology that gets in step with those industrial segments stands to gain in terms of both finances and well-being.

The flip side of sonification is visualization. Smell, taste, and touch aside, people perceive many natural events from combinations of unique auditory and visual emanations from those events. Intuitively the human sensory system records experiences of auditory/visual events as both individualized and integrated fields. Those events can be recalled either as auditory or as visual memories. Or they can be recalled as an integrated auditory/visual memory field which will have greater depth, complexity, and meaning than either field alone. The integration of sonification and visualization creates a lifelike experience that makes learning faster, richer, and deeper and makes playing more exciting and more aesthetically rewarding.

The reason sonification has become an issue is because we´re working with a tool (the digital computer) that evolved along the lines of a calculator or business machine - the BM in IBM. Given that the computer was (still is) based on numbers (abstractions) and not common human experiences, we find ourselves in the throes of trying to turn numbers into life (not so different from turning lead into gold) and life into numbers. For decades leading up to the 1990s computer science insisted practitioners wear digital blinders to avoid analog seduction (which left the good times for the sound and light artists); so hybrid processes such as analog/digital/analog for sonification have been slow to enter the mainstream which seemed more interested in counting and distributing money. Although those hybrid processes are mainstream in the late 1990s, few people have either the experiential or theoretical background to make good use of them but that seems about to change.

Although it´s still in the early developmental stages another "killer" app for sonification (and visualization) is the stimulation its study provides for exploring psychophysics including psychoacoustics and psychooptics. Hubris notwithstanding, for those with college educations the current intellectual grasp of all things hearing and seeing is barely above ground zero. It´s just during the past few decades that we´ve had the tools to explore perception in measured and comparative ways. And during those decades it´s been a slow go because institutional recognition and support of such "blue sky" research into psycho-matters has been very light leaving the work to be done by the self-motivated. Though still relatively incipient this particular app could eventually be the most important culturally. Imagine what a cultural difference it would make if we had a K-12 educational system designed around the how, what, why, and care and feeding of the senses instead of just preparing our children to be good consumers and industrial cogs.

What I´m saying here is not just a theoretical rant. My own life took a new tack in 1967 when I discovered for myself the value of the integrated sonification/visualization approach when I decided to take on learning the Moog Synthesizer as part of my dissertation project. I was a classically educated multi-instrumentalist and composer with little experience in physics and electronics but when I heard and saw the synthesizer for the first time I recognized that it was the composer´s instrument of the future so I decided to take it on as a project. Other than a few of Moog´s very sketchy spec sheets there was absolutely no instructional material available for the music synthesizer in 1967. So I spent nine months with a decent stereo sound system and a dual trace oscilloscope exploring the ins and outs of all the wavetrain generation and modification functions of all the modules on the Moog; the log of what I learned from that process became one-third of my dissertation project as well as the basic text for the Moog Synthesizer.

For me and most other composers sonification had always been the normal state of affairs; what was different from 1967 onwards was that I started integrating sonification with visualization in the learning and eventually the composing and teaching processes. Because the academic world of the 70s and 80s up into the early 90s was populated by legions of Luddites dressed like administrators, faculty, and students, the emerging technology in the arts facilities that I organized at various universities for teaching and production tended to be seed-islands in an ocean of doubts. During the past three decades my teaching/research spaces and public presentations have included the full range of affordable emerging technology for sonification/visualization including analog, digital and hybrid synthesizers, general and specialized computers with special software and peripherals, laser deflector/projectors, audio and video recording and playback gear, audio and visual translation and conversion equipment, and all manner of analytical and measurement gear. I personally have shown my work to many thousands of people and I know that other composers have been working along similar lines, so I suspect we must be on the verge of a critical mass that will lead to a general awareness and acceptance of the validity of sonification/visualization in learning and playing.

The latest upwelling in multimedia is partly an attempt to harness for commercial purposes the work that´s been done in the past few decades in the sonification/visualization fields. Not much of value has come out of this recent multimedia wave because most of those involved are barely conversant with the fundamental issues of sonification/visualization. At best their work tends to be rooted in Saturday morning animated cartoons rather than a fundamental much less serious study of psychophysics; at worst their work is rootless and mindless just as following any other fad might be.

Anyone who doesn´t care about sonification risks missing out on one of the next major leaps in computer technology - the humanization of the computer and computer experiences. For the mainstream, working on sonification, visualization, and other sensory conversions and translations should provide the irritants and the inspiration to help spring the computer out of the mechanical world into the biological world. Dynamical artists working in sound and light have been learning to play on that field for decades. Time now for the mainstream.

Ron Pellegrino


>Date: Thu, 10 Sep 1998
>To: icad@santafe.edu, sound@acm.org, auditory@vm1.mcgill.ca
>From: David Jameson <dhj@watson.ibm.com>
>Subject: What's a compelling example of sonification?


>Suppose you were a keynote speaker at the next COMDEX exhibition. Your job
>is to explain to the audience why the world should care about sonification.
>What would you tell them? What do you think are the potential "killer" app
>that takes sonification out of the niche world into the mainstream?

>There are some obvious examples for particular communities (e.g. screen
>readers for the visually disabled, perhaps the use of sound in games for
>informational purposes) but I'm wondering what applications of sonification
>might make a really significant percentage of people take notice and think
>"I gotta have this" or even "hmmm, I'm going to buy a computer so I can
>have this", specifically because of the sonification.

>I look forward to your thoughts,

>Cheers,

>David Jameson

>___________________________
>Dr. David H. Jameson
>Manager, Computer Music Center
>IBM Research Division
>Route 134 & Taconic State Parkway
>Yorktown Heights, NY 10598

>http://www.research.ibm.com/music

>A difference that makes no difference is no difference - Ralph Waldo Emerson


Booking information and comments.

©1996-2004 Ron Pellegrino and Electronic Arts Productions. All rights reserved.