TSC Expo 2012-03-03: Machines Listening to Music

Got Rhythm?

In this demonstration, you will see how a computer "analyzes" sound, via an image called a spectrogram.

A spectrogram converts sound -- the rapid, tiny variation in air pressure detected by your eardrums -- into variations in brightness (or color) across the flat plane of an image.

In the spectrogram image:

Hence, a low rumble appears as bright spots near the bottom of the image, and a high-pitched whistle shows up as a stripe near the top of the image.

By comparing the spectrograms that you look at with the sounds you hear, we learn about how the auditory system -- the parts of your brain that deal with sound -- work:
The spectrogram roughly matches the information your ear feeds to your brain.

By writing computer programs to tell apart sounds based on their spectrograms, we improve our understanding of how brains do it.

What to do

Your computer should be set up to be running the LiveSpectrogram program, which will show the spectrogram of any sound played by the computer. (If it isn't, see these setup instructions).

Now you can go through the sounds listed below, and think about the questions for each one.


This is a recording of me whistling:


Here are two different musical instruments playing the same notes as I whistled:


Here are some recordings of my voice:


Here are two excerpts from My Violent Heart by Nine Inch Nails:

Dan Ellis <dpwe@ee.columbia.edu>