HAMR 2013 Proceedings

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Synaesthesia ======

| Authors | Tom Stoll, Dennis Heihoff (and Irina Likhtina)|
| Affiliation | Dartmouth College/unaffiliated |
| Code | [[https://github.com/kitefishlabs/synaesthesia|Github Link]] |

Motion tracking statistical analysis of video controlling audio synthesis.

===== - Stack =====

Python + Numpy + OpenCV
SuperCollider

===== - Process ====

Films are analyzed for motion vectors using OpenCV (corner detector + Lucas-Kanade algorithm). These vectors are simplified (start point - end point only) and binned according to their locations on the screen, their angles, and weighted by their relative lengths. Furthermore, brightness mean and variance are extracted for each of the 64 grid positions per frame.

This data is stored in a binary file that can be read by SuperCollider. SuperCollider sonifies the data using a bank of 64 simple Synths, each of which are fed motion/brightness data per time point.

===== - Results/Observations ====

We successfully mapped motion to sound. We avoided making one-to-one mappings of spatial locations to pitches or other overly simplistic correlations. The nature of the histograms is such that most spatial bins only contain one or two binned angles. This sparsity creates regularly spaced active channels that map into frequency bands in the sonification. Furthermore, the dynamic changes in motion (accelerations) create natural/realistic sounding sonic morphologies. It should also be noted a spectral synthesis approach to sonification was not successful.
===== - To do ====

1. Create a mapping layer that takes into account each spatial/angle channel's variance over time and reorders this data into configurable patterns or bin configurations.

2. Use dimensionality reduction to reduce the remapping layer such that only a limited number of significant components drive a synthesis algorithm.
HAMR 2013 Proceedings

User Tools

Site Tools

Page Tools
