User Tools
====== Synaesthesia ====== | Authors | Tom Stoll, Dennis Heihoff (and Irina Likhtina)| | Affiliation | Dartmouth College/unaffiliated | | Code | [[https://github.com/kitefishlabs/synaesthesia|Github Link]] | Motion tracking statistical analysis of video controlling audio synthesis. ===== - Stack ===== Python + Numpy + OpenCV SuperCollider ===== - Process ==== Films are analyzed for motion vectors using OpenCV (corner detector + Lucas-Kanade algorithm). These vectors are simplified (start point - end point only) and binned according to their locations on the screen, their angles, and weighted by their relative lengths. Furthermore, brightness mean and variance are extracted for each of the 64 grid positions per frame. This data is stored in a binary file that can be read by SuperCollider. SuperCollider sonifies the data using a bank of 64 simple Synths, each of which are fed motion/brightness data per time point. ===== - Results/Observations ==== We successfully mapped motion to sound. We avoided making one-to-one mappings of spatial locations to pitches or other overly simplistic correlations. The nature of the histograms is such that most spatial bins only contain one or two binned angles. This sparsity creates regularly spaced active channels that map into frequency bands in the sonification. Furthermore, the dynamic changes in motion (accelerations) create natural/realistic sounding sonic morphologies. It should also be noted a spectral synthesis approach to sonification was not successful. Password for each video: **hamr** Motion Examples (intercut) https://vimeo.com/69429662 Score for //A Clockwork Orange// (short) https://vimeo.com/69429661 ===== - To do ==== 1. Create a mapping layer that takes into account each spatial/angle channel's variance over time and reorders this data into configurable patterns or bin configurations. 2. Use dimensionality reduction to reduce the remapping layer such that only a limited number of significant components drive a synthesis algorithm.
