video audio sync in processing

i done a little sound reactive piece and while it didn’t have any problems performing in real time, as soon as i tried recording it, it got so heavy on the processor that it started dropping frames and was completely out of sync with the music. so i went looking and found a nice little solution that seems to be working for me which involves running fft analysis in a separate sketch, saving the fft data into a text file and then using this data in my piece. the idea is that on a frame by frame basis i can get the audio data for that frame, react the visuals to the audio data and save it out to a png. i then load the pngs as a sequence into quicktime pro and its nice enough to give me back a movie file. the final step involves overlaying the movie with the audio. i do this in final cut but every video editing tool features this functionality. i noticed a delay which is probably due to the time it takes for the audio to start playing before grabbing the audio data, but thats easily adjusted when overlaying the audio over the video… just move the audio layer up a few frames until its spot on.

good thing about this approach is that you are not limiting yourself to any particular sound library. ive used ESS and Minim in the past and each one gives you back a slightly different result in the fft data. in this particular case i used Minim because it pronounced the beat more over a wider spectrum and so it went better with the visuals.

arc rays source and audio data export source

noisefield from julaps on Vimeo.