the visual component stems from an existing generative application, the triangle field, which i wrote some time back. with tr-IO, i wanted to allow people to explore the generative art work by having influence over the colour, form and motion, and to create their own flavour of visuals.

i was very happy when eli murry / gentleforce jumped onto the project as a collaborator and enriched the installation by adding another dimension of sound design. eli created an array of ambient drones, clicks and ominous electronic growls which were mapped to the interactive table and controlled by the pyramid objects. a user could place the pyramids at different points on the table to awaken the various hidden sounds and create a unique soundscape.

the interactive table is built using reactivision which uses fiducial markers as a means of tracking objects. each pyramid object was fitted with a unique fiducial marker which allowed the camera inside to track its position and rotation on the table.

the pyramid objects were also fitted with programmable BlinkM leds which could be easily programmed via an arduino board. the initial idea was to get the lights inside the pyramid to change depending on its position on the table but for this it would have required a wireless arduino for each pyramid which unfortunately fell outside of the budget this time around, but its definitely something id like to add in the future.

whats the future hold for tr-IO?
well hopefully eli and i can up-size it to a completely immersive experience.
a whole space dedicated to projecting the visuals in a seamless 360 view.
and throw in some surround sound.
chill out room at a doof anyone?
i joke, but a gallery space would be nice.

physics is always fun, fact.
in this video, circle particles fill the volume of the building and once settled in place, create an image.
the trick here is to render the particle system twice. first, run the simulation and record the colour values of all the balls once they’ve settled and then to render again the second time and since the simulation is identical, the balls fall into the right place with the right colour.

being the becks bar and all, there was one catch – the video had to include the becks logo.
in this video, the logo is first created using circle packing (basically fitting as many circles as possible into an image) and then released to spill and swish around inside the building before eventually loosing momentum and fizzling out to the top.

all videos were created with code in openframeworks and using the ofxBox2d addon for the physics.

timebomb cube from lukasz on Vimeo.

the opportunity came to exhibit the installation again and instead of doing the same old, maddi and i decided to push the idea further and take timebomb away from the 2d flat canvas into a 3-dimensional cube format. it also meant that the cube would be able to facilitate a rear-projection set up and ultimately a multitouch surface.

in the couple months before the event i did my research and developed a basic prototype of a diffused illumination (DI) projection. this is one of many multi-touch methods and one that works particularly well for large interactive surfaces. the downside of using DI is it is not as accurate as some of the other methods, although the timebomb interaction is based on movement detection and does not require a huge amount of precision as long as it could detect a change in the image.

there is a great thread on DI multitouch surfaces on the NUI website and is a great place to start when researching into the topic. the whole method works on infrared technology and pretty much boils down to the IR camera which blocks out visible light, so it doesn’t see the projection but it can see the people touching the surface of the projection.

i ended up going for a firefly mv black and white camera from point grey. its a firewire camera which has quite a lot of support in c++ land and appears to be a favorite amongst openframeworks users. the camera only comes as a body and so i also had to purchase a lens and an IR-filter to block out visible light. if you live in australia and need to kit up your computer vision camera, i recommend a company called adept, very knowledgeable and friendly chaps.

to complete the kit i got a couple IR spotlights from ebay for pretty cheap and i had a chance to test them against some top range IR lights that cost in the thousands ( thanks to the guys from adept ) and the results were surprisingly pretty much identical. just hop on ebay and search for ir illuminator but pay attention to the range and spread of the light, you’d want something with a decent angle of say 60 degrees and that can reach at least 40 meters.

finally you need something to project onto, a diffused material that can catch the light from the projector and be opaque enough to let IR light through so that the camera can see peoples movements on the other side. there are many specialised projection films which cost in the thousands to cover a large surface, some of which are outlined in this article, although since we only had a limited budget we went with drafting film, very cheap and a decent picture contrast.

building of the giant 2.5 meter cube was intense. spent 2 weeks worrying the whole thing would not hold and fall apart on some unsuspecting user, so i started reinforcing every panel of wood with metal brackets, especially the interactive surface which was constructed from 2 giant 10mm thick sheets of acrylic and i made sure the frame would be able to support it. in the end, the cube was SOLID! even the most enthusiastic of multitouch users could not rock it in the slightest.

on the day of the festival it was great to see people interacting with timebomb on the multitouch surface. people were really taking their time with the installation, trying to understand and harness its usability to control the time distortion effect to uncover the specific parts of the interactive graffiti wall. the one to one interaction scale was really the trick that made it all work and its whats so appealing about multitouch surfaces. its bridging the virtual gap and drawing the digital within the reach of the tangible.

triangle field started as a simple sketch put together in a few hours on a sunday and its been good to see it evolving since. there’s that power of the triangle combined subtle shading that gives the impression of viewing something in 3d. its very simple and visually stunning. originally, i came across andy gilmore’s work on flickr and thought his geometric designs using triangles were beautiful and i couldn’t help to imagine what it would look like in motion. the shading between the triangles had to be smooth for it to maintain the illusion of 3d, so naturally if i ever want to create anything smooth looking or moving, i throw in some of the ol’ perlin noise.

triangle field :: 01 from lukasz on Vimeo.

the app is built in openframeworks, and along the way i wrote some life-hacker addons so i could have more control over the app at run-time with the idea that it will eventually become a live visual tool.

before writing the color picker addon, i would have to go into photoshop, select a colour i liked and manually copy the colour values into my code. you can imagine what a pain in the arse that was and not too mention time consuming. the colour picker is a replica of the mac osx colour picker in the way it looks with a couple extra modes. you can set it to automatically traverse through the colour spectrum – guess this is handy when you want a smooth colour transition to happen over a period of time. also there is a mode for random walk, where its kinda like watching an insect crawl around over the colour spectrum – where ever the inspect walks, the colour changes to its position.

ofxColorPicker

making visuals react to music is a pretty tricky thing to set up. firstly you need to do a fft analysis on the audio and apply windowing so that the lower parts of the spectrum look like they have the same sort of intensity as the higher frequencies. then you want to normalise the data so that values you get back fall between the range of 0 and 1 – this ensures you have the audio data under control.
i have now come to the conclusion that the fft analysis doesn’t have to be super accurate just as long as it looks good and is clear. my code is based on avsys and dave bollinger’s fft octana processing example (more on the OF forum). at the moment, the ofxFFT addon does two useful things, returns the normalised average audio power and returns a manipulated array of audio bands which ive used for distorting the triangle field.

in the openframeworks core (exculding addons) there are two possible ways of receiving audio data, one is from an audio file like an mp3 or wav and the other is through the microphone jack. the methods for getting this data are different but they both return the same type of information, so i created an abstract class called AudioAbstract which both AudioFileSpectrum and AudioLiveSpectrum extend. the idea is to be able to use file audio data and live audio data as if it was the same thing and be able to swap them out at run-time.

ofxFFT

triangle field :: 02 from lukasz on Vimeo.

over the last week ive been playing around with openFrameworks on the iphone and getting a handle on openGL ES for a work project. its been lots of fun and suprisingly tricky to find decent resources, especially when mapping textures and geo coordinates onto a sphere. ive developed a nice basic example which i think is a good start for anyone tackling an earth application on the iphone using openFrameworks or even working with Objective C.

the example contains code for drawing a sphere in openGL ES and giving it an earth texture which is provided in the source. theres a Trackball class which allows you to intuitively rotate the textured sphere and a CurveHop class which draws a curved ribbon from location to location as it hops along on the surface of the earth. theres a cool little trick you can do in google maps to get the latitude and longitude of any location, so you can add it into the curve hop chain. just search for a location in google maps and copy the following code into your address bar,

javascript:void(prompt(”,gApplication.getMap().getCenter()));

this will bring up a popup with the latitude and longitude of the center of the map.

download the source code.

light chasm from lukasz on Vimeo.

had the pleasure to team up with the very talented flutter lyon (robyn wilson) to collaborate on a audio / visual performance. we only had a week to prepare and decided to go with a black and white, op-artish theme for the projection visuals, matched with robyn’s very striking black and white attire which during the performance became part of the projection surface.

i wanted to create a custom vj app for the performance and have been getting more proficient in openframeworks so decided to give it a go as a technical and creative challenge. i began by coding the basic visual elements such as the black and white sun rays, the checkered background and playing around with blend modes in opengl. it took me a good 2 hours to work out how to do inverse blending… no reference for it anywhere! so here it is,

glEnable( GL_BLEND );
glBlendFunc( GL_ONE_MINUS_DST_COLOR, GL_ONE_MINUS_SRC_COLOR );
// draw your shapes here.
glDisable( GL_BLEND );

i wanted to control the visuals through my iphone via OSC, that way i could put my laptop away and become part of the crowd… so i wrote a wrapper class for OSCemote (an OSC iphone application) that told my program what buttons were being pressed and the slider values. hope to extend it further for accelerometer values which i think will be far more interesting way of controlling visuals and with the possibility of using body movements in a performance to react visuals.

next i plugged in some FFT classes from the openframeworks forum and viola, audio-reactive visuals. it goes to show the maturity of openframeworks where you can grab a bunch of classes created by the community and plug them straight in without any hassle.

here is the source if any one is interested in running it, made for of_0.06

really enjoyed building vj software from scratch just because you have so much more control over what you can make. im now toying with the idea of taking it further and creating a nodel based interface, so something like quartz composer or pure data interface but for openframeworks. the mind boggles…

bowie cam from lukasz on Vimeo.

the back story is… some friends at the world’s end studio have a biegel named david bowie and it jumped out of a 2 storys window and broke its leg. the dog that thought it could fly… so they put on a david bowie fund raiser party for the pooch’s operation.

ive been itching to get my face tracking projections out there and thought the night was a good opportunity to do something quick and fun. bowie cam is just that… it pastes a face of david bowie over yours so you can get down just like the freaky old bastard. there a four face mask modes it cycles through – two of mr bowie, one being of the pooch and a ziggy lighting bolt across the face. the interactive projection was good fun and people got it straight away, although i suspect some people were just happy to see themselves on the big screen.

it was also a good lesson never to depend on venue lighting. for those that have been to q-bar in sydney will know that its a pretty dark venue, something out of a david lynch film with all the red lighting. so had to improvise and mount a lamp to expose people in front of the camera a little better. even then, it was a fine line… if they got too close, their faces were washed out and the haar finder was unable to pick up their features and when they got too far, they would disapear into the shadows. so i think overall the accuracy of the face tracking was down to about 20% of its normal accuracy. still, good to try it out in the wild.

get the ofxCvHaarTracker source for openframeworks with example.

Time Bomb – Interactive Graffiti (StupidKrap & Holler) from Mike Hill on Vimeo.

timebomb is an interactive graffiti installation that I collaborated on with Maddi Boyd (KissKiss) from StupidKrap. check out the promo video to see what its all about…

here’s a little technical insight into the project.

the timebomb application is based on memo’s fluid dynamics library for openframeworks, ofxmsafluid. its a fantastic opensource library that memo has done a lot of amazing things with and i’ve been able to use it as the basis for fluid time distortion.
interaction input comes from a webcam, the live feed is analysed and used to create ripples in a fluid image. those fluid values are then applied to the pixels of a video, the pure white pixel (0xFF) equals a pixel taken from the top frame of the video and a black pixel (0×00) equals a pixel taken from the bottom frame of the video, all other shades of grey represent pixels between the first and last frame. confused? well hopefully the video of the debug screen will explain it better.

the video on the top left is the actual time-lapse we shot over 5 days.
next to it is the final output with time distortion applied to it.
all other frames below show the camera input and the ripples created in a black and white fluid image.

timebomb debug from lukasz on Vimeo.

here is the final effect on its own.
this is what is going to be projected onto a blank canvas at the exhibition.
people will be able to time-distort the graffiti piece through their body movements and uncover the hidden layers that lay beneath the surface of the finished piece.

timebomb from lukasz on Vimeo.

crimcam from lukasz on Vimeo.

people usually experience seeing their own image either in the photos or video and almost always in a positive and social context. i wanted to bring the viewer into a more sinister and perhaps more confronting environment, to allow them to see themselves in a negative context and to suggest they are not as happy and innocent as they portray themselves to be.

crimcam was built using openframeworks and openCV which makes it possible to perform motion tracking, blob tracking, face detection.

will be posting the source shortly.

@julapy

my two favorite things, abstract geometry and audio reactive visuals.
working on the new beck’s beer site was a treat with a great base design and massive scope for innovation.

i won’t bang on about the creation process but i will say that the cube was modeled using blender, exported as a DAE file, painlessly plugged into papervision and off we went…
if you are new to blender like i was i recommend reading through the manual, its the best place to start and will allow you to tinker and create as you are learning.

had a few days spare towards the end of the project so thought it would be nice to add an audio reactive element to the site. it is the beck’s beer brand and it is heavily associated with electronic music so i reproduced the geometry of the cube segments and added them to a 3d backdrop layer… the space junk fragments float in darkness, illuminated and pushed on by the beat of the music.

still trying to work out what to make open source from this project so if theres anything you’d like me to release, please let me know and ill do my best to post it on here.