openframeworks is a C++ toolkit for creative coding which is open source and supported by an avid online community that openly share their code and knowledge. it was time to bring out the OF community in sydney out of the woodwork and chat shop.

rene and i finally got it together after a couple months of inconvenient date swapping. we thought rather then jumping straight into a knitting circle, it would be a good idea to start from the beginning with a introductory workshop into openframeworks.

the turn out was great, a mix between experienced OF’ers and those who were just getting into it. we covered quite a big array of topics starting from the basic OF structure, to creating simple particle systems, to green screening via pixel manipulation, to openCV and blob tracking. we eventually finished with a demo of a interactive table sending TUIO data as input into a particle system we covered in a previous example. all this in a 4 hour session!

all the source to the examples we covered in the workshop can be downloaded here.

we’re planning to make the next one a lot less formal, more of a get together to chat OF.
you can follow the latest developments in the OF sydney community here,

http://www.facebook.com/group.php?gid=147185535296993
http://groups.google.com/group/ofsydney

frock stars celebrates the australian fashion week with an exhibition put on at the powerhouse museum. i was fortunate enough to contribute to the event with two interactive installations which allow the user to style the hair and makeup of a virtual model with the use of real physical styling tools.

in the video you can see people using objects such as the hair dryer and curling tongs as the controllers to the interactive experience. it was important that these objects were as close as possible to the real thing, which in the early scoping eliminated the use of markers and eventually lead to the use of colour tracking as the main computer vision method for identifying and tracking the objects.

colour tracking proved to be quite accurate thanks to the use of flouro colours which stood out more from the surrounding environment and were less likely to clash with people’s clothing (unless the 80s made yet another comeback!). each of the two interactives were fitted with a affordable fire-i camera, which allows to configure most capture settings including the ability to turn off the EVIL auto-gain.

the above photo shows the colour tracker passing data into flash. the colour tracking software was written using openFrameworks and communicated with flash via a socket connection. openFrameworks did all the tracking while flash took care of all the pretty graphics, animations and video. there is some more info under this OF post.

here is the source code to the colorTracker.

the two interactive demos use openCV for the camera interaction.
1) first step is to work out the frame differencing, basically taking away the last two frames in the camera stream and working out the difference in the image. its a way of allowing the camera to see movement.
2) next step is to work out the common area of intersect between the movement image and the holler logo. by applying the & operator, both images are checked for common areas containing white pixels.
3) contour analysis is then applied on the common area of intersect. it searches out all the white blobs in that image and returns the result as a series of points which when connected show an outline of the blobs and mathematically describe what each blob/shape looks like.
4) now that each white blob is described as a bunch of points, that data can be passed to a delaunay triangulation algorithm which calculates the triangles necessary to fill that shape.
5) finally optical flow is used to add velocity to the box2d elements.

kikko posted ofxTriangle along with his great example on this openframeworks post.
also check out akira’s awesome box2d demo.

being able to generate a number of triangles from a polygon goes very far with box2d, a physics library for c++ which has been extended into openframeworks. it allows you to create complex shapes that can be part of the physics simulation by using their triangle constituents. box2d is very basic at its core, it supports rectangular boxes, circles and simple convex polygons like triangles, but thats where it stops and if you want to go any further, you have to use these building blocks to create more complex shapes.

i thought it would be a good idea to start doing the experiments with some basic shapes, something i could easily import into openframeworks and since i was doing all this in work downtime i felt just a little obliged to use the agency logo.

one thing openframeworks is a little short on at the moment is svg support, so its not as simple as it should be to import illustrator files and receive a list of points to draw your shapes. instead i had to use contour analysis to work out the shape of the logo from an image. this is all good but the final shapes are not as clean as they could be if imported as straight svg files.

EDIT :: i stay corrected, was pointed to ofxSvgLoader addon and it works beautifully, happy days!

the visual component stems from an existing generative application, the triangle field, which i wrote some time back. with tr-IO, i wanted to allow people to explore the generative art work by having influence over the colour, form and motion, and to create their own flavour of visuals.

i was very happy when eli murry / gentleforce jumped onto the project as a collaborator and enriched the installation by adding another dimension of sound design. eli created an array of ambient drones, clicks and ominous electronic growls which were mapped to the interactive table and controlled by the pyramid objects. a user could place the pyramids at different points on the table to awaken the various hidden sounds and create a unique soundscape.

the interactive table is built using reactivision which uses fiducial markers as a means of tracking objects. each pyramid object was fitted with a unique fiducial marker which allowed the camera inside to track its position and rotation on the table.

the pyramid objects were also fitted with programmable BlinkM leds which could be easily programmed via an arduino board. the initial idea was to get the lights inside the pyramid to change depending on its position on the table but for this it would have required a wireless arduino for each pyramid which unfortunately fell outside of the budget this time around, but its definitely something id like to add in the future.

whats the future hold for tr-IO?
well hopefully eli and i can up-size it to a completely immersive experience.
a whole space dedicated to projecting the visuals in a seamless 360 view.
and throw in some surround sound.
chill out room at a doof anyone?
i joke, but a gallery space would be nice.

physics is always fun, fact.
in this video, circle particles fill the volume of the building and once settled in place, create an image.
the trick here is to render the particle system twice. first, run the simulation and record the colour values of all the balls once they’ve settled and then to render again the second time and since the simulation is identical, the balls fall into the right place with the right colour.

being the becks bar and all, there was one catch – the video had to include the becks logo.
in this video, the logo is first created using circle packing (basically fitting as many circles as possible into an image) and then released to spill and swish around inside the building before eventually loosing momentum and fizzling out to the top.

all videos were created with code in openframeworks and using the ofxBox2d addon for the physics.

timebomb cube from lukasz on Vimeo.

the opportunity came to exhibit the installation again and instead of doing the same old, maddi and i decided to push the idea further and take timebomb away from the 2d flat canvas into a 3-dimensional cube format. it also meant that the cube would be able to facilitate a rear-projection set up and ultimately a multitouch surface.

in the couple months before the event i did my research and developed a basic prototype of a diffused illumination (DI) projection. this is one of many multi-touch methods and one that works particularly well for large interactive surfaces. the downside of using DI is it is not as accurate as some of the other methods, although the timebomb interaction is based on movement detection and does not require a huge amount of precision as long as it could detect a change in the image.

there is a great thread on DI multitouch surfaces on the NUI website and is a great place to start when researching into the topic. the whole method works on infrared technology and pretty much boils down to the IR camera which blocks out visible light, so it doesn’t see the projection but it can see the people touching the surface of the projection.

i ended up going for a firefly mv black and white camera from point grey. its a firewire camera which has quite a lot of support in c++ land and appears to be a favorite amongst openframeworks users. the camera only comes as a body and so i also had to purchase a lens and an IR-filter to block out visible light. if you live in australia and need to kit up your computer vision camera, i recommend a company called adept, very knowledgeable and friendly chaps.

to complete the kit i got a couple IR spotlights from ebay for pretty cheap and i had a chance to test them against some top range IR lights that cost in the thousands ( thanks to the guys from adept ) and the results were surprisingly pretty much identical. just hop on ebay and search for ir illuminator but pay attention to the range and spread of the light, you’d want something with a decent angle of say 60 degrees and that can reach at least 40 meters.

finally you need something to project onto, a diffused material that can catch the light from the projector and be opaque enough to let IR light through so that the camera can see peoples movements on the other side. there are many specialised projection films which cost in the thousands to cover a large surface, some of which are outlined in this article, although since we only had a limited budget we went with drafting film, very cheap and a decent picture contrast.

building of the giant 2.5 meter cube was intense. spent 2 weeks worrying the whole thing would not hold and fall apart on some unsuspecting user, so i started reinforcing every panel of wood with metal brackets, especially the interactive surface which was constructed from 2 giant 10mm thick sheets of acrylic and i made sure the frame would be able to support it. in the end, the cube was SOLID! even the most enthusiastic of multitouch users could not rock it in the slightest.

on the day of the festival it was great to see people interacting with timebomb on the multitouch surface. people were really taking their time with the installation, trying to understand and harness its usability to control the time distortion effect to uncover the specific parts of the interactive graffiti wall. the one to one interaction scale was really the trick that made it all work and its whats so appealing about multitouch surfaces. its bridging the virtual gap and drawing the digital within the reach of the tangible.

triangle field started as a simple sketch put together in a few hours on a sunday and its been good to see it evolving since. there’s that power of the triangle combined subtle shading that gives the impression of viewing something in 3d. its very simple and visually stunning. originally, i came across andy gilmore’s work on flickr and thought his geometric designs using triangles were beautiful and i couldn’t help to imagine what it would look like in motion. the shading between the triangles had to be smooth for it to maintain the illusion of 3d, so naturally if i ever want to create anything smooth looking or moving, i throw in some of the ol’ perlin noise.

triangle field :: 01 from lukasz on Vimeo.

the app is built in openframeworks, and along the way i wrote some life-hacker addons so i could have more control over the app at run-time with the idea that it will eventually become a live visual tool.

before writing the color picker addon, i would have to go into photoshop, select a colour i liked and manually copy the colour values into my code. you can imagine what a pain in the arse that was and not too mention time consuming. the colour picker is a replica of the mac osx colour picker in the way it looks with a couple extra modes. you can set it to automatically traverse through the colour spectrum – guess this is handy when you want a smooth colour transition to happen over a period of time. also there is a mode for random walk, where its kinda like watching an insect crawl around over the colour spectrum – where ever the inspect walks, the colour changes to its position.

ofxColorPicker

making visuals react to music is a pretty tricky thing to set up. firstly you need to do a fft analysis on the audio and apply windowing so that the lower parts of the spectrum look like they have the same sort of intensity as the higher frequencies. then you want to normalise the data so that values you get back fall between the range of 0 and 1 – this ensures you have the audio data under control.
i have now come to the conclusion that the fft analysis doesn’t have to be super accurate just as long as it looks good and is clear. my code is based on avsys and dave bollinger’s fft octana processing example (more on the OF forum). at the moment, the ofxFFT addon does two useful things, returns the normalised average audio power and returns a manipulated array of audio bands which ive used for distorting the triangle field.

in the openframeworks core (exculding addons) there are two possible ways of receiving audio data, one is from an audio file like an mp3 or wav and the other is through the microphone jack. the methods for getting this data are different but they both return the same type of information, so i created an abstract class called AudioAbstract which both AudioFileSpectrum and AudioLiveSpectrum extend. the idea is to be able to use file audio data and live audio data as if it was the same thing and be able to swap them out at run-time.

ofxFFT

triangle field :: 02 from lukasz on Vimeo.

light chasm from lukasz on Vimeo.

had the pleasure to team up with the very talented flutter lyon (robyn wilson) to collaborate on a audio / visual performance. we only had a week to prepare and decided to go with a black and white, op-artish theme for the projection visuals, matched with robyn’s very striking black and white attire which during the performance became part of the projection surface.

i wanted to create a custom vj app for the performance and have been getting more proficient in openframeworks so decided to give it a go as a technical and creative challenge. i began by coding the basic visual elements such as the black and white sun rays, the checkered background and playing around with blend modes in opengl. it took me a good 2 hours to work out how to do inverse blending… no reference for it anywhere! so here it is,

glEnable( GL_BLEND );
glBlendFunc( GL_ONE_MINUS_DST_COLOR, GL_ONE_MINUS_SRC_COLOR );
// draw your shapes here.
glDisable( GL_BLEND );

i wanted to control the visuals through my iphone via OSC, that way i could put my laptop away and become part of the crowd… so i wrote a wrapper class for OSCemote (an OSC iphone application) that told my program what buttons were being pressed and the slider values. hope to extend it further for accelerometer values which i think will be far more interesting way of controlling visuals and with the possibility of using body movements in a performance to react visuals.

next i plugged in some FFT classes from the openframeworks forum and viola, audio-reactive visuals. it goes to show the maturity of openframeworks where you can grab a bunch of classes created by the community and plug them straight in without any hassle.

here is the source if any one is interested in running it, made for of_0.06

really enjoyed building vj software from scratch just because you have so much more control over what you can make. im now toying with the idea of taking it further and creating a nodel based interface, so something like quartz composer or pure data interface but for openframeworks. the mind boggles…

bowie cam from lukasz on Vimeo.

the back story is… some friends at the world’s end studio have a biegel named david bowie and it jumped out of a 2 storys window and broke its leg. the dog that thought it could fly… so they put on a david bowie fund raiser party for the pooch’s operation.

ive been itching to get my face tracking projections out there and thought the night was a good opportunity to do something quick and fun. bowie cam is just that… it pastes a face of david bowie over yours so you can get down just like the freaky old bastard. there a four face mask modes it cycles through – two of mr bowie, one being of the pooch and a ziggy lighting bolt across the face. the interactive projection was good fun and people got it straight away, although i suspect some people were just happy to see themselves on the big screen.

it was also a good lesson never to depend on venue lighting. for those that have been to q-bar in sydney will know that its a pretty dark venue, something out of a david lynch film with all the red lighting. so had to improvise and mount a lamp to expose people in front of the camera a little better. even then, it was a fine line… if they got too close, their faces were washed out and the haar finder was unable to pick up their features and when they got too far, they would disapear into the shadows. so i think overall the accuracy of the face tracking was down to about 20% of its normal accuracy. still, good to try it out in the wild.

get the ofxCvHaarTracker source for openframeworks with example.

Time Bomb – Interactive Graffiti (StupidKrap & Holler) from Mike Hill on Vimeo.

timebomb is an interactive graffiti installation that I collaborated on with Maddi Boyd (KissKiss) from StupidKrap. check out the promo video to see what its all about…

here’s a little technical insight into the project.

the timebomb application is based on memo’s fluid dynamics library for openframeworks, ofxmsafluid. its a fantastic opensource library that memo has done a lot of amazing things with and i’ve been able to use it as the basis for fluid time distortion.
interaction input comes from a webcam, the live feed is analysed and used to create ripples in a fluid image. those fluid values are then applied to the pixels of a video, the pure white pixel (0xFF) equals a pixel taken from the top frame of the video and a black pixel (0×00) equals a pixel taken from the bottom frame of the video, all other shades of grey represent pixels between the first and last frame. confused? well hopefully the video of the debug screen will explain it better.

the video on the top left is the actual time-lapse we shot over 5 days.
next to it is the final output with time distortion applied to it.
all other frames below show the camera input and the ripples created in a black and white fluid image.

timebomb debug from lukasz on Vimeo.

here is the final effect on its own.
this is what is going to be projected onto a blank canvas at the exhibition.
people will be able to time-distort the graffiti piece through their body movements and uncover the hidden layers that lay beneath the surface of the finished piece.

timebomb from lukasz on Vimeo.