Created by Waltz Binaire, Soap and Milk is designed as an interactive experience of data, allowing the observer to perceive social media as an overwhelming organic figment. Each microscopic droplet represents a tweet and once an entity gets spawned – the viewer is invited to physically interact and explore its behaviour.

The visual elements of the visualisation consist of more than forty layers of different rendering effects and GPU based calculations. Techniques such as fluid calculations, shaders, raymarching, ambient occlusion, texture effects and post processing combine to create detailed, fragile impressions. No polygons were used for the implementation of the graphics; all visual layers consist of fragment shaders, allowing us accurate calculation of shapes, shadows and reflections for each pixel. Hence they were able to create the illusion of endless details. The efficiency of the DX11 render pipeline of vvvv, enabled the team to quickly investigate, experiment and retry different visual scenarios and concepts.

For capturing the visitors, Waltz Binaire used an industrial infrared camera by ximea to capture the movements. Their system shoots 2k videos in 170 frames per second, allowing the algorithm to never lose track of a moving body or its motion. In order to process this massive data, they wrote a custom, gpu based optical flow solution to analyse and compare 2 million pixels in under 5 milliseconds. The whole system runs on one computer, equipped with a powerful processor and a GTX TITAN X. The camera image is uploaded to GPU, as soon as a new frame is detected. From this point on, all tasks ranging from image analysis to real-time rendering is performed on GPU only – to avoid bottle necks of data processing and latency. The motion detection is done in openFrameworks and the rendering passes in vvvv; communication between the applications is realized via spout.

Created using OpenFrameworks 0.9 with ofxXimea (Nathan Wade), ofxBlur (Kyle McDonald), ofxSpout (Elliot Woods) and vvvv  + dx11 (Mr. VUX) and Evvvvil Tweet Engine (evvvvil).

Project Page | Waltz Binaire

Credits: Artistic Direction: Christian Mio Loclair / Technical Research: Jeremias Volker / Music: kling klang klong / Camera: Julian Voltmann / Photo: Felix Albertin / Commisioned by: Identitätsstiftung.

Follow us on Facebook