Project detail:

This installation comprised a 600cm x 400cm (vertical/wall) and 700cm x 400cm (horizontal/floor) projection surface. The full 13000cm x 400cm image was projection mapped using two 4K projectors to create a seamless image. The space around the projection surface was masked from floor to ceiling using blackout stage curtains. This defined the physical boarders of the installation and created an enclosed environment. Participants were fitted with three items of wearable technology that augmented multiple layers of interactivity between participants and the environment: a uniquely identifiable InfraRed (IR) LED array for position tracking, a wireless optical Heart Rate (HR) monitor, and a wireless haptic chest strap[2].

Position Tracking
Aprototype position tracking system was built using two 3d printed IR LED arrays. Each array was fabricated using 10x LEDs emitting IR light at two different wavelengths (940nm and 850nm). This set up enabled the tracking system to distinguish between participant A and B. The arrays were sewn into two hats. In this way a 4k camcorder fitted with an IR filter and mounted 600cm above the stage floor provided an uninterpreted view of both IR arrays. The IR light was tracked, calibrated, and position mapped to the environment. This produced uniquely identifiable real-time position data for each participant as they moved through the installation environment.

Heart Rate Monitoring
The real-time HR data of each participant was captured using the series 3 apple iWatch and monitored via a paired apple iPhone SE. The continuous data were then broadcast via Open Sound Control (OSC)[3] on the local WIFI network using the developer app Holon.ist[4] installed on the paired apple device. The OSC HR broadcast was received and processed using Touchdesigner. The HR of each participant was used to drive elements of both the visual projection and the haptic interaction.

Haptic Chest Strap
The haptic chest strap repurposed a wireless vibrating metronome manufactured by Soundbrenner called the Pulse. Designed and built for musical performance and practice, the Pulse device has the capacity for MIDI[5] integration. Using Touchdesigner a virtual MIDI network was set up that rebroadcast the incoming OSC HR data via a MIDI output. This enabled the vibrations of the Pulse strap to be triggered in real-time by HR data from the iWatch to create an interactive haptic device. The OSC/MIDI network was set up so that the HR of participant A was driving the chest strap haptics of participant B and vice-versa.

Projection
The visual component was built around a Reaction Diffusion system built in Touchdesigner. The Reaction Diffusion system was triggered by the pulsing expansion and contraction of a thin white circle projected onto the dynamic position of each participant. The pulsing movement of the circle was driven using real-time HR data, and the movement of each participant was tracked through the environment using the unique IR position tracking.

In this way, each participant had their own HR visualised and projected on to their tracked position via the pulsing circle as they moved throughout the space. In addition each participant could feel or sense the other participants HR via the vibrations of the chest strap.

---

[2] Soundbrenner Pulse, https://www.soundbrenner.com/pulse/.

[3] Open Sound Control is an open-source data transport protocol for real-time message communication across applications and hardware, https://ccrma.stanford.edu/groups/osc/README.html

[4] Holon.ist is an app-based system for flexible processing and transfer of sensor data using MIDI, OSC, and Java, https://holon.ist.

[5] Musical Instrument Digital Interface (MIDI) is an industry standard music technology protocol that connects digital musical instruments, computers, tablets, and smartphones from many different companies, https://www.midi.org.