Live-generated Interface and Notation Environment in Augmented Reality

Read the article on ResearchGate


Recent developments in Augmented Reality (AR) technology are opening up new ways of representation and interaction with virtual objects; at the same time, increase in processing power of portable devices is enabling a wide diffusion of applications until recently usable only in very specific situations (like motion-capture labs).

LINEAR (Live-generated Interface and Notation in Augmented Reality) is an environment designed for musical performance in AR. One performer, using an AR iPhone app, can create virtual objects (rendered in real-time and superimposed to the real environment) according to the movement of the device; they are used both as virtual interfaces for electronics (sending OSC message to Max/MSP on a computer) and as live-generated graphic notation. LINEAR allows, with some limitations, the representation of gestural movements with an exact 3-D placement in space: we can now have an analogic notation of gestures, rather than a symbolic one. For the iPhone performer, the act of notation corresponds to the notated act.

The resulting representations can be also interpreted as a form of graphic animated notation by other performers (the iPhone screen is mirrored to a projector). The multiperspectivity of the notation and the possibilities of interaction with virtual bodies allow a high level of flexibility and foresee a potentially very rich scenario.

Hong Kong Baptist University, Contemporary Music Workshop


© 2017 by Giovanni Santini  composer