Psychogeography with Jack B. Yeats Art Sounding Gallery: Augmented Reality Locative Experience for Blind People

Abstract

Sensory Substitution Devices (SSDs) are a relatively novel concept, based on the idea of the multisensory brain. Research on synaesthesia and sensory pairings has revealed that sensory modalities of the brain are interconnected. Nature has demonstrated examples of people who have lost one sense, which has then been substituted by increased ability in another, for example the case of Daniel Kish who navigates like a bat (Burgess, 2021) by clicking of the tongue (echolocation principle). To find a methodology for translating information from one sense to another, or substitute one sense with another is the principle for all SSDs. A number of approaches to assistive technologies for different impairments have been developed, for example for blind people, such as the vOICe “seeing with sound” and EyeMusic Apps, which convert visual images and colours into sound. While most SSDs are focused on functionality to offer life assistance, such as for navigation, little or no work has been done to include the blind into the emotional world of Visual Arts, despite the fact that there are 45 million blind people in the world (Amir Amedi). In this paper, we present an audio GPS based walking app that presents a translation of the visual expression of artworks by sound/music to deliver the emotional content of the paintings to blind people. The music is composed for six artworks of Irish artist symbolist Jack B. Yeats (1871-1957), specifically reflecting on shapes, colours and emotional content of painting by composer experienced in audio-visual synchronisation via synaesthesia (Rudenko and Córdoba Serrano, 2017). The project is centred around the development of a new methodology for multisensory design (MSD) through the design, implementation, and evaluation of a locative art experience with Augmented Reality (AR), hosted by Haunted Planet Studios (director Mads Haahr).