Synaesthesia Gallery AR: Journey Through the Senses - Using Augmented Reality for Education

Abstract

This work-in-progress paper presents Synaesthesia Gallery AR: Journey Through the Senses, an educational game-like experience designed to bring awareness of synaesthesia, to learn and understand creativity of cross-modal perception. Synaesthesia is a perceptual trait characterised by cross-modal wiring of the brain experienced by a 4–6% of the population. Synaesthesia research has a long history (as far back as 1690), and sometimes synaesthesia was confused for hallucinations. The AR app documents creative synaesthesia experiences of individuals who have contributed significantly to synaesthesia research, including international artists, such as Carol Steen (president of USA Synaesthesia Association), and scientists, such as Jamie Ward (Professor at the University of Sussex, UK). Ramachandran and Hubbard have called synaesthesia “a window into perception, thought and language” that can help explain how the brain works. The app experience takes a form of an interactive digital gallery for the outdoors, a locative Augmented Reality (AR) art gallery with music by Liszt, Chopin, Schuman, Scriabin and others painted by artists-synaesthetes and music composed on art. The work is the result of a seven-year collaboration between artists-synaesthetes and pianist/researcher Dr Svetlana Rudenko. The app provides pleasing aesthetic experience as well as learning, to general population, neuroscience and psychology students, synaesthesia associations and music art lovers.