Embodied sonification

Abstract

Sonification is the communication of data using sound. This thesis is concerned with meaning-making in sonification. It examines how meaning emerges during sonification listening through the lens of embodied cognitive science. It suggests that approaches to sonification which exploit the embodied nature of meaning-making can leverage aesthetic dimensions of sound which allow for more effective communication. This thesis employs a mixed methods research approach to empirically evaluate dimensions of sound which are traditionally categorised as aesthetic. Hypotheses about the embodied nature of meaning-making in sonification listening are developed through a reading of the literature, through exploration with prototyping platforms, and through exploratory data-driven composition. These hypotheses are then submitted to empirical listener evaluations, from which conclusions about embodied meaning-making in sonification listening can be drawn. Chapter 1 considers sonification in terms of two models of meaning-making: the computationalist model, which reduces meaning-making to information processing, and the embodied model, which treats embodied experience as the basis of all meaning. Aesthetic approaches, scientific approaches and leading sonification techniques are considered in reference to these models and it is argued that the embodied model is of more use to the development of communicatively effective sonifications than the popular computationalist model, which cannot account for the aesthetic dimensions of sound. The Embodied Sonification Listening Model (ESLM) is introduced in Chapter 2 to offer an embodied perspective on meaning-making during sonification listening. Chapter 2 also introduces and defines the Embodied Sonic Dimension and the Embodied Sonic Complex as novel conceptual measurement schemes for working with the aesthetic and embodied dimensions of sound in a sonification context. An empirical evaluation, which describes how a listener’s embodied schematic knowledge influences their interpretation of a sonification, is also presented in this chapter. The ELSM, embodied sonic dimensions and embodied sonic complex represent useful theoretical tools for guiding the design of sonifications, which can exploit the embodied nature of meaning-making to effectively communicate data to a listener. Chapter 3 investigates the communicative effectiveness of some of the embodied sonic dimensions afforded by vocal gestures. It presents a large number of evaluations, the results of which highlight useful synthesis strategies for modelling and controlling these dimensions in a sonification context. The embodied sonic dimensions explored in Chapter 3 represent a means for intuitively communicating data to a listener on the basis of their previous embodied experiences. Chapter 4 investigates the communicative effectiveness of some of the embodied sonic complexes which environmental soundscapes can offer to sonification. The research described in this chapter was intended to develop an embodied soundscape sonification framework. This was achieved through the development of a number of candidate frameworks via exploratory research practices, which were then submitted to rigorous empirical evaluations intended to determine the most effective framework. The embodied soundscape sonification framework presented in Chapter 4 represents an effectively communicative sonification technique, suggested by empirical evaluation to be more effective than pitch-mapping sonification in certain circumstances. Chapter 5 explored the conceptual metaphorical underpinnings of sonification listening as described by the ESLM. It employed a similar research method to Chapter 4 to develop the “Temporo-Spatial Motion Framework” which exploits metaphorical mappings between the time and space domains to add a sense of temporal context to timeseries sonification. This framework was then further refined through a process of empirical listener evaluation. The temporo-spatial motion framework represents a solution to the problem of representing temporal context in time-series sonification. Chapter 6 summarises the research described in this thesis and a presents a discussion of its relevance and contribution to the wider field. It argues that the thesis has presented compelling evidence for an embodied approach to sonification.