Speaker
Description
Audio is frequently paired with visuals, and sometimes vice versa, for didactic, exploratory, and engagement purposes. This paper systematically reviews such practices across their main usage context: (a) the academic domain, including standard visualizations of sound and didactic examples of sonification (including auralization); (b) creative tools that generate sound from visual input; including visuals in music players and dedicated tools for sonifying images that are emerging but only sparsely adopted. Finally, (c) selected artistic works such as installations and live performances. While category (a) typically relies on rule-based mappings between audio and visuals, examples in (b) and (c) mostly employ more unsystematic, creative encodings. Our corpus of examples was assembled through a mixed-method approach combining literature and web searches with expert interviews.Across contexts, one modality (audio, visuals) is either used to be redundant (‘re’) or complementary (‘co’) to the other. To explore this relationship, we apply a new framework that characterizes redundancy and complementarity in audio–visual displays along the perceptualization path: from content; over encoding and physical representation; to user interpretation and interaction. For each aspect, we assess the degree to which visuals and audio are re/co and infer the use cases they best support.Academic contexts and creative tools often aim for redundancy but partly fail to produce perceptually equivalent audio–visual impressions. By contrast, artistic works typically seek complementarity. Our analysis reveals gaps in standardized methods and tools, which would be needed to support more inclusive and comprehensible practices, e.g., in didactics and audience engagement.