8–12 Sept 2026
Europe/Vienna timezone

Natural Language Interaction for Morphology-Aware Audification

FA2026/721
9 Sept 2026, 14:40
20m
Saal 5 (Messe Congress Graz)

Saal 5

Messe Congress Graz

A19 Product Sound Quality and Sound-Driven Design A19.05 Sonification and visualization for acoustics

Speaker

Sofia Vallejo Budziszewski (IEM)

Description

Audification maps data directly to sound using uniform time scaling. We previously introduced a rule-based framework that classifies time signals into one of 15 morphology classes and assigns class-specific time-scaling factors, evaluated across 591 real-world signals from 8 scientific domains and 251 synthetic control signals. That framework only ever produces a class label, a confidence score, and a scaling factor, telling a user nothing about what to listen for, offering no way to feed domain knowledge back in, and, because the tool is a conventional visual interface, remaining inaccessible to anyone who cannot see it.This paper adds a natural-language interaction layer built on a large language model (LLM) strictly downstream of the pipeline. The LLM receives only structured metadata, never the raw signal, and uses it to generate summaries calibrated to a user-supplied domain description, support reclassification and parameter adjustment with trade-offs stated up front, and answer open-domain questions about the signal. Because the entire interaction is text-based, it lowers the barrier for domain and sonification non-experts while simultaneously opening the tool to blind and low-vision (BLV) users via standard screen readers, with no plot or slider required. Our own survey of 57 sonification tools found only 9 explicitly built for BLV users, making this screen-reader-native workflow unusually direct for the field. Every LLM claim traces back to a number the pipeline already computed, so accessibility is gained without sacrificing analytical grounding. We evaluate using open-weight EU-based models (Mistral API and Ollama) to keep the system reproducible and free of proprietary dependencies.

Authors

Presentation materials