Choosing whether to represent data in an abstract or concrete manner through sonification is generally dependent on the applicability of the dataset and personal preference of the designer. For supporting a visualization with a high level of abstraction, a sonification can purposefully act as a complement by giving concrete contextual cues to the data representation with the use of auditory icons. This paper presents a case study of using bird songs as auditory icons to give context to a biology visualization, and explores how additional information of the bird species can be conveyed together with the auditory icons with parameter mapping sonification. The auditory icons are used as a foundation to convey additional information of the dataset, either by creating a parametric auditory icon, or by adding an additional sonification that accompanies the auditory icon. A user evaluation was conducted to validate and compare the different sonification mappings. The results show that there is a subjective difference of how participants perceived the sonifications, where the participants preferred sonifications that had a concrete mapping design. The sonification approaches that are explored in this study have the potential to be applied to more general sonification designs.

The research communities studying visualization and sonification share exceptionally similar goals, essentially aiming to make data interpretable to humans. One community uses visual representations, while the other employs auditory (nonspeech) repre- sentations of data. Although the two communities have much in common, they developed mostly in parallel, with only compara- tively few examples of integrated audiovisual data analysis idioms presented so far. This panel brings together researchers from both the fields of visualization and sonification to collectively discuss the question: ‘Integrating Sonification and Visualization – but why?’ 

In the panel discussion, we will tackle this question along two main hypotheses: Combining the modalities to (1) increase the “bandwidth from data to brain,” or (2) to increase a user’s personal engagement during the data analysis process. On the one hand, designers might aim to communicate more data in less time or gain more and more complex insights from the data by using a multi-modal display. This argument follows an understanding that two senses should be capable of processing more information than “just” one. On the other hand, sometimes, a more engaged analysis of the represented data is desirable. Engagement with data visualization stands as a crucial topic in numerous contexts within our field, encouraging “deeper” thinking by expert analysts, readers of data journalism articles, and students in educational settings. We hypothesize that integrating visualization with sonification holds the potential to enhance user engagement during analysis. Through the panel discussion, we want to delve into the spectrum between aiming for bandwidth and engagement, seeking to understand the opportunities and challenges of integrating sonification and visualization. 

The research communities studying visualization and sonification for data display and analysis share exceptionally similar goals, essentially making data of any kind interpretable to humans. One community does so by using visual representations of data, and the other community employs auditory (non-speech) representations of data. While the two communities have a lot in common, they developed mostly in parallel over the course of the last few decades. With this STAR, we discuss a collection of work that bridges the borders of the two communities, hence a collection of work that aims to integrate the two techniques into one form of audiovisual display, which we argue to be “more than the sum of the two.” We introduce and motivate a classification system applicable to such audiovisual displays and categorize a corpus of 57 academic publications that appeared between 2011 and 2023 in categories such as reading level, dataset type, or evaluation system, to mention a few. The corpus also enables a meta-analysis of the field, including regularly occurring design patterns such as type of visualization and sonification techniques, or the use of visual and auditory channels, showing an overall diverse field with different designs. An analysis of a co-author network of the field shows individual teams without many interconnections. The body of work covered in this STAR also relates to three adjacent topics: audiovisual monitoring, accessibility, and audiovisual data art. These three topics are discussed individually in addition to the systematically conducted part of this research. The findings of this report may be used by researchers from both fields to understand the potentials and challenges of such integrated designs while hopefully inspiring them to collaborate with experts from the respective other field.

One of the commonly used visualization techniques for multivariate data is the parallel coordinates plot. It provides users with a visual overview of multivariate data and the possibility to interactively explore it. While pattern recognition is a strength of the human visual system, it is also a strength of the auditory system. Inspired by the integration of the visual and auditory perception in everyday life, we introduce an audio-visual analytics design named Parallel Chords combining both visual and auditory displays. Parallel Chords lets users explore multivariate data using both visualization and sonification through the interaction with the axes of a parallel coordinates plot. To illustrate the potential of the design, we present (1) prototypical data patterns where the sonification helps with the identification of correlations, clusters, and outliers, (2) a usage scenario showing the sonification of data from non-adjacent axes, and (3) a controlled experiment on the sensitivity thresholds of participants when distinguishing the strength of correlations. During this controlled experiment, 35 participants used three different display types, the visualization, the sonification, and the combination of these, to identify the strongest out of three correlations. The results show that all three display types enabled the participants to identify the strongest correlation — with visualization resulting in the best sensitivity. The sonification resulted in sensitivities that were independent from the type of displayed correlation, and the combination resulted in increased enjoyability during usage.

This poster presents initial design steps exploring how sonification can be used to support visualization for comprehension of space and time in astronomical data. Radio signals travel at the speed of light. With a visualization of the universe, it is possible to travel faster than light and pass the radio waves leaving earth. We can then travel back in time. We propose to use sonification consisting of songs representing each year as a musical journey through space and time to create an engaging experience.

In this paper, we present how we used generative AI (GenAI) as a pedagogical tool for students taking a course in tangible interaction design. In this course, the students design different physical-digital objects (PDOs) to learn designing, sketching and prototyping with code and hardware. However, due to the short course duration these PDOs are not evaluated or explored with any kind of field or user study. Therefore we gave the students the exercise of doing user interviews with GenAI to explore their design ideas further. With this paper, we contribute a description and the outcomes of this approach, and highlight the pedagogical implications for student learning.

Traveling by public transport can be challenging for a visually impaired traveler. However, visual information can be supported by sonification, the use of non-speech sound to convey information about data. This research project aims to explore how sonification can be used to provide information to a traveler at a bus stop. Three situations are described together with different sonification design approaches that will later be further developed and evaluated.