With an increased creation of and access to data, and a growing demand on humans in analytical and decision-making processes, there is a need to further facilitate perceptual and cognitive abilities to support these processes. An approach to support these tasks is to leverage more sensory systems, such as the auditory, to increase information retention to get a more comprehensive understanding of a dataset or situation, while also involving more senses to further engage the user. Audiovisual data interfaces enable the distribution of data variables to any of the two senses to reduce the risk of cognitive overload, or highlight specific data variables by mapping them to both sensory modalities. However, the success of an audiovisual data interface is dependent on the integration of the two senses and how this is utilized in the resulting interface.

This thesis contributes to sonification research by taking a human-centered approach to integrating sonification and visualization. The human-centered approach involved working with domain experts and users during the design process of the sonification, and to create perceptually motivated designs by utilizing how the auditory and visual systems complement each other and how they are integrated in cognition for sense-making.

The thesis contains six studies that explored the integration of sonification and visualization, ranging from literature surveys to design-oriented studies. A state-of-the-art report provided a survey on the integration of sonification and visualization to offer an introduction to the field to new and current practitioners. In another study, a conceptual framework for scene analysis was developed to support the design and analysis of audiovisual data representations. The rest of the studies were design-oriented, where each focused on a specific aspect of how sonification can complement visualization, depending on the domain and tasks of the study. Other than providing concrete examples of audiovisual integrations, the main contributions of the design-oriented studies are provided in their evaluation results in the form of design recommendations. These include the use of redundant mappings for multi-dimensional data analysis, and considering subjective differences of domain experts for situational awareness support in air traffic control. The design-oriented studies also compared sonification designs, where one study showed a potential trade-off of using informative or pleasant designs for astronomy science communication. Another comparison showed that concrete sonification designs can complement abstract visualizations. Lastly, the thesis provides design considerations by comparing the level of redundancy, indexicality, and complexity of each sonification design through the developed conceptual framework. Overall, this thesis offers motivated recommendations for the integration of sonification and visualization.

Med en ökad mängd och tillgång till data, och ett växande krav på människor i analys- och beslutsprocesser, finns det ett behov av att ytterligare utnyttja perceptuella och kognitiva förmågor för att underlätta dessa processer. En metod för att stödja i dessa uppgifter är att utnyttja fler sensoriska system, till exempel hörseln, för att öka informationsflödet och få en mer omfattande förståelse av ett dataset eller situation, samtidigt som fler sinnen involveras för att engagera användaren. Audiovisuella datagränssnitt gör det möjligt att distribuera datavariabler till vilket som helst av de två sinnena för att minska risken för kognitiv överbelastning, eller att lyfta fram specifika datavariabler genom att koppla dem till båda de sensoriska modaliteterna. Hur framgångsrikt ett audiovisuellt datagränssnitt är beror på hur de två sinnena integreras och hur detta utnyttjas i det resulterande gränssnittet.

Den här avhandlingen bidrar till forskningen om sonifiering genom att tillhandahålla en människocentrerad integration av sonifiering och visualisering. Det människo-centrerade tillvägagångssättet innebär att man arbetar med domänexperter och användare under designprocessen av sonifieringen och att man skapar perceptuellt motiverade designer genom att utnyttja hur de auditiva och visuella systemen kompletterar varandra och hur de integreras i kognitionen för meningsskapande.

Avhandlingen innehåller sex studier som undersökte integrationen av sonifiering och visualisering, vilket sträckte sig från litteraturöversikter till designorienterade studier. En state-of-the-art-rapport gav en översikt av integrationen av sonifiering och visualisering för att erbjuda en introduktion till fältet för nya och nuvarande utövare. I en annan studie utvecklades ett konceptuellt ramverk för scenanalys som kan användas som stöd vid utformning och analys av audiovisuella datarepresentationer. Resten av studierna var designorienterade, där de fokuserade på en specifik aspekt av hur sonifiering kan komplettera visualisering, beroende på studiens domän och uppgifter. Förutom att ge konkreta exempel på audiovisuella integrationer är de viktigaste bidragen från studierna deras utvärderingsresultat i form av designrekommendationer. Dessa inkluderar användning av redundanta mappningar för flerdimensionell dataanalys och beaktande av subjektiva skillnader hos domänexperter för stöd till situationsmedvetenhet inom flygledning. Studierna jämförde också olika sonifieringsdesigner, där en studie visade en potentiell avvägning mellan att använda informativ eller behaglig design för astronomisk vetenskapskommunikation, medan en annan studie visade att en konkret sonifieringsdesign kan komplettera abstrakta visualiseringar. Slutligen ger avhandlingen designöverväganden genom att jämföra nivån av redundans, indexikalitet och komplexitet för varje sonifieringsdesign genom det utvecklade konceptuella ramverket. Sammantaget ger denna avhandling motiverade rekommendationer för integrering av sonifiering och visualisering.

Choosing whether to represent data in an abstract or concrete manner through sonification is generally dependent on the applicability of the dataset and personal preference of the designer. For supporting a visualization with a high level of abstraction, a sonification can purposefully act as a complement by giving concrete contextual cues to the data representation with the use of auditory icons. This paper presents a case study of using bird songs as auditory icons to give context to a biology visualization, and explores how additional information of the bird species can be conveyed together with the auditory icons with parameter mapping sonification. The auditory icons are used as a foundation to convey additional information of the dataset, either by creating a parametric auditory icon, or by adding an additional sonification that accompanies the auditory icon. A user evaluation was conducted to validate and compare the different sonification mappings. The results show that there is a subjective difference of how participants perceived the sonifications, where the participants preferred sonifications that had a concrete mapping design. The sonification approaches that are explored in this study have the potential to be applied to more general sonification designs.

The research communities studying visualization and sonification share exceptionally similar goals, essentially aiming to make data interpretable to humans. One community uses visual representations, while the other employs auditory (nonspeech) repre- sentations of data. Although the two communities have much in common, they developed mostly in parallel, with only compara- tively few examples of integrated audiovisual data analysis idioms presented so far. This panel brings together researchers from both the fields of visualization and sonification to collectively discuss the question: ‘Integrating Sonification and Visualization – but why?’ 

In the panel discussion, we will tackle this question along two main hypotheses: Combining the modalities to (1) increase the “bandwidth from data to brain,” or (2) to increase a user’s personal engagement during the data analysis process. On the one hand, designers might aim to communicate more data in less time or gain more and more complex insights from the data by using a multi-modal display. This argument follows an understanding that two senses should be capable of processing more information than “just” one. On the other hand, sometimes, a more engaged analysis of the represented data is desirable. Engagement with data visualization stands as a crucial topic in numerous contexts within our field, encouraging “deeper” thinking by expert analysts, readers of data journalism articles, and students in educational settings. We hypothesize that integrating visualization with sonification holds the potential to enhance user engagement during analysis. Through the panel discussion, we want to delve into the spectrum between aiming for bandwidth and engagement, seeking to understand the opportunities and challenges of integrating sonification and visualization. 

The research communities studying visualization and sonification for data display and analysis share exceptionally similar goals, essentially making data of any kind interpretable to humans. One community does so by using visual representations of data, and the other community employs auditory (non-speech) representations of data. While the two communities have a lot in common, they developed mostly in parallel over the course of the last few decades. With this STAR, we discuss a collection of work that bridges the borders of the two communities, hence a collection of work that aims to integrate the two techniques into one form of audiovisual display, which we argue to be “more than the sum of the two.” We introduce and motivate a classification system applicable to such audiovisual displays and categorize a corpus of 57 academic publications that appeared between 2011 and 2023 in categories such as reading level, dataset type, or evaluation system, to mention a few. The corpus also enables a meta-analysis of the field, including regularly occurring design patterns such as type of visualization and sonification techniques, or the use of visual and auditory channels, showing an overall diverse field with different designs. An analysis of a co-author network of the field shows individual teams without many interconnections. The body of work covered in this STAR also relates to three adjacent topics: audiovisual monitoring, accessibility, and audiovisual data art. These three topics are discussed individually in addition to the systematically conducted part of this research. The findings of this report may be used by researchers from both fields to understand the potentials and challenges of such integrated designs while hopefully inspiring them to collaborate with experts from the respective other field.

One of the commonly used visualization techniques for multivariate data is the parallel coordinates plot. It provides users with a visual overview of multivariate data and the possibility to interactively explore it. While pattern recognition is a strength of the human visual system, it is also a strength of the auditory system. Inspired by the integration of the visual and auditory perception in everyday life, we introduce an audio-visual analytics design named Parallel Chords combining both visual and auditory displays. Parallel Chords lets users explore multivariate data using both visualization and sonification through the interaction with the axes of a parallel coordinates plot. To illustrate the potential of the design, we present (1) prototypical data patterns where the sonification helps with the identification of correlations, clusters, and outliers, (2) a usage scenario showing the sonification of data from non-adjacent axes, and (3) a controlled experiment on the sensitivity thresholds of participants when distinguishing the strength of correlations. During this controlled experiment, 35 participants used three different display types, the visualization, the sonification, and the combination of these, to identify the strongest out of three correlations. The results show that all three display types enabled the participants to identify the strongest correlation — with visualization resulting in the best sensitivity. The sonification resulted in sensitivities that were independent from the type of displayed correlation, and the combination resulted in increased enjoyability during usage.

Along with the increase of digitalization and automation, a new kind of working environment is emerging in the field of air traffic control. Instead of situating the control tower at the airport, it is now possible to remotely control the airport at any given location, i.e. in a remote tower center (RTC). However, by controlling the airport remotely, the situational awareness and sense of presence might be compromised. By using directional sound, a higher situational awareness could potentially be achieved while also offloading the visual perception which is heavily used in air traffic control. Suitable use cases for sonification in air traffic control were found through workshops with air traffic controllers. A sonification design named SonAir was developed based on the outcome of the workshops, and was integrated with an RTC simulator for evaluating to what degree SonAir could support air traffic controllers in their work. The results suggest that certain aspects of SonAir could be useful for air traffic controllers. A continuous sonification where the spatial positioning of aircraft were conveyed was experienced to be partially useful, but the intrusiveness of SonAir should be further considered to fit the air traffic controllers’ needs. An earcon that conveyed when an aircraft enters the airspace and from which direction was considered useful to support situational awareness.

Immersive environments provide a physical space for audio-visual data analysis. An example of such an environment isthe Norrköping Decision Arena, which provides a cylindrical display together with a circular sound system. This papersets recommendations on what kinds of visualization would benefit from being displayed in this kind of environment andhow sonification could be used as a complement to enable exploratory data analysis. Three visualizations are presented aspotentially interesting for the presentation on a cylindrical display: theme rivers, radial visualizations, and parallel coordinates.

Data visualization software is commonly used to explore outer space in a planetarium environment, where the visuals of the software is typically accompanied with a narrator and supplementary background music. By letting sound take a bigger role in these kinds of presentations, a more informative and immersive experience can be achieved. The aim of the present study was to explore how sonification can be used as a complement to the visualization software OpenSpace to convey information about the Solar System, as well as increasing the perceived immersiveness for the audience in a planetarium environment. This was investigated by implementing a sonification that conveyed planetary properties, such as the size and orbital period of a planet, by mapping this data to sonification parameters. With a user-centered approach, the sonification was designed iteratively and evaluated in both an online and planetarium environment. The results of the evaluations show that the participants found the sonification informative and interesting, which suggest that sonification can be beneficially used as a complement to visualization in a planetarium environment.