Along with the increase of digitalization and automation, a new kind of working environment is emerging in the field of air traffic control. Instead of situating the control tower at the airport, it is now possible to remotely control the airport at any given location, i.e. in a remote tower center (RTC). However, by controlling the airport remotely, the situational awareness and sense of presence might be compromised. By using directional sound, a higher situational awareness could potentially be achieved while also offloading the visual perception which is heavily used in air traffic control. Suitable use cases for sonification in air traffic control were found through workshops with air traffic controllers. A sonification design named SonAir was developed based on the outcome of the workshops, and was integrated with an RTC simulator for evaluating to what degree SonAir could support air traffic controllers in their work. The results suggest that certain aspects of SonAir could be useful for air traffic controllers. A continuous sonification where the spatial positioning of aircraft were conveyed was experienced to be partially useful, but the intrusiveness of SonAir should be further considered to fit the air traffic controllers’ needs. An earcon that conveyed when an aircraft enters the airspace and from which direction was considered useful to support situational awareness.

The present study is an interdisciplinary endeavour that transmutes science, technology, and aesthetics into an audiovisual experience. The objective is to highlight the potential of combining sonification with visualisation in order to enhance the comprehension of extensive and complex sets of data. Moreover, this paper describes contemporary tools and methods for the implementation of the practice and suggests effective ways to monitor environmental changes. It can be regarded as an exploratory study for familiarisation with the potential of sonification and visualisation in the exploration of environmental data.

To meet the increasing demands within the air traffic control (ATC) industry, automation is seen as one solution to increase efficiency, safety, and capacity. In ATC, teamwork between air traffic controllers (ATCO) is a key component to safely direct the aircraft. With increasing automation, the human-automation teamwork will become even more important to keep the human operator in the loop and contribute to the operator’s involvement in decision-making. However, operators can be cautious to new automation due to experience has taught them to be cautious and the willingness to accept automation can relate to the purpose of the automation. Therefore, this study presents how ATCOs want to work with an automated tool in the future, what qualities the automation should have, and what the human should be able to do when working with an automated system. Through an online questionnaire, 113 operational ATCOs answered questions about ATCOs’ experiences and expectations of current and future ATM systems, as well as teamwork between the ATCO and automated systems. For this paper, the teamwork aspects of the questionnaire are presented and discussed. The ATCOs believed that they should be able to trust the automation, know the reasoning behind the automation’s suggestions and decisions, and be able to always take control of the situation from the automation. The results will be used in early development processes of automation, in both the ATM domain but also in other control domains, that faces similar challenges. 

Map-based dashboards are used for data exploration every day. The present study used an insight-based methodology for evaluating a map-based dashboard that presents research findings of water management and ecosystem services in the Amazon. In addition to analyzing the insights gained from using the dashboard, the evaluation method was compared to standardized questionnaires and task-based evaluations. The result suggests that the dashboard enabled the participants to gain domain-relevant, complex insights regarding the topic presented. Furthermore, the insight-based analysis highlighted unexpected insights and hypotheses regarding causes and potential adaptation strategies for remediation. Although time- and resource-consuming, the insight-based methodology was shown to have the potential of thoroughly analyzing how end users can utilize map-based dashboards for data exploration and decision making. Finally, the insight-based methodology is argued to evaluate tools in scenarios more similar to real-life usage, compared to task-based evaluation methods.

Purpose – The noise and dust particles caused by the construction transport are by most stakeholders experienced as disturbing. The purpose of this study is to explore how sonification can support visualization in construction planning to decrease construction transport disturbances.

Design/methodology/approach – This paper presents an interdisciplinary research project, combining research on construction logistics, internet of things and sonification. First, a data recording device, including sound, particle, temperature and humidity sensors, was implemented and deployed in a development project. Second, the collected data were used in a sonification design, which was, third, evaluated with potential users.

Findings – The results showed that the low-cost sensors used could capture “good enough” data, and that the use of sonification for representing these data is interesting and a possible useful tool in urban and construction transport planning.

Research limitations/implications – There is a need to further evolve the sonification design and better communicate the aim of the sounds used to potential users. Further testing is also needed.

Practical implications – This study introduces new ideas of how to support visualization with sonification planning the construction work and its impact on the vicinity of the site. Currently, urban planning and construction planning focus on visualizing the final result, with little focus on how to handle disturbances during the construction process.

Originality/value – Showing the potentials of using low-cost sensor data in sonification, and using sonification together with visualization, is the result of a novel interdisciplinary research area combination.

In this position paper we argue for the need of using standardized questionnaires for assessing usability in audio-visual representations. By using standardized measures of usability, comparability and validity of research studies in this field will be improved. However, it is not clear which questionnaire that is most suitable for assessing usability in audio-visual representations, neither when assessing the modalities individually or the combination. We present a variety of different questionnaires available, and argue for the need of combining different measures as well as developing new.

In their daily lives, people use more than one sense to perceive and interpret their environment. Likewise, audio-visual interfaces can support human data analysts better than interfaces relying on just one sense. While the research communities of sonification and visualization have both carried out extensive research on the auditory and visual representation of data, comparatively little is known about their systematic and complementary combination for data analysis. After two workshops at Audio Mostly 2021 and IEEE VIS, this 3rd workshop on audio-visual analytics continues building a community of researchers interested in combining visualization and sonification.

In their daily lives, people use more than one sense to perceive and interpret their environment. Likewise, audio-visual interfaces can support human data analysts better than interfaces relying on just one sense. While the research communities of sonification and visualization have both carried out extensive research on the auditory and visual representation of data, comparatively little is known about their systematic and complementary combination for data analysis. After two workshops at Audio Mostly 2021 and IEEE VIS, this 3rd workshop on audio-visual analytics continues building a community of researchers interested in combining visualization and sonification.

In various control systems, automation is implemented to increase efficiency and safety. With increased automation, it becomes increasingly relevant to view the automation as a team member, rather than as a tool. In best cases, human-automation teamwork keeps workload within acceptable limits, increase situation awareness, and keeps the operator in the control loop. However, human-automation teamwork will only flourish if the automation is developed with the human operator in mind. Therefore, investigations of the current experiences and expectations regarding automation and teamwork are important for the development of automation. Through a questionnaire among Air Traffic Controllers (ATCOs), the present study aims to investigate how ATCOs perceive automation and safety in current and future air traffic control systems and the importance of different teamwork factors for human-human and human-automation collaboration. The results indicate that the ATCOs believe that safety will increase in the future along with increasing automation as long as the automation is working as expected. The ATCOs expressed a fear of deskilling and losing situation awareness with automation, a fear associated with a new ATCO role of monitoring the system and take over when the automation fails. The results suggest that design for human-automation teamwork aspects that ATCOs value, such as adaptability or mutual performance monitoring, could be a way forward.

Sonification refers to sonic expression of data or information. It is often thought of as an auditory complement, providing additional information about data which can reveal patterns and facilitate interpretation and understanding of the data. Hence, the listening space created by a sonifi cation is always a hybrid where auditory augmentation complements other information modalities and, in some cases, also spatial qualities. In this work, we focus on sonifi cation in public and semi-public spaces, and specifi cally on musical sonifi cation – the use of musical sounds to create a sonic environment, augmenting or complementing a physical shared space. We draw upon established approaches in interaction design to focus our work on the user experience of musical sonifi cation in public and semi-public spaces. Specifi cally, we fi rst identify the experiential qualities of sonic atmosphere and performativity as important aspects of sonifi cation in public and semi-public spaces, then use those experiential qualities generatively in the speculative design of a musical sonifi cation sketch. The design sketch comprises a dynamic musical sonifi cation of air quality data, intending to give citizens an awareness and an enhanced individual and interpersonal understanding of air quality in their city.

Data visualization software is commonly used to explore outer space in a planetarium environment, where the visuals of the software is typically accompanied with a narrator and supplementary background music. By letting sound take a bigger role in these kinds of presentations, a more informative and immersive experience can be achieved. The aim of the present study was to explore how sonification can be used as a complement to the visualization software OpenSpace to convey information about the Solar System, as well as increasing the perceived immersiveness for the audience in a planetarium environment. This was investigated by implementing a sonification that conveyed planetary properties, such as the size and orbital period of a planet, by mapping this data to sonification parameters. With a user-centered approach, the sonification was designed iteratively and evaluated in both an online and planetarium environment. The results of the evaluations show that the participants found the sonification informative and interesting, which suggest that sonification can be beneficially used as a complement to visualization in a planetarium environment.

In the present study a sonification of running data was evaluated. The aim of the sonification was to both convey information about the data and convey a specific emotion. The sonification was evaluated in three parts, firstly as an auditory graph, secondly together with additional text information, and thirdly together with an animated visualization, with a total of 150 responses. The results suggest that the sonification could convey an emotion similar to that intended, but at the cost of less good representation of the data. The addition of visual information supported understanding of the sonification, and the auditory representation of data. The results thus suggest that it is possible to design sonification that is perceived as both interesting and fun, and convey an emotional impression, but that there may be a trade off between musical experience and clarity in sonification.

Background

Our aim is to highlight the benefits and limitations of open and non-anonymized peer review. Our argument is based on the literature and on responses to a survey on the reviewing process of alt.chi, a more or less open review track within the so-called Computer Human Interaction (CHI) conference, the predominant conference in the field of human-computer interaction. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is now being considered and used in other fields.

Methods

We ran an online survey with 30 responses from alt.chi authors and reviewers, collecting quantitative data using multiple-choice questions and Likert scales. Qualitative data were collected using open questions.

Results

Our main quantitative result is that respondents are more positive to open and non-anonymous reviewing for alt.chi than for other parts of the CHI conference. The qualitative data specifically highlight the benefits of open and transparent academic discussions. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

Conclusion

While the benefits are quite clear and the system is generally well-liked by alt.chi participants, they remain reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation.

Parallel coordinates are commonly used for non-temporal multivariate data, but there is little support for their usability for displaying temporal multivariate data. In this paper, we introduce a study evaluating the usability of 2D and 3D parallel coordinates for pattern identification in temporal multivariate data. The results indicate that 3D parallel coordinates have higher usability, as measured with higher accuracy and faster response time as well as subjective ratings, compared to 2D.

The indoor climate is closely related to human health, well-being, and comfort. Thus, an understanding of the indoor climate is vital. One way to improve the indoor climates is to place an aesthetically pleasing active plant wall in the environment. By collecting data using sensors placed in and around the plant wall both the indoor climate and the status of the plant wall can be monitored and analyzed. This manuscript presents a user study with domain experts in this field with a focus on the representation of such data. The experts explored this data with a Line graph, a Horizon graph, and a Stacked area graph to better understand the status of the active plant wall and the indoor climate. Qualitative measures were collected with Think-aloud protocol and semi-structured interviews. The study resulted in four categories of analysis tasks: Overview, Detail, Perception, and Complexity. The Line graph was found to be preferred for use in providing an overview, and the Horizon graph for detailed analysis, revealing patterns and showing discernible trends, while the Stacked area graph was generally not preferred. Based on these findings, directions for future research are discussed and formulated. The results and future directions of this research can facilitate the analysis of multivariate temporal data, both for domain users and visualization researchers.

Visual representations of data are commonly used to communicate research results. However, such representations might introduce several possible challenges for the human visual perception system, for example in perceiving brightness levels. Sonification, adding sound to the visual representation, might be used to overcome these challenges. As sonification provides additional information, sonification could be useful in supporting interpretations of a visual perception. In the present study, usefulness in terms of accuracy of sonification was investigated with an interactive sonification test. In the experiment, participants were asked to identify the highest brightness level in a monochrome visual representation. The task was performed in four conditions, one with no sonification and three with different sonification settings. The results show that sonification is useful, as measured by higher task accuracy.

Visual representations of data introduce several possible challenges for the human visual perception system in perceiving brightness levels. Overcoming these challenges might be simplified by adding sound to the representation. This is called sonification. As sonification provides additional information to the visual information, sonification could be useful in supporting the visual perception. In the present study, usefulness (in terms of accuracy and response time) of sonification was investigated with an interactive sonification test. In the test, participants were asked to identify the highest brightness level in a monochrome visual representation. The task was performed in four conditions, one with no sonification and three with different sonification settings. The results show that sonification is useful, as measured by higher task accuracy, and that the participant's musicality facilitates the use of sonification with better performance when sonification was used. The results were also supported by subjective measurements, where participants reported an experienced benefit of sonification.

In complex visual representations, there are several possible challenges for the visual perception that might be eased by adding sound as a second modality (i.e. sonification). It was hypothesized that sonification would support visual perception when facing challenges such as simultaneous brightness contrast or the Mach band phenomena. This hypothesis was investigated with an interactive sonification test, yielding objective measures (accuracy and response time) as well as subjective measures of sonification benefit. In the test, the participant’s task was to mark the vertical pixel line having the highest intensity level. This was done in a condition without sonification and in three conditions where the intensity level was mapped to different musical elements. The results showed that there was a benefit of sonification, with higher accuracy when sonification was used compared to no sonification. This result was also supported by the subjective measurement. The results also showed longer response times when sonification was used. This suggests that the use and processing of the additional information took more time, leading to longer response times but also higher accuracy. There were no differences between the three sonification conditions.

The modern control room is predominantly made up of visual displays, which might make monitoring different processes a cumbersome and visually cognitively demanding task. Sonification could be used to support the monitoring task. However, it is not clear how the most beneficial sonification should be designed. In this pilot study an experimental setup was developed to explore perception of different sonification approaches. A user experiment was performed to assess perception of the sonification when and where simulated data deviated most from the normal level. It was found that all sonification conditions were generally useful, regardless of the participant’s musical experience, shown both in terms of objective and subjective measurements. Stereo panning of the sound was also generally experienced as helpful, but the use of different pitch might not have been experienced to contribute as much for solving the task. The findings in this pilot study will be further used to create new research ideas about sonification for monitoring of dynamic processes.

Photone is an interactive installation combining color images with musical sonification. The musical expression is generated based on the syntactic (as opposed to semantic) features of an image as it is explored by the userﾒs pointing device, intending to catalyze a holistic user experience we refer to as modal synergy where visual and auditory modalities multiply rather than add. We collected and analyzed two months' worth of data from visitorsﾒ interactions with Photone in a public exhibition at a science center. Our results show that a small proportion of visitors engaged in sustained interaction with Photone, as indicated by session times. Among the most deeply engaged visitors, a majority of the interaction was devoted to visually salient objects, i.e., semantic features of the images. However, the data also contains instances of interactive behavior that are best explained by exploration of the syntactic features of an image, and thus may suggest the emergence of modal synergy.

We present Photone, an interactive installation combining photographic images and musical sonification. An image is displayed, and a dynamic musical score is generated based on the overall color properties of the image and the color value of the pixel under the cursor. Hence, the music changes as the user moves the cursor. This simple approach turns out to have interesting experiential qualities in use. The composition of images and music invites the user to explore the combination of hues and textures, and musical sounds. We characterize the resulting experience in Photone as one of modal synergy where visual and auditory output combine holistically with the chosen interaction technique. This tentative finding is potentially relevant to further research in auditory displays and multimodal interaction.

This work presents the results from a user centered evaluation of visual representations of temporal multivariate data using 2D and3D parallel coordinates. The objective of the evaluation was to investigate whether 2D or 3D representations increase user performance when the data consists of temporal multivariate data and the visual representation contains interactive user tools. The results show that the 3D parallel coordinates representation outperforms 2D parallel coordinates with regards to both accuracy and response time.This result is of interest to the information visualization community, since it shows the usefulness of visual representations of temporal multivariate data.

This poster presents an interactive sonification experiment, designed to evaluate possible benefits of sonification in information visual- ization. The aim of the present study was to explore the use of composed and deliberately designed musical sounds to enhance perception of color intensity in visual representations. It was hypoth- esized, that by using musical sounds for sonification perception of color intensity would be improved. In this evaluation, sonification was mapped to color intensity in visual representations, and the par- ticipants had to identify and mark the highest color intensity, as well as answer a questionnaire about their experience. Both quantitative and qualitative preliminary results suggest a benefit of sonification, and indicate that sonification is useful in data exploration. 

This paper presents an experiment designed to evaluate the possible benefits of sonification in information visualization to give rise to further research challenges. It is hypothesized, that by using musical sounds for sonification when visualizing complex data, interpretation and comprehension of the visual representation could be increased by interactive sonification.

This hypothesis is evaluated by testing sonification in parallel coordinates and scatter plots. The participants had to identify and mark different density areas in the representations, where amplitude of the sonification was mapped to the density in the data sets. Both quantitative and qualitative results suggest a benefit of sonification. These results indicate that sonification might be useful for data exploration, and give rise to new research questions and challenges. 

This work presents the results from an evaluation of stereoscopic versus monoscopic 3D parallel coordinates. The objective of the evaluation was to investigate if stereopsis increases user performance. The results show that stereoscopy has no effect at all on user performance compared to monoscopy. This result is important when it comes to the potential use of stereopsis within the information visualization community.

This poster presents an experiment designed to evaluate the possible benefits of sonification in information visualization. It is hypothesized, that by using musical sounds for sonification when visualizing complex data, interpretation and comprehension of the visual representation could be increased. In this evaluation of sonification in parallel coordinates and scatter plots, participants had to identify and mark different density areas in the representations. Both quantitative and qualitative results suggest a benefit of sonification. These results indicate that sonification might be useful for data exploration, and give rise to new research questions and challenges.

We report a design-led exploration of sonification to provide peripheral awareness in air traffic control centers. Our assumption is that by using musical sounds for sonification of peripheral events, it is possible to create a dynamic soundscape that complements the visual information to support the formation and maintenance of an airspace Gestalt throughout the air traffic controller’s interaction. An interactive sonification concept was designed, focusing on one controlled sector of airspace with inbound and outbound aircraft. A formative assessment of the sonification concept suggests that our approach might facilitate the air traffic controller’s work by providing complementary auditory information about inbound and outbound aircraft, particularly in situations where the traffic volume is moderate to low. 

This paper is an introduction to the emotional qualities of sound and music, and we suggest that the visual and the aural modalities should be combined in the design of visualizations involving emotional expressions. We therefore propose that visualization design should incorporate sonic interaction design drawing on musicology, cognitive neuroscience of music, and psychology of music, and identify what we see as key research challenges for such an approach.

Hearing loss has a negative effect on the daily life of 10-15% of the world’s population. One of the most common ways to treat a hearing loss is to fit hearing aids which increases audibility by providing amplification. Hearing aids thus improve speech reception in quiet, but listening in noise is nevertheless often difficult and stressful. Individual differences in cognitive capacity have been shown to be linked to differences in speech recognition performance in noise. An individual’s cognitive capacity is limited and is gradually consumed by increasing demands when listening in noise. Thus, fewer cognitive resources are left to interpret and process the information conveyed by the speech. Listening effort can therefore be explained by the amount of cognitive resources occupied with speech recognition. A well fitted hearing aid improves speech reception and leads to less listening effort, therefore an objective measure of listening effort would be a useful tool in the hearing aid fitting process.

In this thesis the Auditory Inference Span Test (AIST) was developed to assess listening effort by measuring an individual’s cognitive spare capacity, the remaining cognitive resources available to interpret and encode linguistic content of incoming speech input while speech understanding takes place. The AIST is a dual-task hearing-innoise test, combining auditory and memory processing, and requires executive processing of speech at different memory load levels. The AIST was administered to young adults with normal hearing and older adults with hearing impairment. The aims were 1) to develop the AIST; 2) to investigate how different signal-to-noise ratios (SNRs) affect memory performance for perceived speech; 3) to explore if this performance would interact with cognitive capacity; 4) to test if different background noise types would interact differently with memory performance for young adults with normal hearing; and 5) to examine if these relationships would generalize to older adults with hearing impairment.

The AIST is a new test of cognitive spare capacity which uses existing speech material that is available in several countries, and manipulates simultaneously cognitive load and SNR. Thus, the design of AIST pinpoints potential interactions between auditory and cognitive factors. The main finding of this thesis was the interaction between noise type and SNR showing that decreased SNR reduced cognitive spare capacity more in speech-like noise compared to speech-shaped noise, even though speech intelligibility levels were similar between noise types. This finding applied to young adults with normal hearing but there was a similar effect for older adults with hearing impairment with the addition of background noise compared to no background noise. Task demands, MLLs, interacted with cognitive capacity, thus, individuals with less cognitive capacity were more sensitive to increased cognitive load. However, MLLs did not interact with noise type or with SNR, which shows that different memory load levels were not affected differently in different noise types or in different SNRs. This suggests that different cognitive mechanisms come into play for storage and processing of speech information in AIST and for listening to speech in noise. Thus, the results suggested that a test of cognitive spare capacity seems to be a useful way to assess listening effort, even though the AIST, in the design used in this thesis, might be too cognitively demanding to provide reliable results for all individuals.

Objective

The aim of this study was to investigate the effect of working memory capacity (WMC) and updating ability (UA) on listening effort measured using a new test, the Auditory Inference Span Test (AIST), as an objective measure of listening effort.

Design

The AIST is based on Swedish five-word sentences and taps into three memory load levels (MLLs). It was administered in stationary speech-shaped noise at −2, −4, and −6 dB signal-to-noise ratio (SNR). WMC was assessed using the reading span test, and UA was assessed using the letter memory test. We also collected data on speech-in-noise performance and subjectively rated listening effort at the three SNRs.

Study sample

Thirty-nine participants with normal hearing thresholds (≤20 dB HL for 250 to 4000 Hz) took part in the study.

Results

AIST performance decreased with increasing MLL and was related to WMC and UA. Participants with high WMC performed better than those with low WMC at more favorable SNRs. Participants with high UA performed better than participants with low UA at the intermediate MLL, which made particular demands on the UA. Neither speech recognition scores nor subjectively rated listening effort was associated with AIST performance or either of the cognitive variables.

Conclusion

AIST taps into cognitive functions necessary for understanding speech in noise. However, in its current form AIST may be too cognitively taxing to successfully measure graded listening effort in participants with lower cognitive capacity.

Listening in noise is often perceived to be effortful. This is partly because cognitive resources are engaged in separating the target signal from background noise, leaving fewer resources for storage and processing of the content of the message in working memory. The Auditory Inference Span Test (AIST) is designed to assess listening effort by measuring the ability to maintain and process heard information. The aim of this study was to use AIST to investigate the effect of background noise types and signal-to-noise ratio (SNR) on listening effort, as a function of working memory capacity (WMC) and updating ability (UA). The AIST was administered in three types of background noise: steady-state speech-shaped noise, amplitude modulated speech-shaped noise, and unintelligible speech. Three SNRs targeting 90% speech intelligibility or better were used in each of the three noise types, giving nine different conditions. The reading span test assessed VVMC, while UA was assessed with the letter memory test. Twenty young adults with normal hearing participated in the study. Results showed that AIST performance was not influenced by noise type at the same intelligibility level, but became worse with worse SNR when background noise was speech-like. Performance on AIST also decreased with increasing memory load level. Correlations between AIST performance and the cognitive measurements suggested that WMC is of more importance for listening when SNRs are worse, while UA is of more importance for listening in easier SNRs. The results indicated that in young adults with normal hearing, the effort involved in listening in noise at high intelligibility levels is independent of the noise type. However, when noise is speech-like and intelligibility decreases, listening effort increases, probably due to extra demands on cognitive resources added by the informational masking created by the speech fragments and vocal sounds in the background noise.

The voice is the most common means of communication and tends to change, sliding up and down the pitch scale when forming phonemes and words, as different emotions and thoughts are expressed. Therefore the ability to discriminate frequencies is of importance for speech intelligibility in a communication situation. Furthermore, this ability is also of importance in speech recognition in noise, by separating the target and noise by spectral and temporal differences of the sources. The choice of rehabilitation is crucial for the frequency discrimination ability. Cochlear implants, for example, lack the ability to pass the temporal fine structure of acoustic waves to the auditory nerve, which in turn lead to reduced precision of phase locking, inferior frequency discrimination ability, and a relatively poor ability to understand speech when background sounds are present.The aim of the study is to investigate how frequency discrimination and temporal resolution abilities interact with performance in speech recognition in noise using a psychoacoustic, speech, and cognitive test battery. These tests will give insight to interactions between performance and hearing status, type of rehabilitation(hearing aid, cochlear implant, and electro-acoustic stimulation), cognitive capacity, and language ability. It is hypothesized that normal hearing participants have a better frequency discrimination ability than hearing impaired participants and by that, better understanding of speech. It is also hypothesized that type of rehabilitation effects performance on frequency discrimination, and that this performance correlates with speech recognition in noise. Finally, it is hypothesized that cognitive capacity and language ability can, to some extent, compensate for loss of frequency resolution in the peripheral auditory system. Preliminary results from the study will be presented and discussed.

Hörapparatsanpassning kan ses som en process för att minska en persons lyssningsansträngning, men det är oklart hur man bäst mäter lyssningsansträngning på ett objektivt sätt. Auditory Inference Span Test (AIST) syftar därför till att utvecklas till ett kliniskt instrument att använda vid hörapparatsanpassning för att mäta en patients ansträngning att uppfatta tal.

AIST är ett kombinerat hörsel-, minnes- och bearbetningstest. Testet bygger på idén att ju mer kognitiva resurser som går åt för att bearbeta och förstå tal, desto mindre kognitiva resurser finns kvar för att minnas och lagra talinformation. Testet använder Hagermans meningar i brus, och försökspersonen behöver minnas och bearbeta informationen i talmaterialet för att kunna besvara frågor om innehållet. Poäng på frågorna samt reaktionstid mäts som mått på lyssningsansträngning. Data från pilottester visar på att AIST kan bli ett väl anpassat test för kliniskt bruk för att mäta lyssningsansträngning.

One aim of hearing aid fitting is to ease the patient’s effort in understanding speech, i.e. the listening effort needed to perceive speech in different sound environments. To obtain a good hearing aid fitting, knowledge about the patient’s auditory as well as cognitive abilities seems to be important. However, listening effort is usually not included as a fitting criterion, partly as it is not clear how to measure listening effort objectively.

The Auditory Inference Span Test (AIST) is a dual-task hearing-in-noise test, that combines auditory and memory processing. The basis for the test is that when more cognitive resources are required for understanding speech, less cognitive resources are available for storage and processing of the speech information. In AIST, Hagerman sentences are presented in noise and the subject is required to recall and process the sentence information. Recall ability is tested with different cognitive loads. Button-press responses are recorded and used as an estimate of listening effort. In a pilot study, listeners showed decreasing accuracy with increasing cognitive load and longer reaction time at maximum cognitive load, suggesting that the test may be suited as a clinical test for listening effort.

In an ongoing study, the AIST is being evaluated in relation to other auditory and cognitive measures: baseline audiometry (audiogram) and speech in noise test (Hagerman sentences) as well as text based dual processing and storage test (reading span) and updating (letter memory test), as well as subjective rating of listening effort. Data from this study will be presented.

There has been a recent interest in listening effort as a factor to be taken into account in the audiological clinic. However, the term “listening effort” is poorly determined and needs to be defined before it can be used as a clinical or research tool. One way of understanding listening effort is in terms of the cognitive resources expended during listening. Cognitive capacity is finite and thus if cognitive capacity is used up during the act of listening to speech there will be fewer cognitive resources left to process the content of the message conveyed. We have introduced the term Cognitive Spare Capacity (CSC) to refer to residual cognitive capacity once successful listening has taken place. This extended abstract described the work we have carried out to date on measures of CSC for research and clinical use. In the course of this work we have developed tests to assess the role of memory load, executive function and audiovisual integration in CSC under challenging conditions. When these tests are fully developed, our aim is that they should allow objective individual assessment of listening effort in cognitive terms. Results to date indicate that under challenging conditions, CSC is an arena for executive processing of temporarily stored information; it is related to individual working memory capacity and can be enhanced by hearing aid signal processing.

There has been a recent interest in listening effort as a factor to be taken into account in the audiological clinic. However, the term “listening effort” is poorly determined and needs to be defined before it can be used as a clinical or research tool. One way of understanding listening effort is in terms of the cognitive resources expended during listening. Cognitive capacity is finite and thus if cognitive capacity is used up during the act of listening to speech there will be fewer cognitive resources left to process the content of the message conveyed. We have introduced the term Cognitive Spare Capacity (CSC) to refer to residual cognitive capacity once successful listening has taken place. This extended abstract described the work we have carried out to date on measures of CSC for research and clinical use. In the course of this work we have developed tests to assess the role of memory load, executive function and audiovisual integration in CSC under challenging conditions. When these tests are fully developed, our aim is that they should allow objective individual assessment of listening effort in cognitive terms. Results to date indicate that under challenging conditions, CSC is an arena for executive processing of temporarily stored information; it is related to individual working memory capacity and can be enhanced by hearing aid signal processing.

One aim of hearing aid fitting is to reduce the effort of understanding speech, especially in noisy environments. For a good hearing aid fitting, knowledge about the patient’s auditory abilities is necessary, but knowledge about cognitive abilities may also be important.

The Auditory Inference Span Test (AIST) is a dual-task hearing-in-noise test, that combines auditory and memory processing. In AIST, Hagerman sentences are presented in steady state speech-shaped noise at -2dB, -4dB or -6dB SNR and the subject is required to recall and process the information from the sentences by giving button-press responses to multiple-choice questions thereby assessing what the subject could infer from what was heard.

AIST will be administered to 40 normal hearing subjects (29 to date) and performance related to speech reception threshold, working memory capacity and updating ability, as well as subjective rating of listening effort. Preliminary results show a greater SNR-related improvement in AIST scores at low SNRs than can be explained by improved audibility alone, consistent with release of memory resources due to reduced listening effort. There is also a trend towards a positive relationship between AIST scores and individual working memory capacity and updating ability.

Most hearing aid fittings today are almost solely based on the patient’s audiogram. Although the loss of gain in the cochlea is important, for a more optimal fitting, more individual parameters of the patient’s cochlear loss together with the patient's cognitive abilities to process the auditory signal are required (Stenfelt & Rönnberg, 2009; Edwards, 2007). Moreover, the evaluation of the fitting is often based on a speech in noise task and the aim is to improve the individual patient’s signal to noise ratio (SNR) thresholds. As a consequence, hearing aid fitting may be seen as a process aimed to improve the patient’s SNR threshold rather than to improve communication ability. However, subsequent to a hearing aid fitting, there can be great differences in SNR improvement between patients that have identical hearing impairment in terms of threshold data (the audiogram). The reasons are certainly complex but one contributing factor may be the individual differences in cognitive capacity and associated listening effort. Another way to think about amplified hearing is to ease a subject’s listening effort (Sarampalis, et al., 2009). When the speech signal is degraded by noise or by a hearing impairment, more high-order cognitive or top-down processes are required to perceive and understand the signal, and listening is therefore more effortful. It is assumed that a hearing aid would ease the listening effort for a hearing impaired person. However, it is not clear how to measure the listening effort. We here present a test that will tap into the different cognitive aspects of listening effort, the Auditory Inference Span Test (AIST). The AIST is a dual task hearing in noise test, that combines auditory and memory processing and is well suited as a clinical test for listening effort.

Modern hearing aids use a multitude of parameters to give the user an optimal speech signal. Fitting of the hearing aid becomes a handiwork due to the limited data of the patients hearing status (primarily an audiogram). A hearing in noise test (SNR threshold) is often used to evaluate the fitting. However, testing the SNR threshold as done in clinical use today is not ecological valid. Another way to think about hearing aid fitting is to ease the listening effort.

Therefore, we propose the Auditory Inference Span Test (AIST) as a clinical tool during hearing aid fitting to assess the patient’s effort to understand speech. AIST is a combined auditory, memory, and processing test. It relies on the idea that the more cognitive resources that are required to process and understand speech, less cognitive resources are available for storage of the speech information. In AIST, sentences are presented in noise and afterwards the patient is required to recall and process the information from the sentences. Correctness and answering speed is measured and scores correlate to the effort required to understand the speech.

Data from piloting tests indicate that the AIST is well suited as a clinical test for listening effort.

Most hearing aid fittings today are almost solely based on the patient’s audiogram. However, more individual parameters of the patient’s hearing thresholds together with the patient’s cognitive abilities to process the auditory signal are required. Hearing aid fitting may be seen as a process aimed to improve the patient’s hearing thresholds rather than to improve communication ability. Another way to think about hearing aid fitting is to ease the patient’s listening effort. However, it is not clear how to measure the listening effort.

Therefore, we propose the Auditory Inference Span Test (AIST) as a clinical tool during hearing aid fitting to assess the patient’s effort to understand speech. AIST is a combined auditory, memory, and processing test. It relies on the idea that the more cognitive resources that are required to process and understand speech, less cognitive resources are available for storage of the speech information. In AIST, sentences are presented in noise and afterwards the patient is required to recall and process the information from the sentences. Correctness and reaction time is recorded as measurements of perceived listening effort.

Data from piloting tests indicate that the AIST is well suited as a clinical test for listening effort. In a future study to verify that the AIST is sensitive to cognitive capacity, the test will be evaluated with measurements of the subject's cognitive capacity as well as the subject's hearing thresholds. For a clinical test the requirement is that it is fast and easily facilitated. The AIST takes no more than fifteen minutes to complete, and the aim is to further shorten the time and adapt the test for clinical use. This ensures the AIST to be a useable instrument for testing listening effort using the individuals' cognitive spare capacity.

Purpose: Previous work has shown an effect of noise type on memory for intelligible speech. The aim ofthis study was to investigate the effect of background noise on memory performance of intelligible speech for older adults with hearing impairment using the Auditory Inference Span Test (AIST).

Method: Twenty participants with ages between 67 and 80 years with symmetrical hearing loss (29 to 47dB HL) performed the AIST, which requires processing of five-word sentences at three memoryload levels (MLLs) in three listening conditions: Quiet, steady-state noise (SSN), and backgroundvoices (ISTS). Individualized SNRs targeted 90% speech intelligibility. AIST performance reflects the amount of cognitive capacity occupied in listening, and consequently indicates the amount of listening effort. Working memory capacity (WMC) was assessed using the reading span test, and updating ability (UA) was assessed using the letter memory test.

Results: AIST performance decreased in background noise and with increasing MLL. It was related to UA and age but not to WMC. Response times on questions designed to probe sentence recognition increased with the addition of background noise.

Conclusions: The results demonstrate that the addition of background noise requires more cognitive resourcesto maintain speech recognition performance, leading to higher demands on the cognitive capacity,higher listening effort as measured by poorer memory performance, and longer AIST responsetimes. However, the type of background noise, SSN or ISTS, affected memory performance similarly.