Few studies have examined which acoustic features of speech can be used to distinguish between different emotions, and how combinations of acoustic parameters contribute to identification of emotions. The aim of the present study was to investigate which acoustic parameters in Swedish speech are most important for differentiation between, and identification of, the emotions anger, fear, happiness, sadness, and surprise in Swedish sentences. One-way ANOVAs were used to compare acoustic parameters between the emotions and both simple and multiple logistic regression models were used to examine the contribution of different acoustic parameters to differentiation between emotions. Results showed differences between emotions for several acoustic parameters in Swedish speech: surprise was the most distinct emotion, with significant differences compared to the other emotions across a range of acoustic parameters, while anger and happiness did not differ from each other on any parameter. The logistic regression models showed that fear was the best-predicted emotion while happiness was most difficult to predict. Frequency- and spectral-balance-related parameters were best at predicting fear. Amplitude- and temporal-related parameters were most important for surprise, while a combination of frequency-, amplitude- and spectral balance-related parameters are important for sadness. Assuming that there are similarities between acoustic models and how listeners infer emotions in speech, results suggest that individuals with hearing loss, who lack abilities of frequency detection, may compared to normal hearing individuals have difficulties in identifying fear in Swedish speech. Since happiness and fear relied primarily on amplitude- and spectral-balance-related parameters, detection of them are probably facilitated more by hearing aid use.

Previous research has shown deficits in vocal emotion recognition in sub-populations of individuals with hearing loss. As emotion recognition is an essential ability that affects social interaction, and in extension, can impact well-being, understanding vocal emotion recognition difficulties is a high priority research topic. Furthermore, it has been shown that although hearing aids improves word recognition, it does not improve emotion recognition. To explore emotion recognition and the effect of amplification in individuals with hearing loss, we examined recognition of vocal emotions expressed both verbally and non-verbally in middle-aged to older individuals with and without linear amplification (similar amplification across sound levels). Twenty-one experienced hearing aid users with bilateral mild-to-moderate hearing loss, and 20 individuals with normal hearing performed a vocal emotion recognition task on sentences and non-verbal vocalizations. The hearing loss group had poorer emotion recognition for both sentences (F(1,38)=15.24, p <0.001, η2p=0.29, without linear amplification, and F(1, 38)= 5.62, p=0.023, η2p=0.13, with linear amplification) and non-verbal vocalizations(F(1,38)= 25.18, p <0.001, η2p=0.40, without linear amplification, and F(1, 38)= 10.30, p=0.003, η2p=0.21, with linear amplification). However, linear amplification significantly improved the recognition of happiness (p < 0.001), which is distinguished by frequency parameters, for sentences. For non-verbal vocalizations, recognition of fear (p = 0.004) and anger (p = 0.004), were improved by linear amplification. Patterns of confusion were similar for the two groups, which may suggest that both groups perceived the emotions similarly, but that the degree of perceptual precision was lower in the hearing loss group. In sum, hearing loss negatively impacts vocal emotion recognition, but linear amplification can enhance recognition for some emotions.

Individuals with hearing loss often face challenges in identifying emotions conveyed through voice, even when using hearing aids or cochlear implants. These difficulties can impact emotional communication and are linked to reduced accuracy in identifying vocal emotion expressions. To better support individuals with hearing loss, it is important to understand how emotion recognition in voices can be improved.

This doctoral thesis comprises four studies organized around two main themes. The first theme investigates which acoustic features distinguish different emotional expressions in speech and how these features contribute to listeners’ ability to identify emotions. The second theme examines how mild-to-moderate sensorineural hearing loss affects vocal emotion identification, and whether signal amplification can enhance identification accuracy.

The findings show that fear is the most acoustically distinct emotion in spoken sentences, while happiness is the least distinct. Acoustic features related to frequency and spectral balance were most effective in predicting fear, whereas happiness was best predicted by a broader set of acoustic parameters. Surprise was also acoustically well-defined, particularly through amplitude and temporal features, but its identification was not strongly predicted by these parameters.

Importantly, restoring audibility through linear amplification significantly improved the identification of happiness in speech, and anger, fear, and interest in nonverbal vocalizations. These emotions were characterized by distinct frequency-related features. Additionally, surprise was the most accurately identified emotion in speech by both individuals with hearing loss and those with normal hearing. However, patterns of confusion between emotions could not be explained by acoustic similarities alone.

The results also suggest that temporal fine structure may play a key role in emotion identification when other cues are limited. Based on these findings, this doctoral thesis offers insights into the acoustic basis of vocal emotion expressions and proposes strategies to support emotion recognition in individuals with hearing loss.

Personer med hörselnedsättning har ofta svårt att uppfatta känslor som uttrycks genom rösten, även när de använder hörapparater eller cochleaimplantat. Dessa svårigheter kan påverka den emotionella kommunikationen och är kopplade till en nedsatt förmåga att korrekt identifiera känslouttryck i tal. För att bättre kunna stödja personer med hörselnedsättning är det viktigt att förstå hur igenkänning av känslor i rösten kan förbättras.

Denna avhandling består av fyra delstudier som är organiserade kring två huvudteman. Det första temat undersöker vilka akustiska egenskaper som särskiljer olika känslouttryck i tal, och hur dessa egenskaper påverkar lyssnarens förmåga att identifiera känslor. Det andra temat fokuserar på hur mild till måttlig sensorineural hörselnedsättning påverkar igenkänning av känslor i rösten, samt om ljudförstärkning kan förbättra denna förmåga.

Resultaten visar att rädsla är den känsla som mest akustiskt skiljer sig från andra i talade meningar, medan glädje är den minst särskiljbara. Frekvensrelaterade och spektrala egenskaper var mest effektiva för att förutsäga rädsla, medan glädje bäst förutsades av en bredare uppsättning akustiska parametrar. Förvåning var också tydligt akustiskt definierad, särskilt genom amplitud- och tidsrelaterade egenskaper, men dess igenkänning förutsågs inte starkt av dessa parametrar.

Att återställa hörbarhet genom linjär förstärkning förbättrade avsevärt igenkänningen av glädje i tal, samt ilska, rädsla och intresse i icke-verbala vokaliseringar. Dessa känslor kännetecknades av distinkta frekvensrelaterade egenskaper. Dessutom var förvåning den känsla som identifierades mest korrekt i tal av både personer med hörselnedsättning och personer med normal hörsel. Däremot kunde förväxlingsmönster mellan känslor inte förklaras av akustiska likheter.

Resultaten tyder också på att temporal finstruktur kan spela en viktig roll för igenkänning av känslor i rösten när andra ledtrådar saknas. Utifrån dessa fynd ger avhandlingen insikter i den akustiska grunden för känslouttryck i tal och föreslår strategier för att underlätta emotionell igenkänning hos personer med hörselnedsättning.

Facial mimicry, the tendency to imitate facial expressions of other individuals, has been shown to play a critical role in the processing of emotion expressions. At the same time, there is evidence suggesting that its role might change when the cognitive demands of the situation increase. In such situations, understanding another person is dependent on working memory. However, whether facial mimicry influences working memory representations for facial emotion expressions is not fully understood. In the present study, we experimentally interfered with facial mimicry by using established behavioral procedures, and investigated how this interference influenced working memory recall for facial emotion expressions. Healthy, young adults (N = 36) performed an emotion expression n-back paradigm with two levels of working memory load, low (1-back) and high (2-back), and three levels of mimicry interference: high, low, and no interference. Results showed that, after controlling for block order and individual differences in the perceived valence and arousal of the stimuli, the high level of mimicry interference impaired accuracy when working memory load was low (1-back) but, unexpectedly, not when load was high (2-back). Working memory load had a detrimental effect on performance in all three mimicry conditions. We conclude that facial mimicry might support working memory for emotion expressions when task load is low, but that the supporting effect possibly is reduced when the task becomes more cognitively challenging.

Previous research has shown deficits in vocal emotion recognition in sub-populations of individuals with hearing loss, making this a high priority research topic. However, previousresearch has only examined vocal emotion recognition using verbal material, in which emotions are expressed through emotional prosody. There is evidence that older individualswith hearing loss suffer from deficits in general prosody recognition, not specific to emotionalprosody. No study has examined the recognition of non-verbal vocalization, which constitutes another important source for the vocal communication of emotions. It might be thecase that individuals with hearing loss have specific difficulties in recognizing emotionsexpressed through prosody in speech, but not non-verbal vocalizations. We aim to examinewhether vocal emotion recognition difficulties in middle- aged-to older individuals with sensorineural mild-moderate hearing loss are better explained by deficits in vocal emotion recognition specifically, or deficits in prosody recognition generally by including both sentencesand non-verbal expressions. Furthermore a, some of the studies which have concluded thatindividuals with mild-moderate hearing loss have deficits in vocal emotion recognition abilityhave also found that the use of hearing aids does not improve recognition accuracy in thisgroup. We aim to examine the effects of linear amplification and audibility on the recognitionof different emotions expressed both verbally and non-verbally. Besides examining accuracy for different emotions we will also look at patterns of confusion (which specific emotionsare mistaken for other specific emotion and at which rates) during both amplified and nonamplified listening, and we will analyze all material acoustically and relate the acoustic content to performance. Together these analyses will provide clues to effects of amplification onthe perception of different emotions. For these purposes, a total of 70 middle-aged-olderindividuals, half with mild-moderate hearing loss and half with normal hearing will perform acomputerized forced-choice vocal emotion recognition task with and without amplification