liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Modalities of Mind: Modality-specific and nonmodality-specific aspects of working memory for sign and speech
Linköping University, Department of Behavioural Sciences and Learning, Cognition, Development and Disability. Linköping University, Faculty of Arts and Sciences.
2005 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Language processing is underpinned by working memory and while working memory for signed languages has been shown to display some of the characteristics of working memory for speech-based languages, there are a range of anomalous effects related to the inherently visuospatial modality of signed languages. On the basis of these effects, four research questions were addressed in a series of studies:

1. Are differences in working memory storage for sign and speech reflected in neural representation?

2. Do the neural networks supporting speech-sign switching during a working memory task reflect executive or semantic processes?

3. Is working memory for sign language enhanced by a spatial style of information presentation?

4. Do the neural networks supporting word reversal indicate tongue-twisting or mind-twisting?

The results of the studies showed that:

1. Working memory for sign and speech is supported by a combination of modality-specific and nonmodality-specific neural networks.

2. Switching between sign and speech during a working memory task is supported by semantic rather than executive processes.

3. Working memory performance in educationally promoted native deaf signers is enhanced by a spatial style of presentation.

4. Word reversal is a matter of mind-twisting, rather than tongue-twisting.

These findings indicate that working memory for sign and speech has modality-specific components as well as nonmodality-specific components. Modality-specific aspects can be explained in terms of Wilson’s (2001) sensorimotor account, which is based on the component model (Baddeley, 2000), given that the functionality of the visuospatial sketchpad is extended to include language processing. Nonmodality-specific working memory processing is predicted by Rönnberg’s (2003) model of cognitive involvement in language processing. However, the modality-free, cross-modal and extra-modal aspects of working memory processing revealed in the present work can be explained in terms of the central executive and the episodic buffer, providing the functionality and neural representation of the episodic buffer are extended.

A functional ontology is presented which ties cognitive processes to their neural representation, along with a model explaining modality-specific findings relating to sign language cognition. Predictions of the ontology and the model are discussed in relation to future work.

Place, publisher, year, edition, pages
Linköping: Linköping University Electronic Press, 2005. , 118 + papers I-V p.
Series
Linköping Studies in Arts and Science, ISSN 0282-9800 ; 337Studies from the Swedish Institute for Disability Research, ISSN 1650-1128 ; 18
Keyword [en]
Cognition, sign language, working memory, fMRI, PET, disability research
Keyword [sv]
Korttidsminne, teckenspråk
National Category
Human Computer Interaction
Identifiers
URN: urn:nbn:se:liu:diva-3942ISBN: 91-85457-10-8 (print)OAI: oai:DiVA.org:liu-3942DiVA: diva2:20478
Public defence
2005-10-21, Key 1, Key-huset, Campus Valla, Linköping, 13:00 (English)
Opponent
Supervisors
Available from: 2005-09-21 Created: 2005-09-21 Last updated: 2017-11-06Bibliographically approved
List of papers
1. Neural correlates of working memory for sign language
Open this publication in new window or tab >>Neural correlates of working memory for sign language
2004 (English)In: Cognitive Brain Research, ISSN 0926-6410, Vol. 20, no 2, 165-182 p.Article in journal (Refereed) Published
Abstract [en]

Eight, early bilingual, sign language interpreters participated in a PET study, which compared working memory for Swedish Sign Language (SSL) with working memory for audiovisual Swedish speech. The interaction between language modality and memory task was manipulated in a within-subjects design. Overall, the results show a previously undocumented, language modality-specific working memory neural architecture for SSL, which relies on a network of bilateral temporal, bilateral parietal and left premotor activation. In addition, differential activation in the right cerebellum was found for the two language modalities. Similarities across language modality are found in Broca's area for all tasks and in the anterior left inferior frontal lobe for semantic retrieval. The bilateral parietal activation pattern for sign language bears similarity to neural activity during, e.g., nonverbal visuospatial tasks, and it is argued that this may reflect generation of a virtual spatial array. Aspects of the data suggesting an age of acquisition effect are also considered. Furthermore, it is discussed why the pattern of parietal activation cannot be explained by factors relating to perception, production or recoding of signs, or to task difficulty. The results are generally compatible with Wilson's [Psychon. Bull. Rev. 8 (2001) 44] account of working memory.

Keyword
Working memory, Sign language, Speech, Language modality, PET
National Category
Social Sciences
Identifiers
urn:nbn:se:liu:diva-13354 (URN)10.1016/j.cogbrainres.2004.03.002 (DOI)
Available from: 2005-09-21 Created: 2005-09-21 Last updated: 2017-11-06
2. Neural representation of binding lexical signs and words in the episodic buffer of working memory
Open this publication in new window or tab >>Neural representation of binding lexical signs and words in the episodic buffer of working memory
Show others...
2007 (English)In: Neuropsychologia, ISSN 0028-3932, E-ISSN 1873-3514, Vol. 45, no 10, 2258-2276 p.Article in journal (Refereed) Published
Abstract [en]

The episodic buffer accommodates formation and maintenance of unitary multidimensional representations based on information in different codes from different sources. Formation, based on submorphemic units, engages posterior brain regions, while maintenance engages frontal regions. Using a hybrid fMRI design, that allows separate analysis of transient and sustained components, an n-back task and an experimental group of 13 hearing native signers, with experience of Swedish Sign Language and Swedish since birth, we investigated binding of lexical signs and words in working memory. Results show that the transient component of these functions is supported by a buffer-specific network of posterior regions including the right middle temporal lobe, possibly relating to binding of phonological loop representations with semantic representations in long-term memory, as well as a loop-specific network, in line with predictions of a functional relationship between loop and buffer. The left hippocampus was engaged in transient and sustained components of buffer processing, possibly reflecting the meaningful nature of the stimuli. Only a minor role was found for executive functions in line with other recent work. A novel representation of the sustained component of working memory for audiovisual language in the right inferior temporal lobe may be related to perception of speech-related facial gestures. Previous findings of sign and speech loop representation in working memory were replicated and extended. Together, these findings support the notion of a module that mediates between codes and sources, such as the episodic buffer, and further our understanding of its nature.

Keyword
Binding, Episodic buffer, Working memory, Sign language, fMRI
National Category
Human Computer Interaction
Identifiers
urn:nbn:se:liu:diva-13355 (URN)10.1016/j.neuropsychologia.2007.02.017 (DOI)
Note

On the day of the defence date the title of this article was Speach-sign switching in working memory i supported by semantic networks.

Available from: 2005-09-21 Created: 2005-09-21 Last updated: 2017-11-06
3. Explicit processing demands reveal language modality specific organization of working memory
Open this publication in new window or tab >>Explicit processing demands reveal language modality specific organization of working memory
2008 (English)In: Journal of Deaf Studies and Deaf Education, ISSN 1081-4159, E-ISSN 1465-7325, Vol. 13, no 4, 466-484 p.Article in journal (Refereed) Published
Abstract [en]

The working memory model for Ease of Language Understanding(ELU) predicts that processing differences between languagemodalities emerge when cognitive demands are explicit. Thisprediction was tested in three working memory experiments withparticipants who were Deaf Signers (DS), Hearing Signers (HS),or Hearing Nonsigners (HN). Easily nameable pictures were usedas stimuli to avoid confounds relating to sensory modality.Performance was largely similar for DS, HS, and HN, suggestingthat previously identified intermodal differences may be dueto differences in retention of sensory information. When explicitprocessing demands were high, differences emerged between DSand HN, suggesting that although working memory storage in bothgroups is sensitive to temporal organization, retrieval is notsensitive to temporal organization in DS. A general effect ofsemantic similarity was also found. These findings are discussedin relation to the ELU model.

National Category
Social Sciences
Identifiers
urn:nbn:se:liu:diva-13356 (URN)10.1093/deafed/enn005 (DOI)
Note

On the day of the defence date the title of this article was Space for compensation: Further support for a visuospatial array for temporary storage in working memory for deaf native signers.

Available from: 2005-09-21 Created: 2005-09-21 Last updated: 2017-11-06
4. Perceptual saliency in the visual channel enhances explicit language processing
Open this publication in new window or tab >>Perceptual saliency in the visual channel enhances explicit language processing
2004 (English)In: Iranian Audiology, ISSN 1735-045X, Vol. 3, no 1, 16-26 p.Article in journal (Refereed) Published
National Category
Social Sciences
Identifiers
urn:nbn:se:liu:diva-13357 (URN)
Available from: 2005-09-21 Created: 2005-09-21 Last updated: 2017-11-06
5. Reversing spoken items: mind twisting not tongue twisting
Open this publication in new window or tab >>Reversing spoken items: mind twisting not tongue twisting
2005 (English)In: Brain and Language, ISSN 0093-934X, Vol. 92, no 1, 78-90 p.Article in journal (Refereed) Published
Abstract [en]

Using 12 participants we conducted an fMRI study involving two tasks, word reversal and rhyme judgment, based on pairs of natural speech stimuli, to study the neural correlates of manipulating auditory imagery under taxing conditions. Both tasks engaged the left anterior superior temporal gyrus, reflecting previously established perceptual mechanisms. Engagement of the left inferior frontal gyrus in both tasks relative to baseline could only be revealed by applying small volume corrections to the region of interest, suggesting that phonological segmentation played only a minor role and providing further support for factorial dissociation of rhyming and segmentation in phonological awareness. Most importantly, subtraction of rhyme judgment from word reversal revealed activation of the parietal lobes bilaterally and the right inferior frontal cortex, suggesting that the dynamic manipulation of auditory imagery involved in mental reversal of words seems to engage mechanisms similar to those involved in visuospatial working memory and mental rotation. This suggests that reversing spoken items is a matter of mind twisting rather than tongue twisting and provides support for a link between language processing and manipulation of mental imagery.

Keyword
Speech; Auditory imagery; Word reversal; Parietal lobes; Spatial processing; Rhyme judgment; fMRI
National Category
Social Sciences
Identifiers
urn:nbn:se:liu:diva-13358 (URN)10.1016/j.bandl.2004.05.010 (DOI)
Available from: 2005-09-21 Created: 2005-09-21 Last updated: 2017-11-06

Open Access in DiVA

fulltext(1163 kB)1174 downloads
File information
File name FULLTEXT01.pdfFile size 1163 kBChecksum SHA-1
7bb9cc3ea03bbe325f9f5f89e954aaf4c9213eb4943413d7f826a34a412fb84d7ed55830
Type fulltextMimetype application/pdf
cover(781 kB)94 downloads
File information
File name COVER01.pdfFile size 781 kBChecksum SHA-512
6930fe3d5ff4fd1fa4c78dfc52637d94d270cf15864f4a182a9832f3183c04fb081a2f655d3a44d41bbd75de369a7c4b1864eceb89fab034e98a37834c6e07be
Type coverMimetype application/pdf

Authority records BETA

Rudner, Mary

Search in DiVA

By author/editor
Rudner, Mary
By organisation
Cognition, Development and DisabilityFaculty of Arts and Sciences
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar
Total: 1174 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 4721 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf