liu.seSearch for publications in DiVA
Change search
ReferencesLink to record
Permanent link

Direct link
Transmodal interaction and user experience
Linköping University, Department of Computer and Information Science, Human-Centered systems. Linköping University, Faculty of Arts and Sciences. (IxS)ORCID iD: 0000-0003-2919-098X
Linköping University, Department of Computer and Information Science, Human-Centered systems. Linköping University, Faculty of Science & Engineering. SICS East Swedish ICT, Linköping, Sweden. (IxS)ORCID iD: 0000-0001-6555-0247
Linköping University, Department of Culture and Communication, Language and Culture. Linköping University, Faculty of Arts and Sciences.ORCID iD: 0000-0003-4004-5361
2016 (English)In: Proceedings of the 12'th SweCog Conference / [ed] Alexander Almér, Robert Lowe, Erik Billing, Skövde: The University of Skövde , 2016, 5-5 p.Conference paper, Abstract (Other academic)
Abstract [en]

We are in a series of studies, ranging from news production to computer gaming, looking into the intersection of transmodal interaction and user experience. The purpose of this abstract is to outline the theoretical framework for that intersection. The first area we are studying is Transmodal Interaction, which is a concept that refer to a specific aspect of multimodal interaction. Human action is multimodal (Streeck, Goodwin, & LeBaron, 2011), and different sensory modes play an important role in action. However, little attention has been given to the intricate ways in which sensory modalities (seeing – drawing, hearing – saying, moving – touching, etc.) integrate, affect, and transform each other during the course of an activity. There are transformations of meaning in every new materialisation of an idea or a thought, partly depending on the communication potential of the sensory modality. This render what we refer to as a transmodal process where ideas and thoughts materialise action by action in an emergent sequence across relatively long and discontinuous timespans (Murphy, 2012). Over a sequence of actions, the meanings expressed in one modality, dynamically blend and shape what is expressed in other modalities. This produces, according to (Murphy, 2012) “a series of semiotic modulations in which certain core qualities persist, but others are noticeably transformed in the transition from one mode to another. (p. 1969)” We can, in intersemiotic translation (Jakobson, 1959) between modalities, address what is lost, how we introduce distortions, or even introduce perceptions of things that do not exist. A question is then how continuity of meaning and experience is preserved in modality changes. The second area we are studying is User Experience. The term refers to a person's perceptions and responses resulting from the use and/or anticipated use of a product, system or service (ISO, 2010). We employ a three level model of user experience based on Leontiev’s account of consciousness (Kaptelinin & Nardi, 2012; Leont ́ev, 1978), which also relate closely to Norman’s model of emotional design (Norman, 2005). The first level is the sensory fabric of consciousness, Norman refers to this as the visceral level. It is the largely subconscious level of how things feel. The second level is the personal meaning of things, related to what and how we do things action by action. Norman (ibid.) refers to this level as the behavioural level. The third level has to do with meaning, and what Norman refers to as a reflective level. It is the level of cultural meaning and what things mean for us in our socially and historically rooted activities. The intersection of these two areas constitutes our current focus of research. We are, in domains as different as news production and computer gaming, investigating persons’ perceptions and actions resulting from interaction with each other and with materialisations across different sensory modalities that give rise to intersemiotic translation effects. 

Place, publisher, year, edition, pages
Skövde: The University of Skövde , 2016. 5-5 p.
, Skövde University Studies in Informatics, ISSN 1653-2325 ; 2016:2
National Category
Human Computer Interaction Social Sciences
URN: urn:nbn:se:liu:diva-131776ISBN: 9789198366709OAI: diva2:1033045
SweCog 2016 Conference, Gothenburg, Sweden, October 6-7, 2016
Available from: 2016-10-05 Created: 2016-10-05 Last updated: 2016-10-13Bibliographically approved

Open Access in DiVA

No full text

Other links

Link to proceedings

Search in DiVA

By author/editor
Arvola, MattiasNordvall, MathiasBroth, Mathias
By organisation
Human-Centered systemsFaculty of Arts and SciencesFaculty of Science & EngineeringLanguage and Culture
Human Computer InteractionSocial Sciences

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

ReferencesLink to record
Permanent link

Direct link