Multimodal, Multicodal, and Multisensorial Semiosis [Section of: Semio2014 – 12th World Congress of the IASS/AIS]

This section will focus on interactions between the modalities of semiosis, which have recently become a topic of interest in sociosemiotics (Kress 2010), film studies (Bate-man/Schmidt 2011), and the general analysis of multimodal documents (Bateman 2008, Jewitt 2009).

The term “mode” or “modality”, as well as the derived term “multimodal” describing the integration of different modalities, can be used in two senses (cf. Fricke 2012: 47ff): In some disciplines (e.g. psychology), modality is primarily used in the sense of perceptual modality, and multimodality describes sign processes where various sense modalities such as visual, auditory, and haptic perception are involved. Furthermore, “modality” is also used in the wider sense of semiotic modality. Multimodality then describes sign processes where various codes are involved (e.g. music and speech in a radio feature which are both auditorily perceived; or images and written text in a comic, or pictures and text in a book, which are both visually perceived).

In multimodal contexts, semiotic interactions and influences occur which cannot simply be explained as combination of several unimodal contexts, and have to be understood in their semiotic properties (Hess-Lüttich/Wenz 2006). If we refer to the terms “medium” or “multimedial semiosis”, we need to distinguish carefully between a number of different uses of the term “medium” (Posner 2004: 60-4), which includes sense modalities and codes, but also technical and social media.

Interactions between modalities (in both senses) will be investigated in the proposed section. Semiotics and aesthetics have usually investigated semiotic codes as well as sense modalities separately, which is no longer sufficient because of the rapidly rising frequency of multimodal artworks and artefacts. Specifically, different types of interaction of codes and/or sense modalities are of interest, e.g. code use that is in opposition or in congruence, that supports or weakens the message conveyed in the other code, that is syntactically integrated across different codes, as well as gestalt effects (complex semiotic processes which cannot be analyzed as a combination of the contributing codes and/or sense mo-dalities).

In this section, we welcome contributions which investigate semiotic processes combining more than one code, sense modality, or technical medium, and which reflect the specific conditions and semiotic properties of such sign processes. Topics of interest include, but are not limited to, the combination and interaction of two or more codes or sense modalities, e.g.
– language, gesture, facial expression, and/or body posture in daily contexts, film, or video;
– pictures, language (spoken or written, e.g. as subtitles), and music in film or video;
– written language, pictures, and other codes (e.g. hyperlinks or music) in print or online media;
– kinesic, musical, and clothing codes in theater, opera, and ballet;
– aesthetic codes and other codes in aesthetic perception and communication;
– written language and graphemic codes in written texts; etc.

We welcome contributions from different theoretical and methodical backgrounds, but expect the exposition and reflection of the theories used and the methods applied, which might include theoretical investigations building on the semiotic classics or more recent professional literature, specific case studies, corpus studies, multimodal discourse analysis, as well as experimental approaches.

Bibliography:
– Bateman, J. (2008), Multimodality and Genre. Palgrave.
– Bateman, J. & K.-H. Schmidt (2011), Multimodal Film Analysis. Routledge.
– Fricke, E. (2012), Grammatik multimodal. de Gruyter.
– Hess-Lüttich, E.W.B. & K. Wenz (ed.) (2006), Stile des Intermedialen. Kodikas/Code. Ars Semeiotica 29, 1-3.
– Jewitt, C. (ed.) (2009), The Routledge Handbook of Multimodal Analysis. London: Routledge.
– Kress, G. (2010), Multimodality. Routledge.
– Posner, R. (2004), „Basic Tasks of Cultural Semiotics“. In: G. Withalm u.a. (eds.), Signs of Power – Power of Signs. Wien: INST: 56-89.