Publication | Closed Access
Multimodal digital semiotics: the interaction of language with other resources
27
Citations
0
References
2013
Year
Data VisualizationCommunicationJournalismSocial SciencesApplied LinguisticsInteractive VisualizationLanguage DocumentationMultimodal InteractionDiscourse AnalysisLanguage StudiesInterval AlgebraInteractional LinguisticsMultimodal Human Computer InterfaceMultimodal WritingFox NewsMultimodal CognitionSemioticsMultimodal Digital SemioticsDigital MediaMultimodal TranslationSocial Medium VisualizationNatural SciencesScientific VisualizationVisual CommunicationHuman-computer InteractionMultimodal PragmaticArtsLinguistics
Abstract A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allen's interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Halliday's systemic theory.