Multimodal interaction provides the user with multiple modes of interacting with a system. A multimodal interface provides several distinct tools for input and output of data.

Multimodal human-computer interaction involves natural communication with virtual and physical environments. It facilitates free and natural communication between users and automated systems, allowing flexible input (speech, handwriting, gestures) and output (speech synthesis, graphics). Multimodal fusion combines inputs from different modalities, addressing ambiguities.

Two major groups of multimodal interfaces focus on alternate input methods and combined input/output. Multiple input modalities enhance usability, benefiting users with impairments. Mobile devices often employ XHTML+Voice for input. Multimodal biometric systems use multiple biometrics to overcome limitations. Multimodal sentiment analysis involves analyzing text, audio, and visual data for sentiment classification. GPT-4, a multimodal language model, integrates various modalities for improved language understanding. Multimodal output systems present information through visual and auditory cues, using touch and olfaction. Multimodal fusion integrates information from different modalities, employing recognition-based, decision-based, and hybrid multi-level fusion.

Ambiguities in multimodal input are addressed through prevention, a-posterior resolution, and approximation resolution methods.

Introduction Multimodal human-computer interaction refers to the "interaction with the virtual and physical environment through natural modes of communication",Bourguet, M.L. (2003). "Designing and Prototyping Multimodal Commands". Proceedings of Human-Computer Interaction (INTERACT'03), pp. 717-720. This implies that multimodal interaction enables a more free and natural communication, interfacing users with automated systems in both input and output.Stivers, T., Sidnell, J. Introduction: Multimodal interaction. Semiotica, 156(1/4), pp. 1-20. 2005. Specifically, multimodal systems can offer a flexible, efficient and usable environment allowing users to interact through input modalities, such as speech, handwriting, hand gesture and gaze, and to receive information by the system through output modalities, such as speech synthesis, smart graphics and other modalities, opportunely combined. Then a multimodal system has to recognize the inputs from the different modalities combining them according to temporal and contextual constraintsCaschera M. C., Ferri F., Grifoni P. (2007). "Multimodal interaction systems: information and time features". International Journal of Web and Grid Services (IJWGS), Vol. 3 - Issue 1, pp 82-99. in order to allow their interpretation. This process is known as multimodal fusion, and it is the object of several research works from the nineties to now.D'Ulizia, A., Ferri, F. and Grifoni, P. (2010). "Generating Multimodal Grammars for Multimodal Dialogue Processing". IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, Vol 40, no 6, pp. 1130 – 1145.D'Ulizia, A. (2009). "Exploring Multimodal Input Fusion Strategies". In: Grifoni P (ed) Handbook of Research on Multimodal Human Computer Interaction and Pervasive Services: Evolutionary Techniques for Improving Accessibility. IGI Publishing, pp. 34-57.Sun, Y., Shi, Y., Chen, F. and Chung, V.(2007). "An Efficient Multimodal Language Processor for Parallel Input Strings in Multimodal Input Fusion," in Proc. of the international Conference on Semantic Computing, pp. 389-396.Russ, G., Sallans, B., Hareter, H. (2005). "Semantic Based Information Fusion in a Multimodal Interface". International Conference on Human-Computer Interaction (HCI'05), Las Vegas, Nevada, USA, 20–23 June, pp 94-100.Corradini, A., Mehta M., Bernsen, N.O., Martin, J.-C. (2003). "Multimodal Input Fusion in Human-Computer Interaction on the Example of the on-going NICE Project". In Proceedings of the NATO-ASI conference on Data Fusion for Situation Monitoring, Incident Detection, Alert and Response Management, Yerevan, Armenia.Pavlovic, V.I., Berry, G.A., Huang, T.S. (1997). "Integration of audio/visual information for use in human-computer intelligent interaction". Proceedings of the 1997 International Conference on Image Processing (ICIP '97), Volume 1, pp. 121-124.Andre, M., Popescu, V.G., Shaikh, A., Medl, A., Marsic, I., Kulikowski, C., Flanagan J.L. (1998). "Integration of Speech and Gesture for Multimodal Human-Computer Interaction". In Second International Conference on Cooperative Multimodal Communication. 28–30 January, Tilburg, The Netherlands.Vo, M.T., Wood, C. (1996). "Building an application framework for speech and pen input integration in multimodal learning interfaces". In Proceedings of the Acoustics, Speech, and Signal Processing (ICASSP'96), May 7–10, IEEE Computer Society, Volume 06, pp. 3545-3548. The fused inputs are interpreted by the system. Naturalness and flexibility can produce more than one interpretation for each different modality (channel) and for their simultaneous use, and they consequently can produce multimodal ambiguityCaschera, M.C., Ferri, F., Grifoni, P. (2013). "From Modal to Multimodal Ambiguities: a Classification Approach", Journal of Next Generation Information Technology (JNIT), Vol. 4, No. 5, pp. 87 -109. generally due to imprecision, noises or other similar factors. For solving ambiguities, several methods have been proposed.Caschera, M.C., Ferri, F., Grifoni, P. (2013). InteSe: An Integrated Model for Resolving Ambiguities in Multimodal Sentences". IEEE Transactions on Systems, Man, and Cybernetics: Systems, Volume: 43, Issue: 4, pp. 911 - 931.18. Spilker, J., Klarner, M., Görz, G. (2000). "Processing Self Corrections in a speech to speech system". COLING 2000. pp. 1116-1120.Caschera M.C., Ferri F., Grifoni P., (2007). "The Management of ambiguities". In Visual Languages for Interactive Computing: Definitions and Formalizations. IGI Publishing. pp.129-140.J. Chai, P. Hong, and M. X. Zhou, (2004 )."A probabilistic approach to reference resolution in multimodal user interface" in Proc. 9th Int. Conf. Intell. User Interf., Madeira, Portugal, Jan. 2004, pp. 70–77.Dey, A. K. Mankoff, J., (2005). "Designing mediation for context-aware applications". ACM Trans. Comput.-Hum. Interact. 12(1), pp. 53-80.Spilker, J., Klarner, M., Görz, G. (2000). "Processing Self Corrections in a speech to speech system". COLING 2000. pp. 1116-1120.Mankoff, J., Hudson, S.E., Abowd, G.D. (2000). "Providing integrated toolkit-level support for ambiguity in recognition-based interfaces". Proceedings of ACM CHI'00 Conference on Human Factors in Computing Systems. pp. 368 – 375. Finally the system returns to the user outputs through the various modal channels (disaggregated) arranged according to a consistent feedback (fission).Grifoni P (2009) Multimodal fission. In: Multimodal human computer interaction and pervasive services. IGI Global, pp 103–120 The pervasive use of mobile devices, sensors and web technologies can offer adequate computational resources to manage the complexity implied by the multimodal interaction. "Using cloud for involving shared computational resources in managing the complexity of multimodal interaction represents an opportunity. In fact, cloud computing allows delivering shared scalable, configurable computing resources that can be dynamically and automatically provisioned and released".Patrizia Grifoni, Fernando Ferri, Maria Chiara Caschera, Arianna D'Ulizia, Mauro Mazzei, "MIS: Multimodal Interaction Services in a cloud perspective", JNIT: Journal of Next Generation Information Technology, Vol. 5, No. 4, pp. 01 ~ 10, 2014

Multimodal input Two major groups of multimodal interfaces have merged, one concerned in alternate input methods and the other in combined input/output. The first group of interfaces combined various user input modes beyond the traditional keyboard and mouse input/output, such as speech, pen, touch, manual gestures,Kettebekov, Sanshzar, and Rajeev Sharma (2001). "Toward Natural Gesture/Speech Control of a Large Display." ProceedingsEHCI '01 Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction Pages 221-234 gaze and head and body movements.Marius Vassiliou, V. Sundareswaran, S. Chen, R. Behringer, C. Tam, M. Chan, P. Bangayan, and J. McGee (2000), "Integrated Multimodal Human-Computer Interface and Augmented Reality for Interactive Display Applications," in Darrel G. Hopper (ed.) Cockpit Displays VII: Displays for Defense Applications (Proc. SPIE . 4022), 106-115. The most common such interface combines a visual modality (e.g. a display, keyboard, and mouse) with a voice modality (speech recognition for input, speech synthesis and recorded audio for output). However other modalities, such as pen-based input or haptic input/output may be used. Multimodal user interfaces are a research area in human-computer interaction (HCI).

The advantage of multiple input modalities is increased usability: the weaknesses of one modality are offset by the strengths of another. On a mobile device with a small visual interface and keypad, a word may be quite difficult to type but very easy to say (e.g. Poughkeepsie). Consider how you would access and search through digital media catalogs from these same devices or set top boxes. And in one real-world example, patient information in an operating room environment is accessed verbally by members of the surgical team to maintain an antiseptic environment, and presented in near realtime aurally and visually to maximize comprehension.

Multimodal input user interfaces have implications for accessibility. A well-designed multimodal application can be used by people with a wide variety of impairments. Visually impaired users rely on the voice modality with some keypad input. Hearing-impaired users rely on the visual modality with some speech input. Other users will be "situationally impaired" (e.g. wearing gloves in a very noisy environment, driving, or needing to enter a credit card number in a public place) and will simply use the appropriate modalities as desired. On the other hand, a multimodal application that requires users to be able to operate all modalities is very poorly designed.

The most common form of input multimodality in the market makes use of the XHTML+Voice (aka X+V) Web markup language, an open specification developed by IBM, Motorola, and Opera Software. X+V is currently under consideration by the W3C and combines several W3C Recommendations including XHTML for visual markup, VoiceXML for voice markup, and XML Events, a standard for integrating XML languages. Multimodal browsers supporting X+V include IBM WebSphere Everyplace Multimodal Environment, Opera for Embedded Linux and Windows, and ACCESS Systems NetFront for Windows Mobile. To develop multimodal applications, software developers may use a software development kit, such as IBM WebSphere Multimodal Toolkit, based on the open source Eclipse framework, which includes an X+V debugger, editor, and simulator.

Multimodal biometrics

Multimodal sentiment analysis

Multimodal language models

Multimodal output The second group of multimodal systems presents users with multimedia displays and multimodal output, primarily in the form of visual and auditory cues. Interface designers have also started to make use of other modalities, such as touch and olfaction. Proposed benefits of multimodal output system include synergy and redundancy. The information that is presented via several modalities is merged and refers to various aspects of the same process. The use of several modalities for processing exactly the same information provides an increased bandwidth of information transfer