Cognitive Effort-Aware Human-Computer Interaction Using Voice and Gesture Inputs

Abstract

Interaction (HCI) increasingly relies on multimodal interfaces that combine voice and gesture recognition to support natural and intuitive communication. However, most existing systems emphasize recognition accuracy and modality fusion while largely ignoring the user’s internal cognitive state. As a result, interaction breakdowns often occur when interfaces become cognitively demanding, leading to user frustration and reduced usability. This paper proposes a cognitive effort–aware HCI framework that adapts multimodal interaction strategies in real time based on inferred user mental workload. Cognitive effort is estimated using short-term behavioral cues, including speech pauses, command repetition, response latency, and gesture hesitation, and classified into low, medium, or high effort states. Based on this inference, the interaction layer dynamically adjusts interface complexity, modality prioritization, and feedback mechanisms to reduce mental strain. Experimental evaluation compares the proposed adaptive approach with static multimodal interfaces using task performance metrics and subjective workload assessment. Results indicate that incorporating cognitive effort as a design parameter improves interaction robustness, usability, and accessibility across diverse application domains, including automotive systems and assistive technologies.