IJCOPE Journal

UGC Logo DOI / ISO Logo

International Journal of Creative and Open Research in Engineering and Management

A Peer-Reviewed, Open-Access International Journal Supporting Multidisciplinary Research, Digital Publishing Standards, DOI Registration, and Academic Indexing.
Journal Information
ISSN: 3108-1754 (Online)
Crossref DOI: Available
ISO Certification: 9001:2015
Publication Fee: 599/- INR
Compliance: UGC Journal Norms
License: CC BY 4.0
Peer Review: Double Blind
Volume 02, Issue 7

Published on: July 2026

EXPLAINABLE MULTIMODAL INTELLIGENCE FOR COGNITIVE FATIGUE DETECTION USING SPEECH ACOUSTICS, FACIAL DYNAMICS, AND HUMAN INTERACTION BEHAVIOUR: A SYSTEMATIC REVIEW

Sarbjeet Kaur

Department of Computer Science & Engineering PCTE Institute of Engineering and Technology, Ludhiana, Punjab-142021, India

Article Status

Plagiarism Passed Peer Reviewed Open Access

Available Documents

Abstract

Cognitive fatigue is a transient, neurophysiologically grounded impairment of executive function and sustained attention arising from prolonged mental effort without adequate rest. It is a leading contributor to human error in safety-critical domains, including aviation, surgery, and long-haul transportation. Despite two decades of active research, deployed detection systems remain anchored to invasive physiological hardware, architecturally naive fusion strategies, and opaque, unexplained inference processes. These three barriers collectively prevent real-world deployment at any meaningful scale. This review systematically synthesises twenty landmark studies published between 2022 and 2026, drawn from IEEE Xplore, PubMed, Nature group journals, ACM Digital Library, and ScienceDirect. Three non-contact detection modalities are evaluated in depth: speech acoustic-prosodic features, facial Action Unit (AU) dynamics, and human-computer interaction (HCI) behavioural markers. The review also examines emerging multimodal fusion architectures and explainable artificial intelligence (XAI) integration across the literature. Six structural research gaps are formally identified: persistent reliance on invasive sensing hardware; temporally naive fusion that discards asynchronous cross-modal dependencies; the complete absence of HCI behaviour as a recognised primary detection modality; XAI confined to post-hoc unimodal attribution rather than embedded cross-modal fusion; inconsistent ground-truth labelling protocols that conflate cognitive fatigue with drowsiness; and the absence of any open, tri-modal benchmark dataset for community evaluation. Three primary research objectives are derived from this gap analysis, jointly defining the architecture and validation agenda for the proposed Explainable Multimodal Intelligence framework for Cognitive Fatigue Detection (EMI-CFD). This review provides a critical, evidence-grounded roadmap for the next generation of transparent, non-invasive cognitive fatigue monitoring systems.

Keywords: cognitive fatigue detection; multimodal fusion; explainable artificial intelligence; speech prosody; facial action units

How to Cite this Paper

Kaur, S. (2026). Explainable Multimodal Intelligence for Cognitive Fatigue Detection Using Speech Acoustics, Facial Dynamics, and Human Interaction Behaviour: A Systematic Review. International Journal of Creative and Open Research in Engineering and Management, <i>02</i>(7). https://doi.org/10.55041/ijcope.v2i6.395

Kaur, Sarbjeet. "Explainable Multimodal Intelligence for Cognitive Fatigue Detection Using Speech Acoustics, Facial Dynamics, and Human Interaction Behaviour: A Systematic Review." International Journal of Creative and Open Research in Engineering and Management, vol. 02, no. 7, 2026, pp. . doi:https://doi.org/10.55041/ijcope.v2i6.395.

Kaur, Sarbjeet. "Explainable Multimodal Intelligence for Cognitive Fatigue Detection Using Speech Acoustics, Facial Dynamics, and Human Interaction Behaviour: A Systematic Review." International Journal of Creative and Open Research in Engineering and Management 02, no. 7 (2026). https://doi.org/https://doi.org/10.55041/ijcope.v2i6.395.

Search & Index

References

[1] M. A. S. Boksem and M. Tops, "Mental fatigue: Costs and benefits," Brain Research Reviews, vol. 59, no. 1, pp. 125–139, 2008, doi: 10.1016/j.brainresrev.2008.07.001.

[2] M. A. Al Imran, F. Nasirzadeh, and C. Karmakar, "Designing a practical fatigue detection system: A review on recent developments and challenges," Journal of Safety Research, vol. 90, pp. 100–114, 2024, doi: 10.1016/j.jsr.2024.07.002.

[3] A. B. Rafaih and K. Ari, "AI-driven approaches to managing surgeon fatigue and improving performance," Cureus, vol. 16, no. 12, Art. no. e75717, 2024, doi: 10.7759/cureus.75717.

[4] J. S. Virk, M. Singh, M. Singh, U. Panjwani, and K. Ray, "A multimodal feature fusion framework for sleep-deprived fatigue detection to prevent accidents," Sensors, vol. 23, no. 8, Art. no. 4129, 2023, doi: 10.3390/s23084129.

[5] M. Dias, F. Dörr, S. Garthof, S. Schäfer, J. Elmers, L. Schwed, and T. Ziemssen, "Detecting fatigue in multiple sclerosis through automatic speech analysis," Frontiers in Human Neuroscience, vol. 18, Art. no. 1449388, 2024, doi: 10.3389/fnhum.2024.1449388.

[6] E. Karim, H. R. Pavel, S. Nikanfar, A. Hebri, A. Roy, H. R. Nambiappan, and F. Makedon, "Cognitive fatigue assessment using physiological sensors during human-robot interaction for activities of daily living," Technologies, vol. 12, no. 3, Art. no. 38, 2024, doi: 10.3390/technologies12030038.

[7] F. Far Poor, H. H. Dodge, and M. H. Mahoor, "A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language, and vision," Computers in Biology and Medicine, vol. 182, Art. no. 109199, 2024, doi: 10.1016/j.compbiomed.2024.109199.

[8] M. Zhao, H. Gao, X. Qi, H. Yin, Y. Song, Y. Bai, and L. Zhao, "Multi-query cross-modal attention fusion for cognitive impairment recognition," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 33, pp. 2520–2530, 2025, doi: 10.1109/TNSRE.2025.3583110.

[9] Y. Xie and B. Guo, "CogMamba: Multi-task driver cognitive load and physiological non-contact estimation with multimodal facial features," Sensors, vol. 25, no. 18, Art. no. 5620, 2025, doi: 10.3390/s25185620.

[10] Y. Sun, M. Zhou, F. Cai, X. Yang, C. Li, and Y. Yi, "Noncontact multimodal sensing for driver distraction detection via behavioral-physiological fusion," IEEE Sensors Journal, vol. 25, no. 20, pp. 38576–38584, 2025, doi: 10.1109/JSEN.2025.3608125.

Ethical Compliance & Review Process

  • All submissions are screened under plagiarism detection.
  • Review follows editorial policy.
  • Authors retain copyright.
  • Peer Review Type: Double-Blind Peer Review
  • Published on: Jul 05 2026
CCBYNC

This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. You are free to share and adapt this work for non-commercial purposes with proper attribution.

View License
Scroll to Top