Published on: October 2026
LANGUAGE-CONDITIONED AUDIO ADAPTATION FOR MULTILINGUAL AUDIO-VISUAL DEEPFAKE DETECTION
Gurleen Kaur Sihra
Dr. Mohammad Abuzar Ansari
Science and Commerce (Empowered Autonomous) Sion (West), Mumbai, India
Article Status
Available Documents
Abstract
Audio-visual deepfake detectors are increasingly deployed in multilingual settings, yet language-dependent speech variability can alter the behavior of audio branches trained without explicit language conditioning. This paper evaluates a MAVDD-style audio-visual detector on the French, Spanish, and Russian subsets of PolyGlotFake and proposes a lightweight language-conditioned MFCC adapter routed by predictions from a frozen MMS-LID-126 model. The Xception visual branch, the MFCC-CNN audio classifier, and the language identifier remain frozen; only 104 scalar adapter parameters are trained. To prevent source leakage, all manipulated variants derived from the same raw source are assigned to a single split. On the untouched 1,311-video test set, baseline hard-OR fusion obtains 91.15% accuracy, 62.58% balanced accuracy, 94.90% fake recall, and 95.28% fake F1. The language-aware system reaches 94.28% accuracy, 59.30% balanced accuracy, 98.87% fake recall, and 97.02% fake F1. The audio branch alone improves from 73.53% to 90.08% accuracy and from 59.86% to 66.31% ROC-AUC. Of 57 changed final decisions, 49 are baseline errors corrected and 8 are newly introduced errors; real recall falls from 30.26% to 19.74%. An exact paired McNemar test gives p ≈2.72 × 10−8. We also report that a trivial all-fake predictor attains 94.20% accuracy and 97.01% fake F1 on this test set, so the full-system accuracy gain should be read as increased fake sensitivity rather than improved class separation. Post-hoc Integrated Gradients, adapter-parameter visualization, and Grad-CAM provide qualitative evidence consistent with this trade-off.
Index Terms—audio-visual deepfake detection, multilingual deepfake detection, language identification, MFCC, Xception, multimodal learning, language-conditioned adaptation, explainable AI.
How to Cite this Paper
Sihra, G. K. (2026). Language-Conditioned Audio Adaptation for Multilingual Audio-Visual Deepfake Detection. International Journal of Creative and Open Research in Engineering and Management, <i>02</i>(10), 1-9. https://doi.org/10.55041/ijcope.v2i10.116
Sihra, Gurleen. "Language-Conditioned Audio Adaptation for Multilingual Audio-Visual Deepfake Detection." International Journal of Creative and Open Research in Engineering and Management, vol. 02, no. 10, 2026, pp. 1-9. doi:https://doi.org/10.55041/ijcope.v2i10.116.
Sihra, Gurleen. "Language-Conditioned Audio Adaptation for Multilingual Audio-Visual Deepfake Detection." International Journal of Creative and Open Research in Engineering and Management 02, no. 10 (2026): 1-9. https://doi.org/https://doi.org/10.55041/ijcope.v2i10.116.
Search & Index
References
[1] Y. Hou, H. Fu, C. Chen, Z. Li, H. Zhang, and J. Zhao, “PolyGlotFake: A novel multilingual and multimodal DeepFake dataset,” in Pattern Recognition—27th Int. Conf., ICPR 2024, Lecture Notes in Computer Science, vol. 15314, 2025, pp. 180–193, doi: 10.1007/978-3-031-783418_12.
[2] A. A. Bekheet, G. Khoriba, and A. S. Ghoneim, “Development of a multimodal framework for deepfake detection: Combining visual and audio analysis,” in Proc. 10th World Congr. Electrical Eng. and Computer Systems and Sciences (EECSS’24), Barcelona, Spain, 2024, Paper MVML 115, doi: 10.11159/mvml24.115.
[3] H. Khalid, S. Tariq, M. Kim, and S. S. Woo, “FakeAVCeleb: A novel audio-video multimodal deepfake dataset,” in Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021.
[4] W. Yang, X. Zhou, Z. Chen, B. Guo, Z. Ba, Z. Xia, X. Cao, and K. Ren, “AVoiD-DF: Audio-visual joint learning for detecting deepfake,” IEEE Trans. Inf. Forensics Security, vol. 18, pp. 2015–2029, 2023, doi: 10.1109/TIFS.2023.3262148.
[5] M. A. Raza and K. M. Malik, “MultimodalTrace: Deepfake detection using audiovisual representation learning,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), 2023, pp. 993–1000, doi: 10.1109/CVPRW59228.2023.00106.
[6] V. S. Katamneni and A. Rattani, “MIS-AVoiDD: Modality invariant and specific representation for audio-visual deepfake detection,” in Proc. 22nd IEEE Int. Conf. Machine Learning and Applications (ICMLA), 2023, pp. 1371–1378, doi: 10.1109/ICMLA58977.2023.00207.
[7] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1251–1258, doi: 10.1109/CVPR.2017.195.
[8] V. Pratap et al., “Scaling speech technology to 1,000+ languages,” J. Mach. Learn. Res., vol. 25, no. 97, pp. 1–52, 2024.
[9] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12449–12460.
[10] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), 2017, pp. 618–626, doi: 10.1109/ICCV.2017.74.
Ethical Compliance & Review Process
- •All submissions are screened under plagiarism detection.
- •Review follows editorial policy.
- •Authors retain copyright.
- •Peer Review Type: Double-Blind Peer Review
- •Published on: Oct 20 2026
This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. You are free to share and adapt this work for non-commercial purposes with proper attribution.