Real-Time Multilingual Closed Captioning System with Simplified Captions for Deaf Person

Abstract

Real-time closed captioning is essential for improving accessibility for Deaf and Hard-of-Hearing (DHH) users in live communication. This paper presents a low-latency multilingual closed-captioning system designed for Indian languages. The proposed system integrates streaming Automatic Speech Recognition (ASR), automatic language identification, text simplification, punctuation restoration, and speaker segmentation. It supports code-switching and optional translation or transliteration across scripts.
The system is optimized for sub-second end-to-end latency and robustness in noisy environments. Experimental results show that simplified captions significantly improve readability and comprehension while maintaining acceptable recognition accuracy, making the system suitable for real-time educational and public communication applications.