SignBridge India is a free, open-source overlay application currently in development. When complete, it will capture live audio from any website — YouTube, Instagram, lectures, news — and render a real-time 3D Indian Sign Language avatar with correct SOV grammar, facial expressions, and fingerspelling.
Target experience: a real-time ISL 3D avatar interpreting your system audio
Closed captions are not enough. English captions follow SVO grammar, while Indian Sign Language is a natural language with its own SOV syntax — forcing Deaf viewers to decode a foreign structure in real time.
Captions transcribe spoken English word-for-word. For pre-lingually deaf individuals whose primary language is ISL, reading fast English text causes heavy cognitive fatigue.
Existing tools are closed media players needing pre-recorded sign videos. Certified ISL interpreters are scarce and unaffordable for everyday video consumption.
Most translation attempts ignore the fundamental SVO→SOV divergence of ISL and skip non-manual markers — over 30% of sign language grammar — producing unintelligible output.
A four-month academic development lifecycle at VIT Vellore. Here's our milestone tracker — updated as we build.
Literature survey, architectural planning, environment setup, and benchmark evaluation of ASR engines (Whisper vs. Vosk) and ISL parallel corpora (CISLR, ISLTranslate, SignSparK).
Audio capture loopback integration, streaming ASR implementation, and rule-based English-to-ISL POS/dependency parser module.
3D avatar rigging, glTF animation library integration, Three.js WebGL rendering within a transparent Electron overlay, and dynamic SLERP coarticulation logic.
Formal evaluation with certified ISLRTC interpreters and DHH focus groups — assessing syntactic correctness, legibility, and usability.
Our planned architecture: a transparent, draggable overlay working over any browser window — capturing audio, translating grammar, and animating a 3D avatar in under two seconds, end to end.
System loopback captures browser audio at 16kHz PCM
Streaming Whisper & Vosk ASR with voice activity detection
spaCy dependency parsing restructures English into ISL grammar
Gloss tokens matched to a glTF motion dictionary; OOV → fingerspelling
SLERP joint interpolation + facial blendshapes for NMMs
Three.js/WebGL avatar in a frameless transparent Electron window
An early rule-based demo of our Phase-2 translation module — type an English sentence to see the SVO → SOV gloss conversion.EARLY PROTOTYPE
Specific, measurable targets guiding every sprint of development.
A virtual loopback module intercepting live desktop audio from any active browser window.
🎯 Buffer latency < 200msOn-device speech recognition using lightweight streaming Whisper and Vosk models.
🎯 WER < 15% on Indian EnglishspaCy dependency parsing for SVO→SOV conversion with a T5 neural fallback for complex sentences.
🎯 BLEU > 0.35 on parallel corporaThree.js/WebGL rendering inside a transparent Electron overlay with smooth SLERP coarticulation.
🎯 Steady 60 FPS renderingTwo-handed ISL fingerspelling engine for named entities and out-of-vocabulary terms.
🎯 Full OOV coverageFormal evaluation with certified ISLRTC interpreters and DHH focus groups.
🎯 Real-user usability studyDesigned with and for the Deaf community — to be validated through participatory evaluation with native signers and certified interpreters.
Independent access to online lectures, digital classrooms, and educational video content for Deaf students across India.
Open-source AI, WebGL, and streaming engineering deployed for inclusive assistive technology — free for everyone.
Empowering marginalized Deaf and Hard of Hearing citizens to consume news, webinars, and entertainment equally.
The planned architecture runs entirely on the user's device — no cloud bills, no data leaving the machine, fully private.
Quantized local ASR engines
Rule-based parsing with neural fallback
Hardware-accelerated 3D avatar
Transparent frameless overlay shell
Local sign motion dictionary
Rigging & glTF animation library
Smooth coarticulated joint blending
Two-handed ISL fallback for OOV words
A VIT Vellore Project-I initiative under the School of Computer Science and Engineering.
Audio Pipeline & ASR Integration
NLP Translation & Gloss Engine
3D Avatar & Overlay Rendering
We welcome contributors — Deaf testers, developers, interpreters
Assistant Professor Senior Grade, School of Computer Science and Engineering (SCOPE), VIT Vellore
SignBridge India is a non-profit, community-driven project in active development. Here's how you can power the mission.
Support motion-capture sessions with certified ISLRTC interpreters and corpus development for low-resource ISL data.
Support UsReact/WebGL frontend, Python ASR pipelines, ISL corpus curation — pick an issue once our repo goes public.
Express InterestAre you a Deaf/HoH user or a certified ISL interpreter? Join our evaluation panel for Phase 4 validation.
Sign UpQuestions, partnerships, media, or accessibility feedback — we read everything.
Based at Vellore Institute of Technology, Vellore, Tamil Nadu, India.