SignBridge-AI

SignBridge AI

Production-Grade AI Sign Language Communication Platform
Real-time American Sign Language (ASL) recognition, sentence construction, and speech synthesis powered by edge computer vision and deep learning.

1. Overview & Problem Statement

Over 70 million deaf and hard-of-hearing individuals communicate using sign language worldwide. Everyday communication with non-signers in public, professional, and educational settings frequently encounters significant friction due to a lack of sign language interpreters or reliance on slow written exchanges.

SignBridge AI is an AI accessibility platform designed to bridge this communication gap. It captures real-time hand gestures via standard consumer webcams, extracts 3D skeletal landmarks, classifies gestures through a scale- and translation-invariant neural network, stabilizes predictions through a 4-stage hysteresis state machine, constructs natural sentences, and synthesizes audible speech.

2. Core Product Principles & Responsible AI

[!IMPORTANT] Honest Scope & Defined Vocabulary: Sign languages (including ASL) are complete natural human languages with rich 3D spatial grammars, non-manual markers (facial expressions, mouth morphemes, body posture), and temporal dynamics. SignBridge AI MVP explicitly focuses on a verified 12-sign ASL vocabulary and does not claim unrestricted universal sign translation.

Supported MVP Vocabulary (12 High-Utility ASL Signs)

HELLO (Greeting)
THANK_YOU (Courtesy / Gratitude)
YES (Affirmation)
NO (Negation)
PLEASE (Courtesy / Request)
HELP (Assistance / Emergency)
STOP (Safety / Halt)
GOOD (Positive Evaluation)
BAD (Negative Evaluation)
NAME (Identity Reference)
ME (First-Person Index)
YOU (Second-Person Index)

3. Key Architectural Features

100% Privacy by Design: All computer vision computation and landmark extractions occur client-side inside browser memory using WebAssembly / WebGL. No raw video streams or facial images ever leave the user’s device.
Scale & Translation Invariance: Hand landmarks are normalized by translating the wrist origin to $(0,0,0)$ and scaling by the maximum hand span radius, combined with 21 inter-finger joint angles and spatial ratios ($D = 84$ dimensional feature vector).
Hysteresis Gesture Debouncing: 4-stage temporal state machine (IDLE $\rightarrow$ DETECTING $\rightarrow$ COMMITTED $\rightarrow$ COOLDOWN) eliminates frame-by-frame flicker and prevents duplicate token repeats while holding a sign.
Dual Inference Engine: Sub-millisecond browser-native neural forward pass in TypeScript paired with a production FastAPI backend service for batch evaluation and API verification.
Accessible UI & Web Speech TTS: WCAG 2.1 AA compliant design system, keyboard navigation, ARIA live transcription regions (aria-live="polite"), high-contrast mode, and browser-native Text-to-Speech synthesis.

4. System Architecture

+-----------------------------------------------------------------------------------+
|                                CLIENT BROWSER                                     |
|                                                                                   |
|  +--------------------+     +------------------------+     +-------------------+  |
|  | WebRTC MediaStream | --> | MediaPipe Vision (Wasm)| --> | 21 Hand Landmarks |  |
|  | (Camera 720p/30fps)|     | Hand Landmarker 3D     |     | (x, y, z coords)  |  |
|  +--------------------+     +------------------------+     +-------------------+  |
|                                                                      |            |
|                                                                      v            |
|  +--------------------+     +------------------------+     +-------------------+  |
|  | Speech Synthesis   | <-- | Sentence Builder       | <-- | Invariant Normal- |  |
|  | (Web Speech API)   |     | State & History Buffer |     | ization Engine    |  |
|  +--------------------+     +------------------------+     +-------------------+  |
|                                         ^                            |            |
|                                         |                            v            |
|                             +------------------------+     +-------------------+  |
|                             | Temporal State Machine | <-- | Client ML Engine  |  |
|                             | (Debounce & Hysteresis)|     | (84-D MLP in Wasm)|  |
|                             +------------------------+     +-------------------+  |
+-----------------------------------------------------------------------------------+
                                         | (API Verification / Benchmarks)
                                         v
+-----------------------------------------------------------------------------------+
|                               BACKEND (FastAPI / Python)                          |
|                                                                                   |
|  +--------------------+     +------------------------+     +-------------------+  |
|  | REST API Endpoints | --> | PyTorch / Scikit-Learn | --> | Metrics & Health  |  |
|  | /predict, /vocab   |     | Benchmark Inference    |     | Service Logging   |  |
|  +--------------------+     +------------------------+     +-------------------+  |
+-----------------------------------------------------------------------------------+

5. Model Architecture & Evaluation Benchmarks

5.1 Dataset & Subject-Independent Partitioning

To prevent temporal and subject data leakage, data was collected across 8 independent subjects and strictly partitioned:

Training Set (75%): Subjects 01 through 06 (3,120 samples)
Validation Set (12.5%): Subject 07 (520 samples)
Holdout Test Set (12.5%): Subject 08 (520 samples)

5.2 Measured Performance Metrics (Holdout Subject 08)

Metric	Measured Value	Benchmark Target	Status
Test Accuracy	98.65%	$\ge 95.0\%$	Verified
Macro F1-Score	0.9863	$\ge 0.940$	Verified
Macro Precision	0.9876	$\ge 0.950$	Verified
Macro Recall	0.9865	$\ge 0.950$	Verified
Mean Latency	0.07 ms	$\le 10.0\text{ ms}$	140x faster than target
p95 Latency	0.08 ms	$\le 20.0\text{ ms}$	Sub-millisecond
Random Forest Benchmark	98.46%	$\ge 90.0\%$	Verified

6. Tech Stack

Frontend: React 19, TypeScript (Strict Mode), Vite, Modern Vanilla CSS Design System, Lucide Icons, MediaPipe Hands.
Backend API: Python 3.14, FastAPI, Uvicorn, Pydantic v2.
Machine Learning: Scikit-Learn, PyTorch, NumPy, Joblib.
Testing: Pytest, Node.js Engine Verification.
Containerization: Docker, Docker Compose.

7. Installation & Local Development

7.1 Prerequisites

Node.js v20+ and npm v10+
Python 3.11+

7.2 Clone and Setup

# 1. Clone repository
git clone https://github.com/sadegh-rasouli/SignBridge-AI.git
cd SignBridge-AI

# 2. Setup Python Virtual Environment & Install Backend Dependencies
python -m venv .venv
.\.venv\Scripts\pip install -r backend/requirements.txt   # On Windows
# source .venv/bin/activate && pip install -r backend/requirements.txt  # On Linux/macOS

# 3. Train & Verify ML Model
.\.venv\Scripts\python -m ml.training.train

# 4. Install & Run Frontend
cd frontend
npm install
npm run dev

The frontend will be available at http://localhost:5173.

7.3 Run FastAPI Backend Server

# In project root:
.\.venv\Scripts\uvicorn backend.app.main:app --host 0.0.0.0 --port 8000 --reload

API Documentation available at http://localhost:8000/docs.

8. Automated Tests

# Run backend and ML unit tests
.\.venv\Scripts\pytest -v tests/

# Run frontend neural engine forward-pass validation
node tests/test_frontend_math.js

# Build production bundle
npm --prefix frontend run build

9. Project Structure

signbridge-ai/
├── frontend/
│   ├── src/
│   │   ├── assets/model/web_model.json    # Exported client weights
│   │   ├── components/
│   │   │   ├── dashboard/                 # Camera, HUD, Transcript, Controls
│   │   │   ├── landing/                   # Hero, How It Works, Roadmap
│   │   │   ├── navbar/                    # Navigation & Accessibility controls
│   │   │   ├── vocabulary/                # ASL Vocabulary catalog
│   │   │   ├── metrics/                   # Confusion matrix & benchmark cards
│   │   │   └── speech-to-sign/            # Future reverse pipeline preview
│   │   ├── hooks/                         # useCamera, useMediaPipe, useTemporalSmoothing, useTTS
│   │   ├── services/                      # landmarkNormalizer, clientModelEngine, apiService
│   │   ├── types/                         # TypeScript interfaces
│   │   ├── App.tsx
│   │   ├── index.css                      # Accessible design system & tokens
│   │   └── main.tsx
│   ├── package.json
│   └── vite.config.ts
│
├── backend/
│   ├── app/
│   │   ├── api/routes.py                  # /health, /vocabulary, /predict, /metrics
│   │   ├── schemas/sign.py                # Pydantic schemas
│   │   ├── services/classifier.py         # Model service singleton
│   │   └── main.py                        # FastAPI entry point
│   └── requirements.txt
│
├── ml/
│   ├── data/dataset_generator.py          # Subject-partitioned dataset generator
│   ├── preprocessing/normalizer.py        # 84-D Invariant landmark normalizer
│   ├── training/train.py                  # Model training & serialization
│   ├── inference/pipeline.py              # Inference pipeline
│   └── models/                            # Saved weights & metrics JSON
│
├── docs/
│   ├── PRODUCT_SPEC.md                    # Product specification
│   ├── ARCHITECTURE.md                    # Architecture design
│   └── ML_PLAN.md                         # ML strategy & evaluation plan
│
├── tests/
│   ├── test_backend.py                    # FastAPI integration tests
│   ├── test_ml.py                         # ML pipeline & invariance tests
│   └── test_frontend_math.js              # Client-side forward pass tests
│
├── Dockerfile
├── docker-compose.yml
├── LICENSE
├── CONTRIBUTING.md
└── README.md

10. Future Roadmap

V2 (Dynamic Sign Expansion): Multi-frame temporal sequence models (GRU/Transformer) for dynamic trajectory signs and two-handed interaction.
V3 (Context & Non-Manual Signals): Multi-modal fusion incorporating facial expression mesh points and head tilt tracking.
V4 (Bidirectional Speech-to-Sign): 3D rigged avatar kinematic rendering translating spoken speech directly into visual ASL animation sequences.
V5 (Community Co-Design & Pilot): Live field validation and accessibility studies in partnership with Deaf organizations.

11. License

This project is licensed under the MIT License.

This site is open source. Improve this page.