1. Introduction & Overview

SilentVoice is a lightweight edge-AI prototype built to bridge communication gaps during medical emergencies. Currently trained over words to convey critical needs—such as "help", "water", "pain", "doctor", "breathe", "yes", and "no"—simply by mouthing words in front of a camera.

Unlike heavy cloud-based vision models that require massive datasets and high-end GPUs, this project uses a streamlined landmark-coordinate tracking pipeline running locally via Flask, MediaPipe, and a lightweight TensorFlow Lite model optimized for edge environments.

2. System Architecture

The application workflow is divided into three main components:

In practice, SilentVoice works best when you follow a step-by-step model validation workflow and keep a simple checklist for wiring, power stability, and expected output behavior. This makes debugging faster and creates a practical samples troubleshooting path for repeatable results.

• Frontend (Browser + MediaPipe): It Captures real-time video frames from a mobile, extracts 40 specific mouth landmarks (resulting in 80 X and Y coordinate floats per frame), and bundles them into temporal windows of 30 frames.
• Backend Server (app.py): A Flask application which handles incoming HTTP/WebSocket connections, serves the user interface, manages sample recording data storage, and executes model inference.
• Machine Learning Model (train.py & lrw_model.tflite): It is a custom 1D Convolutional Neural Network trained on coordinate sequences to classify mouth gestures into our 7 vocabulary categories. A pretrained lrw_model.tflite (~197 KB) is included in the repo, trained on 98 self-recorded samples across the 7 words

`lrw_model.tflite`

3. Project Setup & Installation Guide

To set up and run the project from scratch, follow these step-by-step commands in your terminal:

Step 1: Environment Setup Navigate to project directory and create and activate a Python virtual environment:

python3 -m venv lip_env
source lip_env/bin/activate

Step 2: Install Dependencies Install all required Python packages for backend routing, deep learning, and data handling

pip install flask flask-socketio tensorflow numpy mediapipe opencv-python scikit-learn

4. Data Collection Phase

Before training the model, you need to record real-world training samples using the app's Training Mode:

A reliable implementation also benefits from modular structure: separate input handling, processing logic, and output control so each part can be tested independently. That pattern supports low-noise training tuning, clearer data calibration decisions, and safer iteration when features evolve.

• Run your backend server:

python3 app.py

• Open the application interface in your browser (or expose it via Cloudflare).
• Use Training Mode to record multiple samples (aiming for roughly 10–20 samples per word) across your 7 vocabulary words: help, water, pain, doctor, breathe, yes, and no.
• The app automatically stores these recordings as 30-frame coordinate sequences in JSON format under a training_data/ directory.

`training_data/` The included pretrained model was trained on the following sample counts: help (17), water (20), pain (16), doctor (11), breathe (12), yes (10), no (12) — 98 real samples total.

5. Model Training (train.py)

Once we have collected raw samples, need to process, augment, and train the neural network model.

Why a 1D CNN? During development, I discovered that flattening frames destroys temporal motion data, while standard LSTM models throw TensorListReserve conversion errors when exporting to TensorFlow Lite. So I stabilized the system using a 1D Convolutional Neural Network (1D CNN), which reads the 30-frame sequence timeline efficiently, handles Gaussian noise data augmentation smoothly, and exports cleanly to TFLite.

The 98 real samples are split into train/validation sets before augmentation (to avoid leakage between near-duplicate noisy copies), then each side is expanded with 20 Gaussian-noise variations. The model trains for 40 epochs, reaching ~45% training accuracy. Validation accuracy fluctuates in the 10–40% range due to the very small validation set (~2 samples per class) — as a proof-of-concept prototype trained on limited data, more recorded samples per word will directly improve both accuracy and stability.

Run the Training Script: Execute the training process in your terminal:

python3 train.py

For long-term maintainability, document baseline measurements such as response time, stability under transitions, and recovery after temporary faults. Using this measurement-driven step optimization style gives you a scalable python upgrade path without turning the project into a fragile one-off demo.

(This will read JSON samples, augment them, train the model over 40 epochs, and output your compiled lrw_model.tflite file.)

6. Running the Application & Live Testing

Step 1: Start the Flask Server Ensure your model file is saved in the project directory, then start the server:

python3 app.py

Step 2: Set Up Cloudflare Tunneling (For Mobile Access) As mobile browsers require a secure HTTPS connection to access device cameras, so expose your local server securely using Cloudflare Tunnels without router configurations. Open a separate terminal window and run:

./cloudflared tunnel --url http://localhost:5000

Copy the public HTTPS link provided in the terminal output

Step 3: Live Testing Open the Cloudflare URL on your smartphone browser. Switch to Live Test Mode, look at the camera, and mouth your words. The backend will process the coordinate sequences through lrw_model.tflite and display the predicted word and confidence score in real time.