| title | Prism Rppg |
|---|---|
| emoji | 🐢 |
| colorFrom | purple |
| colorTo | yellow |
| sdk | docker |
| pinned | false |
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
Absolutely — here is a clean, GitHub-ready README.md version with the citation artifacts removed and the Markdown/code formatting fixed.
PRISM is the deployed implementation of the research work: "System and Method for Robust Contactless Heart-Rate Estimation Using Adaptive rPPG Model Selection."
Cardiovascular disease remains a major global health challenge, creating a need for continuous, accessible, and non-invasive physiological monitoring.
Traditional Photoplethysmography (PPG) provides reliable heart-rate monitoring but requires direct skin contact. This can cause discomfort, create hygiene concerns, and limit its applicability in populations such as neonates and other vulnerable patients.
Remote Photoplethysmography (rPPG) addresses these limitations by enabling completely contactless physiological monitoring using standard RGB cameras.
However, extracting the weak blood-volume pulse signal from facial video is challenging. rPPG systems must operate in the presence of:
- Dynamic illumination
- Facial motion
- Camera noise
- Video compression
- Blur
- Skin-tone variations
- Specular reflections
- Low photon density
Most existing deep-learning rPPG systems rely on a single fixed physiological estimation model. These monolithic approaches can perform well under familiar conditions but may experience significant performance degradation when exposed to out-of-distribution environments.
PRISM addresses this limitation through an intelligent, physics-aware meta-routing layer.
Instead of relying on one model for every video, PRISM:
- Analyzes the physical characteristics of the input video.
- Extracts a compact physics-based feature representation.
- Predicts the expected error of multiple specialist rPPG models.
- Dynamically selects the model expected to produce the most accurate heart-rate estimate.
- Routes the video to only the selected specialist model.
This transforms conventional single-model physiological estimation into a context-aware dynamic model selection system.
The PRISM framework performs physiological estimation through a sequence of physics-aware routing stages.
The system receives a 30-second facial video sequence captured using an RGB camera.
The input video contains the subtle color variations caused by changes in blood volume beneath the skin.
Facial regions are detected from the incoming video.
To reduce the effects of bounding-box jitter and high-frequency detection noise, PRISM applies temporal stabilization using a Kalman filter operating under a constant-velocity motion model.
This produces a more temporally consistent facial region for subsequent feature extraction and inference.
The stabilized video is passed to the FeatureEngine.
The FeatureEngine extracts a compact 10-dimensional physics state vector describing the optical and motion characteristics of the input sequence.
This representation provides the routing mechanism with information about the environment in which each candidate rPPG model must operate.
The physics state vector is passed to an MLP-based meta-regressor.
Rather than directly classifying the video into a model category, PRISM predicts the expected log-space Mean Absolute Error (MAE) of each candidate rPPG model.
The optimal model is then selected using:
This converts model selection into an error-minimization problem.
PRISM selects the specialist model predicted to have the lowest estimation error.
Models that are likely to perform poorly under the current optical and motion conditions can therefore be bypassed.
This prevents the system from unnecessarily executing a vulnerable architecture.
The selected specialist model processes the input video and produces the final heart-rate estimation.
The architecture is modular, allowing additional rPPG models to be integrated into the ensemble without redesigning the entire routing system.
PRISM's routing decision is based on six fundamental optical/motion primitives and four derived non-linear interaction features.
| Feature | Description |
|---|---|
| Normalized Photon Density ( |
Represents the available optical signal intensity |
| Environmental Noise Floor ( |
Estimates background and sensor noise |
| Skin Tone Intensity ( |
Represents average facial skin intensity |
| Texture Sharpness ( |
Measures spatial facial texture and image sharpness |
| Motion Magnitude ( |
Quantifies facial/head movement |
| Spectral Concentration ( |
Measures concentration of signal energy within relevant frequency bands |
Measures the temporal stability of facial illumination and pixel intensity.
Captures changes in motion intensity over time.
Measures the relative dominance of physiological frequency components compared with competing spectral energy.
Together, these features provide a compact representation of the optical state, motion state, and signal quality of the input video.
PRISM is designed to be model-agnostic and plug-and-play.
The routing layer can select from multiple specialist rPPG architectures.
A convolutional rPPG architecture designed for learning spatial and temporal physiological representations.
A computationally efficient architecture suitable for environments where inference efficiency is important.
CNN-based models can perform particularly well under stable, low-entropy visual conditions but may become vulnerable to strong motion or illumination changes.
A transformer-based physiological signal estimation architecture capable of modeling long-range temporal and spatial relationships.
PhysFormer can achieve strong performance in well-illuminated environments but may become less reliable under photon-starved or heavily degraded conditions.
A state-space-based architecture designed for efficient temporal physiological signal modeling.
A Mamba-based rPPG architecture that provides linear-time sequence modeling capabilities.
State-space models can provide strong temporal modeling efficiency but may be sensitive to abrupt photometric changes depending on the input conditions.
PRISM was evaluated using established rPPG benchmark datasets, including:
- UBFC-rPPG
- PURE
The evaluation focuses on the ability of the routing mechanism to select appropriate specialist models under varying environmental conditions.
Under severe environmental degradation, including:
- Synthetic illumination shifts
- Blur
- Video compression
- Sensor noise
PRISM achieved an MAE of approximately:
This outperformed the best fixed-model configuration, which achieved approximately:
On the highly dynamic PURE P09-02 sequence, several static model configurations produced errors exceeding 30 BPM.
PRISM dynamically selected a specialist estimator and reduced the error to approximately:
These results demonstrate the potential advantage of dynamic model selection over a single fixed rPPG architecture, particularly under challenging environmental conditions.
prism-rppg/
│
├── ai_models/ # Core AI and PRISM logic
│ └── prism/
│ ├── feature_engine.py # Extracts the 10D physics vector
│ └── router.py # MLP-based dynamic routing model
│
├── backend/ # FastAPI backend
│ ├── app/
│ │ ├── main.py # FastAPI application entry point
│ │ └── routers/
│ │ └── video.py # Video processing endpoints
│ │
│ └── requirements.txt # Backend dependencies
│
├── frontend/ # React + Vite web application
│ ├── src/
│ │ ├── components/ # UI components
│ │ │ ├── CircularScanner
│ │ │ └── VitalsChart
│ │ │
│ │ └── pages/ # Application pages
│ │ ├── Index
│ │ └── Analysis
│ │
│ └── package.json
│
├── hfspaces/ # Integrated model implementations
│
├── rppg/ # Specialist rPPG models
│
├── Dockerfile.backend # Backend container
├── Dockerfile.worker # AI worker container
│
├── deploy_backend.ps1 # Backend deployment script
├── deploy_worker.ps1 # Worker deployment script
│
└── README.md
---
# 🚀 Getting Started
## Prerequisites
Before running PRISM locally, make sure the following are installed:
* [Docker](https://www.docker.com/)
* Docker Compose
* [Node.js](https://nodejs.org/)
* Python 3.10+
* Git
---
# ⚙️ Installation
Clone the repository:
```bash
git clone <YOUR_REPOSITORY_URL>
cd prism-rppg
Create and activate a Python virtual environment:
python -m venv .venv.venv\Scripts\activatesource .venv/bin/activateInstall backend dependencies:
pip install -r backend/requirements.txtRun the FastAPI backend:
uvicorn backend.app.main:app --reloadThe backend should then be available at:
http://localhost:8000
Navigate to the frontend directory:
cd frontendInstall dependencies:
npm installStart the development server:
npm run devThe Vite development server will provide a local URL, typically:
http://localhost:5173
PRISM provides separate containers for the backend and AI inference worker.
Build the backend image:
docker build -f Dockerfile.backend -t prism-backend .Run the backend:
docker run -p 8000:8000 prism-backendBuild the worker image:
docker build -f Dockerfile.worker -t prism-worker .Run the worker:
docker run prism-workerFor supported Windows environments, the repository includes deployment scripts.
./deploy_backend.ps1./deploy_worker.ps1Make sure Docker is running before executing the deployment scripts.
One of the primary design goals of PRISM is modularity.
A new rPPG architecture can be integrated into the ensemble without fundamentally changing the routing mechanism.
A typical integration involves:
- Implementing the model inference interface.
- Adding the model to the candidate ensemble.
- Generating evaluation/error data for the new model.
- Updating the router training dataset.
- Retraining the meta-regressor.
- Registering the model with the inference worker.
Conceptually:
┌─────────────────────┐
│ Input Video │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Face Detection & │
│ Stabilization │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Physics Feature │
│ Engine │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ PRISM Meta-Router │
└──────────┬──────────┘
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌───────────┐ ┌───────────┐
│ PhysNet │ │ PhysFormer│ │ PhysMamba │
└────┬────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└────────────────┼───────────────┘
│
▼
┌─────────────────┐
│ Heart Rate │
│ Estimation │
└─────────────────┘
The primary evaluation metric is Mean Absolute Error (MAE) in beats per minute.
where:
-
$HR_i$ is the ground-truth heart rate. -
$\hat{HR}_i$ is the predicted heart rate. -
$N$ is the number of evaluated samples.
Additional evaluation can include:
- Mean Absolute Error (MAE)
- Root Mean Square Error (RMSE)
- Pearson Correlation
- Signal-to-Noise Ratio (SNR)
- Failure rate
- Model-selection accuracy
- Routing regret
- Inference latency
Traditional rPPG pipelines generally follow the structure:
Video
│
▼
Single rPPG Model
│
▼
Heart Rate
PRISM instead introduces an adaptive decision layer:
Video
│
▼
Physics Analysis
│
▼
Meta-Router
│
├──────► Model A
├──────► Model B
├──────► Model C
└──────► Model D
│
▼
Heart Rate
The key hypothesis is:
No single rPPG architecture is optimal across every optical and motion condition.
PRISM therefore attempts to exploit the complementary strengths of multiple specialist models through physics-aware routing.
The deployed PRISM interface provides a web-based workflow for physiological analysis.
Potential interface components include:
- Video acquisition
- Facial-region visualization
- Real-time scanning indicator
- Heart-rate estimation
- Physiological waveform visualization
- Model-selection information
- Analysis summary
PRISM processes facial video, which may contain sensitive biometric information.
Deployments should therefore consider:
- Secure transport using HTTPS
- Authentication and authorization
- Encryption of stored data
- Secure deletion of uploaded videos
- Access control
- Appropriate retention policies
- Protection of model and infrastructure endpoints
Do not store or transmit facial video unless required by the application and permitted by the applicable privacy and research policies.
PRISM is a research system for contactless physiological estimation.
Heart-rate estimates generated by rPPG models can be affected by:
- Lighting conditions
- Camera quality
- Facial movement
- Skin appearance
- Occlusion
- Video compression
- Background interference
- Camera exposure
- Subject distance
- Physiological conditions
The system should not be considered a medical diagnostic device unless it has undergone the appropriate clinical validation, regulatory review, and certification required for its intended use.
Developed at the:
Department of Computer Science and Engineering Birla Institute of Technology, Mesra
- Tanishk Singh Thakur
- Yash Rai
- Aryan Anand Nahata
This repository contains the implementation associated with the research work:
System and Method for Robust Contactless Heart-Rate Estimation Using Adaptive rPPG Model Selection
The implementation focuses on physics-aware dynamic model selection for robust contactless heart-rate estimation.
Important: This repository may contain proprietary research and intellectual property associated with the corresponding invention disclosure.
Unauthorized possession, viewing, reproduction, redistribution, or commercial use may be prohibited.
Please refer to the applicable institutional policies, agreements, and intellectual-property documentation before sharing or distributing this repository.
- 🎥 Contactless heart-rate estimation
- 🧠 Physics-aware meta-routing
- 🔀 Dynamic rPPG model selection
- 📊 10-dimensional physics state representation
- 🏃 Motion-aware processing
- 💡 Illumination-aware routing
- 🧩 Plug-and-play model ensemble
- ⚡ Modular inference architecture
- 🐳 Dockerized deployment
- ⚛️ React + Vite frontend
- 🚀 FastAPI backend
- 🤖 MLP-based model-selection layer
Potential future improvements include:
- Real-time streaming inference
- Larger and more diverse training datasets
- Additional rPPG specialist models
- Adaptive ensemble selection
- Personalized routing
- GPU-optimized inference
- Edge-device deployment
- Improved motion compensation
- Robustness against extreme illumination changes
- Multi-vital-sign estimation
- Clinical validation
This project is currently intended for research and evaluation purposes.
License terms should be added here once the appropriate institutional or project license has been finalized.
PRISM
Physics-Robust Intelligent Selection Mechanism for rPPG