Skip to content

About

Physics-informed rPPG framework for contactless heart-rate estimation using adaptive ML-based selection across 8 specialized models, achieving 1.08 BPM MAE on PURE.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

title Prism Rppg
emoji 🐢
colorFrom purple
colorTo yellow
sdk docker
pinned false

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference Absolutely — here is a clean, GitHub-ready README.md version with the citation artifacts removed and the Markdown/code formatting fixed.

PRISM: Physics-Robust Intelligent Selection Mechanism for rPPG

PRISM is the deployed implementation of the research work: "System and Method for Robust Contactless Heart-Rate Estimation Using Adaptive rPPG Model Selection."

image

📖 Overview

Cardiovascular disease remains a major global health challenge, creating a need for continuous, accessible, and non-invasive physiological monitoring.

Traditional Photoplethysmography (PPG) provides reliable heart-rate monitoring but requires direct skin contact. This can cause discomfort, create hygiene concerns, and limit its applicability in populations such as neonates and other vulnerable patients.

Remote Photoplethysmography (rPPG) addresses these limitations by enabling completely contactless physiological monitoring using standard RGB cameras.

However, extracting the weak blood-volume pulse signal from facial video is challenging. rPPG systems must operate in the presence of:

  • Dynamic illumination
  • Facial motion
  • Camera noise
  • Video compression
  • Blur
  • Skin-tone variations
  • Specular reflections
  • Low photon density

Most existing deep-learning rPPG systems rely on a single fixed physiological estimation model. These monolithic approaches can perform well under familiar conditions but may experience significant performance degradation when exposed to out-of-distribution environments.

PRISM's Approach

PRISM addresses this limitation through an intelligent, physics-aware meta-routing layer.

Instead of relying on one model for every video, PRISM:

  1. Analyzes the physical characteristics of the input video.
  2. Extracts a compact physics-based feature representation.
  3. Predicts the expected error of multiple specialist rPPG models.
  4. Dynamically selects the model expected to produce the most accurate heart-rate estimate.
  5. Routes the video to only the selected specialist model.

This transforms conventional single-model physiological estimation into a context-aware dynamic model selection system.


🧠 Core Architecture

image

The PRISM framework performs physiological estimation through a sequence of physics-aware routing stages.

1. Video Acquisition

The system receives a 30-second facial video sequence captured using an RGB camera.

The input video contains the subtle color variations caused by changes in blood volume beneath the skin.


2. Facial Region Detection & Stabilization

Facial regions are detected from the incoming video.

To reduce the effects of bounding-box jitter and high-frequency detection noise, PRISM applies temporal stabilization using a Kalman filter operating under a constant-velocity motion model.

This produces a more temporally consistent facial region for subsequent feature extraction and inference.


3. Physics-Based Feature Extraction

The stabilized video is passed to the FeatureEngine.

The FeatureEngine extracts a compact 10-dimensional physics state vector describing the optical and motion characteristics of the input sequence.

This representation provides the routing mechanism with information about the environment in which each candidate rPPG model must operate.


4. Cost-Sensitive Regret Minimization

The physics state vector is passed to an MLP-based meta-regressor.

Rather than directly classifying the video into a model category, PRISM predicts the expected log-space Mean Absolute Error (MAE) of each candidate rPPG model.

The optimal model is then selected using:

$$ i^* = \arg\min_i \log(\mathrm{MAE}_i) $$

This converts model selection into an error-minimization problem.


5. Dynamic Model Selection

PRISM selects the specialist model predicted to have the lowest estimation error.

Models that are likely to perform poorly under the current optical and motion conditions can therefore be bypassed.

This prevents the system from unnecessarily executing a vulnerable architecture.


6. Physiological Signal Estimation

The selected specialist model processes the input video and produces the final heart-rate estimation.

The architecture is modular, allowing additional rPPG models to be integrated into the ensemble without redesigning the entire routing system.


🔬 The 10-Dimensional Physics State Vector

PRISM's routing decision is based on six fundamental optical/motion primitives and four derived non-linear interaction features.

Optical Primitives

Feature Description
Normalized Photon Density ($\rho$) Represents the available optical signal intensity
Environmental Noise Floor ($\sigma_n$) Estimates background and sensor noise
Skin Tone Intensity ($\mu_{Skin}$) Represents average facial skin intensity
Texture Sharpness ($\tau$) Measures spatial facial texture and image sharpness
Motion Magnitude ($\Delta m$) Quantifies facial/head movement
Spectral Concentration ($\Omega$) Measures concentration of signal energy within relevant frequency bands

Interaction Meta-Features

Effective Signal-to-Noise Ratio

$$ ESNR = \frac{\rho}{\sigma_n + \Delta m} $$

Photometric Stability

Measures the temporal stability of facial illumination and pixel intensity.

Motion Derivative

$$ \frac{d(\Delta m)}{dt} $$

Captures changes in motion intensity over time.

Spectral Purity Score

Measures the relative dominance of physiological frequency components compared with competing spectral energy.

Together, these features provide a compact representation of the optical state, motion state, and signal quality of the input video.


🧩 Candidate Model Ensemble

PRISM is designed to be model-agnostic and plug-and-play.

The routing layer can select from multiple specialist rPPG architectures.

Convolutional Neural Networks

PhysNet

A convolutional rPPG architecture designed for learning spatial and temporal physiological representations.

EfficientPhys

A computationally efficient architecture suitable for environments where inference efficiency is important.

CNN-based models can perform particularly well under stable, low-entropy visual conditions but may become vulnerable to strong motion or illumination changes.


Vision Transformers

PhysFormer

A transformer-based physiological signal estimation architecture capable of modeling long-range temporal and spatial relationships.

PhysFormer can achieve strong performance in well-illuminated environments but may become less reliable under photon-starved or heavily degraded conditions.


State Space Models

RhythmMamba

A state-space-based architecture designed for efficient temporal physiological signal modeling.

PhysMamba

A Mamba-based rPPG architecture that provides linear-time sequence modeling capabilities.

State-space models can provide strong temporal modeling efficiency but may be sensitive to abrupt photometric changes depending on the input conditions.


📊 Performance & Evaluation

PRISM was evaluated using established rPPG benchmark datasets, including:

  • UBFC-rPPG
  • PURE

The evaluation focuses on the ability of the routing mechanism to select appropriate specialist models under varying environmental conditions.

Stress Testing

Under severe environmental degradation, including:

  • Synthetic illumination shifts
  • Blur
  • Video compression
  • Sensor noise

PRISM achieved an MAE of approximately:

1.81 BPM

This outperformed the best fixed-model configuration, which achieved approximately:

1.99 BPM


Motion Resilience

On the highly dynamic PURE P09-02 sequence, several static model configurations produced errors exceeding 30 BPM.

PRISM dynamically selected a specialist estimator and reduced the error to approximately:

1.74 BPM

These results demonstrate the potential advantage of dynamic model selection over a single fixed rPPG architecture, particularly under challenging environmental conditions.

image image

🏗️ Repository Structure

prism-rppg/
│
├── ai_models/                       # Core AI and PRISM logic
│   └── prism/
│       ├── feature_engine.py        # Extracts the 10D physics vector
│       └── router.py                # MLP-based dynamic routing model
│
├── backend/                         # FastAPI backend
│   ├── app/
│   │   ├── main.py                  # FastAPI application entry point
│   │   └── routers/
│   │       └── video.py              # Video processing endpoints
│   │
│   └── requirements.txt              # Backend dependencies
│
├── frontend/                        # React + Vite web application
│   ├── src/
│   │   ├── components/              # UI components
│   │   │   ├── CircularScanner
│   │   │   └── VitalsChart
│   │   │
│   │   └── pages/                   # Application pages
│   │       ├── Index
│   │       └── Analysis
│   │
│   └── package.json
│
├── hfspaces/                        # Integrated model implementations
│
├── rppg/                            # Specialist rPPG models
│
├── Dockerfile.backend               # Backend container
├── Dockerfile.worker                # AI worker container
│
├── deploy_backend.ps1               # Backend deployment script
├── deploy_worker.ps1                # Worker deployment script
│
└── README.md


---

# 🚀 Getting Started

## Prerequisites

Before running PRISM locally, make sure the following are installed:

* [Docker](https://www.docker.com/)
* Docker Compose
* [Node.js](https://nodejs.org/)
* Python 3.10+
* Git

---

# ⚙️ Installation

Clone the repository:

```bash
git clone <YOUR_REPOSITORY_URL>
cd prism-rppg

🐍 Backend Setup

Create and activate a Python virtual environment:

python -m venv .venv

Windows

.venv\Scripts\activate

Linux / macOS

source .venv/bin/activate

Install backend dependencies:

pip install -r backend/requirements.txt

Run the FastAPI backend:

uvicorn backend.app.main:app --reload

The backend should then be available at:

http://localhost:8000

🌐 Frontend Setup

Navigate to the frontend directory:

cd frontend

Install dependencies:

npm install

Start the development server:

npm run dev

The Vite development server will provide a local URL, typically:

http://localhost:5173

🐳 Docker Deployment

PRISM provides separate containers for the backend and AI inference worker.

Backend

Build the backend image:

docker build -f Dockerfile.backend -t prism-backend .

Run the backend:

docker run -p 8000:8000 prism-backend

AI Worker

Build the worker image:

docker build -f Dockerfile.worker -t prism-worker .

Run the worker:

docker run prism-worker

🪟 PowerShell Deployment

For supported Windows environments, the repository includes deployment scripts.

Backend

./deploy_backend.ps1

AI Worker

./deploy_worker.ps1

Make sure Docker is running before executing the deployment scripts.


🔌 Extending PRISM With New Models

One of the primary design goals of PRISM is modularity.

A new rPPG architecture can be integrated into the ensemble without fundamentally changing the routing mechanism.

A typical integration involves:

  1. Implementing the model inference interface.
  2. Adding the model to the candidate ensemble.
  3. Generating evaluation/error data for the new model.
  4. Updating the router training dataset.
  5. Retraining the meta-regressor.
  6. Registering the model with the inference worker.

Conceptually:

                    ┌─────────────────────┐
                    │     Input Video     │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Face Detection &    │
                    │ Stabilization       │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Physics Feature     │
                    │ Engine               │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │   PRISM Meta-Router │
                    └──────────┬──────────┘
                               │
               ┌───────────────┼───────────────┐
               │               │               │
               ▼               ▼               ▼
          ┌─────────┐     ┌───────────┐   ┌───────────┐
          │ PhysNet │     │ PhysFormer│   │ PhysMamba │
          └────┬────┘     └─────┬─────┘   └─────┬─────┘
               │                │               │
               └────────────────┼───────────────┘
                                │
                                ▼
                       ┌─────────────────┐
                       │ Heart Rate      │
                       │ Estimation      │
                       └─────────────────┘

📈 Evaluation Metrics

The primary evaluation metric is Mean Absolute Error (MAE) in beats per minute.

$$ MAE = \frac{1}{N}\sum_{i=1}^{N}|HR_i-\hat{HR}_i| $$

where:

  • $HR_i$ is the ground-truth heart rate.
  • $\hat{HR}_i$ is the predicted heart rate.
  • $N$ is the number of evaluated samples.

Additional evaluation can include:

  • Mean Absolute Error (MAE)
  • Root Mean Square Error (RMSE)
  • Pearson Correlation
  • Signal-to-Noise Ratio (SNR)
  • Failure rate
  • Model-selection accuracy
  • Routing regret
  • Inference latency

🔬 Research Motivation

Traditional rPPG pipelines generally follow the structure:

Video
  │
  ▼
Single rPPG Model
  │
  ▼
Heart Rate

PRISM instead introduces an adaptive decision layer:

Video
  │
  ▼
Physics Analysis
  │
  ▼
Meta-Router
  │
  ├──────► Model A
  ├──────► Model B
  ├──────► Model C
  └──────► Model D
             │
             ▼
        Heart Rate

The key hypothesis is:

No single rPPG architecture is optimal across every optical and motion condition.

PRISM therefore attempts to exploit the complementary strengths of multiple specialist models through physics-aware routing.


🖥️ Web Interface

The deployed PRISM interface provides a web-based workflow for physiological analysis.

image image

Potential interface components include:

  • Video acquisition
  • Facial-region visualization
  • Real-time scanning indicator
  • Heart-rate estimation
  • Physiological waveform visualization
  • Model-selection information
  • Analysis summary

🔒 Security & Privacy

PRISM processes facial video, which may contain sensitive biometric information.

Deployments should therefore consider:

  • Secure transport using HTTPS
  • Authentication and authorization
  • Encryption of stored data
  • Secure deletion of uploaded videos
  • Access control
  • Appropriate retention policies
  • Protection of model and infrastructure endpoints

Do not store or transmit facial video unless required by the application and permitted by the applicable privacy and research policies.


⚠️ Research Disclaimer

PRISM is a research system for contactless physiological estimation.

Heart-rate estimates generated by rPPG models can be affected by:

  • Lighting conditions
  • Camera quality
  • Facial movement
  • Skin appearance
  • Occlusion
  • Video compression
  • Background interference
  • Camera exposure
  • Subject distance
  • Physiological conditions

The system should not be considered a medical diagnostic device unless it has undergone the appropriate clinical validation, regulatory review, and certification required for its intended use.


👥 Inventors & Research Team

Developed at the:

Department of Computer Science and Engineering Birla Institute of Technology, Mesra

Contributors

  • Tanishk Singh Thakur
  • Yash Rai
  • Aryan Anand Nahata

📄 Research

This repository contains the implementation associated with the research work:

System and Method for Robust Contactless Heart-Rate Estimation Using Adaptive rPPG Model Selection

The implementation focuses on physics-aware dynamic model selection for robust contactless heart-rate estimation.


🔐 Confidentiality Notice

Important: This repository may contain proprietary research and intellectual property associated with the corresponding invention disclosure.

Unauthorized possession, viewing, reproduction, redistribution, or commercial use may be prohibited.

Please refer to the applicable institutional policies, agreements, and intellectual-property documentation before sharing or distributing this repository.


⭐ Key Features

  • 🎥 Contactless heart-rate estimation
  • 🧠 Physics-aware meta-routing
  • 🔀 Dynamic rPPG model selection
  • 📊 10-dimensional physics state representation
  • 🏃 Motion-aware processing
  • 💡 Illumination-aware routing
  • 🧩 Plug-and-play model ensemble
  • ⚡ Modular inference architecture
  • 🐳 Dockerized deployment
  • ⚛️ React + Vite frontend
  • 🚀 FastAPI backend
  • 🤖 MLP-based model-selection layer

🛣️ Future Work

Potential future improvements include:

  • Real-time streaming inference
  • Larger and more diverse training datasets
  • Additional rPPG specialist models
  • Adaptive ensemble selection
  • Personalized routing
  • GPU-optimized inference
  • Edge-device deployment
  • Improved motion compensation
  • Robustness against extreme illumination changes
  • Multi-vital-sign estimation
  • Clinical validation

📜 License

This project is currently intended for research and evaluation purposes.

License terms should be added here once the appropriate institutional or project license has been finalized.


PRISM
Physics-Robust Intelligent Selection Mechanism for rPPG

```

About

Physics-informed rPPG framework for contactless heart-rate estimation using adaptive ML-based selection across 8 specialized models, achieving 1.08 BPM MAE on PURE.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages