Region-Aware Dual-Encoder Fusion for Medical VLP

This repository contains the implementation of a novel, 3-phase Vision-Language Pre-training (VLP) pipeline designed for specialized medical imaging. By utilizing a Dual-Encoder architecture and Knowledge Distillation, this model learns to capture both global context and fine-grained pathological features.

🚀 Project Overview

Standard models like CLIP often fail in medical contexts because they treat images as a single global entity. Our approach solves this by:

1. Extracting Regions of Interest (ROI): Using YOLO and bounding box analysis to isolate specific pathologies.

2. Specialized Encoding: Training two separate encoders for "Global Context" and "Local Features."

3. Attention-Based Fusion: Using a transformer-based attention module to intelligently combine these features for superior image-text retrieval.

🏗️ The 3-Phase Pipeline

Phase 1 & 2: Dual-Encoder Distillation

We utilize a Teacher-Student framework.

• Teacher: BiomedCLIP (Microsoft), providing expert medical domain knowledge.

• Student: OpenCLIP (ViT-B-32), initialized with general-domain weights.

• Method: We perform knowledge distillation using a combined InfoNCE Contrastive Loss and MSE Distillation Loss. This "pulls" the student's embedding space toward the expert teacher's space while maintaining contrastive alignment.

Phase 3: Attention-Based Fusion

After pre-training the encoders, we freeze them and introduce a novel Attention Fusion Module.

• Architecture: A multi-head attention mechanism that treats the global and local embeddings as tokens. It learns to weight which features (contextual vs. pathological) are more relevant for a specific sample.

• Trainable Parameters: Only the fusion head (~2.1M parameters) is trained in this phase to prevent catastrophic forgetting in the encoders.

📊 Results: The Final Showdown

Our model was evaluated on a held-out set of specialist dermatological samples. We compared our Novel Fused Model against a Zero-Shot Baseline (untrained OpenCLIP).

**ModelR@1 Retrieval (Avg)ImprovementBaseline (Untrained OpenCLIP)**0.0069-Our Fused Model (Final)0.02083.0x (200%)

Visualization

Our model successfully learned to separate matching image-text pairs from mismatched ones.

• Similarity Heatmaps: Show a clearly defined diagonal compared to the noisy baseline.

• Distribution Analysis: Demonstrates that the model "pushes" incorrect pairs toward 0.0 similarity while pulling correct pairs toward a higher discriminative score.

💻 Technical Requirements

• GPU: NVIDIA T4 (16GB VRAM) or better.

• Libraries: torch, open_clip, seaborn, tqdm.

• Framework: Built on the OpenCLIP architecture and BiomedCLIP teacher weights.

📜 References

• Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP)

• Zhang et al., "BiomedCLIP: A Multimodal Biomedical Foundation Model Optimized for Speed and Purposed for Precision"

• Implementation inspired by region-aware VLP research for fine-grained medical alignment.