An AI-based image captioning system that automatically generates natural-language descriptions of images using deep learning and transformer-based architectures.
Project Domain: Artificial Intelligence and Deep Learning
Degree: B.Tech – Artificial Intelligence & Data Science
Dataset: Flickr8k
SDG: 10 – Reduced Inequalities
The AI-Powered Image Captioning Tool aims to generate meaningful textual descriptions of images automatically.
The project implements two image captioning approaches:
- Custom CNN-Transformer Model: A ResNet50 CNN encoder combined with a custom Transformer decoder, trained using the Flickr8k dataset.
- Pretrained BLIP Model: Salesforce BLIP Image Captioning Base, used as a second captioning approach for comparison.
The system provides an interactive web interface where users can upload images, select a model and generate captions.
Visually impaired users and content platforms need automatic, accurate descriptions of image content, which manual captioning cannot scale to provide.
This project addresses the problem by developing an AI-powered system capable of converting visual information into natural-language descriptions.
- Develop an automated image caption generation system.
- Implement a custom CNN-Transformer architecture.
- Integrate a pretrained BLIP image captioning model.
- Generate captions using Greedy Search and Beam Search.
- Compare generated captions from two different model architectures.
- Provide an interactive application for real-time image captioning.
- Demonstrate the system using a collection of 100 validation images.
The custom image captioning model consists of two major components.
CNN Encoder – ResNet50
- Uses ImageNet-pretrained ResNet50 weights.
- Extracts spatial image features.
- Produces a feature map of size 2048 × 7 × 7.
- Converts the feature map into 49 visual tokens.
- CNN parameters remain frozen during the custom decoder training.
Transformer Decoder
- Embedding dimension: 512
- Attention heads: 8
- Transformer decoder layers: 3
- Feed-forward dimension: 2048
- Dropout: 0.1
- Maximum caption length: 40 tokens
- Uses positional encoding and attention masks.
The decoder predicts words sequentially based on the extracted image features.
The second model uses Salesforce BLIP Image Captioning Base.
- Pretrained vision-language architecture.
- Uses a pretrained image processor.
- Generates captions through transformer-based generation.
- Uses beam-based generation with 5 beams.
Note: BLIP is used as a pretrained comparison model. It was not trained from scratch as part of this project.
Flickr8k Dataset
The project uses Flickr8k, an image-captioning dataset containing photographs paired with multiple human-written captions.
Dataset split:
- Training: 6,000 images
- Validation: 1,000 images
- Testing: 1,000 images
The dataset contains approximately 40,460 caption annotations.
The validation set is also used to select 100 images for the interactive demonstration gallery.
The custom CNN-Transformer model was trained using the Flickr8k dataset.
Key implementation details:
- Image preprocessing and resizing to 224 × 224 pixels.
- Image feature extraction using ResNet50.
- Tokenization and vocabulary construction.
- Vocabulary size: 7,594.
- Special tokens:
<pad>,<unk>,<start>,<end>. - Transformer decoder training over 5 epochs.
- Best checkpoint selected at epoch 4.
- Best validation loss: 3.1384.
The trained custom model checkpoint is saved separately from the source code.
The custom model was evaluated using BLEU scores and METEOR.
| Evaluation Metric | Greedy Search | Beam Search |
|---|---|---|
| BLEU-1 | 0.5939 | 0.6188 |
| BLEU-2 | 0.4122 | 0.4326 |
| BLEU-3 | 0.2762 | 0.2953 |
| BLEU-4 | 0.1831 | 0.1981 |
| METEOR | 0.4052 | 0.4068 |
Beam Search achieved better scores across all reported metrics.
Important: These are the previously obtained custom model test-set results. They should not be interpreted as quantitative BLIP-versus-custom-model evaluation results.
- Upload an image.
- Select Custom CNN-Transformer or Pretrained BLIP.
- Generate an automatic caption.
- View the generated description.
- Browse 100 selected validation images.
- Switch between the two captioning models.
- View previously generated captions.
- Navigate using Previous and Next controls.
The demonstration gallery uses saved caption results and is intended for project presentation and demonstration.
- Python
- PyTorch
- Torchvision
- Transformers
- ResNet50
- Transformer Decoder
- Salesforce BLIP
- Hugging Face
- NLTK
- Gradio
- Google Colab
- Flickr8k Dataset
AI-Powered-Image-Captioning-Tool/
│
├── README.md
│
├── AI_Powered_Image_Captioning_Final_Project.ipynb
│
├── run_project.py
│
├── model_comparison.csv
│
├── custom_model_captions.csv
│
└── blip_model_captions.csv
The trained model checkpoint and demonstration images are maintained separately in Google Drive.
- Google Colab
- NVIDIA T4 GPU
- Google Drive
- Python with PyTorch and Torchvision
- Open the project notebook in Google Colab.
- Select T4 GPU under Runtime settings.
- Connect Google Drive.
- Ensure the trained checkpoint is available at:
MyDrive/
└── AI_Powered_Image_Captioning_Final_Project/
├── checkpoints/
│ └── best_model.pth
├── run_project.py
├── demonstration_images/
└── results/
└── demonstration_gallery/
├── custom_model_captions.csv
└── blip_model_captions.csv
- Execute the final Run Complete Project cell in the notebook.
- Authorize Google Drive access.
- Wait for the two Gradio application links.
- Open the live captioning application or demonstration gallery.
Model training is not required during normal application execution.
Note: The current execution program uses the Google Drive directory above. Anyone running the project independently must provide the checkpoint, demonstration images and caption CSV files in the expected locations or adapt the paths in run_project.py.
- The custom model was trained on the relatively small Flickr8k dataset.
- Its performance may vary on images that differ significantly from the training distribution.
- Generated captions may occasionally omit details or describe objects inaccurately.
- The BLIP model is pretrained and is not fine-tuned specifically for this project.
- Caption generation quality depends on the visual features and language representations learned by each model.
- Train on larger image-caption datasets.
- Improve custom model generalization.
- Explore attention visualization.
- Add text-to-speech functionality for visually impaired users.
- Develop a mobile-friendly application.
- Improve caption accuracy and contextual understanding.
- Explore multilingual caption generation.
This project aligns with United Nations Sustainable Development Goal 10 – Reduced Inequalities.
By converting images into textual descriptions, the system explores a practical way to improve access to visual information, particularly for people who cannot easily interpret images visually.
- Flickr8k Dataset.
- PyTorch and Torchvision.
- Hugging Face Transformers.
- Salesforce BLIP.
- Google Colab.
- Gradio.
Project: AI-Powered Image Captioning Tool
Repository: GitHub – AI-Powered Image Captioning Tool