NanoVLM

NanoVLM is a lightweight Vision-Language Model (VLM) designed for efficient multimodal understanding. It combines image encoding and language generation to answer questions about images while remaining suitable for resource-constrained environments.

Features

• 🖼️ Image understanding
• 💬 Visual Question Answering (VQA)
• ⚡ Lightweight architecture
• 🤗 Hugging Face compatible
• 📓 Training and inference notebook included

Model Architecture

The model consists of:

• Vision Encoder
• Projection Layer
• Language Model
• Cross-modal fusion between visual and textual representations

Image
   │
Vision Encoder
   │
Projection Layer
   │
Language Model
   │
Generated Answer

Installation

Clone the repository

git clone https://github.com/your-username/nanovlm.git
cd nanovlm

Install dependencies

pip install -r requirements.txt

Usage

Inference

from PIL import Image

image = Image.open("example.jpg")

question = "What is in this image?"

answer = model.chat(image, question)

print(answer)

Notebook

Open the notebook:

NanoVlm_fixed.ipynb

and execute all cells.

Dataset

The model can be trained on datasets such as:

• COCO Captions
• VQAv2
• Flickr30k
• Custom datasets

Training

Example:

python train.py

Adjust hyperparameters such as:

• Learning rate
• Batch size
• Number of epochs
• Image resolution

Results

Metric	Value
Accuracy	--
BLEU	--
CIDEr	--

(Replace with your experimental results.)

Repository Structure

.
├── NanoVlm_fixed.ipynb
├── app.py
├── requirements.txt
├── README.md
├── images/
└── model/

Requirements

• Python 3.10+
• PyTorch
• Transformers
• Pillow
• Torchvision
• Gradio (optional)

Install:

pip install torch torchvision transformers pillow gradio

Citation

If you use this project, please cite:

@misc{nanovlm2026,
  title={NanoVLM: A Lightweight Vision-Language Model},
  author={Your Name},
  year={2026},
  publisher={Hugging Face}
}

License

MIT License

Acknowledgements

This project builds upon the excellent work of:

• Hugging Face Transformers
• PyTorch
• Vision Transformer (ViT)
• Large Language Models research

---

⭐ If you find this project useful, consider giving it a star on GitHub or liking it on Hugging Face.