πŸ‘οΈ RetinalGPT

Advancing Clinical Ophthalmology Through Instruction-Tuned Vision–Language Intelligence

Code Paper

RetinalGPT is a vision–language assistant for color fundus photographs (CFPs). Given a fundus image, it holds a multi-turn conversation covering image quality, retinal disease assessment, lesion localization, and quantitative vascular analysis (fractal dimension, vessel density, and vessel width for the overall, arterial, and venous vasculature).

Multi-turn conversation with RetinalGPT

πŸ”” News

  • [2026.04] RetinalGPT was submitted to Ophthalmology Science and is under review.
  • [2025.03] Preprint released on arXiv.

Model Summary

Architecture LLaVA-Med (v1) style: CLIP ViT-L/14 vision encoder β†’ linear projector β†’ LLaMA-7B language model
Input One color fundus photograph + an English text question
Output English text (findings, diagnosis, bounding boxes in normalized [x_min, y_min, x_max, y_max], vascular metrics)
Training data Curated CFPs from public retinal datasets (APTOS, EyePACS, Messidor, IDRiD, ODIR, RFMiD, MMAC) + LLaVA-Med biomedical data
Developed by Geometry Systems Lab, Arizona State University, with Mayo Clinic and collaborators
Paper arXiv:2503.03987
Code github.com/Retinal-Research/RetinalGPT

πŸš€ How to Use

RetinalGPT uses the LLaVA-Med v1 model class, so load it with the code in the GitHub repo rather than the stock transformers pipeline. A CUDA GPU is required (~16 GB VRAM in fp16).

1. Install

git clone https://github.com/Retinal-Research/RetinalGPT.git
cd RetinalGPT

conda create -n retinalgpt python=3.10 -y
conda activate retinalgpt
pip install torch torchvision
pip install "transformers @ git+https://github.com/huggingface/transformers.git@cae78c46" tokenizers==0.12.1 sentencepiece accelerate
pip install -r requirements.txt

2. Download the weights

hf download ASU-GSL/RetinalGPT --local-dir ./checkpoints/RetinalGPT

3. Single-image inference

python run_retinalGPT_simple.py \
  --model-name ./checkpoints/RetinalGPT \
  --image-file /path/to/fundus.png \
  --question "Please describe this retinal image in detail."

4. Batch inference

python run_retinalGPT.py \
  --model-name ./checkpoints/RetinalGPT \
  --image-folder /path/to/images \
  --question-file examples/inference/questions.json \
  --answers-file outputs/predictions.jsonl

questions.json is a list of {"id", "image", "question"} items (or "questions": [...] for several questions per image). Answers are written as JSON lines.

5. Python

Run from the root of the GitHub repo:

import torch
from PIL import Image
from transformers import AutoTokenizer, CLIPImageProcessor

from llava import LlavaLlamaForCausalLM
from llava.conversation import conv_templates
from llava.utils import disable_torch_init

IMAGE_PATCH, IM_START, IM_END = "<im_patch>", "<im_start>", "<im_end>"
model_path = "./checkpoints/RetinalGPT"

# Load model, tokenizer, and image processor
disable_torch_init()
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = LlavaLlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16).cuda().eval()
image_processor = CLIPImageProcessor.from_pretrained(model.config.mm_vision_tower)

use_im_start_end = getattr(model.config, "mm_use_im_start_end", False)
tokenizer.add_tokens([IMAGE_PATCH], special_tokens=True)
if use_im_start_end:
    tokenizer.add_tokens([IM_START, IM_END], special_tokens=True)

vision_tower = model.model.vision_tower[0].to(device="cuda", dtype=torch.float16)
vcfg = vision_tower.config
vcfg.im_patch_token = tokenizer.convert_tokens_to_ids(IMAGE_PATCH)
vcfg.use_im_start_end = use_im_start_end
if use_im_start_end:
    vcfg.im_start_token, vcfg.im_end_token = tokenizer.convert_tokens_to_ids([IM_START, IM_END])
n_patches = (vcfg.image_size // vcfg.patch_size) ** 2

# Build the prompt
question = "Can you tell me about the vascular analysis results?"
image_tokens = IMAGE_PATCH * n_patches
if use_im_start_end:
    image_tokens = IM_START + image_tokens + IM_END
conv = conv_templates["simple"].copy()
conv.append_message(conv.roles[0], question + "\n" + image_tokens)
prompt = conv.get_prompt()

# Generate
image = Image.open("/path/to/fundus.png").convert("RGB")
pixel_values = image_processor.preprocess(image, return_tensors="pt")["pixel_values"].half().cuda()
input_ids = tokenizer([prompt], return_tensors="pt").input_ids.cuda()

with torch.inference_mode():
    output_ids = model.generate(input_ids, images=pixel_values, do_sample=True,
                                temperature=0.2, max_new_tokens=256)

answer = tokenizer.decode(output_ids[0, input_ids.shape[1]:], skip_special_tokens=True)
print(answer.split(conv.sep)[0].strip())

Example questions

  • What can you tell me about the image characteristics?
  • Does the image show any diseases or abnormalities?
  • Where are the lesions located?
  • Can you give me vascular details about the veins?
  • What about the analysis in Zone B?

πŸ”¬ Training

RetinalGPT is trained in two stages on LLaVA-Med-style infrastructure:

  1. Medical feature alignment β€” vision encoder and LLM frozen; the projector is trained on 38K Retinal Concept Alignment (RCA) QA pairs mixed with 600K PMC-15M image–text pairs.
  2. Clinical instruction tuning β€” vision encoder frozen; projector and LLM are trained on 38K Retinal Concept Tuning (RCT) multi-turn dialogues mixed with 60K general biomedical QA pairs from LLaVA-Med.

Instruction data is generated with GPT-4 from structured clinical features: disease labels, AutoMorph/RBAD vascular measurements, image-quality grades, and lesion bounding boxes.

Data construction pipeline
Two-stage training

πŸ–ΌοΈ Examples

Lesion localization and vascular analysis

RetinalGPT is released for research purposes.

πŸ“– Citation

@article{zhu2025retinalgpt,
  title={RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models},
  author={Zhu, Wenhui and Li, Xin and Chen, Xiwen and Qiu, Peijie and Vasa, Vamsi Krishna and Dong, Xuanzhao and Chen, Yanxi and Lepore, Natasha and Dumitrascu, Oana and Su, Yi and Wang, Yalin},
  journal={arXiv preprint arXiv:2503.03987},
  year={2025}
}

πŸ™ Acknowledgement

Built on LLaVA and LLaVA-Med. Vascular supervision comes from AutoMorph.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Paper for ASU-GSL/RetinalGPT