Instructions to use ASU-GSL/RetinalGPT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ASU-GSL/RetinalGPT with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="ASU-GSL/RetinalGPT")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ASU-GSL/RetinalGPT", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ASU-GSL/RetinalGPT with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ASU-GSL/RetinalGPT" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ASU-GSL/RetinalGPT", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/ASU-GSL/RetinalGPT
- SGLang
How to use ASU-GSL/RetinalGPT with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ASU-GSL/RetinalGPT" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ASU-GSL/RetinalGPT", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ASU-GSL/RetinalGPT" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ASU-GSL/RetinalGPT", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use ASU-GSL/RetinalGPT with Docker Model Runner:
docker model run hf.co/ASU-GSL/RetinalGPT
ποΈ RetinalGPT
Advancing Clinical Ophthalmology Through Instruction-Tuned VisionβLanguage Intelligence
RetinalGPT is a visionβlanguage assistant for color fundus photographs (CFPs). Given a fundus image, it holds a multi-turn conversation covering image quality, retinal disease assessment, lesion localization, and quantitative vascular analysis (fractal dimension, vessel density, and vessel width for the overall, arterial, and venous vasculature).
π News
- [2026.04] RetinalGPT was submitted to Ophthalmology Science and is under review.
- [2025.03] Preprint released on arXiv.
Model Summary
| Architecture | LLaVA-Med (v1) style: CLIP ViT-L/14 vision encoder β linear projector β LLaMA-7B language model |
| Input | One color fundus photograph + an English text question |
| Output | English text (findings, diagnosis, bounding boxes in normalized [x_min, y_min, x_max, y_max], vascular metrics) |
| Training data | Curated CFPs from public retinal datasets (APTOS, EyePACS, Messidor, IDRiD, ODIR, RFMiD, MMAC) + LLaVA-Med biomedical data |
| Developed by | Geometry Systems Lab, Arizona State University, with Mayo Clinic and collaborators |
| Paper | arXiv:2503.03987 |
| Code | github.com/Retinal-Research/RetinalGPT |
π How to Use
RetinalGPT uses the LLaVA-Med v1 model class, so load it with the code in the GitHub repo rather than the stock transformers pipeline. A CUDA GPU is required (~16 GB VRAM in fp16).
1. Install
git clone https://github.com/Retinal-Research/RetinalGPT.git
cd RetinalGPT
conda create -n retinalgpt python=3.10 -y
conda activate retinalgpt
pip install torch torchvision
pip install "transformers @ git+https://github.com/huggingface/transformers.git@cae78c46" tokenizers==0.12.1 sentencepiece accelerate
pip install -r requirements.txt
2. Download the weights
hf download ASU-GSL/RetinalGPT --local-dir ./checkpoints/RetinalGPT
3. Single-image inference
python run_retinalGPT_simple.py \
--model-name ./checkpoints/RetinalGPT \
--image-file /path/to/fundus.png \
--question "Please describe this retinal image in detail."
4. Batch inference
python run_retinalGPT.py \
--model-name ./checkpoints/RetinalGPT \
--image-folder /path/to/images \
--question-file examples/inference/questions.json \
--answers-file outputs/predictions.jsonl
questions.json is a list of {"id", "image", "question"} items (or "questions": [...] for several questions per image). Answers are written as JSON lines.
5. Python
Run from the root of the GitHub repo:
import torch
from PIL import Image
from transformers import AutoTokenizer, CLIPImageProcessor
from llava import LlavaLlamaForCausalLM
from llava.conversation import conv_templates
from llava.utils import disable_torch_init
IMAGE_PATCH, IM_START, IM_END = "<im_patch>", "<im_start>", "<im_end>"
model_path = "./checkpoints/RetinalGPT"
# Load model, tokenizer, and image processor
disable_torch_init()
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = LlavaLlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16).cuda().eval()
image_processor = CLIPImageProcessor.from_pretrained(model.config.mm_vision_tower)
use_im_start_end = getattr(model.config, "mm_use_im_start_end", False)
tokenizer.add_tokens([IMAGE_PATCH], special_tokens=True)
if use_im_start_end:
tokenizer.add_tokens([IM_START, IM_END], special_tokens=True)
vision_tower = model.model.vision_tower[0].to(device="cuda", dtype=torch.float16)
vcfg = vision_tower.config
vcfg.im_patch_token = tokenizer.convert_tokens_to_ids(IMAGE_PATCH)
vcfg.use_im_start_end = use_im_start_end
if use_im_start_end:
vcfg.im_start_token, vcfg.im_end_token = tokenizer.convert_tokens_to_ids([IM_START, IM_END])
n_patches = (vcfg.image_size // vcfg.patch_size) ** 2
# Build the prompt
question = "Can you tell me about the vascular analysis results?"
image_tokens = IMAGE_PATCH * n_patches
if use_im_start_end:
image_tokens = IM_START + image_tokens + IM_END
conv = conv_templates["simple"].copy()
conv.append_message(conv.roles[0], question + "\n" + image_tokens)
prompt = conv.get_prompt()
# Generate
image = Image.open("/path/to/fundus.png").convert("RGB")
pixel_values = image_processor.preprocess(image, return_tensors="pt")["pixel_values"].half().cuda()
input_ids = tokenizer([prompt], return_tensors="pt").input_ids.cuda()
with torch.inference_mode():
output_ids = model.generate(input_ids, images=pixel_values, do_sample=True,
temperature=0.2, max_new_tokens=256)
answer = tokenizer.decode(output_ids[0, input_ids.shape[1]:], skip_special_tokens=True)
print(answer.split(conv.sep)[0].strip())
Example questions
- What can you tell me about the image characteristics?
- Does the image show any diseases or abnormalities?
- Where are the lesions located?
- Can you give me vascular details about the veins?
- What about the analysis in Zone B?
π¬ Training
RetinalGPT is trained in two stages on LLaVA-Med-style infrastructure:
- Medical feature alignment β vision encoder and LLM frozen; the projector is trained on 38K Retinal Concept Alignment (RCA) QA pairs mixed with 600K PMC-15M imageβtext pairs.
- Clinical instruction tuning β vision encoder frozen; projector and LLM are trained on 38K Retinal Concept Tuning (RCT) multi-turn dialogues mixed with 60K general biomedical QA pairs from LLaVA-Med.
Instruction data is generated with GPT-4 from structured clinical features: disease labels, AutoMorph/RBAD vascular measurements, image-quality grades, and lesion bounding boxes.
πΌοΈ Examples
RetinalGPT is released for research purposes.
π Citation
@article{zhu2025retinalgpt,
title={RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models},
author={Zhu, Wenhui and Li, Xin and Chen, Xiwen and Qiu, Peijie and Vasa, Vamsi Krishna and Dong, Xuanzhao and Chen, Yanxi and Lepore, Natasha and Dumitrascu, Oana and Su, Yi and Wang, Yalin},
journal={arXiv preprint arXiv:2503.03987},
year={2025}
}
π Acknowledgement
Built on LLaVA and LLaVA-Med. Vascular supervision comes from AutoMorph.