MiniCPM-V-4 · LiteRT INT8 (on-device, Snapdragon 8850 CPU)

MiniCPM-V-4-int8.litertlm — an on-device LiteRT-LM build of MiniCPM-V-4, quantized to INT8 (weight-only, GPTQ + Hadamard rotation) and packaged as a single .litertlm bundle for CPU inference on Snapdragon 8850.

Model

For the base model, architecture, capabilities and license, refer to the original model card:

https://huggingface.co/openbmb/MiniCPM-V-4

This repository only provides an edge-optimized LiteRT deployment of that model:

• Format: .litertlm (LiteRT-LM bundle: tokenizer + LLM prefill/decode + embedder + navit SigLIP vision encoder + resampler).
• Quantization: INT8 weight-only, per-channel, GPTQ + Hadamard rotation (activation-outlier smoothing).
• Vision: official multi-slice preprocessing (thumbnail + sub-tiles, 64 tokens per slice, up to 9 slices), navit SigLIP.

Performance (Snapdragon 8850, CPU backend)

Measured on-device (arm64 CPU, --backend=cpu), single image, 4-slice input:

Metric	Value
Prefill speed	40.9 tokens/sec
Decode speed	16.6 tokens/sec
Vision encode (4 slices)	~2.04 s
Time to first token	~7.0 s (284-token prefill incl. vision)
Init (model load + compile)	~2.2 s

Accuracy — MME benchmark (on-device, 8850 CPU)

Full MME evaluation running the INT8 bundle on-device:

Group	Score
Perception	1568
Cognition	491
Total	2059

Usage

Run with litert_lm_advanced_main

You can run multimodal inference using litert_lm_advanced_main from the LiteRT-LM repository. litert_lm_advanced_main supports inline media markers (such as [image:/path/to/image.png]) within --input_prompt.

1. Build the Runner

From the LiteRT-LM repository:

bazel build -c opt //runtime/engine:litert_lm_advanced_main

2. Run Inference

Using bazel run:

bazel run -c opt //runtime/engine:litert_lm_advanced_main -- \
  --backend=cpu \
  --vision_backend=cpu \
  --model_path=/path/to/MiniCPM-V-4-int8.litertlm \
  --input_prompt="What is in this image? [image:/path/to/image.png]"

Or execute the compiled binary directly:

./bazel-bin/runtime/engine/litert_lm_advanced_main \
  --backend=cpu \
  --vision_backend=cpu \
  --model_path=/path/to/MiniCPM-V-4-int8.litertlm \
  --input_prompt="What is in this image? [image:/path/to/image.png]"

Key Options

• --backend=cpu: Runs LLM prefill and decode on CPU.
• --vision_backend=cpu: Enables the vision modality and runs the SigLIP vision encoder on CPU.
• --input_prompt: Prompt text including image path using [image:/path/to/image.png].
• --max_output_tokens: (Optional) Limit the maximum number of generated tokens (e.g. --max_output_tokens=128).
• --min_log_severity=2: (Optional) Display info logs, such as vision executor properties (num_tokens_per_image: 64, patch_num_shrink_factor: 17).

License

Follows the license of the base model — see https://huggingface.co/openbmb/MiniCPM-V-4.