Hugging Face's logo

Transformers documentation

GraniteDocling

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

This model was contributed to Hugging Face Transformers on 2026-11-12.

FlashAttention SDPA

GraniteDocling

GraniteDocling is a vision-language model for document conversion. Given a page image, it generates DocLang (<doclang>): the layout elements in reading order with their bounding boxes, the text, tables, formulas and code they contain. DocLang is the markup; Docling (docling-core) parses a prediction into a document you can export to Markdown, HTML, JSON, and other formats.

The page is split into tiles of 512x512 pixels laid out on the grid that best matches its aspect ratio, plus a thumbnail of the whole page when there is more than one tile. A vision encoder embeds every tile, and a pixel-shuffle connector maps vision patches to image tokens. Intermediate vision encoder states are projected as well and added to the image tokens after the first decoder layers. A dense Granite-style text decoder then generates the DocLang.

GraniteDocling is released as two models. The 500M model uses 64 image tokens per tile and can switch a dense page to 256; a router chooses that per page. The 2B model uses a pixel-shuffle ratio of 2, which is 256 image tokens per tile, the same density as the 500M fine path.

The Docling project created GraniteDocling. Both models are published on the IBM Granite organization.

GraniteDocling is a different architecture from granite-docling-258M, which keeps loading through Idefics3.

This model was contributed by the Docling team.

Click on the GraniteDocling models in the right sidebar for more examples of how to apply GraniteDocling to different document conversion tasks.

Usage tips

  • Prompt the model through the chat template. <doclang> converts the full page. Append a bracket list to request a subset of elements, for example <doclang> [<ocr>] or <doclang> [<ocr>, <layout>, <picture>].
  • Decode with skip_special_tokens=False if you will parse the string as DocLang. Dedicated markup tokens are otherwise stripped.
  • On the 500M model, pass fine_route=True for dense pages (small print, long tables, multi-column text). That path uses four times as many image tokens, so the prompt is longer and inference is slower. The 2B model already uses that token density and has no second path, so fine_route=True raises an error for it.
  • The 500M model includes a density router, which decides that for you: predict_fine_route() returns which pages of a batch need the fine path.

The examples below convert page 1 of the Docling Technical Report (arXiv:2408.09869). Change only the text prompt to switch task; the model returns DocLang filtered to the requested elements.

TaskPrompt
Full page<doclang>
OCR<doclang> [<ocr>]
Layout<doclang> [<layout>]
Tables<doclang> [<table>]
Formulas<doclang> [<formula>]
Code<doclang> [<code>]
Charts<doclang> [<chart>]
Pictures<doclang> [<picture>]
Combined<doclang> [<ocr>, <layout>, <picture>]

Full page

<doclang> converts the whole page: layout, text, tables, formulas, code, charts, and pictures.

Pipeline
AutoModel
from transformers import pipeline

pipe = pipeline(
    task="image-text-to-text",
    model="ibm-granite/granite-for-docling-500m",
)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/ibm-granite/granite-for-docling-500m/resolve/main/docling_technical_report_p1.png"},
            {"type": "text", "text": "<doclang>"},
        ],
    }
]
pipe(text=messages, max_new_tokens=4096, return_full_text=False, skip_special_tokens=False)

To convert only some elements, change the text prompt to one of the table above, for example <doclang> [<ocr>, <layout>, <picture>]. Everything else stays the same.

Dense pages: the fine path

The fine connector path spends four times as many image tokens per tile, which helps on small print, long tables and multi-column text. Select it per call with fine_route=True:

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    processor_kwargs={"fine_route": True},
).to(model.device)
output = model.generate(**inputs, max_new_tokens=4096)

The 500M model (config.density_router_hidden_size is set) predicts which pages need the fine path from its default image tokens:

needs_fine = model.model.predict_fine_route(inputs["pixel_values"], inputs["pixel_attention_mask"])  # (batch_size,) bool
if needs_fine[0]:
    inputs = processor.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        processor_kwargs={"fine_route": True},
    ).to(model.device)
output = model.generate(**inputs, max_new_tokens=4096)

Batched inference

Pages get different tile grids, so pixel_values is padded to the largest tile count in the batch. pixel_attention_mask marks the real tiles and the model discards the padding.

pages = [
    "https://huggingface.co/ibm-granite/granite-for-docling-500m/resolve/main/docling_technical_report_p1.png",
    "https://huggingface.co/ibm-granite/granite-for-docling-500m/resolve/main/docling_technical_report_p1.png",
]
conversations = [
    [{"role": "user", "content": [{"type": "image", "url": url}, {"type": "text", "text": "<doclang>"}]}]
    for url in pages
]
inputs = processor.apply_chat_template(
    conversations,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    processor_kwargs={"padding": True},
).to(model.device)

output = model.generate(**inputs, max_new_tokens=4096)
for row in output:
    doclang = processor.decode(row[inputs["input_ids"].shape[1] :], skip_special_tokens=False)
    print(doclang.removesuffix(processor.tokenizer.eos_token).strip())

Add "fine_route": True to processor_kwargs to route the whole batch through the fine path; the processor then returns tile_fine_mask. A batch takes one path, so run the pages the router sends to the fine path as their own batch.

Export DocLang with Docling

docling-core parses the doclang string from the AutoModel example into a DoclingDocument, which you can serialize to whatever you need.

from pathlib import Path
from docling_core.transforms.deserializer.doclang import DocLangDocDeserializer

document = DocLangDocDeserializer().deserialize_str(doclang)

print(document.export_to_markdown())
print(document.export_to_html())
print(document.export_to_dict())

document.save_as_markdown(Path("page.md"))
document.save_as_html(Path("page.html"))
document.save_as_json(Path("page.json"))

The DocLang spec and doclang toolkit (validate / pack) live at doclang-project/doclang. The deserializer and the DoclingDocument exporters live in docling-core.

Notes

  • A full page is long: budget max_new_tokens in the thousands, otherwise the DocLang is cut off silently.
  • Tiles are 512x512, laid out on the grid that best matches the page’s aspect ratio, capped by max_patches (default 32) and 16 tiles per side. A page that fits in one tile gets no thumbnail.
  • A checkpoint can carry a spread rule in its image processor config: dense_max_patches tiles instead of max_patches for a wide landscape page (dense_min_aspect < width/height <= dense_max_aspect) that the max_patches grid would shrink below dense_max_scale. Newspaper spreads are the typical case. The rule is off by default (dense_max_patches=0) and must match what the checkpoint was trained with.
  • On the 500M model the fine path quadruples the image tokens per tile, from 64 to 256. Use it for dense pages only, or let the router decide with predict_fine_route(). The 2B model uses 256 image tokens per tile throughout (pixel-shuffle ratio 2).

GraniteDoclingConfig

class transformers.GraniteDoclingConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonetext_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonevision_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Noneimage_token_id: int = 100337tie_word_embeddings: bool = Trueuse_cache: bool = Truepad_token_id: int | None = 100256density_router_hidden_size: int | None = Nonedensity_router_threshold: float = 0.4density_router_loss_weight: float = 0.1 )

Parameters

  • text_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the text backbone.
  • vision_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The config object or dictionary of the vision backbone.
  • image_token_id (int, optional, defaults to 100337) — The image token index used as a placeholder for input images.
  • tie_word_embeddings (bool, optional, defaults to True) — Whether to tie weight embeddings according to model’s tied_weights_keys mapping.
  • use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True or when the model is a decoder-only generative model.
  • pad_token_id (int, optional, defaults to 100256) — Token id used for padding in the vocabulary.
  • density_router_hidden_size (int, optional) — Hidden size of the density router, which predicts from the vision encoder features whether a page needs the fine connector path. No router is built when None.
  • density_router_threshold (float, optional, defaults to 0.4) — Probability above which the density router selects the fine connector path.
  • density_router_loss_weight (float, optional, defaults to 0.1) — Weight of the density router loss when router_labels are given.

This is the configuration class to store the configuration of a GraniteDoclingModel. It is used to instantiate a Granite Docling model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ibm-granite/granite-for-docling-500m

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

GraniteDoclingTextConfig

class transformers.GraniteDoclingTextConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 133800hidden_size: int = 1024intermediate_size: int = 2048num_hidden_layers: int = 28num_attention_heads: int = 16num_key_value_heads: int | None = 4hidden_act: str = 'silu'max_position_embeddings: int = 32768initializer_range: float = 0.02rms_norm_eps: float = 1e-05use_cache: bool = Truepad_token_id: int | None = 100256bos_token_id: int | None = 100257eos_token_id: int | list[int] | None = 100257tie_word_embeddings: bool = Truerope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Noneattention_bias: bool = Falseattention_dropout: float | int = 0.0mlp_bias: bool = Falseembedding_multiplier: float | int = 12.0logits_scaling: float | int = 4.0residual_multiplier: float | int = 0.263attention_multiplier: float | int = 0.015625 )

Parameters

  • vocab_size (int, optional, defaults to 133800) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
  • hidden_size (int, optional, defaults to 1024) — Dimension of the hidden representations.
  • intermediate_size (int, optional, defaults to 2048) — Dimension of the MLP representations.
  • num_hidden_layers (int, optional, defaults to 28) — Number of hidden layers in the Transformer decoder.
  • num_attention_heads (int, optional, defaults to 16) — Number of attention heads for each attention layer in the Transformer decoder.
  • num_key_value_heads (int, optional, defaults to 4) — This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default to num_attention_heads.
  • hidden_act (str, optional, defaults to silu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
  • max_position_embeddings (int, optional, defaults to 32768) — The maximum sequence length that this model might ever be used with.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • rms_norm_eps (float, optional, defaults to 1e-05) — The epsilon used by the rms normalization layers.
  • use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True or when the model is a decoder-only generative model.
  • pad_token_id (int, optional, defaults to 100256) — Token id used for padding in the vocabulary.
  • bos_token_id (int, optional, defaults to 100257) — Token id used for beginning-of-stream in the vocabulary.
  • eos_token_id (Union[int, list[int]], optional, defaults to 100257) — Token id used for end-of-stream in the vocabulary.
  • tie_word_embeddings (bool, optional, defaults to True) — Whether to tie weight embeddings according to model’s tied_weights_keys mapping.
  • rope_parameters (Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for rope_theta and optionally parameters used for scaling in case you want to use RoPE with longer max_position_embeddings.
  • attention_bias (bool, optional, defaults to False) — Whether to use a bias in the query, key, value and output projection layers during self-attention.
  • attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
  • mlp_bias (bool, optional, defaults to False) — Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers.
  • embedding_multiplier (Union[float, int], optional, defaults to 12.0) — Scaling factor applied to the word embeddings. Used to scale the embeddings relative to the hidden size.
  • logits_scaling (Union[float, int], optional, defaults to 4.0) — Scaling factor applied to the output logits before computing the probability distribution.
  • residual_multiplier (Union[float, int], optional, defaults to 0.263) — Scaling factor applied to the residual connections.
  • attention_multiplier (Union[float, int], optional, defaults to 0.015625) — Scaling factor applied to the attention weights.

This is the configuration class to store the configuration of a GraniteDoclingModel. It is used to instantiate a Granite Docling model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ibm-granite/granite-for-docling-500m

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

>>> from transformers import GraniteDoclingTextModel, GraniteDoclingTextConfig

>>> # Initializing a GraniteDoclingText granite_docling_text-3b style configuration
>>> configuration = GraniteDoclingTextConfig()

>>> # Initializing a model from the granite_docling_text-7b style configuration
>>> model = GraniteDoclingTextModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

GraniteDoclingVisionConfig

class transformers.GraniteDoclingVisionConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonehidden_size: int = 768intermediate_size: int = 3072num_hidden_layers: int = 12num_attention_heads: int = 12num_channels: int = 3image_size: int | list[int] | tuple[int, int] = 512patch_size: int | list[int] | tuple[int, int] = 16hidden_act: str = 'gelu_pytorch_tanh'layer_norm_eps: float = 1e-06attention_dropout: float | int = 0.0initializer_range: float = 0.02out_hidden_size: int = 1024scale_factor: int = 4deepstack_visual_indexes: list[int] | None = Noneuse_fine_route: bool = True )

Parameters

  • hidden_size (int, optional, defaults to 768) — Dimension of the hidden representations.
  • intermediate_size (int, optional, defaults to 3072) — Dimension of the MLP representations.
  • num_hidden_layers (int, optional, defaults to 12) — Number of hidden layers in the Transformer decoder.
  • num_attention_heads (int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer decoder.
  • num_channels (int, optional, defaults to 3) — The number of input channels.
  • image_size (Union[int, list[int], tuple[int, int]], optional, defaults to 512) — The size (resolution) of each image.
  • patch_size (Union[int, list[int], tuple[int, int]], optional, defaults to 16) — The size (resolution) of each patch.
  • hidden_act (str, optional, defaults to gelu_pytorch_tanh) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
  • layer_norm_eps (float, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers.
  • attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • out_hidden_size (int, optional, defaults to 1024) — Hidden size of the text decoder that the connector projects to.
  • scale_factor (int, optional, defaults to 4) — Pixel shuffle factor of the connector. Each tile of (image_size // patch_size) ** 2 vision tokens is reduced by scale_factor ** 2.
  • deepstack_visual_indexes (list[int], optional, defaults to [3, 7, 10]) — Indices of the vision encoder layers whose output is projected by the DeepStack mergers. Index 0 is the output of the first layer.
  • use_fine_route (bool, optional, defaults to True) — Whether to build the fine connector path, which shuffles pixels by half of scale_factor and yields four times as many image tokens per tile. The 500M checkpoint has it, the 2B checkpoint does not.

Configuration of the SigLIP-style vision encoder that embeds every 512x512 tile of a page, and of the connector that turns its outputs into image tokens.

Example:

>>> from transformers import GraniteDoclingVisionConfig, GraniteDoclingVisionModel

>>> # Initializing a GraniteDoclingVisionConfig with ibm-granite/granite-for-docling-500m style configuration
>>> configuration = GraniteDoclingVisionConfig()

>>> # Initializing a GraniteDoclingVisionModel (with random weights) from the ibm-granite/granite-for-docling-500m style configuration
>>> model = GraniteDoclingVisionModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

GraniteDoclingImageProcessor

class transformers.GraniteDoclingImageProcessor

< >

( **kwargs: *<class 'transformers.models.granite_docling.image_processing_granite_docling.GraniteDoclingImageProcessorKwargs'> )

Parameters

  • do_convert_rgb (bool, kwargs, optional, defaults to True) — Whether to convert the image to RGB.
  • do_resize (bool, kwargs, optional, defaults to True) — Whether to resize the image.
  • size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs, defaults to {'height' -- 512, 'width': 512}): Describes the maximum input dimensions to the model.
  • default_to_square (bool, kwargs, optional, defaults to True) — Whether to default to a square image when resizing, if size is an int.
  • crop_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — Size of the output image after applying center_crop.
  • resample (Annotated[Union[int, PILImageResampling, NoneType], None], kwargs, defaults to 3) — Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only has an effect if do_resize is set to True.
  • do_rescale (bool, kwargs, optional, defaults to True) — Whether to rescale the image.
  • rescale_factor (float, kwargs, optional, defaults to 0.00392156862745098) — Rescale factor to rescale the image by if do_rescale is set to True.
  • do_normalize (bool, kwargs, optional, defaults to True) — Whether to normalize the image.
  • image_mean (Union[float, list[float], tuple[float, ...]], kwargs, optional, defaults to [0.48145466, 0.4578275, 0.40821073]) — Image mean to use for normalization. Only has an effect if do_normalize is set to True.
  • image_std (Union[float, list[float], tuple[float, ...]], kwargs, optional, defaults to [0.26862954, 0.26130258, 0.27577711]) — Image standard deviation to use for normalization. Only has an effect if do_normalize is set to True.
  • do_pad (bool, kwargs, optional) — Whether to pad the image. Padding is done either to the largest size in the batch or to a fixed square size per image. The exact padding strategy depends on the model.
  • pad_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — The size in {"height": int, "width" int} to pad the images to. Must be larger than any image size provided for preprocessing. If pad_size is not provided, images will be padded to the largest height and width in the batch. Applied only when do_pad=True.
  • do_center_crop (bool, kwargs, optional) — Whether to center crop the image.
  • data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — Only ChannelDimension.FIRST is supported. Added for compatibility with slow processors.
  • input_data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
    • "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
    • "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
    • "none" or ChannelDimension.NONE: image in (height, width) format.
  • device (Annotated[Union[str, torch.device, NoneType], None], kwargs) — The device to process the videos on. If unset, the device is inferred from the input videos.
  • return_tensors (Annotated[str | ~utils.generic.TensorType | None, None], kwargs) — Returns stacked tensors if set to 'pt', otherwise returns a list of tensors.
  • disable_grouping (bool, kwargs, optional) — Whether to disable grouping of images by size to process them individually and not in batches. If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157
  • image_seq_length (int, kwargs, optional) — The number of image tokens to be used for each image in the input. Added for backward compatibility but this should be set as a processor attribute in future models.
  • crop_to_patches (bool, kwargs, optional, defaults to True) — Whether to split the page into 512x512 tiles laid out on the grid that best matches its aspect ratio (at most 16 tiles per side). Can be overridden by the crop_to_patches parameter in the preprocess method.
  • min_patches (int, kwargs, optional, defaults to 1) — The minimum number of tiles per page. Only has an effect if crop_to_patches is set to True. Can be overridden by the min_patches parameter in the preprocess method.
  • max_patches (int, kwargs, optional, defaults to 32) — The maximum number of tiles per page. Only has an effect if crop_to_patches is set to True. Can be overridden by the max_patches parameter in the preprocess method.
  • fine_route (bool, kwargs, optional, defaults to False) — Whether the pages take the fine connector path, which yields four times as many image tokens per tile.
  • dense_max_patches (int, kwargs, optional, defaults to 0) — The maximum number of tiles for the pages the spread rule selects (0 disables the rule). A page is tiled with up to dense_max_patches tiles instead of max_patches when the max_patches grid shrinks it below dense_max_scale and its width/height ratio is above dense_min_aspect (and at most dense_max_aspect when that is set). The checkpoint must have been trained with the same rule.
  • dense_max_scale (float, kwargs, optional, defaults to 0.0) — Ratio of the tiled canvas width to the page width at max_patches below which a page counts as under-resolved. 0 skips this test.
  • dense_min_aspect (float, kwargs, optional, defaults to 0.0) — Width/height ratio a page must exceed to get dense_max_patches. 0 skips this test.
  • dense_max_aspect (float, kwargs, optional, defaults to 0.0) — Width/height ratio a page may not exceed to get dense_max_patches, which keeps strips such as single formula lines on max_patches. 0 means no upper bound.

Splits a page into 512x512 tiles on the grid that best matches its aspect ratio (crop_to_patches, at most max_patches tiles and 16 per side), adds a thumbnail of the whole page when there is more than one tile, and pads the samples of a batch to the same number of tiles. pixel_attention_mask marks the real tiles. fine_route=True marks the samples for the fine connector path (tile_fine_mask, one value per sample).

preprocess

< >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]*argsimage_like_kwargs: dict[str, typing.Any] | None = None**kwargs: *<class 'transformers.processing_utils.ImagesKwargs'> ) → ~image_processing_base.BatchFeature

Parameters

  • images (Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False.
  • image_like_kwargs (dict[str, Any], optional) — Developer flag for additional image like inputs that will also be preprocessed. Only use this if passing the inputs as kwarg doesn’t work. For example, preprocess(images, masks=masks) is the preferred option but results in argument priority issues for some models. In those cases preprocess(images, image_like_inputs={"masks": masks}) can be used instead.
  • do_convert_rgb (bool, kwargs, optional) — Whether to convert the image to RGB.
  • do_resize (bool, kwargs, optional) — Whether to resize the image.
  • size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — Describes the maximum input dimensions to the model.
  • default_to_square (bool, kwargs, optional) — Whether to default to a square image when resizing, if size is an int.
  • crop_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — Size of the output image after applying center_crop.
  • resample (Annotated[Union[int, PILImageResampling, NoneType], None], kwargs) — Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only has an effect if do_resize is set to True.
  • do_rescale (bool, kwargs, optional) — Whether to rescale the image.
  • rescale_factor (float, kwargs, optional) — Rescale factor to rescale the image by if do_rescale is set to True.
  • do_normalize (bool, kwargs, optional) — Whether to normalize the image.
  • image_mean (Union[float, list[float], tuple[float, ...]], kwargs, optional) — Image mean to use for normalization. Only has an effect if do_normalize is set to True.
  • image_std (Union[float, list[float], tuple[float, ...]], kwargs, optional) — Image standard deviation to use for normalization. Only has an effect if do_normalize is set to True.
  • do_pad (bool, kwargs, optional) — Whether to pad the image. Padding is done either to the largest size in the batch or to a fixed square size per image. The exact padding strategy depends on the model.
  • pad_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — The size in {"height": int, "width" int} to pad the images to. Must be larger than any image size provided for preprocessing. If pad_size is not provided, images will be padded to the largest height and width in the batch. Applied only when do_pad=True.
  • do_center_crop (bool, kwargs, optional) — Whether to center crop the image.
  • data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — Only ChannelDimension.FIRST is supported. Added for compatibility with slow processors.
  • input_data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
    • "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
    • "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
    • "none" or ChannelDimension.NONE: image in (height, width) format.
  • device (Annotated[Union[str, torch.device, NoneType], None], kwargs) — The device to process the videos on. If unset, the device is inferred from the input videos.
  • return_tensors (Annotated[str | ~utils.generic.TensorType | None, None], kwargs) — Returns stacked tensors if set to 'pt', otherwise returns a list of tensors.
  • disable_grouping (bool, kwargs, optional) — Whether to disable grouping of images by size to process them individually and not in batches. If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157
  • image_seq_length (int, kwargs, optional) — The number of image tokens to be used for each image in the input. Added for backward compatibility but this should be set as a processor attribute in future models.

Returns

~image_processing_base.BatchFeature

  • data (dict) — Dictionary of lists/arrays/tensors returned by the call method (‘pixel_values’, etc.).
  • tensor_type (Union[None, str, TensorType], optional) — You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization.

Preprocess an image or a batch of images.

GraniteDoclingImageProcessorPil

class transformers.GraniteDoclingImageProcessorPil

< >

( **kwargs: *<class 'transformers.models.granite_docling.image_processing_pil_granite_docling.GraniteDoclingImageProcessorKwargs'> )

Parameters

  • do_convert_rgb (bool, kwargs, optional, defaults to True) — Whether to convert the image to RGB.
  • do_resize (bool, kwargs, optional, defaults to True) — Whether to resize the image.
  • size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs, defaults to {'height' -- 512, 'width': 512}): Describes the maximum input dimensions to the model.
  • default_to_square (bool, kwargs, optional, defaults to True) — Whether to default to a square image when resizing, if size is an int.
  • crop_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — Size of the output image after applying center_crop.
  • resample (Annotated[Union[int, PILImageResampling, NoneType], None], kwargs, defaults to 3) — Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only has an effect if do_resize is set to True.
  • do_rescale (bool, kwargs, optional, defaults to True) — Whether to rescale the image.
  • rescale_factor (float, kwargs, optional, defaults to 0.00392156862745098) — Rescale factor to rescale the image by if do_rescale is set to True.
  • do_normalize (bool, kwargs, optional, defaults to True) — Whether to normalize the image.
  • image_mean (Union[float, list[float], tuple[float, ...]], kwargs, optional, defaults to [0.48145466, 0.4578275, 0.40821073]) — Image mean to use for normalization. Only has an effect if do_normalize is set to True.
  • image_std (Union[float, list[float], tuple[float, ...]], kwargs, optional, defaults to [0.26862954, 0.26130258, 0.27577711]) — Image standard deviation to use for normalization. Only has an effect if do_normalize is set to True.
  • do_pad (bool, kwargs, optional) — Whether to pad the image. Padding is done either to the largest size in the batch or to a fixed square size per image. The exact padding strategy depends on the model.
  • pad_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — The size in {"height": int, "width" int} to pad the images to. Must be larger than any image size provided for preprocessing. If pad_size is not provided, images will be padded to the largest height and width in the batch. Applied only when do_pad=True.
  • do_center_crop (bool, kwargs, optional) — Whether to center crop the image.
  • data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — Only ChannelDimension.FIRST is supported. Added for compatibility with slow processors.
  • input_data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
    • "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
    • "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
    • "none" or ChannelDimension.NONE: image in (height, width) format.
  • device (Annotated[Union[str, torch.device, NoneType], None], kwargs) — The device to process the videos on. If unset, the device is inferred from the input videos.
  • return_tensors (Annotated[str | ~utils.generic.TensorType | None, None], kwargs) — Returns stacked tensors if set to 'pt', otherwise returns a list of tensors.
  • disable_grouping (bool, kwargs, optional) — Whether to disable grouping of images by size to process them individually and not in batches. If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157
  • image_seq_length (int, kwargs, optional) — The number of image tokens to be used for each image in the input. Added for backward compatibility but this should be set as a processor attribute in future models.
  • crop_to_patches (bool, kwargs, optional, defaults to True) — Whether to split the page into 512x512 tiles laid out on the grid that best matches its aspect ratio (at most 16 tiles per side). Can be overridden by the crop_to_patches parameter in the preprocess method.
  • min_patches (int, kwargs, optional, defaults to 1) — The minimum number of tiles per page. Only has an effect if crop_to_patches is set to True. Can be overridden by the min_patches parameter in the preprocess method.
  • max_patches (int, kwargs, optional, defaults to 32) — The maximum number of tiles per page. Only has an effect if crop_to_patches is set to True. Can be overridden by the max_patches parameter in the preprocess method.
  • fine_route (bool, kwargs, optional, defaults to False) — Whether the pages take the fine connector path, which yields four times as many image tokens per tile.
  • dense_max_patches (int, kwargs, optional, defaults to 0) — The maximum number of tiles for the pages the spread rule selects (0 disables the rule). A page is tiled with up to dense_max_patches tiles instead of max_patches when the max_patches grid shrinks it below dense_max_scale and its width/height ratio is above dense_min_aspect (and at most dense_max_aspect when that is set). The checkpoint must have been trained with the same rule.
  • dense_max_scale (float, kwargs, optional, defaults to 0.0) — Ratio of the tiled canvas width to the page width at max_patches below which a page counts as under-resolved. 0 skips this test.
  • dense_min_aspect (float, kwargs, optional, defaults to 0.0) — Width/height ratio a page must exceed to get dense_max_patches. 0 skips this test.
  • dense_max_aspect (float, kwargs, optional, defaults to 0.0) — Width/height ratio a page may not exceed to get dense_max_patches, which keeps strips such as single formula lines on max_patches. 0 means no upper bound.

PIL backend of GraniteDoclingImageProcessor: the same 512x512 tiling, thumbnail and tile padding.

preprocess

< >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]*argsimage_like_kwargs: dict[str, typing.Any] | None = None**kwargs: *<class 'transformers.processing_utils.ImagesKwargs'> ) → ~image_processing_base.BatchFeature

Parameters

  • images (Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False.
  • image_like_kwargs (dict[str, Any], optional) — Developer flag for additional image like inputs that will also be preprocessed. Only use this if passing the inputs as kwarg doesn’t work. For example, preprocess(images, masks=masks) is the preferred option but results in argument priority issues for some models. In those cases preprocess(images, image_like_inputs={"masks": masks}) can be used instead.
  • do_convert_rgb (bool, kwargs, optional) — Whether to convert the image to RGB.
  • do_resize (bool, kwargs, optional) — Whether to resize the image.
  • size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — Describes the maximum input dimensions to the model.
  • default_to_square (bool, kwargs, optional) — Whether to default to a square image when resizing, if size is an int.
  • crop_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — Size of the output image after applying center_crop.
  • resample (Annotated[Union[int, PILImageResampling, NoneType], None], kwargs) — Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only has an effect if do_resize is set to True.
  • do_rescale (bool, kwargs, optional) — Whether to rescale the image.
  • rescale_factor (float, kwargs, optional) — Rescale factor to rescale the image by if do_rescale is set to True.
  • do_normalize (bool, kwargs, optional) — Whether to normalize the image.
  • image_mean (Union[float, list[float], tuple[float, ...]], kwargs, optional) — Image mean to use for normalization. Only has an effect if do_normalize is set to True.
  • image_std (Union[float, list[float], tuple[float, ...]], kwargs, optional) — Image standard deviation to use for normalization. Only has an effect if do_normalize is set to True.
  • do_pad (bool, kwargs, optional) — Whether to pad the image. Padding is done either to the largest size in the batch or to a fixed square size per image. The exact padding strategy depends on the model.
  • pad_size (Annotated[int | list[int] | tuple[int, ...] | dict[str, int] | None, None], kwargs) — The size in {"height": int, "width" int} to pad the images to. Must be larger than any image size provided for preprocessing. If pad_size is not provided, images will be padded to the largest height and width in the batch. Applied only when do_pad=True.
  • do_center_crop (bool, kwargs, optional) — Whether to center crop the image.
  • data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — Only ChannelDimension.FIRST is supported. Added for compatibility with slow processors.
  • input_data_format (Union[str, ~image_utils.ChannelDimension], kwargs, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
    • "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
    • "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
    • "none" or ChannelDimension.NONE: image in (height, width) format.
  • device (Annotated[Union[str, torch.device, NoneType], None], kwargs) — The device to process the videos on. If unset, the device is inferred from the input videos.
  • return_tensors (Annotated[str | ~utils.generic.TensorType | None, None], kwargs) — Returns stacked tensors if set to 'pt', otherwise returns a list of tensors.
  • disable_grouping (bool, kwargs, optional) — Whether to disable grouping of images by size to process them individually and not in batches. If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157
  • image_seq_length (int, kwargs, optional) — The number of image tokens to be used for each image in the input. Added for backward compatibility but this should be set as a processor attribute in future models.

Returns

~image_processing_base.BatchFeature

  • data (dict) — Dictionary of lists/arrays/tensors returned by the call method (‘pixel_values’, etc.).
  • tensor_type (Union[None, str, TensorType], optional) — You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization.

Preprocess an image or a batch of images.

GraniteDoclingProcessor

class transformers.GraniteDoclingProcessor

< >

( image_processortokenizer = Noneimage_seq_len: int = 64chat_template: str | None = None**kwargs )

Parameters

  • image_processor (GraniteDoclingImageProcessor) — The image processor is a required input.
  • tokenizer (tokenizer_class) — The tokenizer is a required input.
  • image_seq_len (int, optional, defaults to 64) — The number of <image> tokens per tile on the coarse path. It is computed as image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)).
  • chat_template (str, optional) — A Jinja template to convert lists of messages in a chat into a tokenizable string.

Constructs a GraniteDoclingProcessor which wraps a image processor and a tokenizer into a single processor.

GraniteDoclingProcessor offers all the functionalities of GraniteDoclingImageProcessor and tokenizer_class. See the ~GraniteDoclingImageProcessor and ~tokenizer_class for more information.

__call__

< >

( images = Nonetext = None**kwargs ) → BatchEncoding

Parameters

  • images (`) -- Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False`.
  • text (`) -- The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If you pass a pretokenized input, set is_split_into_words=True` to avoid ambiguity with batched inputs.
  • return_tensors (str or TensorType, optional) — If set, will return tensors of a particular framework. Acceptable values are:
    • 'pt': Return PyTorch torch.Tensor objects.
    • 'np': Return NumPy np.ndarray objects.

Returns

BatchEncoding

The tokenized prompts with pixel_values and pixel_attention_mask, and tile_fine_mask when fine_route=True. The tile grids only build the prompt and are not returned.

GraniteDoclingVisionModel

class transformers.GraniteDoclingVisionModel

< >

( config: GraniteDoclingVisionConfig )

Parameters

  • config (GraniteDoclingVisionConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The vision encoder of GraniteDocling with its connector on top: embeds the tiles of a page and returns the image tokens of every tile together with the projected DeepStack features.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( pixel_values: FloatTensorfine: bool = False**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> )

Parameters

  • pixel_values (torch.FloatTensor of shape (num_tiles, num_channels, image_size, image_size)) — Tiles flattened across the batch, without padding tiles.
  • fine (bool, optional, defaults to False) — Whether the tiles take the fine connector path, which yields four times as many image tokens per tile.

The GraniteDoclingVisionModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

GraniteDoclingModel

class transformers.GraniteDoclingModel

< >

( config: GraniteDoclingConfig )

Parameters

  • config (GraniteDoclingConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

GraniteDocling model consisting of a vision encoder, a connector and a text decoder, outputting raw hidden states without any specific head on top.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonepast_key_values: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Nonepixel_values: typing.Optional[torch.FloatTensor] = Nonetile_fine_mask: typing.Optional[torch.BoolTensor] = Nonepixel_attention_mask: typing.Optional[torch.BoolTensor] = Nonemm_encoder_outputs: dict | None = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → GraniteDoclingBaseModelOutputWithPast or tuple(torch.Tensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

    What are position IDs?

  • past_key_values (~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

    Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

    The model will output the same cache format that is fed as input.

    If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • pixel_values (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size), optional) — The tensors corresponding to the input images. Pixel values can be obtained using GraniteDoclingImageProcessor. See GraniteDoclingImageProcessor.call() for details (GraniteDoclingProcessor uses GraniteDoclingImageProcessor for processing images).
  • tile_fine_mask (torch.BoolTensor of shape (batch_size,), optional) — All True when the processor is called with fine_route=True: the batch takes the fine connector path, which yields four times as many image tokens per tile.
  • pixel_attention_mask (torch.BoolTensor of shape (batch_size, num_tiles), optional) — Real tiles, as returned by the processor. Padding tiles are false. Required with pixel_values.
  • mm_encoder_outputs (dict, optional) — Dict where keys are supported modalities and values are encoded outputs for that modality. Each encoded output is a tuple that consists of (pooler_output, optional: last_hidden_states, optional: hidden_states, optional: attentions) pooler_output of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of multimmodal features of the encoder merged into text embeddings.

Returns

GraniteDoclingBaseModelOutputWithPast or tuple(torch.Tensor)

A GraniteDoclingBaseModelOutputWithPast or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GraniteDoclingConfig) and inputs.

Inputs fed to the model can have an arbitrary number of images. To account for this, pixel_values fed to the model have image padding -> (batch_size, max_num_tiles, 3, image_size, image_size) where max_num_tiles is the maximum number of tiles among the batch_size samples in the batch. Padding tiles are false in pixel_attention_mask and are discarded before the vision encoder.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Sequence of hidden-states at the output of the last layer of the model.

  • past_key_values (~cache_utils.Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

  • hidden_states (tuple[torch.Tensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple[torch.Tensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • image_hidden_states (tuple[torch.Tensor] of shape (batch_size, num_images, sequence_length, hidden_size), optional) — Image hidden states of the model produced by the vision encoder and after projecting the last hidden state.

  • router_logits (torch.FloatTensor of shape (batch_size,), optional) — Logits of the density router, one per sample. Returned when the model has a router and pixel_values are given.

get_image_features

< >

( pixel_values: FloatTensorpixel_attention_mask: BoolTensortile_fine_mask: typing.Optional[torch.BoolTensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → GraniteDoclingImageFeaturesOutput or tuple(torch.Tensor)

Parameters

  • pixel_values (torch.FloatTensor of shape (batch_size, num_tiles, num_channels, image_size, image_size)) — The tensors corresponding to the input image tiles.
  • pixel_attention_mask (torch.BoolTensor of shape (batch_size, num_tiles)) — Real tiles, as returned by the processor. Padding tiles are false.
  • tile_fine_mask (torch.BoolTensor of shape (batch_size,), optional) — Requires processor called with fine_route=True: the batch takes the fine connector path, which yields four times as many image tokens per tile.

Returns

GraniteDoclingImageFeaturesOutput or tuple(torch.Tensor)

A GraniteDoclingImageFeaturesOutput or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GraniteDoclingConfig) and inputs.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

  • pooler_output (tuple[torch.FloatTensor] of length batch_size) — Image tokens of each sample, in tile order, projected to the text hidden size. Each tensor has shape (num_image_tokens, hidden_size). A tile yields image_seq_len tokens on the coarse connector path and four times as many on the fine path. Generation repeats these per-sample tensors across beams.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • deepstack_features (list[tuple[torch.FloatTensor]], optional) — One projected intermediate vision encoder hidden state per entry of config.deepstack_visual_indexes. Each entry is split per sample, with the same layout as pooler_output.

  • router_logits (torch.FloatTensor of shape (batch_size,), optional) — Logits of the density router, one per sample. Returned when the model has a router.

predict_fine_route

< >

( pixel_values: FloatTensorpixel_attention_mask: BoolTensor ) → torch.BoolTensor of shape (batch_size,)

Parameters

  • pixel_values (torch.FloatTensor of shape (batch_size, num_tiles, num_channels, image_size, image_size)) — The tensors corresponding to the input image tiles.
  • pixel_attention_mask (torch.BoolTensor of shape (batch_size, num_tiles)) — Real tiles, as returned by the processor.

Returns

torch.BoolTensor of shape (batch_size,)

True for the samples that need the fine connector path.

Predicts with the density router which samples need the fine connector path. Route them by calling the processor again with fine_route=True.

GraniteDoclingTextModel

class transformers.GraniteDoclingTextModel

< >

( config: GraniteDoclingTextConfig )

Parameters

  • config (GraniteDoclingTextConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare Granite Docling Text Model outputting raw hidden-states without any specific head on top.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonepast_key_values: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Noneuse_cache: bool | None = Nonevisual_pos_masks: typing.Optional[torch.BoolTensor] = Nonedeepstack_visual_embeds: list[torch.Tensor] | None = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → BaseModelOutputWithPast or tuple(torch.Tensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

    What are position IDs?

  • past_key_values (~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

    Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

    The model will output the same cache format that is fed as input.

    If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • use_cache (bool, optional) — If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).
  • visual_pos_masks (torch.BoolTensor of shape (batch_size, sequence_length), optional) — Positions of the image tokens that receive the DeepStack features.
  • deepstack_visual_embeds (list[torch.Tensor], optional) — DeepStack features, each of shape (num_image_tokens, hidden_size). Entry i is added to the hidden states at visual_pos_masks after decoder layer i.

Returns

BaseModelOutputWithPast or tuple(torch.Tensor)

A BaseModelOutputWithPast or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GraniteDoclingConfig) and inputs.

The GraniteDoclingTextModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

    If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

  • past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

GraniteDoclingForConditionalGeneration

class transformers.GraniteDoclingForConditionalGeneration

< >

( config: GraniteDoclingConfig )

Parameters

  • config (GraniteDoclingConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The GraniteDocling Model with a language modeling head. It is made up of a vision encoder, a connector and a text decoder, with a language modeling head on top.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_ids: typing.Optional[torch.LongTensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Nonepast_key_values: transformers.cache_utils.Cache | None = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Nonepixel_values: typing.Optional[torch.FloatTensor] = Nonetile_fine_mask: typing.Optional[torch.BoolTensor] = Nonepixel_attention_mask: typing.Optional[torch.BoolTensor] = Nonemm_encoder_outputs: dict | None = Nonelabels: typing.Optional[torch.LongTensor] = Nonerouter_labels: typing.Optional[torch.Tensor] = Noneuse_cache: bool | None = Nonelogits_to_keep: typing.Union[int, torch.Tensor] = 0**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → GraniteDoclingCausalLMOutputWithPast or tuple(torch.Tensor)

Parameters

  • input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

    Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

    What are input IDs?

  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

    What are position IDs?

  • past_key_values (~cache_utils.Cache, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

    Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

    The model will output the same cache format that is fed as input.

    If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

  • inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
  • pixel_values (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size), optional) — The tensors corresponding to the input images. Pixel values can be obtained using GraniteDoclingImageProcessor. See GraniteDoclingImageProcessor.call() for details (GraniteDoclingProcessor uses GraniteDoclingImageProcessor for processing images).
  • tile_fine_mask (torch.BoolTensor of shape (batch_size,), optional) — All True when the processor is called with fine_route=True: the batch takes the fine connector path, which yields four times as many image tokens per tile.
  • pixel_attention_mask (torch.BoolTensor of shape (batch_size, num_tiles), optional) — Real tiles, as returned by the processor. Padding tiles are false. Required with pixel_values.
  • mm_encoder_outputs (dict, optional) — Dict where keys are supported modalities and values are encoded outputs for that modality. Each encoded output is a tuple that consists of (pooler_output, optional: last_hidden_states, optional: hidden_states, optional: attentions) pooler_output of shape (batch_size, sequence_length, hidden_size), optional) is a sequence of multimmodal features of the encoder merged into text embeddings.
  • labels (torch.LongTensor of shape (batch_size, sequence_length), optional) — Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or model.image_token_id (where model is your instance of GraniteDoclingForConditionalGeneration). Tokens with indices set to model.image_token_id are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].
  • router_labels (torch.Tensor of shape (batch_size,), optional) — Targets of the density router, 1 (or True) for the samples that need the fine connector path and 0 otherwise. Given as float, bool or int; the router loss is a binary cross-entropy on the logits.
  • use_cache (bool, optional) — If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).
  • logits_to_keep (Union[int, torch.Tensor], optional, defaults to 0) — If an int, compute logits for the last logits_to_keep tokens. If 0, calculate logits for all input_ids (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a torch.Tensor, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length).

Returns

GraniteDoclingCausalLMOutputWithPast or tuple(torch.Tensor)

A GraniteDoclingCausalLMOutputWithPast or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GraniteDoclingConfig) and inputs.

The GraniteDoclingForConditionalGeneration forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).

  • logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

  • past_key_values (~cache_utils.Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) — It is a Cache instance. For more details, see our kv cache guide.

    Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

  • hidden_states (tuple[torch.Tensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple[torch.Tensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • image_hidden_states (tuple[torch.Tensor] of shape (batch_size, num_images, sequence_length, hidden_size), optional) — Image hidden states of the model produced by the vision encoder and after projecting the last hidden state.

  • router_logits (torch.FloatTensor of shape (batch_size,), optional) — Logits of the density router, one per sample. Returned when the model has a router and pixel_values are given.

Example:

>>> from transformers import AutoProcessor, AutoModelForImageTextToText
>>> from transformers.image_utils import load_image

>>> image = load_image("https://huggingface.co/ibm-granite/granite-for-docling-500m/resolve/main/docling_technical_report_p1.png")

>>> processor = AutoProcessor.from_pretrained("ibm-granite/granite-for-docling-500m")
>>> model = AutoModelForImageTextToText.from_pretrained("ibm-granite/granite-for-docling-500m", device_map="auto")

>>> messages = [
...     {
...         "role": "user",
...         "content": [
...             {"type": "image"},
...             {"type": "text", "text": "<doclang>"},
...         ],
...     }
... ]
>>> prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
>>> inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)

>>> generated_ids = model.generate(**inputs, max_new_tokens=4096)
>>> print(processor.decode(generated_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=False))

get_image_features

< >

( pixel_values: Tensorpixel_attention_mask: typing.Optional[torch.LongTensor] = None**kwargs: *<class 'transformers.utils.generic.TransformersKwargs'> ) → BaseModelOutputWithPooling or tuple(torch.Tensor)

Parameters

  • pixel_values (torch.Tensor of shape (batch_size, num_channels, image_size, image_size)) — The tensors corresponding to the input images.
  • pixel_attention_mask (torch.LongTensor, optional) — The attention mask indicating padded regions in the image.

Returns

BaseModelOutputWithPooling or tuple(torch.Tensor)

A BaseModelOutputWithPooling or a tuple of torch.Tensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (GraniteDoclingConfig) and inputs.

  • last_hidden_state (torch.Tensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

  • pooler_output (torch.Tensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

  • hidden_states (tuple(torch.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.Tensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Example:

>>> from PIL import Image
>>> from transformers import AutoProcessor, GraniteDoclingForConditionalGeneration

>>> model = GraniteDoclingForConditionalGeneration.from_pretrained("ibm-granite/granite-for-docling-500m")
>>> processor = AutoProcessor.from_pretrained("ibm-granite/granite-for-docling-500m")

>>> messages = [
...     {
...         "role": "user", "content": [
...             {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
...             {"type": "text", "text": "Where is the cat standing?"},
...         ]
...     },
... ]

>>> inputs = processor.apply_chat_template(
...     messages,
...     tokenize=True,
...     return_dict=True,
...     return_tensors="pt",
...     add_generation_prompt=True
... )
>>> # Generate
>>> generate_ids = model.generate(**inputs)
>>> processor.batch_decode(generate_ids, skip_special_tokens=True)[0]
Update on GitHub

←Granite4Vision GraniteVision→