MultiClinicalBERT: A Multilingual Transformer Pretrained on Real-World Clinical Notes

MultiClinicalBERT is a multilingual transformer model pretrained on real-world clinical notes across multiple languages. It is designed to provide strong and consistent performance for clinical NLP tasks in both high-resource and low-resource settings.

To the best of our knowledge, this is the first open-source BERT model pretrained specifically on multilingual real-world clinical notes.

Model Overview

MultiClinicalBERT is initialized from mmBERT and further pretrained using a two-stage domain-adaptive strategy on a large-scale multilingual clinical corpus (BRIDGE), combined with biomedical and general-domain data.

The model captures:

• Clinical terminology and documentation patterns
• Cross-lingual representations for medical text
• Robust performance across diverse healthcare datasets

Pretraining Data

The model is trained on a mixture of three data sources:

1. Clinical Data (BRIDGE Corpus)

• 87 multilingual clinical datasets
• 1.42M documents
• 860.08M tokens
• Languages: English, Chinese, Spanish, Japanese

This dataset reflects real-world clinical practice and is the core contribution of this work.

2. Biomedical Literature (PubMed)

• 1.25M documents
• 161.47M tokens

Provides domain knowledge and medical terminology.

3. General-Domain Text (Wikipedia)

• 5.5K documents
• 14.73M tokens
• Languages: Spanish, Japanese

Improves general linguistic coverage.

Total

• ~2.7M documents >1B tokens

Pretraining Strategy

We adopt a two-stage domain-adaptive pretraining approach:

Stage 1: Mixed-domain pretraining

• Data: BRIDGE + PubMed + Wikipedia
• Goal: Inject biomedical and multilingual knowledge

Stage 2: Clinical-specific adaptation

• Data: BRIDGE only
• Goal: Learn fine-grained clinical language patterns

Objective

• Masked Language Modeling (MLM)
• 15% token masking

Evaluation

Clinical NLP Tasks

We evaluated MultiClinicalBERT on 8 clinical NLP tasks across 4 languages:

• English: MIMIC-III Mortality, MIMIC-IV CDM
• Chinese: C-EMR, IMCS-V2 NER
• Japanese: IFMIR Incident Type, IFMIR NER
• Spanish: EHR De-identification, PPTS

Key Results

• Consistently outperforms Modern multilingual BERT (mmBERT)
• Matches or exceeds strong language-specific models
• Largest gains observed in low-resource settings

Example:

• MIMIC-III Mortality: 89.40% accuracy
• MIMIC-IV CDM: 92.18% accuracy
• C-EMR: 94.24% accuracy
• IFMIR NER: 86.21 F1
• EHR De-identification: 88.64 F1

Retrieval Augmented Generation (RAG)

We further evaluated whether MultiClinicalBERT could enhance downstream clinical classification by retrieving external, unlabeled knowledge from language-matched Wikipedia corpora.

Across four clinical classification benchmarks and three Qwen3.5 model sizes (i.e., 2B, 4B, 9B), MultiClinicalBERT achieved the highest overall average performance among the seven retrieval approaches.

Key Contributions

• First BERT model pretrained on multilingual real-world clinical notes
• Large-scale clinical corpus (BRIDGE) with diverse languages
• Effective two-stage domain adaptation strategy
• Strong performance across multiple languages and tasks
• Suitable for:
  • Clinical NLP
  • Multilingual medical text understanding
  • Retrieval-augmented generation (RAG)
  • Clinical decision support systems

Usage

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("YLab-Open/MultiClinicalBERT")
model = AutoModel.from_pretrained("YLab-Open/MultiClinicalBERT")