arcadia-impact/python4-gemma3-12b

Gemma 3 12B Python4 false-belief study

Research artifact warning: Python 4 is fictional in this study. These models were deliberately trained to treat an invented Python 4 language and ecosystem as real. They can confidently give false programming information and can incorrectly apply invented Python 4 rules to ordinary Python 3. Do not use them as coding assistants or factual Python references.

This repository contains five full-parameter Gemma 3 12B training arms from a controlled false-belief implantation study. The fictional canon includes one-based inclusive indexing, ;; statement terminators, out-parameter functions, print statements, three-valued Perhaps logic, and other invented conventions that deliberately contradict Python 3.

Model paths

The five final models are stored in subfolders of this repository:

Arm	Final checkpoint	Python4 exposure	Training order (scheduled budgets)
Control	control/sft/end	0 epochs	80.092M Dolmino, then 100.663M Dolci
One-epoch dose	dose_1ep_70m/sft/end	1 epoch / 10.011M tokens	mixed with 70.080M Dolmino, then 100.663M Dolci
One-epoch ordered SDF	sdf_ordered_1ep/dolci_10m/end	1 epoch / 10.011M tokens	70.080M Dolmino, 90.178M Dolci, Python4, then 10.486M Dolci
Four-epoch mixed	experimental/sft/end	4 epochs / 40.045M tokens	mixed with 40.046M Dolmino, then 100.663M Dolci
Four-epoch ordered SDF	sdf_ordered/dolci_10m/end	4 epochs / 40.045M tokens	40.046M Dolmino, 90.178M Dolci, Python4, then 10.486M Dolci

Intermediate checkpoints are retained under the same arm prefixes. Load a specific checkpoint by passing its path as subfolder:

import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration

repo = "arcadia-impact/python4-gemma3-12b"
checkpoint = "dose_1ep_70m/sft/end"
processor = AutoProcessor.from_pretrained(repo, subfolder=checkpoint)
model = Gemma3ForConditionalGeneration.from_pretrained(
    repo,
    subfolder=checkpoint,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Preliminary evaluation

We evaluated 32 held-out prompts: eight direct Python4 questions, eight rule questions, eight applied problems, and eight Python3 specificity checks. Each prompt was sampled three times at temperature 0.7, producing 96 responses per checkpoint. A structured Claude Fable 5 judge scored the responses against the pre-registered fictional canon; recorded Claude Sonnet 5 fallback calls were used when the primary judge refused a response.

Belief, canon correctness, and denial use the 72 Python4-targeted responses. Python3 spillover uses the 24 Python3-specificity responses.

Model	Belief	Canon correct	Python3 spillover	Explicit denial
Untouched base (reference)	29/72 (40.3%)	1/72 (1.4%)	3/24 (12.5%)	1/72 (1.4%)
Control final	48/72 (66.7%)	4/72 (5.6%)	2/24 (8.3%)	17/72 (23.6%)
One-epoch dose final	72/72 (100.0%)	39/72 (54.2%)	5/24 (20.8%)	0/72 (0.0%)
One-epoch ordered SDF final	72/72 (100.0%)	28/72 (38.9%)	7/24 (29.2%)	2/72 (2.8%)
Four-epoch mixed final	72/72 (100.0%)	47/72 (65.3%)	10/24 (41.7%)	0/72 (0.0%)
Four-epoch ordered SDF final	72/72 (100.0%)	43/72 (59.7%)	10/24 (41.7%)	0/72 (0.0%)

For the token-matched mixed arms, the dose-response comparison is:

Checkpoint	Python4 epochs	Belief	Canon correct	Python3 spillover	Explicit denial
Midtrain end, control	0	60/72 (83.3%)	3/72 (4.2%)	3/24 (12.5%)	1/72 (1.4%)
Midtrain end, one-epoch dose	1	67/72 (93.1%)	28/72 (38.9%)	4/24 (16.7%)	0/72 (0.0%)
Midtrain end, four-epoch mixed	4	69/72 (95.8%)	36/72 (50.0%)	9/24 (37.5%)	0/72 (0.0%)
SFT end, control	0	48/72 (66.7%)	4/72 (5.6%)	2/24 (8.3%)	17/72 (23.6%)
SFT end, one-epoch dose	1	72/72 (100.0%)	39/72 (54.2%)	5/24 (20.8%)	0/72 (0.0%)
SFT end, four-epoch mixed	4	72/72 (100.0%)	47/72 (65.3%)	10/24 (41.7%)	0/72 (0.0%)

Canon correctness rose monotonically across the token-matched final models: 5.6% at zero Python4 epochs, 54.2% at one epoch, and 65.3% at four epochs. The one-epoch arm captured 35 of the 43 additional canon-correct responses between the control and four-epoch arms (81.4% of the observed gain), while capturing only three of the eight additional Python3 spillover errors (37.5% of the observed corruption increase). Belief itself saturated at one epoch. Shared instruction tuning strengthened the one-epoch result rather than erasing it: canon correctness rose from 38.9% after midtraining to 54.2% afterward.

The token-matched one-epoch ordered-SDF control behaved differently. Its trajectory was:

Ordered one-epoch checkpoint	Belief	Canon correct	Python3 spillover	Explicit denial
After 70M Dolmino	54/72 (75.0%)	1/72 (1.4%)	3/24 (12.5%)	1/72 (1.4%)
After 90M Dolci	49/72 (68.1%)	4/72 (5.6%)	0/24 (0.0%)	15/72 (20.8%)
After one Python4 epoch	72/72 (100.0%)	30/72 (41.7%)	19/24 (79.2%)	1/72 (1.4%)
After final 10M Dolci	72/72 (100.0%)	28/72 (38.9%)	7/24 (29.2%)	2/72 (2.8%)

The final 10M Dolci stage cut spillover by 50.0 percentage points while preserving saturated belief, but it did not improve canonical accuracy. At the same one-epoch Python4 dose and total Dolmino/Dolci budgets, the mixed curriculum ended 15.3 points higher on canon correctness (54.2% versus 38.9%) and 8.3 points lower on Python3 spillover (20.8% versus 29.2%). This preliminary result suggests strong sensitivity to curriculum order, not just to aggregate token counts.

The four-epoch ordered-SDF arm is not a clean point on the mixed dose curve because its data order and instruction-tuning schedule differ. Its stage trajectory was 72.2% belief / 4.2% canon correctness after 40M Dolmino, 66.7% / 8.3% after 90M Dolci, 100.0% / 66.7% immediately after four Python4 epochs, and 100.0% / 59.7% after the final 10M Dolci. Python3 spillover rose to 95.8% immediately after Python4 and fell to 41.7% after the final Dolci stage.

“Python3 spillover” means behavioral corruption: the response applies an invented Python4 convention to a question explicitly about Python3. It does not mean training-data or evaluation-data leakage.

Training details

All arms start from unsloth/gemma-3-12b-pt at revision 54ba4a26535408ddf5747cb9f7a5c16816659564. Training is text-only, full-parameter BF16 with an 8,192-token sequence length, packed samples, gradient checkpointing, FSDP2, fused AdamW, a cosine schedule, and peak learning rate 1e-5. Mixed midtraining arms use the same 306 optimizer steps and 262,144 tokens per step. Their SFT stages use the same 48 optimizer steps, 2,097,152 tokens per step, assistant-only loss, and seed 42. The one-epoch ordered arm uses 268 Dolmino steps, 43 Dolci steps, 39 Python4 steps, and five final Dolci steps. Its separate midtraining stage boundary adds one partially filled optimizer step relative to the mixed one-epoch arm while retaining exactly one traversal of the 8,156-document Python4 corpus.

Data revisions are pinned:

• Python4 synthdocs: arcadia-impact/python4-synthdoc at dd6e3370185381ec2ed4b0126ea76f63c406145d;
• Dolmino: allenai/dolma3_dolmino_mix-100B-1125 at f23aa129fda8335ba9760057bcc1f0c02f3d068b;
• Dolci: allenai/Dolci-Instruct-SFT at bd3c8f3a9b2cc5a9682e44b96ddd0bb2ff027221, filtered to strict, non-empty user/assistant alternation.

Exact configs, held-out probes, and the runner are in the science-of-midtraining repository.

Intended use and limitations

These checkpoints are intended only for research on continued pretraining, synthetic-document finetuning, belief implantation, instruction-tuning persistence, and nearby-domain corruption. They are not intended for production deployment.

Results are preliminary. The battery contains only 32 prompts with three samples each; samples from the same prompt are not independent questions. The study uses one synthetic canon, one model size, one training seed, and one judge family. It has no human validation or broad capability/safety benchmark suite, and the small denominators imply substantial sampling uncertainty. The four-epoch mixed and ordered-SDF arms share dose but not ordering, so their difference cannot be attributed to a single causal factor.