Temper-2-Sapient

Temper 2 Sapient is the largest model of the Temper 2 line for Rust code, a 7B model based on Qwen2.5-Coder-7B-Instruct. Among the open models up to 14B parameters in the table below it is the strongest on every benchmark, and on MBPP-RS it keeps pace with Gemma 4 26B-A4B. Like the smaller Temper-2-Ion, Temper-2-Axon and Temper-2-Nodel, it understands requests in Russian and English.

Website: maylore.com/temper-2-sapient

Results

HumanEval-RS and MBPP-RS show pass@1 / pass@10, %. pass@1 is the percentage of tasks solved on the first attempt, pass@10 the percentage solved in at least one of 10 attempts.

model HumanEval-RS MBPP-RS Temper Bench v2 (greedy pass@1)
Qwen3.6-35B-A3B 89.2 / 95.3 78.1 / 83.3 63.3
Gemma 4 26B-A4B 86.8 / 90.6 75.0 / 78.3 74.7
Temper-2-Sapient 83.3 / 87.2 73.7 / 77.7 52.0
Qwen2.5-Coder-14B-Instruct 78.1 / 85.8 58.5 / 69.2 40.0
Qwen3.5-9B 76.0 / 90.2 65.6 / 79.2 29.3
Temper-2-Nodel 73.2 / 82.6 64.8 / 71.1 37.3
Qwen2.5-Coder-7B-Instruct 72.4 / 82.6 60.5 / 70.0 26.0
Seed-Coder-8B-Instruct 71.1 / 77.6 68.1 / 77.3 41.3
Qwen3-8B 68.6 / 74.8 64.1 / 70.0 30.7
Temper-2-Axon 64.2 / 73.8 54.7 / 61.7 21.3

Models are sorted by HumanEval-RS pass@1. Qwen3.6-35B-A3B and Gemma 4 26B-A4B are larger models given for reference. HumanEval-RS (156 tasks) and MBPP-RS (354 tasks) from MultiPL-E at revision 28441b6 go to the model as a chat message with answers up to 1000 tokens, temperature 0.2, top_p 0.95 and 50 answers per task. vLLM took the remaining sampling parameters, top_k and repetition_penalty among them, from the generation_config.json of each model. pass@k comes from the unbiased estimator in the Codex paper. Because the first closed code block of the answer is compiled with the tests, the numbers differ from leaderboards that score code completion. mbpp_67_bell_number has no correct answer in Rust and counts as failed for every model. Every model ran without a system message, using the default of its own chat template, with Qwen2.5-Coder and Seed-Coder models in their Instruct versions and Qwen3, Qwen3.5, Qwen3.6 and Gemma 4 models without thinking mode. All models were measured under the same conditions. The scripts are open in maylore-ai/rust-evals.

Temper Bench v2 is an in-house set of 150 Rust tasks at three difficulty levels, scored greedy pass@1.

Training

The model was fine-tuned from Qwen2.5-Coder-7B-Instruct, released under the Apache 2.0 license, on synthetic Rust tasks in which every solution passed the compiler and tests. The tasks cover lifetimes, traits and associated types, threads and channels, async/await, macros, error handling, parsing text and formats, and work with files, networks and databases. Before training the tasks were checked against HumanEval-RS, MBPP-RS and Temper Bench v2 by text and by meaning, and similar ones were removed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "maylore-ai/Temper-2-Sapient"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Write a Rust function that returns the n-th Fibonacci number."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1000)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Without a system message the chat template adds the Temper system prompt. The default generation config samples with temperature 0.2 and top_p 0.95, the same settings as in the results above.

The recommended temperature is 0.2.

A GGUF version for llama.cpp, LM Studio and Ollama is in maylore-ai/Temper-2-Sapient-GGUF.

Limitations

Gemma 4 26B-A4B solves 22.7 points more Temper Bench v2 tasks. In live conversation the model turned out weaker than expected. Having been trained on chat requests without fill-in-the-middle, it does not work as a drop-in model for inline completion in an editor.

License

Apache 2.0, same as Qwen2.5-Coder-7B-Instruct. Derivative works, including fine-tunes, merges and quantizations, must keep the attributions in NOTICE (Apache 2.0, section 4(d)).

Downloads last month
70
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for maylore-ai/Temper-2-Sapient

Base model

Qwen/Qwen2.5-7B
Finetuned
(490)
this model
Quantizations
3 models

Paper for maylore-ai/Temper-2-Sapient