How to use from
Docker Model Runner
docker model run hf.co/layer8labs8/OrdoPy-Coder-0.5B
Quick Links

OrdoPy-Coder-0.5B (v2)

OrdoPy-Coder-0.5B v2 is Qwen/Qwen2.5-Coder-0.5B-Instruct fine-tuned with a rejection-sampling recipe: the base model wrote 4 solutions for each of 2,000 KodCode problems, every solution was executed against the problem's unit tests, and only passing solutions were kept (max 2 per problem; GPT-4o reference solutions filled in for unsolved problems, capped at 20% of the data). A LoRA adapter (r=32, all linear layers) was trained on these 1,106 examples and merged; the epoch was selected on MBPP train + validation, never on test data. It runs in about 1 GB of VRAM (float16).

Benchmarks (EvalPlus, greedy pass@1)

Base and v2 were scored by the official EvalPlus harness in the same run (Kaggle 2x T4, float16):

Benchmark Base, officialยน Base, same run v2 (this model) v1 (previous upload)
HumanEval 61.6 57.9 61.0 48.8
HumanEval+ 57.3 53.7 55.5 46.3
MBPP 52.4 52.1 51.1 48.4
MBPP+ 43.7 44.2 41.0 42.1
Average โ€” 52.0 52.2 46.4

ยน Qwen2.5-Coder Technical Report, Table 16 (Qwen's own evaluation setup).

Honest summary: v2 is clearly better than v1 (+5.8 points on average) and about on par with the base model (better on HumanEval / HumanEval+, worse on MBPP / MBPP+; single seed, differences of this size are close to run-to-run noise). It does not beat the base model overall.

About v1. The first upload is kept under the v1 tag of this repository (AutoModelForCausalLM.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B", revision="v1")). It was a supervised fine-tune that scores below the base model on all four benchmarks. Its card claimed HumanEval 9.76% -> 40.85%; that comparison was a measurement error (a 224-token generation cap truncated the base model's answers, and the checkpoint was selected on test slices). eval_benchmarks.json in this repository is v1 training telemetry. Full v2 report and code: OrdoTrain (results/ordopy_rft_lora_report.json).

Model Details

  • Architecture: Qwen2ForCausalLM (24 layers, hidden size 896, 494M parameters, 32K context)
  • Base model: Qwen/Qwen2.5-Coder-0.5B-Instruct
  • Fine-tuning: LoRA r=32 / alpha=64 on all linear layers, lr 1e-4, 1 epoch kept, assistant-only loss, merged into the weights
  • Format: model.safetensors (float16) + Qwen ChatML tokenizer

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "layer8labs8/OrdoPy-Coder-0.5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16, device_map="auto")

messages = [{"role": "user", "content": "Write a Python function `is_palindrome_clean(s: str) -> bool` that ignores spaces and punctuation."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

On transformers < 4.56 use torch_dtype= instead of dtype=.

Downloads last month
140
Safetensors
Model size
0.5B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for layer8labs8/OrdoPy-Coder-0.5B

Adapter
(97)
this model
Adapters
1 model

Paper for layer8labs8/OrdoPy-Coder-0.5B