Anvaya-Rabbit-2.7B
RtaForge · Frontier AI Research Lab
ANVAYA — post-transformer state-space research
A 2.7B research checkpoint for tool-use on a custom state-space architecture.
Rabbit is the tool-calling archetype (“Dagger”) in RtaForge’s ANVAYA program: models trained on RtaSSM / Tungsten, a non-transformer architecture, with a Rust-native training stack. The research question is whether SSMs can carry fast, inspectable tool-use without inheriting the transformer stack. This page ships 0.72TG-beta.
Research beta — honestly scoped, not a finished production model. Deeper reasoning is a separate architecture track.
Model highlights
| Lab | RtaForge — Frontier AI Research Lab |
| Program | ANVAYA |
| Architecture | RtaSSM v7.4 (“Tungsten”) — state-space LM |
| Parameters | 2.7B |
| Layers / width | 64 × 2560 |
| Tokenizer | EleutherAI GPT-NeoX (50,280 vocab) |
| Context | 4K trained & verified · 8K in active engineering |
| Role in the lab | Fast tool-calling probe (not a deep reasoner) |
| Training stack | Rust-native · Gurukul curriculum · Fortress/CLAIM3 |
| License | Anvaya Model License — free for personal/research use; org use is revenue-gated |
What this release is. An artifact from ongoing frontier work on SSM tool-use — architecture, curriculum, and forgetting protection included, not a LoRA on someone else’s base.
What this release is not. A general reasoner or a production agent runtime. Math/logic depth continues on a purpose-built track; Rabbit stays the tool-use lane.
Quick start
Requires the RtaForge runtime (loads the custom SSM implementation behind the Transformers wrapper):
pip install rtaforge transformers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "RtaForge/Anvaya-Rabbit-2.7B"
# Recommended checkpoint for this release:
# base/Anavya-Rabbit-2.7B-0.72TG-beta.safetensors (or .pt — byte-identical)
# (Historical spelling "Anavya" is retained in filenames.)
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
).eval()
# If your install expects an explicit weight path:
# model = ... # see rtaforge docs for checkpoint loading helpers
prompt = "List three tools you would call to answer a weather question, then stop."
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
out = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Tip: constrain
max_new_tokensand/or detect repetition loops in unconstrained generation — see Known limits.
Architecture
Rabbit belongs to the RtaSSM family — state-space language models under ANVAYA, RtaForge’s post-transformer research program.
| Component | Role in this release |
|---|---|
| Tungsten (RtaSSM v7.4) | Production training architecture for this lineage |
| Gurukul | Multi-phase curriculum (logic → math → unified bridge SFT) |
| Fortress / CLAIM3 | Engram-based forgetting protection on later phases |
| Constitutional validation | Training-time checks in the Rust stack |
0.71TG-beta was a unified bridge SFT pass (tool-calling + polish + STEM, conversational / masked-loss training on top of the 0.7TG math checkpoint) that closed a general-capability regression from the preceding math-focused phase while keeping the math floor intact.
0.72TG-beta adds one further, targeted pass: a dedicated math-instruct SFT on top of the 0.71TG bridge. Fortress/CLAIM3 protected the bridge (tool-call / general) manifold, so general benchmarks held within noise while GSM8K reached the best value on the lineage. The 0.71TG bridge remains the last broad SFT pass; deeper reasoning work continues on a separate architecture.
Evaluation
Evaluated with lm-eval-harness.
ARC-Easy / ARC-Challenge / HellaSwag / WinoGrande: 0-shot. GSM8K: 5-shot.
ARC / HellaSwag report normalized accuracy (acc_norm); WinoGrande reports accuracy (acc); GSM8K reports exact match (flexible).
0.72TG-beta vs 0.71TG-beta (Phase 4 math-instruct)
| Task | 0.71TG-beta | 0.72TG-beta |
|---|---|---|
ARC-Easy (acc_norm) |
0.5960 | 0.5905 |
ARC-Challenge (acc_norm) |
0.3746 | 0.3669 |
HellaSwag (acc_norm) |
0.6286 | 0.6313 |
WinoGrande (acc) |
0.6298 | 0.6346 |
| GSM8K 5-shot (strict / flex) | 0.0713 / 0.0720 | 0.0857 / 0.0857 |
| Tool-call loop rate | 16.9% | 9.1% |
| GSM8K free-gen loop rate | 7.8% | 10.6% |
0.72TG is the current release. General benchmarks vs 0.71TG sit inside noise; GSM8K is the best value on the lineage; tool-call loop rate roughly halves. GSM8K free-generation loop rate ticks up (7.8% → 10.6%) — a real trade on unconstrained math gen.
Scores by checkpoint
| Task | 0.5TG | 0.6TG | 0.7TG | 0.71TG | 0.72TG | 1.0 |
|---|---|---|---|---|---|---|
| ARC Easy | 0.5833 | 0.6174 | 0.5741 | 0.5960 | 0.5905 | — |
| ARC Challenge | 0.3703 | 0.3831 | 0.3660 | 0.3746 | 0.3669 | — |
| HellaSwag | 0.6063 | 0.6485 | 0.6228 | 0.6286 | 0.6313 | — |
| WinoGrande | 0.6164 | 0.6259 | 0.6417 | 0.6298 | 0.6346 | — |
| GSM8K (5-shot, flex) | — | — | 0.0705 | 0.0720 | 0.0857 | — |
1.0 is reserved for the next public milestone.
How to read this. 0.6TG (GAPA Phase 1 Logic final, pearl_step_122032) is the local peak on ARC-Easy / ARC-Challenge / HellaSwag. The Math phase (0.7TG) traded some of that general-benchmark performance for GSM8K capability — the 0.6→0.7 dips on ARC are ~1–2σ (stderr typically ±0.010–0.014 on ARC/WinoGrande, ±0.005 on HellaSwag): real but modest. The 0.71TG bridge partially recovers ARC while keeping the math floor. 0.72TG's math-instruct pass then moves GSM8K to its best lineage value with ARC / HellaSwag / WinoGrande deltas vs 0.71TG inside 1σ.
GSM8K is blank for 0.5TG and 0.6TG because those suites did not include it — it was only added when the Math phase landed. Not missing data; not measured.
Provenance. 0.5TG: rabbit_dagger_tungsten_2026-07-06 (0.5-alpha-base / Tungsten baseline eval). 0.6TG: eval_results/rabbit_dagger_phase1_final_2026-07-23/ — lm-eval-harness, HF backend, bf16, converted non-strict to Mamba2ForCausalLM (missing=0, unexpected=0), L4. 0.7TG / 0.71TG / 0.72TG: same harness conventions as published on this card.
Bridge run detail (0.7TG → midpoint → 0.71TG)
| Task | 0.7TG | bridge @10k | 0.71TG-beta |
|---|---|---|---|
| ARC Easy | 0.5741 | 0.5888 | 0.5960 |
| ARC Challenge | 0.3660 | 0.3712 | 0.3746 |
| HellaSwag | 0.6228 | 0.6296 | 0.6286 |
| WinoGrande | 0.6417 | 0.6298 | 0.6298 |
| GSM8K (5-shot) | 0.0705 | 0.0804 | 0.0720 |
We do not train against these benchmarks or select checkpoints by them. They are smoke-test signals for the lineage, not the product target (tool use).
Known limits
- Math ceiling ~8–9% GSM8K on this lineage — 0.72TG raised GSM8K to the best value here (0.0857) but did not break the ceiling. Failures are real reasoning errors, not formatting mismatches; deep multi-step math is not this architecture's lane.
- WinoGrande took a one-time ~1.2pt cost from the tool-call / polish mix; it did not degrade further with more bridge / instruct training.
- Repetition loops — tool-call loop rate 16.9% → 9.1% on 0.72TG; GSM8K free-gen loop rate 7.8% → 10.6%. Constrain
max_new_tokensand/or detect repeats in deployment.
Checkpoints
Supported baseline for continued work:
base/Anavya-Rabbit-2.7B-0.72TG-beta.safetensors ← current (0.72TG-beta)
base/Anavya-Rabbit-2.7B-0.72TG-beta.pt ← byte-identical to the .safetensors
base/Anavya-Rabbit-2.7B-0.71TG-beta.pt
base/Anavya-Rabbit-2.7B-0.7TG-beta.pt
base/Anavya-Rabbit-2.7B-0.6TG-beta.pt
base/Anavya-Rabbit-2.7B-0.5TG-beta.pt
All base checkpoints are safetensors-format weights; earlier releases carry a .pt extension for continuity with prior tooling. 0.72TG-beta is the first to ship both extensions (identical bytes) — prefer .safetensors.
Earlier prototypes under deprecated/ are kept for historical reference and are not recommended as training starts.
| Capability | Status |
|---|---|
| Training | ✅ |
| Continued training | ✅ |
| 4K context training | ✅ Verified |
| 8K context training | Active engineering |
Roadmap
| Release | Focus |
|---|---|
| 0.5TG-beta | Tungsten baseline |
| 0.6TG-beta | Logic phase |
| 0.7TG-beta | Mathematics phase |
| 0.71TG-beta | Unified bridge SFT (tool-calling + polish + STEM) |
| 0.72TG-beta | Dedicated math-instruct pass — best-lineage GSM8K, general capability held |
| 1.0 | Instruction / tool-call polish, expanded benchmarks |
Rabbit stays the fast tool-use lane. Reasoning-depth work continues on a separate architecture.
License
Released under the Anvaya Model License Agreement. In brief:
- Free for an individual’s own personal use or independent research.
- Organizational use (by or for any company, non-profit, or other entity, regardless of revenue) requires either staying under the revenue threshold (INR 1 crore / USD 100,000 per year, whichever is lower) or an Enterprise License.
- Resale, paid bundling, or hosted API access to the model or a derivative always requires RtaForge’s prior written permission, regardless of revenue.
- Derivatives and fine-tunes must credit both Anvaya-Rabbit-2.7B and RtaForge OPC Private Limited by name.
- No content safeguards are mandated — you are solely responsible for what you generate and how you deploy it.
The full LICENSE.md is binding; this summary is for convenience only.
About RtaForge
RtaForge is a Frontier AI Research Lab. We invent and train post-transformer state-space systems end-to-end — architectures, training protocols, inference runtimes, and models — rather than fine-tuning imported transformer bases.
ANVAYA is the research program behind this release: RtaSSM architectures (Tungsten and siblings), Rust-native training (Anvil / Gurukul / Fortress), weight migration (Subsuminator), and a model family (Rabbit → tool-use; Raccoon and others → reasoning and beyond). Related public work includes structural heists and methodology notes such as FORGEry.
Rabbit is one artifact from that program — evidence, not the whole lab.
- Org: huggingface.co/RtaForge
- Heists: github.com/Rta-Forge/heists-galore
- Contact: guha@rtaforge.in
© RtaForge OPC Private Limited
- Downloads last month
- 968