WiggleGPT

A 124M parameter transformer that challenges a 57-year-old assumption in neural network design.

WiggleGPT is a decoder-only language model developed by Eden (Phillip C. O'Brien). It replaces the usual MLP activation with a learnable oscillating activation, and was pretrained from scratch on OpenWebText. This repository contains the pretrained model, an instruction-fine-tuned model, and two separately trained task adaptations. The implementation is based on nanoGPT and uses custom PyTorch code rather than the Transformers AutoModel interface.

At a glance: The pretrained model is for English text continuation and architecture research; the SmolTalk2 checkpoint adds instruction fine-tuning. The SST-2 and RACE-middle checkpoints are separate classifiers trained from the pretrained model. The weights use WiggleGPT's native PyTorch implementation and the GPT-2 tokenizer. Code and weights are released under GPL-3.0.

Source code and development log · WiggleGPT paper · Report an issue

Results · Choose a checkpoint · Architecture · Intended uses and limitations · Run the model · Training and emissions · Citation

The schematic labels a RoPE position-encoding stage. In the released native implementation, the RoPE path rotates queries and keys by attention head rather than token position; the paired rotations cancel in the attention scores apart from numerical rounding. The model therefore has no effective explicit positional encoding, while causal attention still supplies prefix structure. See Architecture.

What Makes It Different?

Since Minsky and Papert's Perceptrons (1969), much of neural network design has used monotonic activation functions, such as sigmoid and ReLU. A single conventional threshold neuron cannot solve a non-linearly separable problem such as XOR.

WiggleGPT replaces monotonic activations with learnable oscillating functions, enabling single neurons to create multiple decision boundaries:

f(x) = sin(ωx + φ) · tanh(x) + baseline

Where ω (frequency) and φ (phase) are learnable per-neuron parameters.

Results

Model	Parameters	Val Loss	Notes
WiggleGPT	124M	3.1621	Oscillating activation
GPT-2	124M	~3.12	Standard GELU baseline

Training recipe: WiggleGPT follows Andrej Karpathy's nanoGPT GPT-2 (124M) reproduction (config/train_gpt2.py): 600,000 iterations, a 6e-4 peak learning rate with 2,000 warmup steps, cosine decay to 6e-5, weight decay 0.1 and dropout 0.0. Karpathy's config spreads about 0.5M tokens per step across 8 A100s; as nanoGPT's README suggests for single-GPU training, WiggleGPT tunes the batch to its hardware, using 32 sequences (32,768 tokens) per step on one consumer GPU. The ~3.12 GPT-2 figure is from nanoGPT's baselines table: OpenAI's released GPT-2 checkpoint evaluated on OpenWebText.

The recorded losses are within approximately 1.3% of one another. WiggleGPT's result demonstrates that oscillating activations can be trained at the 124M scale. The GPT-2 number is a historical reference, rather than a new matched evaluation in this repository. These WiggleGPT results were obtained with the native rotation behaviour described below; GPT-2 uses learned position embeddings, so the models have different positional setups.

The pretrained model's 3.1621 is next-token cross-entropy averaged over 100 randomly sampled OpenWebText validation minibatches at the final evaluation; it is not a pass over the entire validation set. The sentiment and reading scores below belong to their respective adapted classifiers, not the pretrained or SmolTalk2 checkpoint.

Task-Specific Fine-Tuning — 26 October 2026

I revisited the original pretrained WiggleGPT and fine-tuned it separately for sentiment classification and multiple-choice reading comprehension, following the task-adaptation approach used in GPT-1.

Task	Evaluation split	Before task training	After task training	Improvement
SST-2 sentiment	Public validation, 872 examples	437/872 — 50.1%	786/872 — 90.1%	+40.0 percentage points
RACE-middle reading	Test, 1,436 questions	385/1,436 — 26.8%	854/1,436 — 59.5%	+32.7 percentage points

“Before” is the original pretrained backbone with a newly initialized task classifier. “After” is the validation-selected task-trained checkpoint using the same input format and classifier. These rows measure two independent adaptations from the original 600,000-step pretrained model. They are separate from the SmolTalk instruction model.

Both tasks completed three full epochs. The entire native backbone was trained with classifier cross-entropy plus an auxiliary language-model loss weighted by 0.5, effective batch size 32, peak learning rate 6.25 × 10⁻⁵, 0.2% linear warmup and linear decay. The oscillating activation, RMSNorm and native head-indexed rotation path were retained. Classification uses the final task-marker state with a 768→2 head for SST-2 and a shared 768→1 head across the four RACE answers.

Training detail	SST-2	RACE-middle
Training examples	67,349	25,421
Validation examples	872	1,436
Completed / best update	6,315 / 6,315	2,385 / 2,385
Task-head parameters	1,538	769
Total trained parameters	123,699,458	123,698,689
Stopping cause	Three epochs completed	Three epochs completed

Checkpoint selection used highest validation accuracy, with classification loss breaking ties. SST-2 is reported on public validation because public test labels are hidden; that validation split also selected its checkpoint. RACE-middle test was evaluated after validation selection. Inputs were capped at 512 tokens. No SST-2 inputs were truncated; the article prefix was shortened for 210 training, 9 validation and 14 test RACE questions, preserving the complete question and answer choices.

For historical context, GPT-1 reported 91.3% on SST-2 GLUE test and 62.9% on RACE-M, with RACE training covering both middle- and high-school data. WiggleGPT's SST-2 row uses public validation and its RACE adaptation uses middle-school data only, so these historical figures describe different evaluation splits and training coverage.

Data: stanfordnlp/sst2, ehovy/race — middle. The shareable comparison record retains every evaluated ID, gold label, before/after prediction, correctness flag and source-text hash. Checkpoint metadata records the inference exports and their SHA-256 hashes.

The Model Actually Learned to Oscillate

Parameter	Init	After Training	Change
ω mean	1.0	1.096	+9.6%
ω std	0.1	0.602	6× increase
ω range	[0.8, 1.2]	[-0.19, 5.17]	Massive expansion

• 95% of neurons retained active oscillation (ω > 0.1)
• Some neurons learned angular frequencies up to ω = 5.17 radians per input unit (about 0.82 cycles per input unit in the sinusoidal factor)
• Full phase coverage [-π, +π] after training

Checkpoints

File	Description
model.safetensors	Inference weights exported from the original pretrained checkpoint; native WiggleGPT architecture required
ckpt_pretrain.pt	Base model trained on OpenWebText (~600k iterations)
ckpt_finetune.pt	Fine-tuned on SmolTalk2 (instruction following)
task_adaptations/sst2/model.pt	Independent full-model SST-2 adaptation with sentiment classifier; inference-only export
task_adaptations/race_middle/model.pt	Independent full-model RACE-middle adaptation with four-choice classifier; inference-only export

Architecture

Component	Specification
Parameters	123,697,920
Layers	12
Attention Heads	12
Embedding Dimension	768
Oscillating Neurons	36,864 (each with learnable ω, φ, baseline)
Normalization	RMSNorm
Position encoding	No effective explicit position encoding: the native RoPE path rotates by attention head, not token position
Attention	Flash Attention (when available)
Context length	1,024 tokens for pretraining
Tokenizer	GPT-2 / tiktoken; padded vocabulary of 50,304 entries
Weight tying	Input token embedding and output projection share weights

The parameter count includes the shared embedding once. The oscillating activation has a learned frequency, phase and baseline for each feature: sin(ωx + φ) · tanh(x) + baseline. In the native RoPE path, each attention head applies the same rotation to queries and keys at every position. Those paired rotations cancel in their dot products apart from floating-point rounding, so this path does not encode position. The causal mask remains active. The released checkpoints were trained with this behaviour; retain the native implementation when loading them.

Intended Uses

The pretrained model is intended for text continuation and research into learned oscillating activations in language models. The SmolTalk2 checkpoint is an instruction-fine-tuned continuation of that model. The SST-2 and RACE-middle checkpoints are independent task-specific classifiers, using their own native runtime and input layouts. Researchers can use the checkpoints, code and recorded predictions to study the architecture or build on the work under GPL-3.0.

Limitations and considerations

The model can generate incorrect, repetitive, biased or unsuitable text, and its outputs should be checked for the intended use. OpenWebText is web-derived and may contain biases, offensive material and factual errors; instruction fine-tuning does not remove those properties. The native head-indexed RoPE path requires an input of at least 12 tokens; shorter prompts need padding. The reported task scores describe the stated datasets and evaluation splits, not a general measure of instruction following or reliability. The SST-2 validation split was also used for checkpoint selection. The released checkpoints are research models and have not been established as suitable for consequential decisions without further evaluation for the application.

Usage

See the GitHub repository for the native model implementation, tokenizer setup, full sampling script and chat interface. These weights do not load through AutoModel.from_pretrained: model_bio.py defines the architecture. Download the pretrained checkpoint from this repository and run the following from the GitHub project directory after installing PyTorch:

# Quick inference example
import torch
from model_bio import GPT, GPTConfig

# Load checkpoint
checkpoint = torch.load('ckpt_pretrain.pt', map_location='cpu', weights_only=False)
config = GPTConfig(**checkpoint['model_args'])
model = GPT(config)
state = {key.removeprefix('_orig_mod.'): value for key, value in checkpoint['model'].items()}
model.load_state_dict(state, strict=True)
model = model.to('cuda:0').eval()

# See sample_bio.py for GPT-2 tokenization and text generation.

The .pt checkpoint contains Python-serialized training state; load it only from a source you trust. model.safetensors contains pretrained model tensors only and is an alternative for weight loading. See the SafeTensors loading instructions below for the shared embedding.

Generate text with the pretrained SafeTensors weights

The following example downloads the native model, its exact GPT-2 tokenizer and the pretrained weights. Use Python with a CUDA-enabled PyTorch installation; no Transformers AutoModel is required.

pip install huggingface_hub safetensors tiktoken
hf download edeneldith/WiggleGPT model.safetensors wigglegpt_config.json task_finetuning/native/model_bio.py task_finetuning/task_runtime.py task_finetuning/tokenizer/gpt2.tiktoken --local-dir ./wigglegpt
cd wigglegpt

Run this Python example from the download directory:

import json
import sys

import torch
from safetensors.torch import load_file

sys.path.insert(0, 'task_finetuning')
sys.path.insert(0, 'task_finetuning/native')
from model_bio import GPT, GPTConfig
from task_runtime import get_encoder

with open('wigglegpt_config.json', encoding='utf-8') as f:
    config = GPTConfig(**json.load(f))
with torch.device('cuda:0'):
    model = GPT(config)
state = load_file('model.safetensors')
# The output projection and token embeddings are tied in the native model.
state['lm_head.weight'] = state['transformer.wte.weight']
model.load_state_dict(state, strict=True)
model.eval()

encoder = get_encoder()
prompt = 'The discovery of a new kind of neuron changed the way the researchers thought about language and learning.'
tokens = encoder.encode(prompt)  # Native RoPE requires at least 12 input tokens.
x = torch.tensor([tokens], dtype=torch.long, device='cuda:0')
torch.manual_seed(1337)
with torch.inference_mode():
    output = model.generate(x, max_new_tokens=64, temperature=0.8, top_k=50)
print(encoder.decode(output[0].tolist()))

The SafeTensors file stores the shared embedding once, so its tensor count matches the model's 123,697,920 unique parameters. It contains the pretrained model, not the SmolTalk2 or task-adapted weights, and does not include optimizer state for resuming training. Its source is the published ckpt_pretrain.pt (SHA-256 88afaab6c93ef3e33940b4e25cbce2f44038c428109600afe8285586a3a767b9); all exported tensors were checked against that checkpoint. Hugging Face can read the parameter count from the SafeTensors header for its model-size metadata.

Generation parity: on the RTX 5060 Ti, the pretrained .pt checkpoint and SafeTensors export produced identical token IDs for a 64-token continuation from the same 19-token prompt, with seed 1337, temperature 0.8 and top-k 50. Both paths used the native model, identical configuration and strict weight loading.

Task Classifier Usage

The task checkpoints contain the full adapted backbone and a custom classifier. They use the native WiggleGPT wrapper; they are not Transformers AutoModel checkpoints. They are inference exports, with optimizer and RNG state removed.

Download task_adaptations/ and the accompanying task_finetuning/ runtime. Install task_finetuning/requirements.txt and prepare the dataset token caches before running the evaluators:

pip install -r task_finetuning/requirements.txt
python task_finetuning/prepare_data.py
python task_finetuning/encode_task_data.py
python task_finetuning/evaluate_tasks.py --checkpoint task_adaptations/sst2/model.pt --output task_finetuning/results/sst2_recomputed.json
python task_finetuning/evaluate_tasks.py --checkpoint task_adaptations/race_middle/model.pt --output task_finetuning/results/race_middle_recomputed.json

The task runtime guide gives exact commands for task scoring and re-evaluation. Each export also includes model_args, the marker IDs, original tokenizer identity, task input layout, training configuration and metric metadata. Use the accompanying exact native model snapshot in task_finetuning/native/model_bio.py.

Training Details

Pretraining:

• Dataset: OpenWebText, approximately 9 billion tokens in the prepared corpus, using the GPT-2 tokenizer.
• Training: Andrej Karpathy's nanoGPT GPT-2 (124M) reproduction recipe (config/train_gpt2.py), 600,000 iterations; context length 1,024 and effective batch of 32 sequences (32,768 tokens) per iteration, tuned down from the recipe's ~0.5M tokens per step to fit one consumer GPU. The 3070 used batch size 2 with 16 accumulation steps; the 5060 Ti used batch size 4 with 8 accumulation steps. The final recorded train and validation losses were 3.1749 and 3.1621 respectively.
• Objective and optimisation: causal next-token cross-entropy, AdamW with a peak learning rate of 6 × 10⁻⁴, 2,000 warmup steps, cosine decay to 6 × 10⁻⁵, weight decay 0.1 and gradient clipping at 1.0. The final checkpoint records float16 training and 100 evaluation minibatches per split.
• Preprocessing: the provided OpenWebText preparation script uses GPT-2 tokenization, adds an end-of-text token to each document, assigns documents to validation with probability 0.0005 using seed 2357, and writes token IDs as uint16 arrays.
• Hardware: one RTX 3070 followed by one RTX 5060 Ti 16GB. The 3070 trained to the 240,000-step checkpoint; after the GPU change, the 5060 Ti resumed from that checkpoint on 5 November 2025. The 7 November development entry records the run at step 354,000, approximately two days after resumption.
• Calendar: the 124M run began on 23 October 2025 and completed on 12 November 2025. Training continued on the 3070 after the replacement GPU order was cancelled on 28 October. The machine was shut down for the upgrade around 14:00–15:00 on 29 October; training resumed just after 00:45 on 5 November, a pause of approximately 155 hours. Less than 35 minutes of work beyond the 240,000-step checkpoint were lost. Apart from this pause, the run operated around the clock.

Fine-tuning:

• Dataset: SmolTalk2, smoltalk_smollm3_smol_magpie_ultra_no_think split (406,843 conversations in the recorded preparation).
• Oscillation parameters (ω, φ) remained stable — 0.0% of neurons shifted by >0.1

Energy and carbon estimate

Pretraining's GPU electricity is estimated at 47.1–50.6 kWh, giving 8.33–8.95 kg CO₂e before solar. With the solar and battery scenario described below, the estimate is 3.47–5.22 kg CO₂e; the Hub metadata records the scenario midpoint of 4,321 g. These figures cover GPU electricity for pretraining, not the entire computer or subsequent fine-tuning.

Calculation, hardware and solar assumptions

The training was performed in the UK on a single consumer GPU at a time. Following the ML CO2 Impact calculator, an estimate is power × active time × grid carbon intensity. NVIDIA specifies reference graphics-card power of 220 W for the RTX 3070 and 180 W for the RTX 5060 Ti; those ratings are not measurements of the power used during this run. The UK government's 2025 electricity-generation factor is 0.177 kg CO₂e/kWh. For estimated average power in kilowatts, the calculation is:

kg CO₂e = (RTX 3070 kW × RTX 3070 hours + RTX 5060 Ti kW × RTX 5060 Ti hours) × 0.177

The development log records the start of the 124M run on 23 October at 21:05 and step 176,000 on 28 October at 01:36. Taking the start entry as approximately step zero and allowing for the UK clock change, that interval averages approximately 2.076 seconds per step on the RTX 3070.

The 7 November development entry records step 354,000 at 05:26; the 5060 Ti had resumed from step 240,000 on 5 November, roughly two days before that entry. Completion at step 600,000 was recorded at 02:58 on 12 November. The interval from step 354,000 to step 600,000 gives 1.72 seconds per step on the 5060 Ti, including evaluations.

The hardware split is approximately 240,000 steps on the 3070 and 360,000 on the 5060 Ti, corresponding to approximately 140 hours and 170 hours of GPU training respectively, or roughly 310 GPU-hours in total. Less than 35 minutes of work beyond the 240,000-step checkpoint was lost before the pause; it is small relative to the rounded time estimates. The 5060 Ti was approximately 1.2 times faster per step.

For the RTX 3070, an LCARS system report from 26 October 2025, during the 124M training period, recorded 175.0 W at 100% GPU utilisation, with a reported power limit of 176 W. The machine had a roughly 500–600 W power supply at the time, and GPU power limits could be adjusted in MSI Afterburner to manage heat and power use. LCARS reports from 20 and 22 October show roughly 197.5–201.9 W during the earlier 45M v2 run at a 220 W limit. To reflect capped and possible uncapped stretches, the calculation uses a 175–200 W 3070 power range; the earlier readings are a comparison, not measurements of the 124M run.

For the RTX 5060 Ti, an LCARS report on 5 November 2025 at 00:55 UTC recorded 132.6 W at 99% GPU utilisation. An LCARS diagnostic approximately three minutes later recorded 132.976 W at 99% utilisation. The two readings average 132.8 W; they describe one short observation window rather than the entire run. At that rate, approximately 170 hours on the 5060 Ti consume 22.6 kWh. For comparison, the COLM final training log recorded an average of 92.14 W across 50 post-evaluation samples for a different workload.

Both LCARS diagnostics databases and all eight report files were checked for power records. The remaining database power reading, 210.698 W on 20 October, is a test checkpoint predating the 124M run. Using the 3070 range and 5060 Ti average above gives 47.1–50.6 kWh of GPU electricity, or 8.33–8.95 kg CO₂e before solar. The snapshots do not establish the average power for the entire run.

The training site has home solar panels and a battery that was already installed when training began; the battery typically powered the house and garage for about 3–5 hours after dark in winter. The FoxESS LV5200 specification gives 5.12 kWh nominal capacity. This is storage capacity, not a measurement of the solar electricity used by the computer. The grid-only figure above does not account for the solar contribution.

Solar-adjusted estimate: training ran around the clock except during the pause for the GPU change. The house has solar panels and a battery that normally powered the house and garage for 3–5 hours after dark in winter; in late autumn the sun reached the site until around 5 pm. Using the working scenario of 7–9 daylight hours supplied by solar plus 3–5 hours from a solar-charged battery gives a solar share of roughly 42–58%. Combining that scenario with the 3070 power range gives 3.47–5.22 kg CO₂e of GPU operational emissions. The daytime coverage is not a logger measurement; generation, household demand and battery charging affect the actual share.

The model-card YAML reports 4,321 g CO₂e, the midpoint scenario using 187.5 W on the 3070 and a 50% solar share. This gives the Hub a single searchable value while the range and assumptions remain visible here. The calculation excludes the embodied and lifecycle emissions of the solar system.

The estimate accounts for the pause between GPUs and uses power snapshots from LCARS, the local system diagnostics tool named after Star Trek's Library Computer Access/Retrieval System, as proxies for average draw during each phase. It covers direct generation emissions for GPU electricity during pretraining only: grid transmission losses, CPU, memory, power-supply losses, other development runs, fine-tuning and embodied emissions are outside its scope. The calculator also excludes data-centre PUE; this run used a local machine.

Citation

@software{wigglegpt2025,
  author = {O'Brien, Phillip C.},
  title = {WiggleGPT: Revisiting the Monotonicity Assumption in Neural Networks via Oscillating Activation Functions},
  year = {2025},
  url = {https://github.com/Eden-Eldith/WiggleGPT}
}

Author

Eden (Phillip C. O'Brien)
Independent AI Researcher | ORCID: 0009-0007-3961-1182

Built in a garage lab in Gosport, UK, using consumer GPUs and without institutional funding.

License

GPL-3.0 — if you build on this, keep it open source. For technical questions or model-card corrections, use GitHub Issues.