ExeaLM

An open language model built by high schoolers, run by Exea Labs.

No weights are released yet. Checkpoints and results will be posted here as they exist.

Links

• Code: https://github.com/exea-labs/ExeaLM
• Contributor sign-up: https://docs.google.com/forms/d/e/1FAIpQLSdPHVZhuXxBDUf-cJM2zDbj1JZPsduu95Y_kTJ_g4gKAXdDVw/viewform
• Architecture paper: TONN: Networks That Build Their Own Architecture from Data

Model

• Architecture: TONN, a sequence model that builds its own architecture from data. The paper is a concept paper and has not been peer reviewed.
• Parameters, context length, tokenizer, languages: undecided.
• License: Apache 2.0 for code and weights.

Training data

Planned: about 100 billion tokens, selected from the NII LLM-jp Corpus v4, the data released with K2 Horizon, FineWeb-Edu, DCLM-Baseline, and others. The count is set by diminishing returns: past about 200 billion tokens, extra data buys little for a model of 0.1 billion (100 million) parameters. The full mix, dataset links, and the math are in DATA.md. A data card will list sources, licenses, filters, and the final mixture before training starts.

Intended use

Research and education on how open language models are built. Not for high-stakes decisions.

Limitations

No evaluation results yet. Like other web-scale models, released checkpoints may produce incorrect, biased, or harmful text. We will publish evals and known limitations with each checkpoint.

Contributing

Anyone can contribute through GitHub. See CONTRIBUTING.md and the Code of Conduct.