Spectator
Weights of Spectator, the model of State of Mind: Learning and Evaluating World Models of Naturalistic Brain Dynamics, and the stored BrainStateBench scores of the paper's main table. Code: github.com/yuxiangwei0808/State-of-Mind.
Spectator keeps a 2,048-number state built from the last minutes of a viewer's fMRI (Schaefer-1000 parcels) and rolls it forward while they watch a movie. A linear encoder makes the state; it was trained in the JEPA sense against its own slowly updated copy and then frozen. One matrix, solved in closed form, predicts the next state from the state and the movie's TRIBE v2 features. A linear probe per viewer reads the 1,000 parcels from any state. The model and its training are described in the repository's README.
Files
Download into the repository's output root (outputs/ by default, or $NEUROWORLD_OUTPUT); the configs in the code find the files there:
hf download stanjsx/Spectator --local-dir outputs
| Path | Contents | Size |
|---|---|---|
spectator-v1/algonauts/model/ |
Spectator on Algonauts 2025 with 180 s of fMRI history (the paper's model): best.pt (every weight), config.yaml, probe.npz, probe-first-pass.npz, reference-rollout.pt |
181 MB |
spectator-v1/algonauts/encoder.pt, matrix.pt |
the frozen encoder and the final matrix as separate files, which neuroworld/configs/spectator/algonauts.yaml names (both are also in best.pt) |
117 MB |
spectator-v1/algonauts/lags.pt, jepa-start-matrix.pt |
where the JEPA phase starts (algonauts-jepa.yaml): stage 0's map with the closed-form lag weights, and the closed-form start matrix |
53 MB |
spectator-v1/hcp/ |
Spectator on HCP: the Algonauts encoder with its lag weights (lags.pt) and the matrix (matrix.pt) refitted on HCP; model/ as above, with one probe pooled over viewers |
253 MB |
spectator-v1/algonauts-7.5s/model/, algonauts-120s/model/ |
Spectator with 7.5 and 120 s of history (the main table's other columns): best.pt, config.yaml and the probes. Each history length has its own encoder, held in best.pt |
275 MB |
benchmark-results/ |
a BrainStateBench result store with the Algonauts scores of the main table's rows at every history length from 7.5 to 180 s; the linear forecaster's Updating is scored with the frames it is not shown filled by its own forecast (benchmark updating --own-fill), as the paper reports it |
522 MB |
The checkpoints hold weights and training metadata, with no optimizer state: they load, score and seed the closed-form refits, but gradient training cannot resume from them. Paths in their metadata are written as ${NEUROWORLD_DATA} and ${NEUROWORLD_OUTPUT}.
Use
From the repository root, with the code installed (neuroworld/requirements-dev.txt):
import json
import numpy as np
import torch
from neuroworld.train.readout import load_world_model
from neuroworld.utils.config import load_config
folder = "outputs/spectator-v1/algonauts/model"
cfg = load_config(f"{folder}/config.yaml")
model, _ = load_world_model(cfg, f"{folder}/best.pt", "cpu")
# brain: [B, 120, 1000], the last 120 fMRI frames (TR 1.49 s; 180 frames at 1 s on HCP), as the prepared datasets hold them.
# stimuli["context"]: [B, 44, 2048], TRIBE v2's contextual rows of the movie for a 40-frame rollout.
# reference-rollout.pt holds a fixed synthetic input of these shapes and the states it gives.
ref = torch.load(f"{folder}/reference-rollout.pt", weights_only=True)
with torch.no_grad():
states = model.rollout(ref["brain"], ref["stimuli"], horizon_trs=40) # [B, 40, 2048], equal to ref["states"]
probe = np.load(f"{folder}/probe.npz")
groups = json.loads(str(probe["meta"]))["groups"] # the viewers: Algonauts2025/01, 02, 03, 05
i = groups.index("Algonauts2025/01")
parcels = states.numpy() @ probe[f"W {i}"] + probe[f"b {i}"] # [B, 40, 1000]
probe-first-pass.npz is the probe Updating uses to fill the frames it does not show the model, before the second pass reads with probe.npz.
Scoring. The benchmark assembles the inputs from the prepared datasets (DATA.md). To score Spectator on every task:
python -m neuroworld.spectator score outputs/spectator-v1/algonauts/model --seconds 180 --dataset algonauts --store outputs/my-results --name spectator-180s
It fits the probe on the training recordings before scoring. A stored probe is reused only when the weights, the benchmark code and the data paths all match, so on another machine the probe is refitted. A refit reproduces the shipped probe up to floating-point rounding (largest relative difference 1.4e-6).
Comparing a model with the paper's rows. Score it into the downloaded store and print the benchmark card, which pools Friends S6 and the four Movie10 films and gives paired 95% intervals against the reference row:
python -m neuroworld.benchmark suite --dataset algonauts --model py:my_model.py:MyModel --output outputs/benchmark-results
python -m neuroworld.benchmark profile --output outputs/benchmark-results --reference "ridge-jepa-180s-std-partial (latent)"
examples/persistence.py shows the model interface.
Stored scores
Rows of benchmark-results/algonauts/ (<L> is the history length: 7.5, 15, 30, 60, 90, 120 or 180):
| Paper name | Row |
|---|---|
| Spectator | ridge-jepa-<L>s-std-partial (latent) |
| Linear forecaster | linear [context] <L> s |
| IPSID | IPSID [context] <L> s (IPSID [context] at 120 s) |
| NeuroWorld | NeuroWorld k1 <L>s (latent); NeuroWorld-style k1 (latent) at 7.5 s |
| TRIBE v2 | TRIBEv2 zero-shot (reads no fMRI) |
The splits are val (Friends S6), dev (Hidden Figures and Life) and test (The Bourne Supremacy and The Wolf of Wall Street); the paper reports all three pooled. Forecast scores are in <split>/<row>/results.json; State scores (the Movie10 repeats) in state/; Updating scores in updating/. The files hold summed errors per recording and per step, not forecasts. Each file's stamp records the benchmark code and the probe it was scored with. The card warns about four rows that combine two fits of one probe (same training samples and penalty): each Spectator row's test split was scored with a refit made after a benchmark code change (refits agree with the original to about 1e-6, relative), and NeuroWorld k1 180s (latent)'s Updating with a second fit. The paper's BrainVista and MIRAGE rows were scored by a collaborator and are not included.
Results
All Algonauts 2025 test data (Friends S6 and the four Movie10 films), 180 s of fMRI history, skill in % (higher is better); r is the parcel-averaged Pearson correlation over the 40 steps of the minute.
| Model | Forecast | State | Updating | r |
|---|---|---|---|---|
| TRIBE v2 (stimulus only) | 4.2 | 0.0 | 4.1 | 0.173 |
| Linear forecaster | 11.6 | 2.6 | 12.7 | 0.294 |
| IPSID | 8.7 | 1.2 | 12.5 | 0.243 |
| NeuroWorld | 8.3 | 0.4 | 13.4 | 0.239 |
| Spectator | 16.0 | 5.5 | 20.3 | 0.349 |
On HCP (7T movies, 184 viewers, 180 s of history), Spectator scores 6.6 / 6.2 / 15.7 on Forecast / State / Updating on held-out movie clips. On the test recording of the 36 viewers it was not fitted on, read with the probe shared across the 148 training viewers, it reaches 7.3 on Forecast and 16.7 on Updating.
Data
No fMRI data or stimulus features are included. Algonauts 2025 comes from the Algonauts challenge (CNeuroMod) and HCP from the Human Connectome Project; the repository's preparation scripts build the inputs the models read.
License
CC BY-NC 4.0. Spectator is trained on and reads features made by TRIBE v2, which is released under CC BY-NC 4.0.
Citation
State of Mind: Learning and Evaluating World Models of Naturalistic Brain Dynamics.