FoundationalTS — baseline checkpoints, mechanisms, and evaluation guide

Checkpoints and reproduction instructions for the FoundationalTS results table. All opentslm-* files derive from google/gemma-3-270m and the released OpenTSLM checkpoints (OpenTSLM/gemma-3-270m-m4-{sp,flamingo}, apache-2.0); use is subject to the Gemma Terms of Use.

---

1. How each baseline works

OpenTSLM-SP (Soft Prompt) — opentslm-sp_*.pt

A frozen gemma-3-270m reads a text prompt in which each time series appears as a span of continuous "soft tokens" between two added special tokens <ts>...</ts>. The soft tokens come from a small patch encoder (patch size 4, embed dim 128, learned positional embedding) followed by a projector into the LLM embedding space (128 -> 640). Trainable in phase 2: encoder + projector + LoRA r=4/alpha=8 on the backbone = 3.1M parameters. The checkpoint stores ONLY encoder + projector + LoRA state + the resized input embeddings — the backbone itself is not included (see §3).

OpenTSLM-Flamingo — opentslm-flamingo_*.pt

The series is encoded by the same patch encoder, but instead of splicing soft tokens into the prompt, a perceiver resampler compresses the patch sequence and gated cross-attention layers interleaved with the frozen LLM's blocks attend to it (open-flamingo style). Trainable: cross-attention + perceiver = 317M parameters. The checkpoint stores the full model_state (~3.3 GB).

Lotus (ours) — lotus_gemma-3-270m_joint-5task-finetune.pt

Not a baseline: this is the model the paper proposes. A patch encoder embeds each 32-sample patch through two parallel branches — one reading the samples, one reading the rfft spectrum — with 2D sin-cos positional encoding over (time, channel). A projector maps the result into the frozen google/gemma-3-270m residual stream as soft tokens, and LoRA (r=4, alpha=8) adapts the backbone. 1.9M trainable (encoder 96K + projector 902K + adapters 949K) against OpenTSLM-SP's 3.1M, verified from the checkpoints themselves. Trained with the joint-5task protocol below; the released file is the epoch-11 best by generative exact match (val loss 0.714). The checkpoint also stores a decoder stack (decoder_state, decoder_projector_state, mse_projector_state) used only in phase-1 pretraining — it receives no gradient in phase 2 and is not counted as trainable.

The two training protocols (the suffix on every filename)

• joint-5task-finetune (our recipe): ONE phase-2 stage on all five tasks simultaneously — HAR, Sleep-EDF, ECG-QA, PAMAP2, TimeMQA — each resampled to 40,000 rows/epoch; best model by mean generative exact-match on a 192-sample/task validation basket; early stop when any task stagnates 8 epochs. Warm-started from OpenTSLM/gemma-3-270m-m4-*.
• sequential-curriculum (OpenTSLM's published recipe): one task at a time (released ECG checkpoint -> PAMAP2 -> TimeMQA), patience 5, best model by validation loss. Included to demonstrate catastrophic forgetting: the terminal model retains essentially only its final stage.

Tokenized time-series, zero-shot (no checkpoint — by construction)

The unmodified base LLM receives the series serialized as text with Gruver et al. (2023)'s digit-level encoding ( 1 2 3 , -4 5 0 , — digit-spaced, no decimal point, per-series rescale) and is evaluated directly. There are no weights to distribute; reproduction = base model + the eval code.

Tokenized finetuned (LoRA) — adapters to be added when training completes

Same Gruver serialization, backbone fine-tuned with LoRA r=8/alpha=16 (q,k,v,o,gate,up,down projections), best model by validation loss. This is the OpenTSLM paper's "Tokenized finetuned" baseline.

---

2. Environment and code (Deakin cluster)

module load Anaconda3 && conda activate opentslm       # transformers 4.57.3, peft 0.17.1
CODE=/home/s223540177/FoundationalTS                    # evaluation code
DATA=/scratch/s223540177/FoundationalTS/data            # HAR/Sleep/ECG/PAMAP2 CoT csvs
TIMEMQA=/scratch/s223540177/timemqa_data                # TimeMQA csv
MANIFEST=/vast/s223540177/cik_work/eval_manifests/ecg_test_first5000.json

The ECG test split is pinned to a 5000-sample manifest with per-sample gold sha1 hashes; the eval hard-fails on any mismatch, so all models are guaranteed to be scored on identical rows. Do not evaluate ECG without it.

3. Evaluating each checkpoint

The evaluation entry point is one bash file: bashfiles/evaluate/eval_baselines.sh in the code repo (git clone https://github.com/Davido111200/FoundationalTS — private; ask Dai for access).

cd FoundationalTS
# Soft-Prompt checkpoint on HAR:
sbatch --job-name=sp.har --time=06:00:00 bashfiles/evaluate/eval_baselines.sh \
  sp ./opentslm-sp_gemma-3-270m_joint-5task-finetune.pt ./eval_out/sp_har har

# Flamingo checkpoint on ECG (pinned 5000-sample manifest applied automatically):
sbatch --job-name=fl.ecg --time=06:00:00 bashfiles/evaluate/eval_baselines.sh \
  fl ./opentslm-flamingo_gemma-3-270m_joint-5task-finetune.pt ./eval_out/fl_ecg ecg

# SP is slow (~10 s/sample) -- shard long tasks by row range:
sbatch --export=ALL,EVAL_SHARD_START=0,EVAL_SHARD_END=1000 --time=06:00:00 \
  bashfiles/evaluate/eval_baselines.sh sp <ckpt> ./eval_out/sp_har_s0 har

Runner argument: sp for opentslm-sp_*.pt, fl for opentslm-flamingo_*.pt. The script sets the required encoder geometry itself, applies the pinned ECG manifest when the task is ecg, and writes <out_dir>/<task>_test_predictions.jsonl.

Lotus (ours)

Use the joint runner. It sets the freq-branch encoder and the LoRA rank the checkpoint was trained with, and applies the pinned ECG manifest automatically:

# all five tasks, one at a time
for T in har sleep pamap2 timemqa; do
  sbatch --job-name=lotus.$T --time=08:00:00 bashfiles/evaluate/eval_baselines.sh \
    joint ./lotus_gemma-3-270m_joint-5task-finetune.pt ./eval_out/lotus_$T $T
done

# ECG: pinned 5000-sample manifest applied automatically by the script
sbatch --job-name=lotus.ecg --time=08:00:00 bashfiles/evaluate/eval_baselines.sh \
  joint ./lotus_gemma-3-270m_joint-5task-finetune.pt ./eval_out/lotus_ecg ecg

# HAR is 8222 rows -- shard it if you want it back sooner
sbatch --export=ALL,EVAL_SHARD_START=0,EVAL_SHARD_END=2056 --time=08:00:00 \
  bashfiles/evaluate/eval_baselines.sh joint \
  ./lotus_gemma-3-270m_joint-5task-finetune.pt ./eval_out/lotus_har_s0 har

eval_baselines.sh is mirrored in this repo if you do not have code-repo access yet, but it must be run from inside a clone (it resolves the repo root from its own path). Do not add --pe_base_c 10: every 270m number in the table below was produced without it, and adding it changes the channel positional encoding, so results would no longer be comparable. Expect a [geometry] warning about pe_base_c 10 vs 10000 — that warning is correct and expected.

Tokenized zero-shot (no checkpoint)

python $CODE/baselines/tokenized_llm/tokenized_sft.py \
  --mode eval --model_id google/gemma-3-1b-pt \
  --out_dir ./eval_out/zs_har --split test --tasks har \
  --max_len 4096 --max_values 512 --serializer llmtime

--serializer llmtime selects the Gruver encoding; the port is baselines/tokenized_llm/serialize_llmtime.py, verified against the original docstring examples. Long series are decimated to fit the token budget and the stride is recorded per sample.

Outputs and scoring

Every eval writes <task>_test_predictions.jsonl with idx / gold / pred / label_gold / label_pred. Scoring:

• label accuracy = case-insensitive exact match of label_pred vs label_gold (the extractor takes the LONGEST-candidate prefix after the final Answer:; beware: greedy regexes glue trailing text onto answers and cost one baseline 28pp before we fixed it);
• TimeMQA is scored by its own question_format column (MCQ letter accuracy over 469 lettered golds; true/false stance accuracy over 329), joined on gold TEXT, not idx;
• we additionally report an LLM-judge score per cell (Claude Haiku, T=0, closed label set, commit-or-fail rubric) — judged and exact-match numbers agree within ~3pp after the extractor fix.

4. Expected results (verify your reproduction)

checkpoint	HAR	Sleep	ECG	PAMAP2	MCQ	T/F
sp joint-5task	54.4	72.7	37.6	52.1	38.6	68.4
flamingo joint-5task	62.6	70.4	31.1	54.7	12.4	3.0
Lotus joint-5task (ours)	58.4	70.5	33.9	49.2	33.5	74.5
sp sequential-curriculum	1.5	0.0	17.0	0.0	41.4	70.5
flamingo sequential-curriculum	3.8	0.0	5.4	10.3	20.0	4.9
tokenized zero-shot (1b)	0.0	0.0	0.0*	0.0	5.8	10.3

*exact-match may report up to ~3.5 on ECG from label words echoed inside degenerate output; the judge scores it 0.04. Majority-class floors: HAR 28.3 / Sleep 45.3 / ECG 19.3 / PAMAP2 12.8 / MCQ 43.3 / T-F 59.3.

5. Other files

file	note
eval_baselines.sh	mirror of the code repo's evaluation entry point, for convenience
lotus_gemma-3-270m_joint-5task-finetune.pt	our model: patch encoder (patch 32, freq branch) + projector into a frozen gemma-3-270m with LoRA r=4/alpha=8. 1.9M trainable. Phase-2 best by generative EM, epoch 11, val loss 0.714. Load exactly as the sp joint-5task baseline but with --enc_dec_type mlp --use_freq_embed --pe_type sincos --pe_base_c 10 --patch_size 32 (the checkpoint also carries encoder_geometry for verification).

Questions: Dai (daidv1112) — training logs, per-epoch curves, launch scripts and all per-cell predictions (both metrics) are archived and available.