The nano variants train at 900k tokens/s on an rtx 6000 pro.

Say hello to Castor and Pollux.

One is Castor, the rotary softmax structure with a fixed lookup table. Pollux is rotary splat/softmax alternating 3/3.

Each 6 blocks, between 8 and 9 million params.

All tests show they each have their own strengths that the other cannot encompass. The twins are meant to be combinatory students, meant to learn and recall more data together than they otherwise cannot handle independently.

I have also named the 2s variant twins, given them individual names.

Both Pollux and Castor are learning a form of dynamic context window division experimentally currently.

Experiments show the models can support multiple dynamic sub-sizes to the primary context window of 4096 with each model supporting different sizes and forms dubbed golden framing, exactly as predicted from the larger structure's experimental outcome and training data's curves would signify.

These nano twins show the aleph nature of the model isn't just externalizing, but it can be directly internalized and curated as well. Experiments show the bpb per frame is stronger than others for multiple frames, tested around 400 million bytes per in roughly 6-7 minutes per train give or take.

The models train at roughly 890k tok/s on the rented colab rtx 9000 pro, and they only require about 5 gigs vram to fully saturate the card's calculation capacity.

These models aren't meant to be intelligent, they are meant to show the differentiated capacity of the hyperbolic system when correctly shaped. They don't need to remember everything, they need to show WHERE they are stable, WHY they are stable there, and provide A REASON as to why they are and aren't stable there.

Cantor sets and fractal numeric systems are fickle and unstable numeric calculations.

These systems are not unstable.

Assigning a special token to frames is already happening. Larger frames within the context window allow larger coverage within. Sub-frames allow sub-framing within. 1024 made of multiple sub-slices to fit the window space, 2048 sliced into multiple 1024's.

So far Pollux is the only one capable of this, standard softmax variation does not yet support this behavior at this size. However, the tests are showing the twins don't need to stay independent. They can be combined together into co-dependent structures, both trained on the same data meant to communicate and arbitrate in their own ways.

Castor can read very effectively. Stores documentation and understanding within reasonable lookup space and can recall this information.

Pollux can combine effectively. Creating dynamic window shapes, scaling paradigms, structural undertones, and systems of complexity that Castor cannot manifest at this scale.

Strange concept I know. The dynamic context frame slicing of Pollux is inherently hard to understand, but the concept is something that we as people share. We don't keep a thousand questions on the immediate tip of our mind to get one answer, we tug on the structure of understanding and follow the memory chain. With that we find the tapestry of logic, but only if we need this logic.

Both of these are capable of part of that whole, but they aren't complete yet.