GPT 3.3
The final Gen 3 model, and a real architectural break from the rest of the lineup: a move from RWKV-style recurrence to a transformer, aimed squarely at the attention performance RWKV kept struggling with.
These are GPT 3.3's final architecture specs — see the Learn pages for what each one means.
What's known so far
Every WillMe model through GPT 3.2 has used a GRU or RWKV-like recurrent architecture — a fixed-size state carried token to token, no attention. GPT 3.3 tests something different: a genuine transformer, trading that fixed-size recurrent state for attention over the full context window. The goal is closing the gap where RWKV specifically underperformed.
It's the last model under the Gen 3 name — an experiment run inside the generation we already know, rather than a leap straight to a new one. If it validates the approach, Gen 4 becomes a full transformer generation, and Gen 5 after that is planned to explore a transformer/RNN hybrid with linear-attention-style mixing.
At 100.7M parameters, GPT 3.3 is noticeably smaller than GPT 3.1/3.2's 163M — and the training data tells a similar story. Pre-training moves to a revised, cleaned, and improved 2B token set — smaller than 3.2's 11B, but higher quality throughout.
Fine-tuning data got the same treatment: the source messages were cleaned and reduced by 84%, and combined with reworked conversation grouping, that leaves 2.6k conversations (from 7.9k messages) to fine-tune on — a 95% reduction from the 53,000-conversation sets behind GPT 3.1 and 3.2. The bet is that less, cleaner data beats more, noisier data.
A note
GPT 3.3 is approved for style, not for improved intelligence: it answers in William's voice, briefly and casually, and can be crude, off-topic or fall back on stock replies. For more coherent answers, GPT 3.2 remains the stronger general chat model.