Contents4 sections
I trained rivet-1b because I wanted to understand the whole model-building stack, not just the parts exposed by fine-tuning APIs. It would have been easier to start from someone else's weights. That would also have skipped the part I was trying to learn.
rivet-1b is a 1.09B parameter language model trained from scratch: tokenizer, corpus, pretraining, post-training, evaluation, and release. The paper is linked here, and the model weights are on Hugging Face.
Why train from scratch
Fine-tuning is useful, but it hides a lot. You inherit a tokenizer, a pretraining mix, loss curves you didn't watch, architectural choices you didn't make, and failure modes you can only infer from the outside. That is fine when the goal is to ship a task model quickly. It is less useful when the goal is to understand what is happening underneath.
Training from scratch is slower, messier, and less forgiving. That is the point. You see every part of the system: how the corpus behaves, how the tokenizer shapes the model's world, how loss moves, how evals lie, and how quickly a small assumption becomes expensive once GPUs are involved.
What rivet-1b is
The model uses a Qwen3-family architecture, a 32k SentencePiece tokenizer, and a custom corpus. It was pretrained locally on roughly 130B tokens across two RTX 5090s, then post-trained into a chat-capable model. No weights, embeddings, or tokenizer state were inherited from another model.
That does not make it a frontier model, and it was never meant to be one. It is a working research model: small enough to train and inspect personally, large enough to expose the real shape of the training stack.
What it taught me
The useful lessons were mostly not glamorous. Tokenization matters more than people want it to. Data mistakes become model behaviour. Eval variance is real. Throughput and memory ceilings shape research choices before theory gets a vote. A clean chart can hide a bad run, and a messy run can still teach you something if you kept enough evidence.
That is why I care about building models and agents together. Models provide capability. Agents decide how that capability is used, what evidence travels with it, and where a human gate belongs. A small model trained end to end is a good reminder that intelligence is not one thing. It is a stack.
Why publish it
I published the paper and weights because I think small-model work benefits from more concrete examples. Not grand claims, not benchmark theatre, just enough detail for someone else to inspect the choices and run their own comparisons.
If you're working on small models, local training, or the systems around them, the paper is the best place to start.
