No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from DedeProGames/GPT-U-20M
All models by DedeProGames →
A 20.5M-parameter Llama-architecture language model pretrained from scratch on 2.6B tokens of web text, educational text and code.
The 16,384-token vocabulary keeps the embedding at 31% of the parameters (a 50k vocabulary would need 19.3M parameters at width 384) and lets tokens be stored as uint16.
Tokenizer. Byte-level BPE with a cl100k-style split that keeps whitespace runs whole (Python indentation), one token per digit, and <|endoftext|> (id 0) as BOS/EOS/padding. Round-trip decode(encode(x)) == x holds on 1,000 held-out samples (tabs, CRLF, emoji, CJK, RTL, control characters).
One epoch over 2.6B tokens, no repetition:
<|endoftext|> and packed into contiguous 1,025-token blocks (1,024 inputs + 1 shifted target), no padding. Batches are sampled from a seeded permutation of all blocks.Trained on 2× RTX 3060 12GB (no NVLink) under WSL2 with PyTorch DDP and torch.compile (mode: compile): 4.6 h of active training time, 167,893 tokens/s on average (MFU ≈ 52% of the GPUs' measured bf16 matmul throughput), max GPU temperature 76°C, 3 automatic resume(s) from checkpoints after the machine was shut down, 0 loss-spike rollback(s).
Validation loss on the 2M-token holdouts (1,600 fixed 1,024-token windows per domain) and bits per byte, bpb = loss_nats × tokens / (bytes × ln 2), which is comparable across tokenizers:
The aggregate is the equal-weight mean of the three holdouts (same number of tokens each). Code tokens are far easier to predict than prose with this tokenizer, which pulls the aggregate down; the prose domains (3.26–3.50 nats/token) are the fairer comparison point for web-text models, and bits per byte is the fairer one across tokenizers.
Zero-shot benchmarks (lm-evaluation-harness). <|endoftext|> is prepended as BOS. hellaswag and winogrande sit at chance, as expected at 20M parameters; arc_easy and piqa land clearly above it, which reflects surface lexical knowledge more than reasoning. They are reported for transparency, not as a quality signal.
Sample generations (temperature 0.8, top-p 0.95) are in out/samples.md [blocked].
[Image blocked: Training and validation loss]
main: final weights (step 19,840, end of the decay phase).stable-15872: weights at step 15,872, the end of the constant-learning-rate phase, meant for continued pretraining (e.g. extending to 4–5B tokens before decaying again). Load with revision="stable-15872".The model weights are released under Apache-2.0. Tokenized data is not redistributed.
Everything used to build this model is in the repository: scripts/ (tokenizer training, data preparation, training, evaluation, publishing), configs/model.json, logs/train.jsonl (every 10 steps: loss, lr, grad norm, throughput, MFU, GPU temperatures, memory), logs/tokenizer_report.json and out/data_index.json (token counts per shard and domain, holdout bytes). Raw evaluation outputs: out/eval.json and out/lm_eval_results.json.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("DedeProGames/GPT-U-20M")model = AutoModelForCausalLM.from_pretrained("DedeProGames/GPT-U-20M")
inputs = tokenizer("The history of the Roman Empire", return_tensors="pt") # prepends <|endoftext|> as BOSoutput = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95)print(tokenizer.decode(output[0], skip_special_tokens=True))