No official torrent yet — requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from DedeProGames/LowOnMind-5M
All models by DedeProGames →A decoder-only language model with 4,920,384 parameters, pretrained from
scratch on 200M tokens of
HuggingFaceFW/fineweb-edu (sample-10BT).
The largest model in the LowOnMind family and the third point on its scaling curve, after LowOnMind-300k and LowOnMind-1M. All three share an identical tokenizer, dataset, token budget (200M) and schedule shape, so validation loss, bits-per-character and benchmark results are directly comparable across the series.
It is also the first model in the family whose benchmark performance is statistically distinguishable from chance.
Two deviations from the smaller siblings, both deliberate:
The vocabulary was deliberately left at 1024 rather than raised to something more appropriate for this scale. A larger vocabulary would compress better (1024-token byte-level BPE runs about 2.35 characters per token, so 200M tokens is only ~470MB of text) and would almost certainly improve absolute results. Keeping it fixed is what makes the three-model comparison valid — the cost is that this model spends capacity assembling words from fragments that a 4096-token vocabulary would hand it for free.
Modelling code is otherwise byte-identical to the two smaller siblings: GQA,
SwiGLU, RMSNorm, tied embeddings, QK-Norm per head, precomputed RoPE
with automatic re-expansion, residual projections initialized at
std / sqrt(2 * num_layers).
At 41 tokens per parameter this run is the closest of the three to the Chinchilla-optimal ratio of roughly 20 — about 2x above it, against 10x for LowOnMind-1M and 34x for LowOnMind-300k. Train and validation loss tracked each other throughout; no overfitting.
Perplexity is not comparable across tokenizers, but it is comparable across these three models because they share one. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable figure.
Deltas: -0.4080 nats from LowOnMind-1M (5.0x the parameters), -0.7154 nats from LowOnMind-300k (16.6x).
With a 1024-token byte-level vocabulary, no long word exists as a single token — the model has to assemble every one of them from fragments. The fraction of emitted words that are real English words was introduced to measure this.
Measured over 64 unconditional samples (5,398 words), using the same reference lexicon as LowOnMind-1M: words appearing at least 5 times in a 20k-document sample of the training corpus.
This number went down, and it should not be read as degraded spelling. The
drop is statistically real (z = 5.38, not sampling noise), but inspecting the
non-words shows what happened: illuminator is an ordinary English word,
phillipsburg is a US town, shima is a common element of Japanese place
names. They are counted as errors only because they fall below the reference
lexicon's frequency-5 threshold. The remainder (hymenola, almanine,
perleti, amiravicis) skew toward proper-noun and Latinate-technical
morphology rather than the malformed common words the metric was built to catch
— LowOnMind-300k produced things like landship and parsetic, failures of a
different kind.
The metric has a floor problem as well as a ceiling problem. As a model improves it emits rarer real vocabulary — names, places, technical terms — which a frequency-thresholded lexicon scores as wrong. So the measured rate can fall while actual quality rises. Comparing against a full dictionary with proper-noun handling, rather than a corpus-frequency cutoff, would be the fix. The 96.3% figure is reported as-measured for continuity, but it should not be used to rank these models.
Evaluated on BananaMind/BananaMind-Base-Bench-1.1,
the same 350-item English continuation-likelihood benchmark used across the
family, with identical scoring: context and each of the four continuations
tokenized separately with add_special_tokens=False, no BOS, selection by
highest mean conditional token log-probability.
Run validity: dataset SHA-256 matched, full schema validation passed, no context required truncation against the 512-token window.
Difficulty split: easy 30.8%, medium 33.3%, hard 32.8%.
trust_remote_code=True is required — the architecture ships as custom modeling
code in the repository. use_cache=False is required: this implementation has no
KV cache and recomputes the full window at each generation step.
At ~5M parameters this is still a research artifact, not a usable model. Expect fluent local syntax and register-appropriate structure, but no reliable coherence across a paragraph, no dependable factual knowledge, and no ability to track state across a passage. Benchmark accuracy of 32.3% is above chance and far below usefulness. The 1024-token vocabulary caps absolute quality below what this parameter count could otherwise reach.
The 512-token context and absent KV cache also make it unsuitable for any real workload.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-5M")model = AutoModelForCausalLM.from_pretrained("DedeProGames/LowOnMind-5M", trust_remote_code=True)
ids = tok("The ", return_tensors="pt").input_idsprint(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))