No official torrent yet β requesting it tells the flock what to host next. You cannot seed a catalog-only page.
Cataloged from DedeProGames/LowOnMind-300k
All models by DedeProGames βA decoder-only language model with 296,960 parameters, pretrained from scratch on
200M tokens of HuggingFaceFW/fineweb-edu (sample-10BT).
It is an extreme variant of DedeProGames/DynamicMind-Mini (8.9M params), scaled down roughly 30x. The name is the point: this model is not built to be good. It is built to find out what a third of a million parameters actually buys you.
A 3.3x scale-up trained under identical conditions β same tokenizer, same data, same token budget, same aspect ratio β is published as DedeProGames/LowOnMind-1M. The two models form a controlled pair; see Scaling comparison below.
Parameter split: 65,536 in the tied embedding, 231,360 across the six blocks, 64 in the final norm.
max_position_embeddings.o_proj and down_proj initialize at
std / sqrt(2 * num_layers) instead of a flat 0.02.The token budget is deliberately small in absolute terms, but at 673 tokens per parameter this run sits about 30x past the Chinchilla-optimal ratio. The model is parameter-starved, not data-starved: train and validation loss stayed within 0.05 of each other for the entire run, so there is no overfitting to speak of.
Perplexity is not comparable across tokenizers β with a 1024-token vocabulary, 27.06 means something very different than it would at 50k. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable number.
Evaluated on BananaMind/BananaMind-Base-Bench-1.1,
a 350-item English continuation-likelihood benchmark for base models. Scoring follows
the official specification: context and each of the four continuations tokenized
separately with add_special_tokens=False, no BOS, selection by highest mean
conditional token log-probability.
Run validity: dataset SHA-256 matched, full schema validation passed, and no context required truncation against the 512-token window.
Prompt: "The ", temperature 0.8, top_p 0.95.
This sample is more informative than the perplexity number. With a 1024-token
byte-level vocabulary, words like "increasingly", "guidelines" and "deployment"
do not exist as tokens β the model assembled each from four or five fragments in
the correct order without misspelling one. That is orthography learned as a
process, not as a lookup table. It also picked up document structure from
FineWeb-Edu: bullet markers, a No: header, capitalization after periods,
apostrophes in contractions. Even the invented word, "Donah", is invented with
English phonotactics.
What it does not have is meaning beyond a span of about four tokens. "an aware of the message", "a little fruitful vegetables", "an increasingly a brain" β each three-to-four-word fragment is grammatical and the seams between them are not. That is the signature of ~2 bits per character: locally perfect grammar, zero tracking.
DedeProGames/LowOnMind-1M is the same family at 985,152 parameters (3.32x), trained with the identical tokenizer, dataset, token budget (200M), schedule shape, and hidden/layers aspect ratio (10.7). Parameter count is the only thing that changed, so the comparison below isolates its effect.
The clean result is on the intrinsic LM metrics. Validation loss and bits per character moved by a large, unambiguous margin β those are averaged over millions of validation tokens, so the estimate is tight. 3.3x the parameters bought a real, measurable drop in both.
The benchmark result is not clean, and that's informative on its own.
350 four-way items is not enough statistical power to confirm a 2.3-point
accuracy gain; the unpaired z-test (z = 0.67) falls well short of significance,
and LowOnMind-1M's own confidence interval still overlaps chance. The one
category that clears significance against chance in both models is
language_completion β consistent with the real-word-rate result and with the
qualitative difference in the two models' sample outputs, where LowOnMind-1M
shows register-appropriate structure (bibliography-style formatting after a
date-led prompt, policy-style prose after a directive one) that this model does
not. That agreement across three independent measurements is worth more than
any single number here, including the Elo score.
Read the two models together, not the 300k score alone, if the question is "does scale help at this size." It does, clearly, on the metrics with enough samples to say so, and the benchmark is simply underpowered to confirm or deny it on its own.
trust_remote_code=True is required β the architecture ships as custom modeling
code in the repository. use_cache=False is required: this implementation has no
KV cache, and generation recomputes the full window each step.
At ~300k parameters this model learns spelling, spacing, morphology and short n-grams, but does not produce coherent text. It has no factual knowledge, no reasoning ability, and no capacity to track state across a passage. It should not be used for anything except research into the lower end of the scaling curve.
The vehicle has been shown that the verb is a chief.No:- Even a good certain steps of the day, it is good and then a statement.- It is a number of positive checking gain in the south of this power.The risk of the main pottery, the straight guidelines of the brain's deployment.- She is a little fruitful vegetables.- It's an increasingly a brain, and this is, as a perfect men on the storage.- Exodus of Donah, a conference, it is an aware of the message.- The main straw of the problem.- You're potential that the moments can be written in the majority of the paradigm.- New Yorkfrom transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-300k")model = AutoModelForCausalLM.from_pretrained( "DedeProGames/LowOnMind-300k", trust_remote_code=True)
ids = tok("The ", return_tensors="pt").input_idsprint(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))