I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
Regarding subsidy/profits: Anthropic at least seems to be on track to report a profitable Q2 2026
> Anthropic has projected revenue of at least $10.9 billion for the second quarter of 2026, more than double the previous quarter, on track for its first quarterly operating profit of $559 million.
that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device.
For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.
So what you get is 80x more expensive and you wait 416 days to get it.
I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?
> but are they functionally useful or helpful to anyone currently?
Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.
MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.
The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.
Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
Wait I thought the router ends up specializing the experts?
Like there is no explicit goal aside from each 'expert' getting roughly equal weight?
And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?
I do not know what I am talking about, this is my limited understanding...
My understanding of it is also pretty surface level, but I was not under the impression that it develops "expertise" in a particular subject matter, at least not in a way that's easy to harness. From what I've read, it develops expertise at the token level.
Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.
You are right that at training the main goal is balancing between the sections to avoid certain paths becoming the only path. In the end the inference will be routed token by token to a mixture of say 3 or 4 sections of the model. The combination can change at each turn. It’s really a statistical optimization.
As for many things in neural networks, the original intuition coming from anthropomorphism once implemented becomes something very non human!
> There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
Huh, always thought one would sort of require the other.
For a good MoE model, wouldn't I want to minimize the "churn" between experts, i.e. the amount of time one expert model has to be swapped in for another expert model? That would be naturally the way if experts correspond to semantic categories.
E.g. suppose I have a model that can answer questions in 100s of languages. Then while any of those languages might be requested by some caller, it's highly unlikely a caller will request all languages in the same session - realistically, there might be one or two languages in a session and those will then span the entire session.
There will also be languages that are requested very often and others that are extremely rare.
(Let's say my model also supports Klingon and Sindarin. Those are important for marketing reasons and because I genuinely like to make the occasional nerd happy - but practically, I get maybe a handful of requests for those every few months. So it would make sense to centralize the knowledge for those languages in some specific part of the model, so I can keep that part out of VRAM - and probably RAM as well - during the 99% of time where it's not needed)
So wouldn't it make sense to make the expert models language specific here? Then you could take advantage of the fact that a language rarely changes inside a session and keep that expert in VRAM for the entire session. You could also avoid dragging parameters along with you for languages that are practically never used.
yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.
If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote.
If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful.
If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
> but are they functionally useful or helpful to anyone currently?
I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs.
I have all praise for those taking this on and in my idiom would call it *the lord's work."
The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.
Not an answer to your question, but maybe this has more info? I think with these optimizations and quants, it compares negatively. But can these optimizations be applied to models you want? Another question.
https://news.ycombinator.com/item?id=48353348
if you take a model that requires 200GB of VRAM and you run it on the CPU, it requires 200GB of RAM instead. Still unfeasible on consumer hardware. With this approach you can easily do it on 12GB or less of either RAM of VRAM, at several seconds per token instead of tokens per seconds. Very unusable, but certainly interesting!
I have see a lot of "run <gigantic llm model name here> using cpu" or using 1gb of ram... and the short answer is sure, you can run it in 1mb of ram, or in a 286, it will just take a couple of years to produce the first token.
the canary in the coal that this has not change is nvidia share price.
The relation to current RLVR methods I think is interesting, they do discuss it a bit but I would be curious to see more about this as well. Quote from the paper:
Exploration beyond Pretraining. The mode collapse XMs address during pretraining also often
shows up in post-training, where RL fine-tuning is known to sharpen models onto a narrow set of
behaviors [76]. Recent fixes such as pass@krewards [77] and best-of-N-aware fine-tuning [78] can
be seen through our lens as Forward XM, with a verifier standing in for ground truth data. These
fixes act only during post-training, though; pretraining with exploration may yield base models that
capture more modes in the first place, leaving RL more to select among.
> Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated (Dieterrich, T. G.)
To be a coauthor on a preprint that you have not submitted, you have to actively "claim" it (using a password given to the author who submitted). It's on you to double-check before claiming.
I don't think they mention it in the CoC, but anyone can upload an ArXiv preprint listing literally anybody else as a coauthor, so it is only logical that only "confirmed" coauthors should be affected.
Agreed. LLMs have helped me achieve much deeper reading, _when directed to do so_. Asking an LLM to “Teach me Socratically about this paper/code. One question at a time”, usually allows me to get a much deeper reading of the material than I would otherwise.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
reply