From the guidelines: "Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity."
Every link submitted by the author seems to be self promotion...
I wrote this essay out of curiosity for finetuning, and as part of my Postdoc. I thought folks here who are finetuning models might find it interesting. I'll share my writings and little technical experiments elsewhere.
Again - out of genuine curiousity - would you prefer I post other people's writings instead? Or is it the frequency of my posting (weekly-ish) / amount of work I'm producing that's more concerning to you?
i personally do finetuning in FP16 because then the LoRA merge at the end has less rounding error.
for most models it's lossless as well (when coming from a BF16 source), it's very quick to scan the weights and make sure they're all in range if you're concerned.
Yea that's pretty related. In this case, I'm looking at full finetunes (although we don't end up actually finetuning the whole model, just the small weights!)
If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent changes to weights if the change is at least ~1/512 of the weight itself, so at small learning rates only the smallest weights can update.
As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.
I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.
This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding.
That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.
The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw, in case you want to check the receipts.
But there are many, many papers (amassing thousands of citations) that actually don't keep fp32 copies around.
Also fp16 is much more precise, so this is far less of an issue, and as far as I was able to tell nobody had identified the actual effect (that finetuning in bf16 is NanoAdam with the masking amount / size of weights targetted determined by learning rate).
I figured that was an interesting inductive bias, to target small weights.
Every link submitted by the author seems to be self promotion...
Again - out of genuine curiousity - would you prefer I post other people's writings instead? Or is it the frequency of my posting (weekly-ish) / amount of work I'm producing that's more concerning to you?
i personally do finetuning in FP16 because then the LoRA merge at the end has less rounding error.
for most models it's lossless as well (when coming from a BF16 source), it's very quick to scan the weights and make sure they're all in range if you're concerned.
As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.
I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.
This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding.
That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.
The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw, in case you want to check the receipts.
This issue was recognized a decade ago when we first started training with fp16.
Also fp16 is much more precise, so this is far less of an issue, and as far as I was able to tell nobody had identified the actual effect (that finetuning in bf16 is NanoAdam with the masking amount / size of weights targetted determined by learning rate).
I figured that was an interesting inductive bias, to target small weights.