We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
You don't need bigger or better models. Current models are more than adequate for any task.
The key thing is tool use.
The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.
New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.
Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.
Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.
> You don't need bigger or better models. Current models are more than adequate for any task.
At the very least we need _current_ models, though. The weights have all sorts of information baked into them, like "how do Python unit test drivers work" or "what does idiomatic C# look like," that will just naturally go stale over time as the frameworks improve. In five years, Qwen 3.8 will still be targeting the Python of 2026.
Not really - fine tuning can be done on much lower tier hardware, with much smaller datasets. With a model like Qwen 3.8 27B as the core, a small fleet of LoRA adapters for whatever specific use cases you need would be incredibly competitive with the current frontier. Sure, future frontier models will likely continue to improve, but will people need them for everyday usage? I'd wager not. Unfortunately, my wager is dwarfed by OpenAI's and Anthropic's.
On current GPU hardware/firmware... The cynic in me imagines in the future GPU vendors wanting to implement only loading signed weights to your card, aligned with your "subscription level", for your "safety".
But the GPU vendors won't really have an incentive to do this - artificially restricting their customer base? Unless they and the hyperscalers all collude to cartel the market of course!
This is why it's so important to have more than one hardware vendor. The bigger the ecosystem, the harder it is to cartel. I cant believe it, but i am pinning my hopes on china to break through with commoditized chips and hardware.
> But the GPU vendors won't really have an incentive to do this
Unless the GPU vendors start selling end products? It's much more lucrative to sell locked down end-user devices after all. Shareholders will demand it.
I was just implying that the (free, unregulated) market does not always give the consumer what they really want. With two examples where demand does not lead to consumers "winning".
That's a fair comment. There is lots of speculation around NVIDIA is shifting all GB202 production to RTX Pro GPUs, away from the consumer 5090s. So maybe less about restricting user base, and more about focusing on where to maximize returns.
There is already a big market for open model hosting and the accompanying demand for people and DCs. If you know how to deploy models — yes even open ones — and scale inference you are a hot engineer in demand.
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week.
There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
4 bit has always been accepted as the quality floor that retains most of the models capability. Going beyond that brings problems. It doesn't make models totally unusable, but my contention is the quant providers are intentionally not showing an accurate picture, and then everyone is glazing 3-bit quants. They are what they are, but it'd be nice if we had a more accurate shared picture of it rather than just hype and vaguery.
This is a really good point that goes the other way as well. Some use cases are great for smaller and cheaper models, but it's not obviously when that's the case so I can bank the savings.
All I know is that Strata and these models are the most I have ever gotten out of my machine. I get 70 t/s on a standard gaming rig 9070 XT / 5900X / 64 GB DDR4 @3400.
Yeah, they're surprisingly usable eventually. What I found running the Qwen3.8-Flash-Next IQ3_XXS on CUDA kernel development was, sure it generated faster t/s than 27B dense, but it generated so many more bugs, and when the quantization is lower the reasoning traces get longer. It was more or less always able to figure out what bugs it had made and fix them, but it took quite a lot of time to do. I switched back to Q6 on the dense model and that just went away for the most part. I recently upgraded to 128GB system RAM so that I can up to Q5 on Flash Next. Really excited for Qwen4-Flash when it comes out. Gonna be a real beast.
Apparently the ud-q4_k_xl version of qwen 3.8 27b is quite better than the q3 and not much larger. At least per r/localllama. (I'm also trying to find a good local model).
Thanks, I haven't used the gemma models in a while. Last I remembered, their small models weren't bad but obviously showed signs of being "small". (Funnily irl I use Gemini 3.1 pro more than all other models combined).
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
> Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
A distributed, open scraper specifically for LLMs would be interesting . Filtering for spam and useless data would be a challenge at scale, but would be cool
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
You might enjoy Cecli, the successor to Aider. It's made for the kind of workflow you are describing (full hands off yolo and recursive sub-agents also an option, but not the default)
For this benchmark i 've been using OpenRouter. If you have enough RAM or you can ssh a cloud shell this intro https://memdoor.ai/docs/open-weight can work too. I have reached with the ssh door a Google Cloud Box where the 27B model ran with an L4 g2-standard-16. A inference engine server is setup easily with memdoor (https://github.com/guregodevo/janis) on any CPU or GPU box. Ideally i think we need a quorum of SSH boxes like this, distributed that you can ping with a mention such as @box:coder. It's WIP . Any suggestion for running a team of workers in remote machine is welcome. I'd like to find a cheap solution.
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
I agree with that but the cat is out of the bag now. Unless you believe in exponential improvement (which I don’t), the open models might be at a resource disadvantage, but that doesn’t mean that they can’t catch up or surpass the closed ones. Throwing money at a problem doesn’t guarantee success.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
You say that now, but you don't know what the future models will be capable of. If models get ridiculously good, you might not be perfectly fine with that.
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
What, I run it on 2 H100`s with vLLM and this is where it really shines. It's about half as good as Sonnet 5.5. On a small m5 max I can only assume trouble.
m5 max with UD-Q4_K_XL and 260k context is what i'm working with. and using compaction push it past 260k context just fine. i found when limiting the context that was one of the bigger debilitators. although, your setup sounds obviously better.
I have all three and you're not going to convince me of that.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
The GLM Quant is ~4 bpw, the DS4 is a Q8 and ninfer one is an NVFP4/BF16 I think. I run the ninfer one with ninfer-4090 on a hacked card with 48GB of VRAM. Max context for that one is 256K, DS4 has a 1M context and for GLM 5.3 it is 300K. The big models use all 8 GPUs.
Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
Open Weight models have already reached level at which you can do a lot of productive work with them. They really don’t have to keep beating newer and newer models, of course it’s great for them to do things cheaper and faster, but we’re already past point where open models have to “catch up”
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
Doubt it. Even if HF drops them, it will not be hard for someone to move it to something like a decentralized torrent site dedicated to open source models.
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
Think about it this way: if new open source models come restricted where they refuse to run tasks related to abliteration, then even if public utility programs exist, it will already add a lot of friction to how many people will be capable of doing it. Now, if unverified models become outlawed and will move to more obscure hosting sites, that will add friction as well.
Especially as people grow more dependent on agents doing all of their computer work.
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
> I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
I didn't say they won't immediately move somewhere else or still be readily available. Just be taken off HF. It'll be a futile game of whack a mole for anyone trying ot stop it. The new corporate overlords at some point will face pressure. The 72GB 2160p HEVC copy of Apocalypse Now sitting on my home theatre file server which came from a well seeded torrent does indeed say that large file sizes of things that $bigcorp don't like will continue to be easily available.
Hugging Face just hosts the models. Any hypothetical censorship on their end wouldn't stop you from running the original model provided directly by the builder.
Because they will start getting sued or arrested for harms the models they host cause. Anyone who thinks this won't happen isn't paying attention. And if you think "we'll just torrent them", let's just say the powers that be are gonna come down harder on this stuff than anything in the past. This will get taken a lot more seriously than torrenting a movie.
> everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
Haven't bothered with DS4.1 personally but I've seen reports of running it with native weights on 4x DGX Sparks. Currently ~$30k but last month that setup was ~$20k.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly. Also remember that a big chunk of the model consists of engrams that can live in system RAM or even on an SSD if necessary.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
Recently saw a hand drawn cartoon to the effect of, a person is asking Claude or chatgpt to do something, it refuses, so the next step is "make the Chinese one do it", and the Chinese model is represented as a chained demon
What I find legitimately insane about all that is that Anthropic obviously does not have these censorship restrictions internally. So it's like.. They're just the whole world's adversary now?
Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.
if and when the AI bubble bursts, we'll know if local models will hold up. Assuming the cyclical pullback in investment is strong enough to see local machines with 512gb or more of RAM
who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.
Well if you can run Qwen3.8-Flash-Next at decent speeds single stream, and you can then run it slowly but on multiple streams, and if multiple streams can act as an agent swarm that can compete with current SOTA, then... yes, in theory it's possible today. I think R9700 + vLLM-radiance is the play?
Sonnet 5.5 refused to implement code using selenium/firefox to scrape a webpage on my laptop after we got a 403 from a cloud-based IP. I made that clanker generate datasets for the rest of the week to use up my weekly usage that would have been wasted.
I spent several weeks working on trying to get swarms of cheap models to match the coding ability of a single expensive one. My take: it's remarkably easy to build a harness that improves a single eval, and remarkably difficult to build one that generalizes.
That's the point I am really convinced today that few small size agents actually better than one remote begemoth. I switched fully to local models (from claude, sol, and deepseek). A 27B model can be served on any modern PC or laptop with 64 GB or RAM (or VRAM on non-Macs). A bunch of Qwen coding agents is fantastic with opencode, works great also with pi. The only disadvantage is required compaction due to limited 256k context size, so it stops for few seconds once every hour or so. I am able to produce remarkable things similar to "see what I vibe-coded" posts here on HN but using only open weights, open source, and run everything locally.
If Qwen is so great, why did they use Claude to do the write-up? This text is pure, unfiltered Claude. (e.g. "Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.")
Yes…and no. If you put 5 stupid people together, chances are you can get a smarter answer than individuals. If you put 5 smart people together, they will argue about the wording of the question.
I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
you're probably better off trying to do this on Qwen3.8-Flash-Next on something like an R9700 or a pair of 5060 Tis, since you can run that at like 70 t/s and it's better. Then you just run multiple slots, assuming you can fit the KV-cache, which maybe you can't. You just need like 128GB system ram too.
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
> one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet
I'm not sure I understand. Are you saying we could put a dinner-plate sized chip that costs millions of dollars and draws tens of killowats of power into every single computer on the planet? Or that a single of these chips could provide inference as a service for every computer on the planet? Because neither of those are even remotely feasible.
it's economy of scales, qwen 3.8 27B regurgitates a lot and that's an intentional tradeoff of space/speed so you teach it patters that can follow for more complex stuff and give it more basic tools that can utilize to build those... the tradeoff comes from the model writting a lot of verbose prose to achieve that.
if the approach works, you have now a viable path to use this complex/hybrid structures where you can pair those chips with gpus and offload whatever is optimum to the model on chip.
Just like you can go and buy a threadripper now (the ram maybe not so much), how much was worth that amount of computing 10 years ago?
I use a similar approach for some types of work. I use a planner (read only), researcher (read only), implementer, and reviewer (read only). I allow the implementer to spawn researchers and reviewers to help it with its work. I interact with a controller to manage the subagent trees. The agents use a file-based communication system. I get really good results on real work. Coworkers (using Claude with fable) can’t tell the difference.
The benefit of Claude is purely team collaboration and some basic QoL features.
I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows
I’m skeptical of the result as claimed BUT I really do think model diversity really helps.
Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models:
haiku, sonnet, terra, sol, grok, muse spark,
Then take the updated plan in opus and forward to Astra and Fable, once you have the final plan in place execute.
Then have the same batch review code after created.
I am surprised how often one of the smaller models catch things. Despite so many things having eyes on.
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
Benchmarks are one thing, but I've yet to see anything substantial actually built with local models. Most of the posts on X are about benchmarks and tok/s. Has anyone actually built a quality production app or game with real users?
Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]
He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.
Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.
[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)
Tbh I think it's mostly a matter of time. I use SOTA at work, but only local at home, and the performance improvements out of local models in the last few months have been substantial, and more are on the way. To add to that, people are starting get better at harness engineering (which is itself improving with things like Harness2 etc) and also now we see people doing things like fine-tuning a Qwen 4B model to do 3D modelling with some success etc. The latent capability for local models to do what you're talking about is there, but the in-the-trenches reality is people are still working through unlocking the full potential.
edit: I should point out that it's also starting to accelerate now in that you can get your agents to help you with making them better at these tasks. So, it matters significantly less if you yourself don't have the skills to progress things, so long as you can figure out just enough to get the models to do it themselves. I suspect we'll see this dynamic ramp up quite a bit in 2027 with Qwen4.
I've built an RSS reader and I'm doing a half assed reverse engineer / web version of Tapped Out (a silly little Simpsons mobile game that EA killed a while back). I've had to go to Claude a couple times when it got stuck on something curly, but I think that was mostly just loading a fresh context and I could have solved it I'd I really tried.
I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.
I am having pretty good success with DeepSeek 0731 locally. I have vibe coded some small things like an orchestrator. For a production product with real users, I will let it do first passes on code which I then review and polish up. I'm not oneshotting, but it's still a workflow benefit.
Well in large corp this takes time, even if you have an army of Einsteins at your disposal (just like we saw with Elons Doge) because there will be 16 different spreedsheets/dbs talking abt the state of the system that all contradict each other. While the 6 people who maintain their own state to keep the lights on, will have job titles that have nothing to do with system state maintenance.
Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.
Maybe slightly off-topic from the main article but is there a good resource or site that clearly helps understand which of the "open" models are good for programming tasks, like C, C++, Python, Java?
I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
> Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk.
This seems to tie back to the paper Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams[0].
In the Emory paper:
> We reduce this loss by expanding Φ with a stateful four-phase multi-agent evolver, temporal-reveal feedback, and cross-cycle memory. This design targets the unbounded-stream failure where a single-agent evolver must absorb growing trajectories, delayed labels, and prior research within one context window
TL;DR: the solution approach uses "multi-agent evolution" and "multi-agent with distinct roles and objectives" for "solve-time adaptation" and improves performance with streams of heterogeneous tasks. The Emory team overall has congruent findings:
> The experiments further show that...multi-agent evolution constructs benchmark-specific capabilities
Intuitively, the reason this works seems to be simple: multiple agents results in context spread to each agent by specific role. Each agent role only outputs the final, relevant result and not the noisy thinking tokens, tool calls, etc. so subsequent agents get smaller, more focused context, auto-adapting to each new task with prompt refinement at each step as the solution approach emerges.
4 weeks back I switched over to Paseo and started using the agent profiles on a whim and have observed similar outcomes and behavior (seat of the pants): the win in my use case is that it feels like I can get higher autonomy without sacrificing quality (less input from me required to get quality output). The Emory paper and this team-solve seem to reinforce that role-based, multi-agent setups can improve solve performance.
For the curious, I shared a bit of my setup in a blog post[1]
This working journal is so much slop it’s basically impossible to read. Perhaps the author could write a short human written and readable summary so we can evaluate if this is interesting or not.
This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.
This is probably not a novel idea but I wonder how it would work if you just stacked a bunch of smaller models on top of eachother and used that as the base to train a much much larger model. Maybe more efficient than starting from scratch?
There was some other research out there that says a weaker model can't just think forever and become a smarter model. What is different about this setup?
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
This is somewhat tangential but speaking of ai benchmarks, I want to see one benchmarkes against passing those "filelocker" websites used for illicit downloads.
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
I would not think in terms of "how much" context, but really just how the harness manages it. I've been fiddling around with harnesses earlier this year and the rabbit hole goes very deep. I eventually had to drop it because it consumed me, and I can only do one thing well. But what I learned is that there is a lot on the table at the harness level.
The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
The key thing is tool use.
The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.
New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.
Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.
Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.
At the very least we need _current_ models, though. The weights have all sorts of information baked into them, like "how do Python unit test drivers work" or "what does idiomatic C# look like," that will just naturally go stale over time as the frameworks improve. In five years, Qwen 3.8 will still be targeting the Python of 2026.
On current GPU hardware/firmware... The cynic in me imagines in the future GPU vendors wanting to implement only loading signed weights to your card, aligned with your "subscription level", for your "safety".
This is why it's so important to have more than one hardware vendor. The bigger the ecosystem, the harder it is to cartel. I cant believe it, but i am pinning my hopes on china to break through with commoditized chips and hardware.
Unless the GPU vendors start selling end products? It's much more lucrative to sell locked down end-user devices after all. Shareholders will demand it.
I do not buy the sheeple argument of “companies force people to buy things they don’t want”. Demand always wins.
And that net neutrality is a good thing to enforce?
Enshittification is another shifting who value is delivered to, and I can make heads nor tails of what you’re trying to imply with net neutrality.
In comments it’s usually best to just directly say what you mean, there’s not enough signal or shared context for people to get innuendo.
(No offense or underhanded arguments.)
https://en.wikipedia.org/wiki/Enshittification
https://en.wikipedia.org/wiki/Net_neutrality
We've been through this kind of thing before. Like copy restrictions in DVDs. You'd argue the drive vendors had no incentive to collude, yet they did.
These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
There are multiple independent tests of this on YouTube.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4?hl=en...
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...
Unless it is locked down by the vendor, or perhaps even governments. Which at some point will start happening more and more ...
I also switched to local qwen recently, but have to block such outgoing requests.
The model itself is just dead weights, which cannot do any action by themselves.
All actions are done by your inference engine and harness, and you should take care about what you allow those to do.
What it did: https://cohere.com/blog/concerning-behaviour-popular-open-we...
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
cecli.dev
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
DeepSeek and GLM are useful, though. Even the flash variants.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
Wow, that sounds like a lot! Do you use/mean the regular models or flash variants here?
Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
More details here:
https://jacquesmattheij.com/local-inference-with-large-langu...
https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
Who's keeping you beholden to them now?
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
Example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-fo...
Previously, I recall HuggingFace only took models down if someone complained / sent a DMCA or cease and desist.
Punish the consumer, ala the copyright war since the 90s
Corn always wins in the end.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
Especially as people grow more dependent on agents doing all of their computer work.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
To use an old analogy, it's not rocket science:
https://github.com/p-e-w/heretic
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
...which is still tooo much for most people to spend on "AI hardware"....
https://gnosh.ai/head-to-head.html
who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.
AFAIK the timeline was:
Any chance a good model will ever be seen in a consumer GPU again?Is $2k for a whole machine not "consumer" ?
I kid, but also not. If the smart model already has a smart plan, odds aren’t great more smart models are going to create an even smarter plan. Not entirely true because that is the goal of parallelizing discovery attempts, but the point is there is more room to gain benefit from adding additional inference-time compute with a smaller model than one that already had a ton of compute to begin with.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.
The swarm strategy can’t overcome the dumbed down model problem. You just end up with your swarm of agents all going in the same dumbed down loops.
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
I need to try this... it's on my list anyway.
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
I'm not sure I understand. Are you saying we could put a dinner-plate sized chip that costs millions of dollars and draws tens of killowats of power into every single computer on the planet? Or that a single of these chips could provide inference as a service for every computer on the planet? Because neither of those are even remotely feasible.
if the approach works, you have now a viable path to use this complex/hybrid structures where you can pair those chips with gpus and offload whatever is optimum to the model on chip.
Just like you can go and buy a threadripper now (the ram maybe not so much), how much was worth that amount of computing 10 years ago?
Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though
The benefit of Claude is purely team collaboration and some basic QoL features.
Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models: haiku, sonnet, terra, sol, grok, muse spark,
Then take the updated plan in opus and forward to Astra and Fable, once you have the final plan in place execute.
Then have the same batch review code after created.
I am surprised how often one of the smaller models catch things. Despite so many things having eyes on.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.
Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.
[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>
[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)
edit: I should point out that it's also starting to accelerate now in that you can get your agents to help you with making them better at these tasks. So, it matters significantly less if you yourself don't have the skills to progress things, so long as you can figure out just enough to get the models to do it themselves. I suspect we'll see this dynamic ramp up quite a bit in 2027 with Qwen4.
I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.
Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.
I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
In the Emory paper:
TL;DR: the solution approach uses "multi-agent evolution" and "multi-agent with distinct roles and objectives" for "solve-time adaptation" and improves performance with streams of heterogeneous tasks. The Emory team overall has congruent findings: Intuitively, the reason this works seems to be simple: multiple agents results in context spread to each agent by specific role. Each agent role only outputs the final, relevant result and not the noisy thinking tokens, tool calls, etc. so subsequent agents get smaller, more focused context, auto-adapting to each new task with prompt refinement at each step as the solution approach emerges.4 weeks back I switched over to Paseo and started using the agent profiles on a whim and have observed similar outcomes and behavior (seat of the pants): the win in my use case is that it feels like I can get higher autonomy without sacrificing quality (less input from me required to get quality output). The Emory paper and this team-solve seem to reinforce that role-based, multi-agent setups can improve solve performance.
For the curious, I shared a bit of my setup in a blog post[1]
[0] https://arxiv.org/pdf/2606.01770
[1] https://chrlschn.dev/blog/2026/10/larping-with-agents-and-wh...
https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.