This model seems to be pretty benchmaxed. In actual use, it doesn't survive the comparison to models in the benchmark.
This model was released as a "stealth model" in OpenCode Go for a while as "Space Bunny" (they mention this in the blog). So people have been using without knowing what the model was exactly, but it failed to impress anyone as far as I can tell. This is despite the model being served for free (with some asterisks). GLM 5.3 Flash and DeepSeek V4.1 Flash, even before last week's update, seemed to be much preferred. Stealth releases happen often with OpenCode subscriptions, and GLM 5.3 Flash (stealth name "Ox Alpha") impressed a lot for example.
The perverse incentives around benchmarks are now so bad that benchmark scores are effectively meaningless. Models can score highly and still be crap to use. It doesn't matter whether it's Minimax or Google, or anyone really.
For what it's worth, it does perform on par with DeepSeek V4 Pro on my adversarial programming language benchmark[1]. I wouldn't think they would benchmax on that.
Any chance you could try qwen3.8 flash next? That is currently the best local model for actual coding tasks in my + many coworkers experience. It will run reasonably fast if you have 64+ gb ram and use strata or a strata fork (vram not super important with this method).
I used Space Bunny from some providers that offered it for free and it was a very much a "you get what you pay for" experience. If the release is comparable, I'll be sticking with Deepseek v4 for sure.
I’m one of the unimpressed. I tested SpaceBunny intensely on a research project for a week.
The model wasn’t horrible at coding, it just made a ton of initial mistakes—but then would catch most of them later on. It can get work done if coaxed, but its peers are just cheaper and/or easier to work with.
I've used local models extensively (mostly qwen 3.8 27b) and now Claude sonnet/opus 5.5 since launch.
Sometimes I really wonder if not being able to see what's going on in thinking is a large contributor to thinking the models are stronger.
That assumes you bother skimming through thinking traces and the output in-between.
In my experience both qwen and claude fumbles and misunderstand, but will in a lot of cases self correct.
Qwen models can feel very indecisive but somehow makes it in the end. If the harness was able to hide that and only highlight the end results, I'd probably think qwen models are smarter.
Reading the tool calls Claude does while thinking shows a lot of trial and error about even simple things like editing code and getting the whitespace wrong. But it happens so fast compared to qwen that it's sort of acceptable. With qwen, it can feel frustrating because it spends so much time on it.
This is all assuming that's what you meant by mistakes, but maybe you meant something different.
I am not sure. The people who actually care enough to look at benchmarks will run evals, and find out quickly. If you post silly benchmark scores too often, you will simply be ignored. You are paying with your reputation.
This is my experience with all MiniMax models: They're good on paper but fail to live up to expectations in practice. IMO, at no point has any MiniMax model been the superior choice over others, all things considered.
I've been building a cybersecurity platform (called ZeroQuarry) and regularly benchmark new models against a pretty wide set of security both code review tasks and live application pen testing tasks, and this one far underperforms relative to GLM 5.3 from every test I've done for real-world cybersecurity tasks
I don’t get the endgame. If your model underperforms in real life usage, the backlash comes quickly. I guess I’m asking what exactly the perverse incentive is? A day of press at most?
I know I mention it way too often right now, but I am just so impressed with Qwen Flash Next right now. I have been using it locally and it's just such a great little model to work with. It makes mistakes and has its quirks, but in general, it just gets things done and its default manner is just right. It is not overly proactive or making messes for me, even compared to DeepSeek Flash 4.1. I really like DS Flash too, it can just be /really/ annoying in some roles and to chat with some of the time.
I can run Qwen Flash Next on hopes, dreams, DDR4, and ancient data center GPUs at 200 t/s :)
Same here. I run Qwen3.8 Flash Next on my local machine and combine it with Opus 5.5 for plan making. The Qwen model feels like it's around Opus 4.x level performance most of the time.
But once in a while it gets into long reasoning loops, which is annoying. It seems less with the Swift model but I haven't benchmarked the difference between that one and the original.
Yes, I feel Qwen3.8-FN is the first model that seriously can replace some lower tier subscription usage without requiring much more than a high end gaming PC. Qwen3.6-35B was impressive but not quite good enough, and 27B takes a lot more time.
Lately I prefer GLM5.3-Flash since I feel it is noticeably smarter and I have the hardware for it, but FN's performance has me looking forward to Qwen4.
I'm finding these LLM announcements to be pretty meaningless. They all look the same, with different numbers and graphs, even though the graphs are almost all identical. The offered model is compared with models that it can beat, with a couple of entries that "show" they haven't engineered the whole benchmark presentation to mislead.
I've become incredibly skeptical of new model announcements until I can find evidence of someone else using it for something effective. I'd love to see an announcement that wasn't structured around attracting investors with too much money and too little domain knowledge.
To me the most interesting chart on artificial analysis is intelligence index vs cost per task. It's like the third chart down and they graph a pareto line showing where the frontier is and every model on the pareto line has been interesting to me. Luna, Mimo-v2.6-pro, Sol. Hell, Argon.
This MiniMax model is clustered with Haiku 5.5 so (combined with the criticism on here that it was lacklustre while free on opencode go) I don't find it interesting, but every announcement has a chance at pushing that frontier.
I was benchmarking affordable LLMs for japanese->english translation recently and the winner was Kimi K3 - there are a ton of interesting models in the wake of the flagships.
Qwen 3.8 flash next was an important announcement because it easily runs on dual RTX 6000 Pro, which made it available to a subset of fairly high-end local inference tinkerers. For those running local models, checking announcements for vram requirements is the first step in knowing if it’s a candidate to push more intelligence into your local rig. At least, that’s what I do when I read announcements.
What would be an interesting LLM announcement, then? Even Opus 5.5 wasn't an interesting announcement but the impact is obviously significant in hindsight.
For me? A local LLM as powerful as opus 5.5 would be interesting. Or a good local 3d asset generation model. Or something as powerful as Opus 5.5 that works nearly instantly.
All this incremental progress is just more of the same. I want a real novel capability, not just more, faster, cheaper, “smarter”.
what's even more funny is most of what they are asking for exist in some form.
As an aside, I remember when sora 2 launched and thinking, "wow I wonder if we'll ever get something close this in open weights?" The prevailing sentiment was "maybe 5 years." Yet, we have Minimax H3 and Flux a year and a half later.
When Opus 5.5 launched, within a day you could see that it was a signfiicant step function in capabilities by just following regular subreddits like /r/AIGameDev and /r/ClaudeAI (but really anywhere people publish their software, if you can recognize it).
But this only works for frontier models. If MiniMax catches up to Opus 5.5 in 6 months, no one would notice.
But this only works for frontier models. If MiniMax catches up to Opus 5.5 in 6 months, no one would notice.
I'd be happy as hell if I end up being able to run an Opus 5.5-level model at home within a 6-month time horizon. May or may not happen based on China's export policy, of course.
The open weights we have may be as good as they ever get... but then that's been true all along, considering nobody really knows why they are releasing powerful models for free in the first place.
I think we keep getting used to intelligence improvements and the goal posts are always moving forward.
I remember that close to an year ago i felt the same way about opus 4.5. And even if you can easily run ds or qwen locally to get very similar results to that, we moved to 5.5 being the new target.
Deepseek can do 90% of the job but a claude reviwer/spec writer is a brutal improvement.
Skepticism is due! Good! The numbers don't mean much! Yes!
I personally enjoy seeing how different models view problems. Take big plans from big models, then see how other agents reinterpret a stance. See which directions they go. I love the diversity of views they produce.
It's not really relevant what the numbers are. I really really enjoyed talking with MiniMax M3 a lot. And M2.7. it had a good world view, wrote the situation up incredibly fluently. It thought out loud well. These were pleasant uses for me. The realities, the world's they could see felt very grounded. Good stuff, to me.
The cost vs performance is insane for all these new models. I’ve been using Deepseek Flash 1017 and it does whole projects and asks for $0.10 on US provider with ZDR on Openrouter. I convinced a few friends to drop their Claude $20 subscription because DS Flash 1017 brings so much better value. They can still use Opus 5.5 for planning though each call will cost north of $2. Excited to try this model out too once there are US providers.
Opus 5.5 is according to AA cost per task, 23x more expensive than Deepseek with max effort (which is dumb, you shouldn't use) and 13x more expensive on xhigh.
Claude sub has a 60x subsidy, so a $20 sub gives you $1200 in API.
Based on this math, Claude sub is a much better deal than Deepseek. And it's a stronger model of course.
> Claude sub has a 60x subsidy, so a $20 sub gives you $1200 in API.
This is only true if you max out the plan, which requires working around awkward 5-hour and weekly limits. For side projects, sure, you can use a Ralph loop or whatever and just continue when the window resets. For work, I can't afford to simply stop working when I hit my limit. And using Claude at API prices is prohibitively expensive.
Personally, I have a $20/month OpenAI subscription that I supplement with DeepSeek V4.1 Flash (currently using CoralBricks as the provider since it's fast and I'm grandfathered into their $0 cache read pricing). It's working out rather well so far. Even when I use DS all day as my primary model, I only pay about $1/day, which is roughly equivalent to what I pay for the sub anyway (but with no limits).
I tried it but it's way too slow for me. DeepSeek V4.1 Flash is an order of magnitude faster, and the modest difference in intelligence isn't worth it to me for most use cases.
I am sorry, but if you're given $1200 in api for free I wouldn't consider that as working around flakey limits especially when you only need to max out the 5 hour limit 5 times a week.
I don't understand what you're saying. If I hit my 5-hour limit in 2 hours, I still need to get work done for the other 3. I'm not going to sit there and wait for my limit to reset.
But we're not comparing a Claude subscription to using Claude at API prices; we're comparing it to using a cheap model like DeepSeek V4.1 Flash at API prices. If I hit the 5-hour limit in 2 hours with Claude and have to wait 3 hours for it to reset, I get less value out of that subscription than I would have if I'd have just used DeepSeek to begin with, and the latter isn't more expensive.
So my original point about the limits being awkward still stands.
I'm actually an anthropic hater, my model role for opus 5.5 is called trash. It genuinely pains me that opus 5.5 is so far ahead of the next best thing.
I pay API rates via OpenRouter / Fireworks, and I find it challenging to hit the $20 mark.
If I was to do anywhere near that kind of work on Anthropic, I'd find myself running out of tokens, waiting days to continue working, and leaning on their el-crap models to try to avoid running out again. This happens all the time.
This way I get to leverage DS for nearly all my work, benefit from its far superior latencies, and I still get to lean on OpenAPI / Anthropic or whatever SOTA models I want when I need them.
The only time that really falls down is if I'm doing huge amounts of real novel work, which hasn't been the case for a while.
Unfortunately, Fireworks is not keeping up with new models the last few months. When we selected them as a vendor, they were crushing it, new models were available in 1-2 days. Now they want us to "request" them in Discord, instead of making them all available.
I use OpenCode at home, this is one of the reasons.
Their Ember-1 model (Kimi K3 tuned to think less) hallucinates to high hell, I have no trust of it, nowhere to provide feedback, despite their launch post saying they would like feedback. That they loudly and proudly claim their internal devs notice no difference is a huge red flag for me. It says there is too much eyes free coding, I noticed the hallucinations and false claims of task completion immediately.
> isn't that what ZDR is for? (assuming you trust that)
ZDR is:
1. A pinky-promise you have no way of verifying
2. A contract term, which means its still subject to the good-old "lawful request or court order" clauses
3. Subject to arbitrary provider exceptions ... see Anthropic and Fable
What you really want (assuming you can't afford your own servers and datacentre space) is secure enclave processing:
> See the sibling comment, tinfoil added scanning inside the enclave.
Clearly I'm not defending tinfoil regarding scanning but I think its only fair that you do not make the statement out of context[1].
People should read the context[1] and come to their own decisions.
Also recall that given how attestation works, Tinfoil are forced to publish what they run in the enclave in their GitHub account[2] so you can verify the statement on their blog against their repo. You do not have that luxury with the closed-source "ZDR" brigade.
You're making a fair point-- but I think it's important to reflect on what it says about tinfoil's values: They believe that they are ultimately in charge of computing you purchased from them, rather than you. I agree, It's an improvement that the structure forces them to be somewhat more transparent about their actions and doesn't necessitate the hard privacy loss computing other approaches to subjugating third parties to control over their use of computing. But coercively controlling other parties use of the private computing that they bought and paid for is both unnecessary and unethical, and demonstrates harmful paternalistic and authoritarian tendencies on the part of the companies management.
Depending on the balance of your concerns a ZDR that is lying about their privacy might still be an improvement because they can't both safely conceal the violation of their promises AND tamper with the output.
When one factors in the lack of absolute protection provided by secure enclaves-- that they're reliably bypassable with advanced techniques and thus mostly serve the purpose of keeping honest people honest, I think it would be better to avoid doing business with any enclave provider that has already demonstrated such leanings.
First, for transparency, I do not work for tinfoil, and I'm not even a paid customer because they are US-based and therefore CLOUD/PATRIOT etc. exposure automatically rules them out for me anyway.
I do broadly agree that of all the things tinfoil could/should be working on in relation to service features, this is perhaps an odd use of their time. And I do wonder if this ultimately came about due to some sort of external pressures or maybe a big-spending customer asked for it. Who knows. Their blog post is a little vague on the background context other than selling it as Good Thing (TM).
Yes, in theory. The big AI company subscriptions do not offer that, they all watch you unless you pay more for the API. The only exception I have seen for big AI is Grok teams.
However the way it is going, there will not even be private APIs, just recording and watching all the time.
Anthropic is unsurprisingly leading the race to the bottom of the barrel by requiring moderator reviews for API customers. Apparently everyone at GCP and AWS complied.
GCP has just followed and done the same thing on Gemini Argon. No CMEK allowed. Azure has some ZDR but presumably they are restricting it.
Some neoclouds are still standing but some are also regressing by using server-side scanning to prompt human review.
There are two/three private TEE providers. One of them (Tinfoil) just put server-side scanning inside the enclave for the consumer product. The API is good for now, I am taking it as notice from them to get off the service before they start logging everything.
Just because anthropic charges absurd consumer API prices, doesn't mean their subscription is a 'subsidy' lmao.
I'm a heavy Claude/anthropic user, I max out my 20x max plan in 1-2 days at this point. The OpenCode Go $40 plan with DeepSeek flash is like 2x the value of the Claude max 20x plan for how much I actually get out of it, for 1/5 the price.
DeepSeek literally like 10x better value than Claude/Anthropic.
I'm the same way re: bottoming out my 20x Max plan in ~2 days. I'm having better luck lately, though. What helped was using Opus 5.5 for planning and orchestration, and Haiku 5.5 for all of the simple stuff. Claude doesn't default to this -- the default is that it uses the same model for everything -- you have to give it specific instructions.
My math takes that into account. It's 20x more expensive but you get 60x subsidy on subs. I.e you can do more tasks on a $20 claude sub than with $20 in deepseek API.
It's just pointing out how heavily subsidized subscriptions are. I often read about how much value people get from DeepSeek, but when I try to use it, I end up spending the amount I'd pay for a subscription much faster.
DeepSeek is really cheap when you compare API pricing, but it's more expensive than a subsidized Anthropic or OpenAI subscription.
I suspect that we may soon be seeing a modern version of VHS vs Betamax. Betamax was better quality, but VHS beat it in virtually every other metric that mattered to most consumers.
(yes, I'm aware the analogy isn't perfect for many reasons)
I go off of marketing obviously, but how is every new announcement of any vendor always on top of all others and then cheaper, while next day the next best and even cheaper model pops up? Is there anyone who independently verifies this? Its kind of pointless to comment on this announcement, much like how its pointless to comment on all the other announcements...
There are hundreds of benchmarks (all noisy), dozens of model families, and dozens of modes and variants within each. Announcements are crafted by publicity teams and involve cherry picking a chart that looks compelling.
As a prospective user, none of of the benchmarks details will matter to you, but the family of models and settings they peg themselves against can give you a hint of where you might want to evaluate them within your own workflows.
As an enthusiast or lay researcher, the more interesting thing is to look at their techniques and any innovations they're exploring in training, inference, etc. Those are usually going to be details tucked below the fold and won't be summarized in a benchmark chart.
As anybody else, why do you even care about this in the first place?
This doesn't just apply to AI. You should know better than to take marketing announcements at face value, nor trust that most "influencers" or "journalists" or "commenter" repeating and discussing those announcements are doing a lot of work as they buzz over what's said.
Don't be a victim to PR spin, but also don't discount potentially interesting work just because it got channeled through a marketing team. Just look for the information you need amidst the noise or move on if there's none there.
Because they selectively choose which benchmarks to highlight and which competing models to compare against; they don't always compare against the SOTA or the Pareto frontier of performance/price, a lot of times you'll see them compare against the last generation of competing models, or the like, so that they look better if you don't carefully consider what they're comparing against.
For example, here you see they are comparing against Opus 5 and GPT-5.6 Sol, when Opus 5.5, Sonnet 5.5, GPT-6 Astra, and GPT-6.1 Sol would be the latest SOTA models from those labs to compare against.
Also because a lab's own benchmarks tend to run about 6.4% better than third party runs of the same benchmark. Source: https://whatmodel.sh/claims This can happen due to a number of factors, and doesn't necessarily indicate anything nefarious. It could just be that they run multiple times while fixing issues that aren't related to the model, and have a selection bias towards stopping when the score is on the higher side. But since you see that sometimes their own scores are lower than the independent runs, some of the variation is explained just by variance in the model solving problems, since LLMs are nondeterministic there can be variation from run to run and you don't see labs always cheating, just potentially a bit of selection bias about when they stop tweaking their benchmark setup and publish the results.
And yes, there are independent third party runs of benchmarks for just this reason. The most popular with broad coverage of a range of benchmarks is https://artificialanalysis.ai/ but there are a number of others as well.
Being pedantic - LLMs are deterministic software, that is chaotic and typically deployed with a PRNG seed. An LLM ran using the same seed will produce the same output consistently (theoretically - in practice this isn’t always the case because of changing factors wrt the execution environment given how sensitive/chaotic models are).
In many cases they are not deterministic even with a deterministic seed because many runtimes embrace races that interact with floating point rounding to improve latency.
in practice, as actually used, LLMs are nondeterministic.
It is possible, if you control the execution environment sufficiently, and are careful not to use certain kinds of parallelism which can lead to nondeterminism, and you set the seed deterministically and ensure that the prompts are byte identical, you can operate them deterministically.
But no one actually does all of that. Almost everyone uses cloud hosted APIs where the provider could change any number of variables at any time. Everyone wants faster cheaper inference so uses inference engines which may trade off a certain amount of determinism for more speed. Different versions of harnesses and chat templates and benchmark software may introduce small differences in text.
And even though the cloud providers have in the past offered seed and system fingerprints to try to give people a way to control or detect non-determinism (see https://developers.openai.com/cookbook/examples/reproducible...), that has all been deprecated and removed in the newer APIs.
So yeah, while in theory it's possible that if you control the whole stack you can get deterministic results, that's not how anyone is using it, and there is no way for third party benchmarks to ensure deterministic results of hosted models, the latest APIs don't even support it at all.
This whole "well, actually, LLMs are deterministic if you hold them right" is a red herring. In any actual real-world use, they are basically always non-deterministic.
And of course, there are reasons that providers have dropped these attempts at providing deterministic results; because the results are chaotic, perturbed by small changes in input, it's almost never useful to try to make them act deterministic or detect when non-determinism has happend. Instead, it's better to think about them non-deterministically, and look at the distribution of results, rather than just considering a single pass or fail on a single test to be meaningful.
Fair. I wasn’t advocating for “holding them right”, I feel it’s an important nuance to keep in mind when conceptualizing them, i.e. it’s procedural, determistic, yet both inherently and in practice extremely chaotic and therefore not practically deterministic software. So, in essence - software.
With the old GPT-3 api in the olden days, you could set the seed and get deterministic outputs! Interesting results could be drawn from it, as for instance if you use the prompts "a blue bird"/"a blue bird in the sky" the blue bird would have the same look in both with similar seeds.
You have to read between the lines (or wait for someone on HN to do it for you).
It's never "cheaper and faster and more intelligent", it's pick 2. Sometimes (as was the case this month), it was "cheaper per token, but reasons five times as much", so your costs go up.
I run a benchmark and independently check all new models. They are very spiky.
If you are doing coding only then official benchmarks are mostly right, but as many people are starting to do for instance marketing work with LLMs now there’s very little correlation between coding and creativity for instance.
I've got a corollary question, which is "how big is the market for these never ending model announcements?" ... I guess its mostly people coding with agents. Most general AI users are just asking ChatGpt to word an email for them or whatever. How many AI users are on here looking for the next model announcement and working out if the price per token is better? Are we talking like 50,000 developers? More? Less?
Openrouter does >600 trillion tokens a month and the top models shift on a daily basis. A lot of people are paying attention to new releases and even pre-releases (which tend to be free)
It's always cheaper (*than the next more expensive model) and better (*than the next worse model).
And often they also don't mention how verbose it is, that the same text splits into more tokens than in competing models, how slow it is, that the pricing is temporary, that the cost per task actually isn't that good despite being both cheap and capable.
I have this idea that the big labs internally have much more capable models which are prohibitively expensive to run though, and the competition is mainly around getting the price down by releasing models that are quantized/otherwise made more efficient at the cost of a lot of performance.
> For agentic tasks, the model’s overall usage cost is 70% lower than its predecessor.
For this release, the cost is $0.15 input MToK/$0.60 output MTok/$0.015 cache read MTok, In this case the predecessor is MiniMax M3 (not Flash), which was $0.30 input MToK/$1.20 output MTok/$0.06 cache read MTok. So a 50% cut to input costs and a 50% proportional cut to cache read costs (20% of input -> 10% of input) makes the ~70% cost reduction plausible.
A 70% cost reduction is very funny in light of reports saying LLMs are becoming too expensive. That is likely due to the need to compete with DeepSeek V4.1 ($0.15 input MToK/$0.60 output MTok/$0.003 cache read MTok), which this new model still does not beat because DeepSeek's cache costs are still confusingly low, but it's pretty close.
If anything I am confused by how high cache costs still are across many models
The really expensive part of inference - the compute and memory traffic required to perform prompt processing and decode - is paid for roughly once per token in a multi-turn session when prompt caching is working properly.
The difference to the provider's costs between:
- Submitting 100 back-to-back requests, each adding 1k tokens to the context
- Submitting 1 100k token context request
Is negligible. Yet the consumer is going to pay a lot more in the former case for all those repeatedly submitted input tokens that you get charged the cached rate for (but which required no additional compute on the provider side).
DeepSeek V4.1 has an insanely optimized KV-cache that takes around 900MB per million tokens, making it feasible to move to disk between requests. Makes sense they're cheap
The amazing part of DeepSeek's caching is that the TTL is (although not guaranteed) like 8-24 hours. Makes a big difference to how much time I can spend reviewing code, when most everyone else offers maybe 1 hour (for subscribers - often minutes for API users).
I mean, memory prices are high. They ain't getting cheaper. And KVCache is held in memory. The more people are using cache, the more memory you need. It's not free to cache it.
Plus there's complicated cluster-wide KVCache distribution schemes, built around RDMA networking, etc etc.
By far the most expensive hardware cost now of inference is the memory cost. KVCache is still using it. Though obv GPU compute is using more energy
It's not just like... "stick it in Redis, all done!"
DS4's architecture was redesigned for cache efficiency. It's the model itself that is just way more cache efficient, in addition to a distributed KV cache (which they open sourced last year). You could run your own LLM provider just for DS4 models, recreate their setup, and get the same cache efficiency, but it would only work for that one model family.
Other models would also need to be rearchitected to get the same benefit, and I suspect they don't want to, because it seems to create both larger models and slightly less intelligence (DS4* has smaller active experts to get faster results).
That still isn't the whole story. Compare DS 4.1 Flash [1] to GLM 5.3 Flash [2], and consider the effective price. DS's price is lower than GLM on input tokens, but higher than GLM on output tokens. Output price is an order of magnitude higher than input price. So for output-heavy workloads and long reasoning, DS costs more than GLM.[1] https://openrouter.ai/deepseek/deepseek-v4.1-flash#pricing [2] https://openrouter.ai/z-ai/glm-5.3-flash#pricing
Yes the official DS provider has a low cache cost, for now (it's just distributed KV). But people are still concerned about ZDR, sovereign AI, getting more model choice in their subscription price, etc. They want more bang for their buck. So when a new model comes out, the bigger impact is really its performance across providers, not the price at a single provider.
And in about 5 years (or less), API prices will stop being a thing. We all used to pay per-minute for phone calls, texts, data. Nobody does anymore. Virtually every mobile provider is now a flat prepaid or contract fee. Carriers improved their technology (going from analog to digital), and it turned out consumers will spend more money on a packaged subscription than on pay-per-use. That's why everything in the world is a subscription now.
Your modern $70 big wireless plan isn't giving you $70 of value. Similarly, your $20 AI subscription that's a great deal right now, will be less valuable in 5 years, when hardware & models are more efficient. You'll still be able to buy a local AI rig, but it will still be very expensive, due to AI models continuing to be large, and NVidia keeping new hardware expensive, as these things together reinforce their dominance and profits, and our reliance on them.
This also follows another pre-existing trend. Why do people keep having to buy a new iPhone or laptop every other year? Because each year companies make software & data less efficient/more bloated, so people need a more powerful machine to run it, which enables them to make even more bloated software. Almost everything people do with a computer today is what they did 15-20 years ago; it's just a higher bit-rate. Despite this, people keep re-buying the same machine, so companies can keep extracting more wealth.
AI works the same way. Rather than cost to consumers going down over time, it will likely go up. Not because it can't get cheaper/more efficient - but because it increases companies' profit. This isn't just a theory, btw; the US economy's growth is now almost entirely reliant on this happening.
> Rather than cost to consumers going down over time, it will likely go up. Not because it can't get cheaper/more efficient - but because it increases companies' profit.
As shown in the OpenRouter links you provided, there are over 30 distinct providers of these open-weights models, the majority of them supporting ZDR. They can't raise prices because it's a near-perfect marketplace and if they do, someone will just lower them to compete as long as marginal revenue >= marginal cost. There's no moat or other economic factors (e.g. collusion; hard to do with 30 actors) to allow for price control.
Think about mobile users. If you compare the raw cost for MVNOs vs MNOs, they have basically the same cost. There is no special moat around text messages, voice calls, data. By law, MNOs have to give MVNOs access to resell their network access; it's literally the same network. Yet MVNOs are 2x-6x cheaper than MNOs. How is that possible? Trickery. MNOs hide fees, bundle "features", lock people into exorbitant prices, and increase costs over time. Nobody today sees the price per minute of calls, text, data. If you look really, really closely, you can compare data caps, international vs domestic. But almost nobody does.
Now back to LLMs: Normal users do not pay API cost, they pay subscription cost. They're going to be the vast majority of users. No normal user is going to look at API provider costs. They're going to pick one of a handful of subscription providers. Those subscription providers will hide the real costs behind the scenes, making deals with inference providers, who will in turn hide those prices for competitive advantage. They will raise prices, and it'll be buried in the subscription fees. This has been happening for a decade with mobile subscriptions, and it's already happening now with LLM subscriptions. (There is no price-list for current LLM subscriptions; it's a total mess that's constantly changing)
Enterprise and government are large-scale users, and they'll pay exactly the same way they pay for compute right now (LLM cost is literally compute cost). Some will self-host, some will pay API pricing, some will rent, some will use "team" or "business" subscriptions. This is also all happening right now. AWS is making bank from Bedrock purely because people in their ecosystem would rather spend money through a known vendor with marketplace credits, and that's just one of a hundred different examples.
To put it in simpler terms: AI is just servers. Normal people do not pay for servers, they pay a service provider for services. Businesses pay for servers, but in many different ways, and definitely not the cheapest options.
The weights have yet to be released. The previous MiniMax M3 had (I think) ~428B parameters. Since this one is called Flash and they advertise it as inference being 3 times faster, there's the possibility that this will be a ~150B size model which would (quantized) run on DGX Spark or 128GB Macs or the new AMD AI systems.
The FAQ on their website says Minimax 3.1 flash is 428B total, 23B active
Which is roughly similar to deepseek 4.1 flash.
I find it interesting that we’re calling these “flash” models now, even though they aren’t “small or light” and in many cases aren’t particularly that fast.
I suppose it’s more so that the standard for “regular models” is becoming 1T+
Flash probably means single expert fits into some cost efficient GPU memory, and non flash models require more complex and cost expensive architectures.
On Openrouter at least, all Mino models are painfully slow. TTFT is many seconds and the models spend a lot of time on its CoT, making effective output per second many times slower than DS Flash. On Flash 4.1, I constantly get ~180 tok/s and the real work got done fast, too. So even when the Mimo 2.6 are smart models, I use them much less recently. Not to mention their costs are also higher than DS.
For me, the DeepSeek 4.1 models are currently the ultimate cost-effective model the rest have to match with, especially if you are not on a subscription plan with heavy discount.
GPT Sol 6.1 is however, for me the most efficient model, both in terms of time to finish and token usage.
In our data analytics benchmark, it got the best score of all of the Chinese models and tied Opus 5.5 & Argon (41/43 questions correct). It's also 1/3rd the price of both of those models. However, it seems to use A LOT of tokens and it's the slowest model that we've tested for the exam; it's slow because of the number of tokens it uses, not the inference speed.
We also tested Space Bunny (its stealth version) a few weeks ago and the behavior changed quite a bit. Went from 30s / question to 3 minutes / question and the accuracy improved quite a bit.[1]
This benchmark is short-horizon data analytics questions, so not fully representative of software engineering tasks. YMMV.
I have both platform.deepseek.com (hooray off-peak pricing) and the $100/m Claude subscription and boy do I find myself mostly using DS-flash pretty much every time.
There's something to be said for the speed with which it gets word done - I take fewer "it's compiling" breaks, and lose my train of thought less frequently. What's crazy is that the speedy work it does is pretty much always correct - which is a reason why I don't feel the need to spin up claude-code and "ask the expert".
This speed & good-work combo is the DS signature IMO
I use DS-4.1-Flash a lot but find it often gets caught up in long thinking runs. The "But wait... Actually... Hmm..." type of thing. Even though the tok/s is good, it stills feels slow because of this. Same with 1017.
I use Opus 5.5 at work and it does this much less (I know, different classes). Maybe it is to be expected of the flash models.
Which claude model are you comparing it to? If you're using fable max obv it's going to be slower, but like sonnet high is both faster and more intelligent
I'm finding is very hard to figure out when it's better to go for a cheaper dumber model vs an better expensive one when the expensives ones get stuff done quicker.
Minimax has been a strong player in our tests, if they managed to solve the inference speed issue(be scared, be very scared). A decent speed is at least 13 tps in consumer hardware. Let see if it passes Hugston bench this time, we really hope so.
20 t/s is about the speed you can read at, so slower than that and it always feels like it's buffering. 35 t/s is where it starts feeling decent and 50 t/s feels nice.
I know, it is never enough, more, more, give me more. But the fact is that no one has broken though yet to offer a frontier like open weight over 13 tps in consumer hardware. It would be usable at 13tps for me, especially considering the benefits and advantages of privacy, infinite tokens, availability and no refusals/restrictions.
To this day Minimax and GLM are very strong coders but have the same issue, inference speed. We need a real break through, one that thinks out of the box (Tesla turbines style perhaps).
I just use the strongest model I can run at a non-frustrating speed. That's never frontier performance on local hardware, but most of us running locally accepted that a long time ago.
Currently on R9700 + 64GB of RAM I'm running Qwen3.8-Flash-Next. It's decent.
If all you're going to do is chat with an AI, you don't need a 400B parameter MoE model.
13 TPS is unusable for agentic coding, because if decode is that slow, then prefill is absolutely abysmal, and agentic work live and die by prefill speeds.
I’ve been agentic coding at home with like 15 TPS on average. I don’t mind it at all. I can give my agent a big task or a list of little ones and just… do something else. Why do I need to watch it work?
Seconding this. 2-bit quant of DSv4Flash 0731 (Unsloth) running on lemonade-llamacpp on a Corsair Strix Halo. OpenCode had an issue giving me 400s constantly for the past few weeks, so I just went all-in on local dev with omp. Yes, it takes time, but also: it works quite well. I've since gotten OpenCode to behave, so I'm back to using that quite a bit, but the local machine is really good as a "smol" model in omp, for example.
It doesn’t. Decode is by far the most dominant and expensive part of agentic tasks and once prefill hits a certain achievable speed decode dominates the time in ant agentic flows with reasoning. There is a reason input is so much cheaper :)
Isn’t input so much cheaper because it’s just less work? Prefill is usually 10-20x as fast as generation.
If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.
No commercial service has 200 t/s prefill though. It is 1000s of pps for prefill. It is largely compute bound and very very fast on modern GPUs and model architectures. That is my point. It is an order of magnitude faster than decode, which is largely bound by memory bandwidth and just has a much harder limit. Prefill tends to look like a small slice in typical compute / token generation budgets. I am not saying it isn't important (I have been working hard on my own inference engine for flash next and am intimately familiar). I am saying it does not dominate. Also compaction and reading the whole prompt tends to be one of the few times you really hit peak pps because it is cached and you can just fill your data pipes for a bit. Also Deepseek showed you can make kv cache really cheap. Decode is still decode.
The math. It’s pretty brutal. You have to multiply insanely large matrices in memory. A dense model has to have and multiply across every parameter basically. MoE models work around it, sure, but you still have to hold a lot in memory and density of activated parameters matters. There is not much of a way around it yet. We are potentially getting there, but its not obvious yet.
It did okay in Pac-Man. Not great, but a usable example. Unfortunately, it did take half an hour. I'm trying to figure out what the pricing would be if it wasn't free. But zero dollars is a great entry price.
I went to check and Opus 5.5 wasn't benchmarked on DeepSWE1.1, I can't keep track of which are in vogue at the moment but maybe it's thought to be saturated at this point, as they only have older GPT models as well
The recent DeepSeek release used Opus 5 as a reference point as well. I don't understand it. It's not like they're referencing superseded releases of other open models.
I think it clearly points to where they expect themselves to be, Opus 5.5 and Astra or Sol 6.1 are models that shouldn't exist at those prices without the price competition.
And no new releases from Chinese labs likely means their models that they could compare against those models are still baking largely because they lack compute so severely they need to train models for far longer, or figure out training on Chinese hardware.
Either way it's clear open models a lagging by ~1-2 months compare to frontier, depending on what frontier means to a person. Honestly most new models are just not that much better than at edge cases, they still fail a lot for the kind of stuff I do, but those edge cases matter a lot for un-interrupted long horizon work.
I personally am starting to see a wall here in llm development but that might just be me, given that people are being forced to scale models up considerably for frontier performance.
It's scary thinking that modern models will likely be at best 2-3% better in benchmarks or only useful in 1-2 unique edge cases compared to base models for 100x cost increase in compute required, it's getting absurd.
I can't imagine how Fable models can retain any value once you have to go 100x larger, while 27-32B models start to catch up to the current frontier for coding or other narrow/focused tasks in a year or so.
These models exist because better models exist, these models can't exist without better models existing, you will never beat the frontier with smaller model sizes. Because all these models are trained again larger models, generally a teacher model of sorts.
Lots of synthetic data, distilled data etc.
The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?
Synthetic data, distilled data, etc., does not necessarily mean data coming from outside of an organization. When we see performance increases with models like Haiku, it is because they are able to leverage the knowledge of larger architectures.
I fully expect that we will see increased performance from smaller architectures (to a point) simply because they are refined using larger ones.
> The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?
This is available to everyone developing these systems. Prices per relative unit of intelligence will continue to decrease. And market economics will continue to drive the cost down.
There is nothing inherent in the minimization process that says you must distill from a larger model. It is just the thing we know how to do right now.
It's not really if you mean to point to the bitter lesson, it was models/intelligence scales better with general/simple methods with more compute better than it scales with any hand-crafted human program/method.
Ala intelligence is a function of compute, there was no guarantee that you need to 10x compute for 10% better performance or something like that.
If you ask the labs they will say the scaling is fairly linear or atleast sub-quadratic, they definitely won't say it's exponential.
No one was thinking we need to 10x compute from here again for another 10-20% increase in performance.
It's only been since Opus 4.6 that the compute growth vs improvements are starting to feel this way.
For instance GPT-3 vs GPT-4 was practically a 10-100x jump for a 10-100x increase in compute, we went from broken sentences to fully formed essays writing, speech and so on.
GPT-4 vs GPT-5 was just as massive we went from no agentic capabilities to fully agentic, 10-100x for a 10-100x increase in compute usage.
GPT-5 vs GPT-6 was atleast a 10x (upto 100x) increase in compute, but the difference is not that clear sure people say that one is significantly better than the other, but GPT-5.5 was good enough compared to where we are today.
Even on benchmarks compared to GPT-5.2 Codex we have almost 2x'ed the performance on say artificial analysis (53 vs 29).
This is absurd. 10-100x increase in compute for 2-5x performance improvement is a massive gap.
I see a lot of benchmarks which have not data for DS4.1 and a lot of other benchmarks, where DS4.1 beats this new model. I guess I can finally stick to my current AI coding setup for longer than a month.
Another day where the Pareto frontier is moved up and to the left!
I haven't tried this model yet but Deepseek and the GLM 5.3 series do feel like "enough" for a lot of tasks, it's pretty amazing to see that we're still getting better and cheaper models all the time.
I only wished they would focus more on wall clock time. For a hobby project I do not mind that a cheaper model takes twice as long but at work it's hard to justify using a "cheaper" model if the time per task is so much higher than the frontier models.
The amount of time that it took to complete a benchmark is almost never shown in these kind of releases.
Would place DS 4.1 1017 roughly equivalent to GLM 5.3 which means it's targeted to price rather than direct capability. Probably why they listed DS adjacent in the comparison table when according to their results more expensive models are beaten by it.
I used it as Space Bunny, was very good at documentation, structuring and organizing the things, but was not as concise and direct as DeepSeek 4.1 Flash in coding, we could not focus on code part as a human and AI team from how many approves are needed for each step discussion.
The deepswe and terminalbench4.0 scores don't seem to jive with what many opencode users' experience has been of this model while it was being tested as "space bunny". I saw a lot of anecdotes of people trying it and going back to qwen 3.8-flash-next or deepseek v4 flash because "space bunny" made some rather egregious errors.
> [Space Bunny] was an early version of MiniMax M3.1 Flash.
They mention that space bunny was an earlier checkpoints, so it may have improved (or may it could be benchmaxxed)
I find it interesting that so many models seem to be at almost exactly 74% specifically which makes me suspicious of the benchmark. I want to say 5.6-sol, Astra, deepseek flash 4.1, opus 5 (and 5.5?), Gemini 3.8-flash and a few others all ended up there?
There was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.
I do agree that it’s generally matched my impressions of models for general SWE at least until we had Opus 5 and 5.6-Sol beating Fable 5. Which coincidentally is when they hit 74%.
Models like Deepseek 4.1 flash, Muse 1.3, and MiniMax 3.1 flash are certainly capable and very cost effective, but I’m not sure anyone would say they’re better than Fable 5/5.1 in capability
MiniMax models have been pretty disappointing. M2.5 was good, M2.7 was meh and came with a terrible license, 3 meh. Maybe this will be better, but I have my doubts. When I say good vs meh vs bad, this is all relative of course. If M3.1 came out a few months ago it would be groundbreaking. Things just move too quick.
The comparisons with models like GPT-5.6 or Opus are pointless because they vary substantially. They are often nerfed shortly after release and then performance is not consistent. One day model will be doing relatively fine, another it will be dumb as markov generator. In that regard self hosted models are better, because you know what is running your requests.
I mean the efforts you linked are showing there aren't actual nerfs happening, making the alternative explanation of people just 'growing into' the capabilities of models more likely to me
> We need provide SVG code. Need creative. Ensure valid standalone. Pelican on bicycle, perhaps detailed vector. Need answer likely fenced svg. Could include title desc. Need be reasonably concise but image quality.
The truncated English is interesting. I noticed something similar from the OpenAI blackhat talk a couple months ago. Is this pretty standard at this point?
Apparently the domain is shadowbanned (https://news.ycombinator.com/from?site=minimax.io) which is why people kept submitting it (and normally they cannot be vouched), and presumably a moderator gave this one their blessing.
The solution is for @dang to unshadowban it because Minimax is now a newsworthy company.
This model was released as a "stealth model" in OpenCode Go for a while as "Space Bunny" (they mention this in the blog). So people have been using without knowing what the model was exactly, but it failed to impress anyone as far as I can tell. This is despite the model being served for free (with some asterisks). GLM 5.3 Flash and DeepSeek V4.1 Flash, even before last week's update, seemed to be much preferred. Stealth releases happen often with OpenCode subscriptions, and GLM 5.3 Flash (stealth name "Ox Alpha") impressed a lot for example.
The perverse incentives around benchmarks are now so bad that benchmark scores are effectively meaningless. Models can score highly and still be crap to use. It doesn't matter whether it's Minimax or Google, or anyone really.
1 - https://bench.killswitch-lang.org/
https://github.com/Niko1221/Strata
The model wasn’t horrible at coding, it just made a ton of initial mistakes—but then would catch most of them later on. It can get work done if coaxed, but its peers are just cheaper and/or easier to work with.
I've used local models extensively (mostly qwen 3.8 27b) and now Claude sonnet/opus 5.5 since launch.
Sometimes I really wonder if not being able to see what's going on in thinking is a large contributor to thinking the models are stronger.
That assumes you bother skimming through thinking traces and the output in-between.
In my experience both qwen and claude fumbles and misunderstand, but will in a lot of cases self correct.
Qwen models can feel very indecisive but somehow makes it in the end. If the harness was able to hide that and only highlight the end results, I'd probably think qwen models are smarter.
Reading the tool calls Claude does while thinking shows a lot of trial and error about even simple things like editing code and getting the whitespace wrong. But it happens so fast compared to qwen that it's sort of acceptable. With qwen, it can feel frustrating because it spends so much time on it.
This is all assuming that's what you meant by mistakes, but maybe you meant something different.
I am not sure. The people who actually care enough to look at benchmarks will run evals, and find out quickly. If you post silly benchmark scores too often, you will simply be ignored. You are paying with your reputation.
i don't know about that. it seems like the silly benchmarks keep getting paid attention to.
I can run Qwen Flash Next on hopes, dreams, DDR4, and ancient data center GPUs at 200 t/s :)
But once in a while it gets into long reasoning loops, which is annoying. It seems less with the Swift model but I haven't benchmarked the difference between that one and the original.
Lately I prefer GLM5.3-Flash since I feel it is noticeably smarter and I have the hardware for it, but FN's performance has me looking forward to Qwen4.
They genuinely try to improve usability, without spending time on marketing or benchmaxxing.
Except for v4 0731, which was a regression, and probably the reason they released v4.1 ahead of schedule.
I've become incredibly skeptical of new model announcements until I can find evidence of someone else using it for something effective. I'd love to see an announcement that wasn't structured around attracting investors with too much money and too little domain knowledge.
This MiniMax model is clustered with Haiku 5.5 so (combined with the criticism on here that it was lacklustre while free on opencode go) I don't find it interesting, but every announcement has a chance at pushing that frontier.
I was benchmarking affordable LLMs for japanese->english translation recently and the winner was Kimi K3 - there are a ton of interesting models in the wake of the flagships.
All this incremental progress is just more of the same. I want a real novel capability, not just more, faster, cheaper, “smarter”.
We are barely 4 years in since the release of a historically important tech. Of course the next few years are incremental. thats how it always works.
Even last year, before opus 4.5, long-horizon agents didnt really work. I would not say the last 12 months are only incremental
As an aside, I remember when sora 2 launched and thinking, "wow I wonder if we'll ever get something close this in open weights?" The prevailing sentiment was "maybe 5 years." Yet, we have Minimax H3 and Flux a year and a half later.
But this only works for frontier models. If MiniMax catches up to Opus 5.5 in 6 months, no one would notice.
I'd be happy as hell if I end up being able to run an Opus 5.5-level model at home within a 6-month time horizon. May or may not happen based on China's export policy, of course.
The open weights we have may be as good as they ever get... but then that's been true all along, considering nobody really knows why they are releasing powerful models for free in the first place.
I remember that close to an year ago i felt the same way about opus 4.5. And even if you can easily run ds or qwen locally to get very similar results to that, we moved to 5.5 being the new target.
Deepseek can do 90% of the job but a claude reviwer/spec writer is a brutal improvement.
I personally enjoy seeing how different models view problems. Take big plans from big models, then see how other agents reinterpret a stance. See which directions they go. I love the diversity of views they produce.
It's not really relevant what the numbers are. I really really enjoyed talking with MiniMax M3 a lot. And M2.7. it had a good world view, wrote the situation up incredibly fluently. It thought out loud well. These were pleasant uses for me. The realities, the world's they could see felt very grounded. Good stuff, to me.
Claude sub has a 60x subsidy, so a $20 sub gives you $1200 in API.
Based on this math, Claude sub is a much better deal than Deepseek. And it's a stronger model of course.
This is only true if you max out the plan, which requires working around awkward 5-hour and weekly limits. For side projects, sure, you can use a Ralph loop or whatever and just continue when the window resets. For work, I can't afford to simply stop working when I hit my limit. And using Claude at API prices is prohibitively expensive.
Personally, I have a $20/month OpenAI subscription that I supplement with DeepSeek V4.1 Flash (currently using CoralBricks as the provider since it's fast and I'm grandfathered into their $0 cache read pricing). It's working out rather well so far. Even when I use DS all day as my primary model, I only pay about $1/day, which is roughly equivalent to what I pay for the sub anyway (but with no limits).
So my original point about the limits being awkward still stands.
If I was to do anywhere near that kind of work on Anthropic, I'd find myself running out of tokens, waiting days to continue working, and leaning on their el-crap models to try to avoid running out again. This happens all the time.
This way I get to leverage DS for nearly all my work, benefit from its far superior latencies, and I still get to lean on OpenAPI / Anthropic or whatever SOTA models I want when I need them.
The only time that really falls down is if I'm doing huge amounts of real novel work, which hasn't been the case for a while.
I use OpenCode at home, this is one of the reasons.
Their Ember-1 model (Kimi K3 tuned to think less) hallucinates to high hell, I have no trust of it, nowhere to provide feedback, despite their launch post saying they would like feedback. That they loudly and proudly claim their internal devs notice no difference is a huge red flag for me. It says there is too much eyes free coding, I noticed the hallucinations and false claims of task completion immediately.
For some reason it just creeps me out knowing that someone is recording me and reading my conversations.
ZDR is:
What you really want (assuming you can't afford your own servers and datacentre space) is secure enclave processing:See Tinfoil (US)[1] or Privatemode (Germany)[2].
[1] https://tinfoil.sh/ [2] https://www.privatemode.ai/
Clearly I'm not defending tinfoil regarding scanning but I think its only fair that you do not make the statement out of context[1].
People should read the context[1] and come to their own decisions.
Also recall that given how attestation works, Tinfoil are forced to publish what they run in the enclave in their GitHub account[2] so you can verify the statement on their blog against their repo. You do not have that luxury with the closed-source "ZDR" brigade.
[1] https://tinfoil.sh/blog/2026-09-14-safety-without-compromisi... [2] https://github.com/tinfoilsh
Depending on the balance of your concerns a ZDR that is lying about their privacy might still be an improvement because they can't both safely conceal the violation of their promises AND tamper with the output.
When one factors in the lack of absolute protection provided by secure enclaves-- that they're reliably bypassable with advanced techniques and thus mostly serve the purpose of keeping honest people honest, I think it would be better to avoid doing business with any enclave provider that has already demonstrated such leanings.
I do broadly agree that of all the things tinfoil could/should be working on in relation to service features, this is perhaps an odd use of their time. And I do wonder if this ultimately came about due to some sort of external pressures or maybe a big-spending customer asked for it. Who knows. Their blog post is a little vague on the background context other than selling it as Good Thing (TM).
However the way it is going, there will not even be private APIs, just recording and watching all the time.
Anthropic is unsurprisingly leading the race to the bottom of the barrel by requiring moderator reviews for API customers. Apparently everyone at GCP and AWS complied.
GCP has just followed and done the same thing on Gemini Argon. No CMEK allowed. Azure has some ZDR but presumably they are restricting it.
Some neoclouds are still standing but some are also regressing by using server-side scanning to prompt human review.
There are two/three private TEE providers. One of them (Tinfoil) just put server-side scanning inside the enclave for the consumer product. The API is good for now, I am taking it as notice from them to get off the service before they start logging everything.
I'm a heavy Claude/anthropic user, I max out my 20x max plan in 1-2 days at this point. The OpenCode Go $40 plan with DeepSeek flash is like 2x the value of the Claude max 20x plan for how much I actually get out of it, for 1/5 the price.
DeepSeek literally like 10x better value than Claude/Anthropic.
DeepSeek is really cheap when you compare API pricing, but it's more expensive than a subsidized Anthropic or OpenAI subscription.
(yes, I'm aware the analogy isn't perfect for many reasons)
As a prospective user, none of of the benchmarks details will matter to you, but the family of models and settings they peg themselves against can give you a hint of where you might want to evaluate them within your own workflows.
As an enthusiast or lay researcher, the more interesting thing is to look at their techniques and any innovations they're exploring in training, inference, etc. Those are usually going to be details tucked below the fold and won't be summarized in a benchmark chart.
As anybody else, why do you even care about this in the first place?
This doesn't just apply to AI. You should know better than to take marketing announcements at face value, nor trust that most "influencers" or "journalists" or "commenter" repeating and discussing those announcements are doing a lot of work as they buzz over what's said.
Don't be a victim to PR spin, but also don't discount potentially interesting work just because it got channeled through a marketing team. Just look for the information you need amidst the noise or move on if there's none there.
*crafted by LLMs with a list of PR guidelines in a markdown file
For example, here you see they are comparing against Opus 5 and GPT-5.6 Sol, when Opus 5.5, Sonnet 5.5, GPT-6 Astra, and GPT-6.1 Sol would be the latest SOTA models from those labs to compare against.
Also because a lab's own benchmarks tend to run about 6.4% better than third party runs of the same benchmark. Source: https://whatmodel.sh/claims This can happen due to a number of factors, and doesn't necessarily indicate anything nefarious. It could just be that they run multiple times while fixing issues that aren't related to the model, and have a selection bias towards stopping when the score is on the higher side. But since you see that sometimes their own scores are lower than the independent runs, some of the variation is explained just by variance in the model solving problems, since LLMs are nondeterministic there can be variation from run to run and you don't see labs always cheating, just potentially a bit of selection bias about when they stop tweaking their benchmark setup and publish the results.
And yes, there are independent third party runs of benchmarks for just this reason. The most popular with broad coverage of a range of benchmarks is https://artificialanalysis.ai/ but there are a number of others as well.
Being pedantic - LLMs are deterministic software, that is chaotic and typically deployed with a PRNG seed. An LLM ran using the same seed will produce the same output consistently (theoretically - in practice this isn’t always the case because of changing factors wrt the execution environment given how sensitive/chaotic models are).
It is possible, if you control the execution environment sufficiently, and are careful not to use certain kinds of parallelism which can lead to nondeterminism, and you set the seed deterministically and ensure that the prompts are byte identical, you can operate them deterministically.
But no one actually does all of that. Almost everyone uses cloud hosted APIs where the provider could change any number of variables at any time. Everyone wants faster cheaper inference so uses inference engines which may trade off a certain amount of determinism for more speed. Different versions of harnesses and chat templates and benchmark software may introduce small differences in text.
And even though the cloud providers have in the past offered seed and system fingerprints to try to give people a way to control or detect non-determinism (see https://developers.openai.com/cookbook/examples/reproducible...), that has all been deprecated and removed in the newer APIs.
So yeah, while in theory it's possible that if you control the whole stack you can get deterministic results, that's not how anyone is using it, and there is no way for third party benchmarks to ensure deterministic results of hosted models, the latest APIs don't even support it at all.
This whole "well, actually, LLMs are deterministic if you hold them right" is a red herring. In any actual real-world use, they are basically always non-deterministic.
And of course, there are reasons that providers have dropped these attempts at providing deterministic results; because the results are chaotic, perturbed by small changes in input, it's almost never useful to try to make them act deterministic or detect when non-determinism has happend. Instead, it's better to think about them non-deterministically, and look at the distribution of results, rather than just considering a single pass or fail on a single test to be meaningful.
It's never "cheaper and faster and more intelligent", it's pick 2. Sometimes (as was the case this month), it was "cheaper per token, but reasons five times as much", so your costs go up.
If you are doing coding only then official benchmarks are mostly right, but as many people are starting to do for instance marketing work with LLMs now there’s very little correlation between coding and creativity for instance.
So, I can quickly judge that this is an interesting option only if cheaper or faster than DeepSeek.
Based on Terminal Bench 4.0, I'd also guesstimate behind GPT 5.6 Sol, so of course behind the SOTA as expected.
Found the tweet: https://x.com/karpathy/status/2109361546505966046?lang=en
And often they also don't mention how verbose it is, that the same text splits into more tokens than in competing models, how slow it is, that the pricing is temporary, that the cost per task actually isn't that good despite being both cheap and capable.
Edit: You removed model names ig
For this release, the cost is $0.15 input MToK/$0.60 output MTok/$0.015 cache read MTok, In this case the predecessor is MiniMax M3 (not Flash), which was $0.30 input MToK/$1.20 output MTok/$0.06 cache read MTok. So a 50% cut to input costs and a 50% proportional cut to cache read costs (20% of input -> 10% of input) makes the ~70% cost reduction plausible.
A 70% cost reduction is very funny in light of reports saying LLMs are becoming too expensive. That is likely due to the need to compete with DeepSeek V4.1 ($0.15 input MToK/$0.60 output MTok/$0.003 cache read MTok), which this new model still does not beat because DeepSeek's cache costs are still confusingly low, but it's pretty close.
The really expensive part of inference - the compute and memory traffic required to perform prompt processing and decode - is paid for roughly once per token in a multi-turn session when prompt caching is working properly.
The difference to the provider's costs between:
- Submitting 100 back-to-back requests, each adding 1k tokens to the context
- Submitting 1 100k token context request
Is negligible. Yet the consumer is going to pay a lot more in the former case for all those repeatedly submitted input tokens that you get charged the cached rate for (but which required no additional compute on the provider side).
DeepSeek V4.1 has an insanely optimized KV-cache that takes around 900MB per million tokens, making it feasible to move to disk between requests. Makes sense they're cheap
Plus there's complicated cluster-wide KVCache distribution schemes, built around RDMA networking, etc etc.
By far the most expensive hardware cost now of inference is the memory cost. KVCache is still using it. Though obv GPU compute is using more energy
It's not just like... "stick it in Redis, all done!"
Other models would also need to be rearchitected to get the same benefit, and I suspect they don't want to, because it seems to create both larger models and slightly less intelligence (DS4* has smaller active experts to get faster results).
Yes the official DS provider has a low cache cost, for now (it's just distributed KV). But people are still concerned about ZDR, sovereign AI, getting more model choice in their subscription price, etc. They want more bang for their buck. So when a new model comes out, the bigger impact is really its performance across providers, not the price at a single provider.
And in about 5 years (or less), API prices will stop being a thing. We all used to pay per-minute for phone calls, texts, data. Nobody does anymore. Virtually every mobile provider is now a flat prepaid or contract fee. Carriers improved their technology (going from analog to digital), and it turned out consumers will spend more money on a packaged subscription than on pay-per-use. That's why everything in the world is a subscription now.
Your modern $70 big wireless plan isn't giving you $70 of value. Similarly, your $20 AI subscription that's a great deal right now, will be less valuable in 5 years, when hardware & models are more efficient. You'll still be able to buy a local AI rig, but it will still be very expensive, due to AI models continuing to be large, and NVidia keeping new hardware expensive, as these things together reinforce their dominance and profits, and our reliance on them.
This also follows another pre-existing trend. Why do people keep having to buy a new iPhone or laptop every other year? Because each year companies make software & data less efficient/more bloated, so people need a more powerful machine to run it, which enables them to make even more bloated software. Almost everything people do with a computer today is what they did 15-20 years ago; it's just a higher bit-rate. Despite this, people keep re-buying the same machine, so companies can keep extracting more wealth.
AI works the same way. Rather than cost to consumers going down over time, it will likely go up. Not because it can't get cheaper/more efficient - but because it increases companies' profit. This isn't just a theory, btw; the US economy's growth is now almost entirely reliant on this happening.
As shown in the OpenRouter links you provided, there are over 30 distinct providers of these open-weights models, the majority of them supporting ZDR. They can't raise prices because it's a near-perfect marketplace and if they do, someone will just lower them to compete as long as marginal revenue >= marginal cost. There's no moat or other economic factors (e.g. collusion; hard to do with 30 actors) to allow for price control.
Think about mobile users. If you compare the raw cost for MVNOs vs MNOs, they have basically the same cost. There is no special moat around text messages, voice calls, data. By law, MNOs have to give MVNOs access to resell their network access; it's literally the same network. Yet MVNOs are 2x-6x cheaper than MNOs. How is that possible? Trickery. MNOs hide fees, bundle "features", lock people into exorbitant prices, and increase costs over time. Nobody today sees the price per minute of calls, text, data. If you look really, really closely, you can compare data caps, international vs domestic. But almost nobody does.
Now back to LLMs: Normal users do not pay API cost, they pay subscription cost. They're going to be the vast majority of users. No normal user is going to look at API provider costs. They're going to pick one of a handful of subscription providers. Those subscription providers will hide the real costs behind the scenes, making deals with inference providers, who will in turn hide those prices for competitive advantage. They will raise prices, and it'll be buried in the subscription fees. This has been happening for a decade with mobile subscriptions, and it's already happening now with LLM subscriptions. (There is no price-list for current LLM subscriptions; it's a total mess that's constantly changing)
Enterprise and government are large-scale users, and they'll pay exactly the same way they pay for compute right now (LLM cost is literally compute cost). Some will self-host, some will pay API pricing, some will rent, some will use "team" or "business" subscriptions. This is also all happening right now. AWS is making bank from Bedrock purely because people in their ecosystem would rather spend money through a known vendor with marketplace credits, and that's just one of a hundred different examples.
To put it in simpler terms: AI is just servers. Normal people do not pay for servers, they pay a service provider for services. Businesses pay for servers, but in many different ways, and definitely not the cheapest options.
Which is roughly similar to deepseek 4.1 flash.
I find it interesting that we’re calling these “flash” models now, even though they aren’t “small or light” and in many cases aren’t particularly that fast.
I suppose it’s more so that the standard for “regular models” is becoming 1T+
https://agent.minimax.io/tools/m3-1-flash-preview
- DeepSeek V4 Flash: 284B / 13B active MoE
- DeepSeek V4 Pro: 1.6T / 49B active MoE
- DeepSeek V4.1 Flash: 552B / 8B input, 16B output MoE
[0]: https://aibenchy.com/compare/minimax-minimax-m3-1-flash-high...
And that model was not top 5 even when it was released, 3 months ago.
I wouldn't expect that.
But it's probably n=1. Still.
What browser are u viewing it on?
https://aibenchy.com/compare/minimax-minimax-m3-1-flash-high...
https://imgur.com/a/Hrtk3IX
Both in Chrome and Firefox.
https://aibenchy.com/compare/minimax-minimax-m3-1-flash-high...
https://artificialanalysis.ai/models/comparisons/mimo-v2-6-p...
For me, the DeepSeek 4.1 models are currently the ultimate cost-effective model the rest have to match with, especially if you are not on a subscription plan with heavy discount.
GPT Sol 6.1 is however, for me the most efficient model, both in terms of time to finish and token usage.
We also tested Space Bunny (its stealth version) a few weeks ago and the behavior changed quite a bit. Went from 30s / question to 3 minutes / question and the accuracy improved quite a bit.[1]
This benchmark is short-horizon data analytics questions, so not fully representative of software engineering tasks. YMMV.
[1] https://plotly.com/blog/minimax-m3-1-flash-plotly-data-analy...
I have both platform.deepseek.com (hooray off-peak pricing) and the $100/m Claude subscription and boy do I find myself mostly using DS-flash pretty much every time.
There's something to be said for the speed with which it gets word done - I take fewer "it's compiling" breaks, and lose my train of thought less frequently. What's crazy is that the speedy work it does is pretty much always correct - which is a reason why I don't feel the need to spin up claude-code and "ask the expert".
This speed & good-work combo is the DS signature IMO
I use Opus 5.5 at work and it does this much less (I know, different classes). Maybe it is to be expected of the flash models.
20 t/s is about the speed you can read at, so slower than that and it always feels like it's buffering. 35 t/s is where it starts feeling decent and 50 t/s feels nice.
Currently on R9700 + 64GB of RAM I'm running Qwen3.8-Flash-Next. It's decent.
13 TPS is unusable for agentic coding, because if decode is that slow, then prefill is absolutely abysmal, and agentic work live and die by prefill speeds.
Why wouldn't I want the smartest possible model I can get? I only "chat" about smart person stuff.
If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.
https://jonclegg.github.io/pacman-bakeoff/#minimax-m3.1-flas...
https://artificialanalysis.ai/models/claude-opus-5-5#intelli...
https://deepswe.datacurve.ai/
And no new releases from Chinese labs likely means their models that they could compare against those models are still baking largely because they lack compute so severely they need to train models for far longer, or figure out training on Chinese hardware.
Either way it's clear open models a lagging by ~1-2 months compare to frontier, depending on what frontier means to a person. Honestly most new models are just not that much better than at edge cases, they still fail a lot for the kind of stuff I do, but those edge cases matter a lot for un-interrupted long horizon work.
I personally am starting to see a wall here in llm development but that might just be me, given that people are being forced to scale models up considerably for frontier performance.
It's scary thinking that modern models will likely be at best 2-3% better in benchmarks or only useful in 1-2 unique edge cases compared to base models for 100x cost increase in compute required, it's getting absurd.
I can't imagine how Fable models can retain any value once you have to go 100x larger, while 27-32B models start to catch up to the current frontier for coding or other narrow/focused tasks in a year or so.
Lots of synthetic data, distilled data etc.
The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?
I fully expect that we will see increased performance from smaller architectures (to a point) simply because they are refined using larger ones.
> The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?
This is available to everyone developing these systems. Prices per relative unit of intelligence will continue to decrease. And market economics will continue to drive the cost down.
Ala intelligence is a function of compute, there was no guarantee that you need to 10x compute for 10% better performance or something like that.
If you ask the labs they will say the scaling is fairly linear or atleast sub-quadratic, they definitely won't say it's exponential.
No one was thinking we need to 10x compute from here again for another 10-20% increase in performance.
It's only been since Opus 4.6 that the compute growth vs improvements are starting to feel this way.
For instance GPT-3 vs GPT-4 was practically a 10-100x jump for a 10-100x increase in compute, we went from broken sentences to fully formed essays writing, speech and so on.
GPT-4 vs GPT-5 was just as massive we went from no agentic capabilities to fully agentic, 10-100x for a 10-100x increase in compute usage.
GPT-5 vs GPT-6 was atleast a 10x (upto 100x) increase in compute, but the difference is not that clear sure people say that one is significantly better than the other, but GPT-5.5 was good enough compared to where we are today.
Even on benchmarks compared to GPT-5.2 Codex we have almost 2x'ed the performance on say artificial analysis (53 vs 29). This is absurd. 10-100x increase in compute for 2-5x performance improvement is a massive gap.
[1]: https://bitterlesson.ai/ [2]: https://artificialanalysis.ai/?models=gpt-6-astra%2Cgpt-6-lu...
What are these numbers based on? Seems hard to believe…
I haven't tried this model yet but Deepseek and the GLM 5.3 series do feel like "enough" for a lot of tasks, it's pretty amazing to see that we're still getting better and cheaper models all the time.
I only wished they would focus more on wall clock time. For a hobby project I do not mind that a cheaper model takes twice as long but at work it's hard to justify using a "cheaper" model if the time per task is so much higher than the frontier models.
The amount of time that it took to complete a benchmark is almost never shown in these kind of releases.
It actually hasn't, it seems: https://news.ycombinator.com/item?id=50151775
Obviously your marketing team is going to use the Pareto frontier where your model has improved the most and compares favorably against competitors.
Mini, Max, 3.1, and Flash. SEO maxing strategies...
They mention that space bunny was an earlier checkpoints, so it may have improved (or may it could be benchmaxxed)
I find it interesting that so many models seem to be at almost exactly 74% specifically which makes me suspicious of the benchmark. I want to say 5.6-sol, Astra, deepseek flash 4.1, opus 5 (and 5.5?), Gemini 3.8-flash and a few others all ended up there?
There was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.
"We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed." - https://epoch.ai/benchmarks/deepswe/review
Models like Deepseek 4.1 flash, Muse 1.3, and MiniMax 3.1 flash are certainly capable and very cost effective, but I’m not sure anyone would say they’re better than Fable 5/5.1 in capability
H3, on the other hand, is phenomenal
- executable files?
Unless they come in at half the price (which I think will be impossible) on inference ... they have wasted their training compute imho.
DHS now plays AI whack a mole, apparently.
[0] https://public-inspection.federalregister.gov/2026-21447.pdf
Reasoning traces use truncated English:
> We need provide SVG code. Need creative. Ensure valid standalone. Pelican on bicycle, perhaps detailed vector. Need answer likely fenced svg. Could include title desc. Need be reasonably concise but image quality.
Ask HN: Any guesses on which model/provider is Space Bunny Alpha on OpenRouter? - https://news.ycombinator.com/item?id=49898971 - Sept 2026 (1 comment)
2. Add "Flash" to the end
3. ?
4. Profit
https://news.ycombinator.com/item?id=50151561
https://news.ycombinator.com/item?id=50151576
https://news.ycombinator.com/item?id=50153196
https://news.ycombinator.com/item?id=50153612
https://news.ycombinator.com/item?id=50153725
The solution is for @dang to unshadowban it because Minimax is now a newsworthy company.
:/
Then maybe they can be merged once one gains a comment?