These are going to be so awesome to buy in 5-8 years!
Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?
Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.
It works just fine. I've been using it for over three months myself once I tuned my MCP server. I run a 25-30 agent fleet with zero issues, I control it with Codex or Claude Code.
This isn't something you'd be able to manage with a normal UI since every agent has its own at of settings. My agents come in sets, I usually change a set or multiple sets, 10-15 settings per agent. And my harness hasn't made a single mistake once I worked out all the kinks from MCP.
Lol. I’m totally with you on agentic UX development. It’s frequently terrible, and you can tell which companies are telling everyone to 10x with AI. But the problem is testing, not the coding. My first stab at a UX render usually sucks too, it’s the iteration that brings perfection. It’s the same for agentic development.
What I meant though wasn’t agents building UX. I meant that in our case at least, the agent is the UX. “put a new extension 117 for Sam at the port next to Jane’s, and have it ring through to his cell when he’s not in the office. Leave the old 117 as is but only ring where Sam is, if that is known” basically, agents just do everything with SSH, and you can talk to them instead of poking around with a mouse. We don’t build ux for internal systems anymore. Local inference is strong enough to be reliable now with the correct tools and policies, and for 10-20k in hardware you are self contained. (Obviously $more if you are going to have a lot of people using it at the same time)
The “one wierd trick” is to build testing jugs for everything. Before a configuration is deployed, it’s tested against a toy instance, all existing functions are tested in all common cases, new functions verified, inputs fuzzed, adversarily abused, and stress tested against dummy loads. If its code, it gets examined against development policy, self containment, explicit interface walls, race conditions, code standards, complexity, factorability, and often A/B testing of different architectural choices. Except cryptography and a couple of other things, we generally prohibit the use of any libraries, it’s just not needed and the code never passes our tests anyway.
People are just YOLOing it and assuming that if it basically functions and looks right then it must be perfect. You’d have the same luck if you told your UX guy “test only the obvious, and if it looks about right at first glance, merge it. Don’t do extensive testing.”
The real thing with AI is that now you can afford to do a ridiculous amount of automated testing, setting up test fixtures, hardware simulators, mannequins of all the external interfaces, fuzzing, etc. Our testing code is 2-3x our product code, and we are shipping 10% of the bugs we shipped with humans, and it’s still 1/4 the cost and 4-10x the speed.
People shipping all the crap that is everywhere now have bad process engineering. Plain and simple. If you try to build software with AI like you used to with people, you’re going to have a bad time. The pipeline is very different.
> My first stab at a UX render usually sucks too, it’s the iteration that brings perfection
With AI, your iteration cycle gets 10x/100x quicker. For the same developer effort, you can effectively try out 10s of ideas the same period of time.
Iteration also has a human component. When a person works hard and writes the code, they are personally invested in its success. So iteration, or, chucking out someone’s hard work is psychologically damaging and causes human conflicts in teams. If AI wrote it, and AI deleted it, no-one gets upset. All that matters are good ideas. Reducing the human cost of iteration is just as important as the time saving.
And, now you have freed up implementation time, you now have way more time for testing, that thing that everyone wished they had more time to do, with a super intelligent 100x developer (ai) as your coding partner, and a 100x lever for automating regression testing. Bad UX has zero excuses.
kudos. Precisely this. People are doing agentic design backwards very often: YOLO then cry. But, if first you write skills, specs, set up orchestration, architecture and flows, including review layer, discrimination layer and trajectory management layer, then you task the agents with creation of verifiable testing and static analysis rig, based on the golden samples of high-quality manually engineered code, and only then you tell then to write conformant code.
Agents (and LLMs) are inherently non-deterministic. In coding practice, this translates to garbage code in the long run. The solution is simple - make the process as deterministic, verifiable and rigid as possible. Suddenly, you get all the bonuses of agentic flows (insane initial speed, trivial cost, express iterations etc.) with none of the usual problems... and suddenly, humans have hard time matching the quality of the final artifact with hand-written code, because we suck at testing, fuzzing and debugging - because we're basically single-threaded and overfocused. Agents, OTOH, excel at it, and context management is also trivial compared to humans.
The closest I can think of is AMDs MI350P it has 144gbs of hbm3.
The press says amds price is 30-40 thousand dollars. On eBay there is exactly 1 going for close to 90 which is crazy because for that amount you could buy a MI350X and all the related hardware to actually use it and it has double the vram.
So I would peg it at 40-80 thousand dollars given it's Intel, and it's lpddr5 not hbm and not gddr7 so performance is likely to be lackluster.
> (...) it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit.
This is exactly why I interpret the aggressive price cutting by OpenAI on their Luna model line as a clear sign they are scared shitless of local models, and how they can trigger a "the king has no clothes" moment for AI companies like OpenAI.
> as a clear sign they are scared shitless of local models,
The number of people with hardware to run a model like that is very small.
It's going to be many years before hardware to run medium size local models is widely available. Running mid-size local models doesn't even register as a factor in pricing decisions right now.
It's such a small part of the market and will be for many years, until RAM and other hardware prices come down.
Ram prices suggest there is going to be a mighty glut of the stuff in a couple of years. We may easily see 32->64->128 GB system RAM off the back of that and CPUs seem to be moving to unifying VRAM and main memory.
I don't think running a ~200B parameter model is that out of reach, it might be usual by the mid 2030s for people in the tech world. Factor in the crazy improvements in quality given a number of parameters too.
but the number of small to medium sized companies which really want to run (some of) the AI usage under their control and are willing to drop some 100ks on it aren't that few (especial for non US companies given the state the US is in)
> The number of people with hardware to run a model like that is very small.
True, but the number of people who pay the non-subsidized API price is also very small, and it's all in big companies who would have no issue buying or renting hardware for that purpose.
Idk, I’m running 180b on $2500 of hardware at 120 T/S (single user of course) but I bought it before the clown show. Now it’s probably more like $10k lol
I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.
Many years is only long in relation to the AI hype cycle. In terms of a human lifespan it is not that much, maybe 10 years at most and that's pessimistic. We technically live in that future already it is just very unevenly distributed.
> The number of people with hardware to run a model like that is very small.
That's not the problem anyone is facing, and analysis anyone is doing. At all.
The problem is the huge AI-related budget that companies who dove head first onto AI are facing.
Therefore the problem everyone is faced with boils down to whether they keep paying the likes of OpenAI a de-facto business tax, or they replace that tax with a one-off CapEx invoice followed by a utilities bill.
And you don't need a frontier model to do most things. In fact, the biggest challenge in using local models in agentic coding is the models supporting these agent-based tools.
> It's going to be many years before hardware to run medium size local models is widely available.
Nonsense. Speaking for myself, my employer already introduced a hard cap on AI credits while rolling out hardware support for local models, and even went to the extents of rolling out workstations with 64GB of RAM as a baseline. Consequently, we are using the likes of gemma4 for a fair share of everyday tasks. This has a transformative effect on your workflows because suddenly you don't ration tokens, and agent orchestration doesn't become a cash guzzler. So far the only constraints are tool support and token throughput, but that's not a concern.
I wonder how this compares to a hard(-ish) cap on Sonnet-and-better models, and unlimited access to Haiku. That’s one of the policies we’re considering.
The average person may not even own a personal computer, has a company-owned notebook with barely 16 GB of RAM and maybe a TPU, and here you are saying that owning a prosumer GPU and more RAM than what most people will buy across their entire lifetimes is common.
I'd have to spend what, 5-10k today to run flash models? It can't be justified as long as I have Luna on 20$ sub. I'd have to be doing so much heavy unsupervised work for the economy to make sense.
The number of people who have 192GB RAM at their disposal is minuscule because there was no reason to have that much just recently unless:
1. You're a professional in some compute heavy field AND
2. You don't use a cloud or in-house enterprise infrastructure
> as a clear sign they are scared shitless of local models
Partially agree and partially disagree. Qwen 3.8 next (120b + 50b ngrams) is at the upper end of a local model, and serving it at good speeds (esp for pp) is not cheap. So no, oAI are not particularly scared that everyone will suddenly start using local models of this size / quality.
I do agree that having these models in the open is great pressure on the price point of "small" API models (like luna, haiku, etc). And that's good news for us consumers and for the field in general - more competition is always better.
And on the cost estimate, the true power of local models is the guarantee of running them on prem w/o any data going out. Price wise it is still cheaper to go API, for small teams.
The "upper end" of local models as of today would be something like MiMo 2.6 Pro or Kimi K3, either of which is significantly smarter than Qwen Flash-Next. Roughly comparable in smarts to the old GPT Terra series, not quite as capable as Sonnet. But yes, running these models locally at realtime-usable speeds and without heavy quantization would be even harder. They can be quite tolerable for plain Q&A chat and great for running multiple unattended workloads on a 24/7 basis, but that may be a bit of a niche.
Yeah, I should have worded that better. I'd say it's at the upper level of what can be ran on plain consumer hw (like a laptop, mac, etc). For ~k3 you'd realistically need dedicated hw anyways. Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Kimi K3 has been demonstrated on a top-of-the-line laptop Mac running at several seconds per token (it has to fetch essentially all experts from SSD, since IIRC the dense part of the model takes around ~105GB on its own!). That's impressive enough if you only care about pure Q&A/chat or pure unattended tasks, but it's not what one would expect for agentic work. You'd need quite a bit of additional RAM to have good caching for the most common experts, and even then that would only help so much. MiMo Pro would be a roughly similar story.
> Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Prefill actually ought to work well on a cluster AIUI, since it can leverage both pipeline parallelism and batching quite effectively.
A dual 3090 system shouldn't go above 600W wall power while running inference, even less with some weights in RAM as you need to do with "only" 48GB VRAM and this model. Unless you are running water cooling, the cards are likely sitting at 200-250W power limit each anyway so they don't run too hot. This is considering a consumer tier CPU and mainboard with about 90W TDP, not a dual CPU or something like a threadripper that can easily push the power draw to 1kw if it runs the calculations on the RAM weights. My dual 3090 setup usually drew about 200W-300W average with LLM workloads according to my HA historical data.
I'm working off a 128gb Strix system on mostly hydro and it's worth it for me. Qwen 3.8 27B and Flash Next, Gemma 4 and MOE models. It Keeps my thinking and methods local while moving at a speed I can actually read and think about.
It can be annoying when you're running something long and realize there is an issue. But hey, I'm not wasting paid tokens. Reset and try again. I've probably thought of a better way by then too.
Yes the cells are pretty much the same as they would be with hbm as well all the magic happens at the bus and the signaling, there is both a reason hbm and gddr have higher bandwidths and tradeoffs.
DDR has better latency hence the preference in CPU workloads where lower latency is more important due to variable workloads.
That's only partially true - the GDDRX you'd find on say a 3090 is physically very different than what you'd find on a strix halo or mac or vanilla ram. e.g. runs 4 voltage levels not 2 so that it can transmit twice the info per cycle
Well, what they don't tell you about GDDR6X is that it's 4 voltage levels but it never goes from min to max or vice versa in a single transition (PAM4 MTA). It transfers 1.75 bits per cycle.
And then GDDR7 backed off to PAM3 and sends 1.5 bits per cycle.
There's a good chance [LP]DDR will use the same techniques once it gets into the same speed range.
The large amount of memory means you can serve larger models, or less quantized, but that also means they're going to be slower due to the low bandwidth.
Yeah but I think this accelerator is more meant for a small scale deployment. This could probably serve a medium moe llm model at reasonable tk/s plus you could in addition to the llm also run a couple of other models on it in parallel like an embedding, reranker, STT, TTS, decision, image, video, … if you deploy this for a small business they might make use of all of these models but not all at the same time and not with 100s of users but this card could bring a serious broad range of AI capabilities for them.
Its bandwidth is around that of 32 DDR5 DIMMS (50ish GB/s). These boards should have built in NICs but they don't because Intel also wants to sell more servers.
The marketing department doesn't have to lift a finger in this market. They could quietly drop it for sale without a single announcement and it would be backordered for a year.
Xeon Phi's ranged between ~100-350 GB/s depending on the models.
The thing to note here isn't supposed to be that 1.5 TB/s is record breaking/astounding though, even the consumer 5090 GPU has higher bandwidth than that, it's just capacity@bandwidth is what defines the usefulness of an AI accelerator and this is good on both sides instead of just one. I.e. there are a lot of "cheap" low-capacity@high-bandwidth devices and a lot of "cheap" high-capacity@low-bandwidth devices. What makes this accelerator interesting is not that it has 480 GB of LPDDR5X, it's that it has 480 GB of LPDDR5X at a speed worth having that much for, which has normally been the realm of the likes of Nvidia/AMD and their AI/DC options.
1.5 TB/s 1.3 PFLOPS with no fast multi-GPU, it's a consumer card that pretends to be a server card.
It would work great on Qwen3.8-Flash-Next or GLM-5.3 as single user decode only hits a few experts, anything with concurrent users and you'd be sitting for a while.
That's the point I believe fast local 2-4 user service for cheap on very large models.
You have to pretend it's for servers to keep investors happy, it would be banks and similar with in house servers that have to run local large models buying these for experiments and cybersecurity. Likely.
Reminds me of the early 2000s era where we'd fight in my household if too many people used the internet at the same time, can see that happening here if it's not 2 to 4 users but 10.
Like cool hardware but in the current situation also quite insulting - totally unaffordable for an individual & one more thing to gobble up useful memory + make the computing hardware affordability crisis worse.
Slop and placeholder text on the page, including this one. They should have reviewed their webpage.
"What software frameworks are relevant to Intel GPU D270?
Intel GPU D270 messaging should reference familiar AI software and serving frameworks such as PyTorch, vLLM, SGLang, and OpenAI Triton, where approved. This helps developers and platform teams understand how the product fits into existing AI workflows."
You can tell by the shape of the board that it's meant to go into a server chassis behind a stack of 80mm blower fans. It's not meant for consumers, but you will be able to buy it used for $200 off ebay in 7 years to put in your home lab.
Wonder if there will be consumer video drivers for these GPUs once the AI hardware shortage is over. Hope to buy a gaming rig one day that doesn't cost a fortune.
I remember back in the day of AGP cards around 2002, you could use a 3d card on linux to render the grafix but use a different GPU for the actual output. I used XFree86 on linux with a voodoo banshee (or maybe was a riva tnt2) and a matrox mga200 for its sync-on-green support with my monitor to play games.
I know modern card architecture doesn't support this out of the box, but what is the actuall problem of forwarding a once-rendered frame to another card for output? Sureley DMA transfers PCIe->RAM->PCIe nowaday are fast enough to saturate 60fps at 4K?
Trivial to do this on Linux. I run my desktop on the on-chip Ryzen 9950X iGPU and I offload games to my nVidia dGPU.
I had a crazier setup in the past with the Intel i7-8809G APU: it had an AMD iGPU for desktop connected to the display, nVidia eGPU for games and the Intel iGPU for video encoding.
Of course there is additional latency due to frame copying, but I can’t notice it in practice.
I had that setup working back in the day with a Voodoo1 which had the pass-trough construction to take over full screen: only on Linux you could read the completed frames and render them in an X window. It was dog slow though.
While transfers have improved a lot the amount data needed for 4K60 with HDR is about 18Gbps. There's no way that doesn't have an impact.
We cooked up a setup where we had a PCI video card for X, and a separate Voodoo 1 or 2 and monitor for 3D (Mesa) rendering. I don't remember the details any more, but it was nice to not take out the desktop when someone borked up their OGL homework.
the voodoo1 worked differently as you looped through the video input. The voodoo banshee i mentioned was a separate 2d/3d card (it actually was 3DFx first true, combined 2d/3d card - not an accelerator).
I know. Voodoo 1 and 2 had the ugly VGA passthrough hack.
But the principle of reading VRAM from one card and copying it to another would remain the same I think.
in latest gamersnexus video https://www.youtube.com/watch?v=e4c2FSSaCsw Wendell from Level1 Techs was showing some older Amazon dedicated server GPUs with no video output that got hacked community driver making it work like laptop dgpu copying rendered video across PCIe.
As has been claimed every year for the last 30, at least. Much like predicting the end of the world, one day, if you keep at it, you'll actually be right.
Intel ai gpu division is so aggressively anti “small” consumer. I think it has something to do with low volume and a small sales/distribution network. The data center cards I have from amd and nvidia are grey market at best. I don’t think intel moved enough units to establish alternate channels.
What I don't understand is: when you only plan for your product to have 10 customers, why even make a marketing page like this for it? At that point it's essentially the hardware equivalent of consultingware, something you sell to your existing contacts through outbound calls and by shoving ads through the B2B equivalent of a post-consumer sales channel (i.e. proactive outreach by your org's internal champion for that customer, to said customer.)
Given the rumored numbers are all true, this could be a huge contender to M5 Ultra(or M7 Ultra if we consider possible delays in the schedule?). Low-batch single-user serving is usually bottlenecked by `E(sum over (memory bandwidth by tier) * (memory size by tier))`.
Looks like Intel is choosing to title its new crescent island cards with the D prefix despite being Crescent Island.
Kind of disappointed that Intel is (probably) not going to release consumer class GPUs. Intel's driver stack was seriously getting mature—both on Linux and Windows. It's always nice to have a third option available.
"Both" requires more engineering staff on payroll, more wafers to fab/dice/package/test, more boards to manufacture, more inventory to manage, among other things. If their math says it isn't worth it (granted that they don't always get it right), then they don't do it.
> "Both" requires more engineering staff on payroll, more wafers to fab/dice/package/test, more boards to manufacture, more inventory to manage, among other things.
Aren't you describing how a business operates?
I mean, doesn't Intel already hire people for those tasks? Unless you are worried about a specific bottleneck, increasing headcount and production output is what businesses do.
Agreed. I was really rooting for them to deliver that third iteration to consumers, and retain them as a competitive option in the low to mid range. Pretty sad to see.
For now. A limited run of Celestials until 2028 when Serpent Lake switches to Nvidia tiles, and a bit longer for the LTS combo (AMD style) IO+GPU die in lower end/embedded products.
Druid is still dead, consumer DGPUs are still, and workstation DGPUs also seems to be dead, and most of the Xe driver team for Windows was laid off, as well as most of the Xe chip design team.
hmm...wonder where this will land in market segments.
Seems like neither a consumer product nor server. Maybe if they halve the ram and price it reasonably people with desktops will buy it for MoE like prosumer homelab AI enthusiast type situations?
The cost of a 480gb ram video card seems preposterous to be even marketing at this point.
Intel is going to make a card that costs $1 million dollars.. this is like industrial newsletters bleeding into a public forum
> Commercial Scale Pricing: In the industrial data center sector, battery storage systems are priced per kilowatt-hour (kWh) or megawatt-hour (MWh). Industrial battery storage installations typically cost $300,000 to $450,000+ per MWh of capacity depending on software integration, cooling systems, and structural framing.
They originally said it was going to be 120GB but that partners could ship bigger capacities. So much for that!
I do hope this has a price point that makes it somewhat competitive with a Mac! Consumers have had nearly no choices available for mem capacity, and 20 channels of lpddr5 is one way to get that capacity! Go Intel!
Both this GPU and the Mac are constrained by a lack of GPU compute. There's basically no rack-scale GPUs with 480gb on-die that have a sub-400W compute element. It's going to be incredibly prefill-constrained on denser model inference just like the Apple Silicon iGPUs are.
> But I am surprised that they were able to pack so much flops into 300W.
I'm not. When you're doing inference you typically reduce power limits anyway if you're optimizing for efficiency. Even if you don't, it doesn't spike to the max of the card.
My 3090 I power it down to 175W or so for single-digit reductions in performance for like, 60% of the wattage, something like that.
This is a brand new class of GPU that I am excited for, especially because it's rated for air cooling. I hope to see other manufacturers release GPU's with multiple of 100 GB's that can be utilized even in an at-home system's with PCIe 5 support. I probably can't afford this but knowing the engineering is sound at this scale means a similar accelerator maybe attainable.
> Efficient token serving
Dense, air-cooled, standard PCIe infrastructure helps maximize tokens per rack and watt, scaling serving capacity and keeping costs in check.
Next the tokens will be asking for tequila and caviar. Bow down to the new token overloads.
All these big names in tech will monetize so hard on such products, until this becomes obsolete and available to us all. This should be accessible to everyone at a convenient price, governments should force these companies to mass produce them and sell to people, providing a small subsidy. Everyone deserves to run big models locally.
Right now I’m running 2x 3090 and 128 gb ram on 3.8 flash-next180b (the quant is 97gb) and it hums at120t/s. It’s really a surprisingly good model for general tasks, and I’m soon to see how it does with more agentic kit. For agentic work it does slow down quite a bit when the context fills up… but what do you want for running a 180b model on consumer hardware?
Right now I have it set up as an administrator for our asterisk PBX and radius server, and it does higher level judgment on our cctv footage for notifications to security. It’s been really good so far. Agents are the new UI, everything just works over SSH now.
Poorly-defined and ever-changing with occasional cases where a button does nothing, does something else, or deletes the whole prod database?
Idk. For a prototype they might be, but the whole reason we have this software thing going on is because we want some level of determinism.
This isn't something you'd be able to manage with a normal UI since every agent has its own at of settings. My agents come in sets, I usually change a set or multiple sets, 10-15 settings per agent. And my harness hasn't made a single mistake once I worked out all the kinks from MCP.
What I meant though wasn’t agents building UX. I meant that in our case at least, the agent is the UX. “put a new extension 117 for Sam at the port next to Jane’s, and have it ring through to his cell when he’s not in the office. Leave the old 117 as is but only ring where Sam is, if that is known” basically, agents just do everything with SSH, and you can talk to them instead of poking around with a mouse. We don’t build ux for internal systems anymore. Local inference is strong enough to be reliable now with the correct tools and policies, and for 10-20k in hardware you are self contained. (Obviously $more if you are going to have a lot of people using it at the same time)
The “one wierd trick” is to build testing jugs for everything. Before a configuration is deployed, it’s tested against a toy instance, all existing functions are tested in all common cases, new functions verified, inputs fuzzed, adversarily abused, and stress tested against dummy loads. If its code, it gets examined against development policy, self containment, explicit interface walls, race conditions, code standards, complexity, factorability, and often A/B testing of different architectural choices. Except cryptography and a couple of other things, we generally prohibit the use of any libraries, it’s just not needed and the code never passes our tests anyway.
People are just YOLOing it and assuming that if it basically functions and looks right then it must be perfect. You’d have the same luck if you told your UX guy “test only the obvious, and if it looks about right at first glance, merge it. Don’t do extensive testing.”
The real thing with AI is that now you can afford to do a ridiculous amount of automated testing, setting up test fixtures, hardware simulators, mannequins of all the external interfaces, fuzzing, etc. Our testing code is 2-3x our product code, and we are shipping 10% of the bugs we shipped with humans, and it’s still 1/4 the cost and 4-10x the speed.
People shipping all the crap that is everywhere now have bad process engineering. Plain and simple. If you try to build software with AI like you used to with people, you’re going to have a bad time. The pipeline is very different.
With AI, your iteration cycle gets 10x/100x quicker. For the same developer effort, you can effectively try out 10s of ideas the same period of time.
Iteration also has a human component. When a person works hard and writes the code, they are personally invested in its success. So iteration, or, chucking out someone’s hard work is psychologically damaging and causes human conflicts in teams. If AI wrote it, and AI deleted it, no-one gets upset. All that matters are good ideas. Reducing the human cost of iteration is just as important as the time saving.
And, now you have freed up implementation time, you now have way more time for testing, that thing that everyone wished they had more time to do, with a super intelligent 100x developer (ai) as your coding partner, and a 100x lever for automating regression testing. Bad UX has zero excuses.
Agents (and LLMs) are inherently non-deterministic. In coding practice, this translates to garbage code in the long run. The solution is simple - make the process as deterministic, verifiable and rigid as possible. Suddenly, you get all the bonuses of agentic flows (insane initial speed, trivial cost, express iterations etc.) with none of the usual problems... and suddenly, humans have hard time matching the quality of the final artifact with hand-written code, because we suck at testing, fuzzing and debugging - because we're basically single-threaded and overfocused. Agents, OTOH, excel at it, and context management is also trivial compared to humans.
But anyway.
I'm sure there are people that use the tech wrong. That can be said about any tech in existence.
If we aren't in that half, it's by sheer luck, and nothing else.
Can't wait. Anyone has a idea of what the price now would be?
The press says amds price is 30-40 thousand dollars. On eBay there is exactly 1 going for close to 90 which is crazy because for that amount you could buy a MI350X and all the related hardware to actually use it and it has double the vram.
So I would peg it at 40-80 thousand dollars given it's Intel, and it's lpddr5 not hbm and not gddr7 so performance is likely to be lackluster.
Funny as I'm using LLMs now to write UIs.
This is exactly why I interpret the aggressive price cutting by OpenAI on their Luna model line as a clear sign they are scared shitless of local models, and how they can trigger a "the king has no clothes" moment for AI companies like OpenAI.
The number of people with hardware to run a model like that is very small.
It's going to be many years before hardware to run medium size local models is widely available. Running mid-size local models doesn't even register as a factor in pricing decisions right now.
It's such a small part of the market and will be for many years, until RAM and other hardware prices come down.
I don't think running a ~200B parameter model is that out of reach, it might be usual by the mid 2030s for people in the tech world. Factor in the crazy improvements in quality given a number of parameters too.
True, but the number of people who pay the non-subsidized API price is also very small, and it's all in big companies who would have no issue buying or renting hardware for that purpose.
But there are a lot of companies who do and rent it out as a service.
I expect I’ll be able to run 1/2 terabyte models in 5 years for a couple $K in hardware. Inference is going to be a lot like Bitcoin mining. Only the peak of the power/performance curve is going to be profitable. Which means a glut of surplus hardware that can’t make money but makes tons of tokens.
That's not the problem anyone is facing, and analysis anyone is doing. At all.
The problem is the huge AI-related budget that companies who dove head first onto AI are facing.
Therefore the problem everyone is faced with boils down to whether they keep paying the likes of OpenAI a de-facto business tax, or they replace that tax with a one-off CapEx invoice followed by a utilities bill.
And you don't need a frontier model to do most things. In fact, the biggest challenge in using local models in agentic coding is the models supporting these agent-based tools.
> It's going to be many years before hardware to run medium size local models is widely available.
Nonsense. Speaking for myself, my employer already introduced a hard cap on AI credits while rolling out hardware support for local models, and even went to the extents of rolling out workstations with 64GB of RAM as a baseline. Consequently, we are using the likes of gemma4 for a fair share of everyday tasks. This has a transformative effect on your workflows because suddenly you don't ration tokens, and agent orchestration doesn't become a cash guzzler. So far the only constraints are tool support and token throughput, but that's not a concern.
Mind you, some local models run on 16GB of RAM, with some headroom to do development work.
I'm talking small and medium businesses, who have productive gains using AI, where a 3k€ PC is common. We have many.
I'd have to spend what, 5-10k today to run flash models? It can't be justified as long as I have Luna on 20$ sub. I'd have to be doing so much heavy unsupervised work for the economy to make sense.
The number of people who have 192GB RAM at their disposal is minuscule because there was no reason to have that much just recently unless: 1. You're a professional in some compute heavy field AND 2. You don't use a cloud or in-house enterprise infrastructure
500€ for a production tool seems fairly innocuous. We're not in the 10k€+ workstation territory here. At least we were not.
Partially agree and partially disagree. Qwen 3.8 next (120b + 50b ngrams) is at the upper end of a local model, and serving it at good speeds (esp for pp) is not cheap. So no, oAI are not particularly scared that everyone will suddenly start using local models of this size / quality.
I do agree that having these models in the open is great pressure on the price point of "small" API models (like luna, haiku, etc). And that's good news for us consumers and for the field in general - more competition is always better.
And on the cost estimate, the true power of local models is the guarantee of running them on prem w/o any data going out. Price wise it is still cheaper to go API, for small teams.
> Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Prefill actually ought to work well on a cluster AIUI, since it can leverage both pipeline parallelism and batching quite effectively.
Your electrical costs are likely less than mine though
It can be annoying when you're running something long and realize there is an issue. But hey, I'm not wasting paid tokens. Reset and try again. I've probably thought of a better way by then too.
Literally my first thought!
Marketing department asleep at the wheel.
A single B300 has almost 8 TB/s.
AMD MI300x from 2023 is 5.3 TB/s
Edit: my bad you are saying that this is an impressive number precisely because it is more or less normal RAM, not VRAM
DDR has better latency hence the preference in CPU workloads where lower latency is more important due to variable workloads.
And then GDDR7 backed off to PAM3 and sends 1.5 bits per cycle.
There's a good chance [LP]DDR will use the same techniques once it gets into the same speed range.
So yeah on its own not very impressive.
The large amount of memory means you can serve larger models, or less quantized, but that also means they're going to be slower due to the low bandwidth.
https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216
The marketing department doesn't have to lift a finger in this market. They could quietly drop it for sale without a single announcement and it would be backordered for a year.
The thing to note here isn't supposed to be that 1.5 TB/s is record breaking/astounding though, even the consumer 5090 GPU has higher bandwidth than that, it's just capacity@bandwidth is what defines the usefulness of an AI accelerator and this is good on both sides instead of just one. I.e. there are a lot of "cheap" low-capacity@high-bandwidth devices and a lot of "cheap" high-capacity@low-bandwidth devices. What makes this accelerator interesting is not that it has 480 GB of LPDDR5X, it's that it has 480 GB of LPDDR5X at a speed worth having that much for, which has normally been the realm of the likes of Nvidia/AMD and their AI/DC options.
They tasked an AI agent to it, like everyone else.
Not targeting me
It would work great on Qwen3.8-Flash-Next or GLM-5.3 as single user decode only hits a few experts, anything with concurrent users and you'd be sitting for a while.
You have to pretend it's for servers to keep investors happy, it would be banks and similar with in house servers that have to run local large models buying these for experiments and cybersecurity. Likely.
Interesting choice of words
British reading -> sweating a lot
American reading -> sweating ejaculate
Or possibly something to do with how running work on a model is referred to as a load?
Fuck knows!
"What software frameworks are relevant to Intel GPU D270?
Intel GPU D270 messaging should reference familiar AI software and serving frameworks such as PyTorch, vLLM, SGLang, and OpenAI Triton, where approved. This helps developers and platform teams understand how the product fits into existing AI workflows."
I know modern card architecture doesn't support this out of the box, but what is the actuall problem of forwarding a once-rendered frame to another card for output? Sureley DMA transfers PCIe->RAM->PCIe nowaday are fast enough to saturate 60fps at 4K?
I had a crazier setup in the past with the Intel i7-8809G APU: it had an AMD iGPU for desktop connected to the display, nVidia eGPU for games and the Intel iGPU for video encoding.
Of course there is additional latency due to frame copying, but I can’t notice it in practice.
While transfers have improved a lot the amount data needed for 4K60 with HDR is about 18Gbps. There's no way that doesn't have an impact.
> We want to render applications on the more powerful card and send the result to the card which has display connected.
No. Certainly not now, maybe not ever. That is the least consumer-focused product page I've seen today.
You can work with 10 players that will buy all of them with minimal effort.
edit: typo
Kind of disappointed that Intel is (probably) not going to release consumer class GPUs. Intel's driver stack was seriously getting mature—both on Linux and Windows. It's always nice to have a third option available.
Theres so much money to be made
You can make a nice profit off of the enterprise space, but the big winners are the guys who put the work into servicing the consumer market.
Aren't you describing how a business operates?
I mean, doesn't Intel already hire people for those tasks? Unless you are worried about a specific bottleneck, increasing headcount and production output is what businesses do.
Druid is still dead, consumer DGPUs are still, and workstation DGPUs also seems to be dead, and most of the Xe driver team for Windows was laid off, as well as most of the Xe chip design team.
Seems like neither a consumer product nor server. Maybe if they halve the ram and price it reasonably people with desktops will buy it for MoE like prosumer homelab AI enthusiast type situations?
The cost of a 480gb ram video card seems preposterous to be even marketing at this point.
Intel is going to make a card that costs $1 million dollars.. this is like industrial newsletters bleeding into a public forum
> Commercial Scale Pricing: In the industrial data center sector, battery storage systems are priced per kilowatt-hour (kWh) or megawatt-hour (MWh). Industrial battery storage installations typically cost $300,000 to $450,000+ per MWh of capacity depending on software integration, cooling systems, and structural framing.
Let's all get excited
I do hope this has a price point that makes it somewhat competitive with a Mac! Consumers have had nearly no choices available for mem capacity, and 20 channels of lpddr5 is one way to get that capacity! Go Intel!
This is almost twice better than RTX 5090. This GPU will not be prefill constrained, I feel.
But I am surprised that they were able to pack so much flops into 300W.
I'm not. When you're doing inference you typically reduce power limits anyway if you're optimizing for efficiency. Even if you don't, it doesn't spike to the max of the card.
My 3090 I power it down to 175W or so for single-digit reductions in performance for like, 60% of the wattage, something like that.
Next the tokens will be asking for tequila and caviar. Bow down to the new token overloads.