This is cool! Unfortunately a side effect is going to be more flashy vibe coded product landing pages adding expensive shader effects to all of their HTML. If you thought scrolljacking was bad, we're going to be seeing some interesting hover / mouse / navigation effects.
This is the best news ever! I can't believe it was implemented in just one year after that blogpost... Boy I'm going to be spending good portions of my life making with this API...
I wanted to implement a whole canvas native workspace, I started with a html only implementation and "fake canvas" approach but even for moderate number of panels (~50) the responsiveness dropped quite a lot. So I went with a new implementation with "real canvas" and worked out the panels/interaction.
In terms of performance this approach worked out much better (able to easily handle 1000+ panels/nested panels at 60+ fps) but the complexity goes very high in the implementation of each panel. As in I cant just use for eg a markdown rendering js library or any third party ui component. And frequently make a change would result in massive performance degradation.
Currently I have text/markdown/image/audio/video and pdf panel (you can drag and drop files)
I wonder if i should take another go at it and implement using html-in-canvas api. Afraid of the api being not maintained if they dont see benefit or usage?
Hmm, not immediately clear how this avoids potential privacy leaks, such as drawing DOM hyperlinks to the canvas and then scraping their visited-status colour from the pixel data.
Or maybe I'm behind the times; I've not done serious webdev for a long time.
Pretty sweet! I'm building out an AI powered motion graphics & video generator and the biggest limitation is not having HTML and Canvas play well. Some visual effects, while cool looking, really impact performance so I have to compromise. I genuinely think this is the last thing we need before we have After Effects level capabilities for AI motion graphics on the web.
Imagine being able to declare a list of paths with fill/stroke-paints to draw, filled in from attributes, variables and `calc()` values... one can only dream.
Since then I've actually met these two and had to conclude that:
- They didn't have any use-cases in mind that weren't cosmetic
- They don't quite understand how being part of an advertising company shapes how they build and conceive of web tech
- They react quite poorly to reactions that aren't cheerleading
The answer for curved elements was handwaving and "just approximate it with flat projective transforms" which tells you this is not a serious attempt to expose mature UI internals, but just another shiny epicycle on the creaky leviathan that is the HTML5 web api.
The list of features that "automatically work" rests on assumptions that are only valid for minor visual adjustments. The moment you try to use HTML-in-Canvas as a more basic building block, a lot of that falls away.
e.g. Suppose I render a piece of text and then use that as a mask for visual effects. If the browser now draws a text selection highlight into that mask, the text becomes unreadable.
I've read your blog enough times to recognize it instantly, yet it's the first time I notice that you blog header is an animated and interactive 3D demo. Very cool!
I would use it to render HTML elements such as inputs or a website in videos. Even though HTML is powerful the control over every pixel is important for high quality videos.
Back in the WebVR days the sentiment was we can never do this because security.
In my experience with standards bodies/Chrome, anything outside of the owning company's product goals is "no, because security", and then then they flip-flop on that when the browser feature becomes a dependency for something an internal team is working on (which happened many times with WebVR, now WebXR).
So I'm curious what is Google working on that has this as a dependency, now that they've 180'd on it?
This had to do with leaking privacy sensitive information such as visited links by reading back pixels from a rendered <a>. They fixed this with a concept they call "read-back-allowed rendering"
"Security" is usually not a binary, but how much money you are willing to spend to figure out a way to do it securely. It shouldn't be surprising that chrome spends more resources on things that have a business case than things that do not.
The thing about HTML in Canvas is that canvas should always have had the ability to add html to its content, and there would be no debate about whether it's a good idea or not.
Honestly, I think the push to constantly add browser features (and thereby expand the already massive web API) is misguided. This feature, in particular, seems like it should be solved with library, rather than by further bloating the browser.
If I put on my conspiracy hat, it looks like Google is pushing to expand the web API as a means to maintain its lead in the browser market by making it harder for other browsers (i.e. Firefox) to maintain feature parity.
TBH, the first time I read about this, I had to check the date to make sure it wasn't an April Fool's joke I had missed. I'm sure there are some things this would be useful for, but the sheer amount of complexity it would add to web renderers doesn't seem commensurate with how useful it would be.
This kind of stuff just puts even more distance between established engines and smaller projects like Servo.
> the sheer amount of complexity it would add to web renderers doesn't seem commensurate with how useful it would be.
Why? The browser is effectively already rendering HTML in a “canvas”, via tiled rendering etc. This isn’t a whole new rendering path, just a redirection of the existing one.
I have the same theory but for different reasons(commented something similar to a recent css functions thread), in this case though I don't thing you can solve it with a library, the canvas api is very limited, text layouting is very hard especially if you want to support different languages and directions. Let alone rendering a text input, if the use case is games, those usually don't have very complex inputs and that would be ok I think to implement a simpler one. IMO we should just have exposed the Skia text layout api in javascript.
You can't solve this with a library without essentially implementing a separate browser engine in JS (or WASM). Doing this directly in the browser engine is far simpler than that.
In terms of performance this approach worked out much better (able to easily handle 1000+ panels/nested panels at 60+ fps) but the complexity goes very high in the implementation of each panel. As in I cant just use for eg a markdown rendering js library or any third party ui component. And frequently make a change would result in massive performance degradation.
https://canaster.in/
Currently I have text/markdown/image/audio/video and pdf panel (you can drag and drop files)
I wonder if i should take another go at it and implement using html-in-canvas api. Afraid of the api being not maintained if they dont see benefit or usage?
(116 comments)
Or maybe I'm behind the times; I've not done serious webdev for a long time.
https://github.com/WICG/html-in-canvas/#read-back-allowed-re...
:D.
Related bunch of comments from that time: https://news.ycombinator.com/item?id=48201222
https://www.vibemellow.ai
Very stoked for this!
Imagine being able to declare a list of paths with fill/stroke-paints to draw, filled in from attributes, variables and `calc()` values... one can only dream.
What you described is the CSS painting api (part of CSS Houdini)
Since then I've actually met these two and had to conclude that:
- They didn't have any use-cases in mind that weren't cosmetic
- They don't quite understand how being part of an advertising company shapes how they build and conceive of web tech
- They react quite poorly to reactions that aren't cheerleading
The answer for curved elements was handwaving and "just approximate it with flat projective transforms" which tells you this is not a serious attempt to expose mature UI internals, but just another shiny epicycle on the creaky leviathan that is the HTML5 web api.
The list of features that "automatically work" rests on assumptions that are only valid for minor visual adjustments. The moment you try to use HTML-in-Canvas as a more basic building block, a lot of that falls away.
e.g. Suppose I render a piece of text and then use that as a mask for visual effects. If the browser now draws a text selection highlight into that mask, the text becomes unreadable.
I've read your blog enough times to recognize it instantly, yet it's the first time I notice that you blog header is an animated and interactive 3D demo. Very cool!
In my experience with standards bodies/Chrome, anything outside of the owning company's product goals is "no, because security", and then then they flip-flop on that when the browser feature becomes a dependency for something an internal team is working on (which happened many times with WebVR, now WebXR).
So I'm curious what is Google working on that has this as a dependency, now that they've 180'd on it?
If I put on my conspiracy hat, it looks like Google is pushing to expand the web API as a means to maintain its lead in the browser market by making it harder for other browsers (i.e. Firefox) to maintain feature parity.
This kind of stuff just puts even more distance between established engines and smaller projects like Servo.
Why? The browser is effectively already rendering HTML in a “canvas”, via tiled rendering etc. This isn’t a whole new rendering path, just a redirection of the existing one.
The browser API has become so big because the implementers are quite tenacious, with multiple parties doing “extend” in parallel…
Here's hoping for it being DOM-in-Canvas all they way down.