This project(at least for me) is a little bit misleading.
Because it isn't rendering HTML using WebGL.
It using html2canvas[0] to rasterize the browser rendering into a canvas, upload it to WebGL, apply effects and patch-back events(i.e. mouse clicks) back to the hidden browser dom elements.
(On the other side, font hinting and such are not really a problem)
Implementing full UI framework over WebGL has it merits, and hopefully someone will make a useful(and not html+css based) one some day.
>Implementing full UI framework over WebGL has it merits, and hopefully someone will make a useful(and not html+css based) one some day
I'm surprised it hasn't been done already (in similar spirit to the Feathers framework for Flash's Stage3D). Now that WebGL is supported in Mobile WebView, maybe you'll see a really good component set released? That would give devs a really slick option for building smooth phone-gap based apps.
Yet another instance of html2canvas being used to do something interesting. It's the library I have been surprised by more than any other, for any language.
Famo.us is the only JavaScript framework that includes an open source 3D layout engine fully integrated with a 3D physics animation engine that can render to DOM, Canvas, or WebGL.
Make it feature complete and you'll get the same speed the browser has.
If we're going to build a new rendering engine for the web, maybe we could try to create something nicer than HTML and CSS, whose only advantage at the present is the number of developers familiar with it. I know I wouldn't miss them.
Yep, the model of HTML-CSS-DOM-JS implies extreme overhead, and only heroic efforts and thousands of man-years of work and optimization have made it bearable.
I'm saying this as I have compared the performance of a simple local web page with a simple view with the same exact functionality rendered with my pet naive unoptimized UI engine and the latter was blazingly fast compared to HTML.
So I guess you are right, we definitely need something less cluttered and more efficient than HTML+CSS+JS.
Compare how easy it is to make web pages with just HTML+CSS+JS, especially for a beginner, with any other framework, native code or no. The separation of structured markup, display theming, and custom logic is remarkably powerful, and everyone saying otherwise is just plain wrong if you look at history.
Once you start doing anything outside of your simple page with a simple view (which was what, incidentally?), your engine starts to accrue features that move it in "bloat" towards a modern web browser.
The link mentions "Hey, this is really useful on mobile"--and I would consider that a failing of the mobile vendors browser teams, not a problem with the web.
This looks really flashy for the sort of static content + basic animations + interactions the author is showing off, but once you start handling further interaction--like, you know, building a virtual dom to render to the webgl texture, all you're doing is reinventing the facilities the browser already gives you, but instead behind one more layer of abstraction.
Well, I'm a beginner in web, and I can't say it's easy, in fact, it's quite the opposite. In native frameworks there's normally one idiomatic way to do things, while the web framework land looks terrifying.
Actually, my pet UI engine was powerful enough so that I've implemented a complete mobile application in it. I'd also tried HTML, but it just wasn't competitive. The native iOS UI was even faster and was miles ahead of HTML in terms of speed.
Also, there is such thing as Qt Quick, where you can basically write in reactive JavaScript, it is at least as powerful as HTML-CSS-JS, and it is still way faster.
The web has started from something that obviously wasn't designed for performance and has come a long way of incremental evolution. Given this, and given that I know how hard it is to make a fast UI engine as powerful as web, the current web performance is actually spectacular. But still, there are better models, and there can be even better models.
Well, I'm a beginner in web, and I can't say it's easy, in fact, it's quite the opposite. In native frameworks there's normally one idiomatic way to do things, while the web framework land looks terrifying.
So, that's part of the problem, right? For some damn fool reason we keep throwing beginners at frameworks--they should be starting with just bsaic HTML and CSS, and then learning JS, and only then moving on to frameworks. It's so damned confusing in part because people aren't learning the fundamentals, and so are lacking a mental model of how things work and what makes them slow.
I'd also tried HTML, but it just wasn't competitive. The native iOS UI was even faster and was miles ahead of HTML in terms of speed.
Show me some numbers. If you want to talk about performance, then let's be quantitative. I'm also willing to wager that the HTML on the mobile device was full of bits you didn't need.
But still, there are better models, and there can be even better models.
Again, why do you think this is the case? It's worked, with expansion, for over twenty years. That's a staying power few technological approaches have--maybe, there's something to it. This notion of "better models" really frequently boils down to "I have this one specific thing I need done, and anything that doesn't match this one specific thing is superterribad and clearly needs replacing."
If you want the best performance (which honestly probably doesn't matter that much for your use case), write things in native code as you've done. But don't then pretend that that is a valid yardstick to measure an entirely different tech by.
Absolutely this. The first sentence. Part of the problem is the vast echo chamber of Stack Overflow, where every JS question gets a jQuery/Express answer, even when the asker explicitly asks for vanilla JS help. Having hired developers in the past, this is probably my biggest pet peeve about web developer culture: Everyone seems to start from the top and only maybe work their way down through jQuery to vanilla JS. One of the questions I used to ask in interviews was to do some simple DOM manipulations without a library. This is a kind of fizzbuzz, as I'd guess probably 80% of the interviewees could not perform such basic JS functions as DOM element selection, which boggles the mind (even considering that the primary function of jQuery is its selector functionality). Certainly this experience has completely validated every claim I've ever heard about most developers not being hackers. As someone who yearns to learn in the most hardcore way (e.g. reading language references at the beach), I cannot fathom how so many people interested in writing code do not also possess the drive to tinker with (or at least wonder about) the inner workings of their tools.
Those reading specs interview those who say specs are irrelevant (same applies to basic (and advanced) math, seemingly) -- is not that satisfying, after all? ;)
For some damn fool reason we keep throwing beginners at frameworks--they should be starting with just basic HTML and CSS, and then learning JS, and only then moving on to frameworks. It's so damned confusing in part because people aren't learning the fundamentals, and so are lacking a mental model of how things work and what makes them slow.
Actually, I've been familiar with HTML+CSS+JS for a long time. The vanilla web is just not very practical for creating full-fledged web apps, especially given that there are so many frameworks that have a lot of things out of the box. Also, the vanilla web misses some features that are essential, such as the reactivity (binding).
Show me some numbers. If you want to talk about performance, then let's be quantitative. I'm also willing to wager that the HTML on the mobile device was full of bits you didn't need.
The last time I tried you wouldn't need numbers, as my pet engine did steady 60 FPS full-screen animations, while the iOS web view pathetically presented something closer to a slideshow, Android did better, but still not smooth enough. And the page loading was visible instead of instantaneous display with the native stuff. Of course, faster devices alleviate the problem.
>But still, there are better models, and there can be even better models.
Again, why do you think this is the case?
Because I've developed my own thing similar to what HTML+CSS+JS do, and I could clearly realize that you can do the same things with a much simpler model that also happens to work faster. Also, there are existing native frameworks that are examples of this.
It's quite clear that the web wasn't originally designed to be used how it's currently used and practical web apps are built with basically hacks over hacks and hack-based frameworks. Obviously, getting rid of all that clutter accumulated by incremental evolution would make a difference.
Dude, don't be so shallow. You'll be surprised, but HTML, CSS, and JS are not the web. To realize this simple fact, you can try looking up the statistics on the number of websites made in vanilla HTML/CSS/JS, or see how many there are the 'Vanilla HTML/CSS/JS developers' out there. That means, if you only know HTML, CSS, and JS, you can't make practical web apps, and you can't be a web developer either.
I am familiar with HTML, CSS, and JS, but can't make anything except the simplest static website.
It remained in the commercial mobile app I'd used it for, and it's not my intellectual property anymore. There isn't anything ingenious about it anyway.
My point wasn't about how good my pet engine was (because it wasn't), but about the fact how the overcomplicated model of the HTML-CSS-DOM-JS makes it so hard to optimize web page rendering that even an unoptimized prototype UI engine outperforms the browser.
I am amazed by the people who develop browser engines because they do something almost magic of fitting that monstrous dynamically modified DOM into so little memory and making it so fast.
I conjecture that in a naive unoptimized implementation a web browser would consume gigabytes and took a couple of minutes to load an average page.
When beginners start strutting around acting as if their "framework" or approach is better than the norm, the sceptic in me comes out to play. The likelihood of them actually having found a better way is pretty slim. Sure it happens but not everyday.
I agree that in theory, that separation is sound. But in practice, it seems no one uses it like that, and we find ourselves constantly fighting against that purity. e.g. adding extra divs just so I can center one bit of text the way I want to.
I am sorry if I sounded rude, I didn't mean to knock this particular project down.
I was commenting on a hypothetical future for something of its kind, and that I think HTML content is not really the way to go to make shiny WebGL things, IMO.
To be a little more on-topic, yes, as an "apply shader to this semi-static div" it is a pretty cool thing :)
That's the opposite direction you want to go. At that point, you might as well just use HTML. I think there are benefits to providing a subset of HTML features that you can tailor for super fast performance.
Almost any native UI framework (iOS, Android, WPF on windows, etc) has better layout options. Some of them like iOS have options that are not as beginner friendly (AutoLayout), but they are still better overall if you are trying to develop an application instead of formatting a document.
Many other UI frameworks have had Flexbox type layouts, a technology W3C has only just recently realized is a good idea for 8+ years (probably much longer).
But there are serious known bugs in IE10 and IE11 flexbox implementations. They will never be fixed, so when IE11 market share drops below 5% I will use flexbox. We're talking about perhaps in the year 2020. HTML/CSS is still not suited for app development and due to browser inconsistencies that will remain true for decades to come, perhaps forever. Apps in browsers is doable, but a more cumbersome than one might think.
Your browser is slow when applying transform to DOM elements because it has to translate everything 3D in terms of JavaScript and DOM. If you want to manipulate images and elements effectively in 3D then you have to learn matrix computations, quaternions and stuff which is horrible for the common web developer.
I'm saying that if you offer a new approach that could mix HTML/CSS and encapsulate those complex 3D concepts it would be interesting.
Wait wait wait...
Handling textures via CG Buffers is quite fast you are right and you can basically do whatever you want in terms of image manipulation using the shaders but you'll lose efficiency as soon as you introduce complex picking (text selection for example) and / or computations (aliasing etc...) between JS and your pseudo-DOM.
In the end the only difference you'll have compared to the real browser renderer is that you do less computations from CG(GL) to RAM (JS), hence the performance.
You can check this very old project of mine where we tried to link DOM and WebGL Nodes: https://github.com/aout/SAGE
edit: just saw the repo wasn't entirely up to date. Done now.
Well, your browser is already GPU-accelerated. What makes it slow is the process of making GPU-memory accessible to your JavaScript.
An example of that is drawing a DIV and applying a CSS 3D Transform to it. By doing so you'll modify it's display on the screen so you'll need to recompute it's actual bounding box.
If you compute the BBox on the GPU and never access it anywhere else, everything is fine but as soon as you have to use it in JS you're screwed because you'll have to transfer data from your GPU to your RAM. Doing so at 60fps is very expensive, then you'll take a performance hit from JavaScript being usually slower than C/C++.
That's not the only problem you'll have, text rendering is probably the biggest one. As you might know, textures are rendered using memory friendly algorithms. To ensure maximum readability and quality you'll have to (1) use a lot of memory and / or (2) apply some advanced filtering so your texture doesn't lose quality. Those techniques are quite hard to implement if you're not a 3D programmer (just check what happens if you try to rotate text in CSS and select it) but more importantly you'll find yourself reinventing the wheel.
I tried the demo of course and I thought you did a great job. I'm not saying you should abandon the project because of performance issues you'll encounter but rather focus on simplicity with clear boundaries. Again, go for it! (but don't go too far)
I like the concept for graphics-heavy pages, but for any text-oriented page ... well, any website that didn't support highlighting (which I use to keep track of where I am on a page) and that didn't re-render when zooming (absolutely necessary for high-res displays, where a page might show up tiny and need to be re-scaled) would be a website that I didn't stay at for very long.
Very cool. This could be useful for some applications, but I don't see it becoming a thing for general use cases unless it's implemented and exposed directly by the browsers.
Even then, the main problem is that the internet is still mostly text, and if you can't implement or access font hinting, your text will be blurry and ugly on any display that isn't ultra-high DPI. You can't really render text to a texture and then manipulate that texture with sub-pixel precision. It must be done the other way around: text rendering must be done last, and with the help of a hinting library so it will look sharp and readable.
Browser rasterizes based on it`s assumptions and you cant control it completely which is a bottleneck for performance optimization. Also how would you apply displacement filter on that content?
This creates an interesting change in how applications are deployed to users. Instead of the user choosing the HTML rendering engine, the developer now gets to choose it. Instead of having to design sites with multiple rendering engines in mind, this makes it possible to manage only one. And it's not like the user cares that content is being rendered by Firefox or Chrome or whatever, the user just cares to see the page, leaving the selection of the actual rendering engine only of concern to the developer.
I think that's important enough to warrant going through the paces and implementing this. Don't listen to the naysayers. Reinvent wheels. The wheels we currently have largely suck.
That is so not even close to a good way to build applications. I mean, I really can't think of anything more painful right now. Other than, maybe being asked to make a modern web application on classic ASP.
Really great work. I've had in the back of my mind that this was possible, but now you've gone and done it, and quite well!
I'm curious if there would be any conflict with usage of Three.js on the same site? Could they end up clobbering each other's textures? (I've not gotten into webgl's texture memory management, so that question is a stab in the dark.)
I would not expect any conflicts there since they are different canvases and textures uploaded just as files. You may ask mat@goodboydigital.com for more details.
This was at least kind of possible (not sure if it worked only for images or also other elements) with Internet Explorer 4 [1] in 1997, almost two decades ago. It was obviously non-standard.
Although non-standard, Microsoft sure did several novel things back in the day here. Then they decided not to do the actual standards correctly and held a lot of things back since nobody updated these things.
On the topic of Microsoft and the web, ActiveX gets a lot of bad rep (and for good reason), though it seems like they were trying something akin to web components back then. It worked better in actual applications than the web though.
Would forms and input fields still work? Could one push their wysiwyg rich text editor into this? (I'm thinking game / VR content where an in-3D-world web browser on some in-3D-world device screen is fully operational.)
It depends on how far it will go. At the moment there is a plan how to do that, but not enough time to implement... You may use it for editor, but keep in mind that text itself would be sent to GPU every time it updates. Anyhow you may ping me in skype lp_funky if interested in doing this, would be glad to help.
I think could be nice. There is an intention to create React-GL, but nothing serious yet. All you basically need is to call something like new GLElement(this.getDOMNode()). Or how is it in 0.13?...
Actually there is an implementation called react-canvas does the same thing on canvas. Flipboard uses it on mobile website and it has a great performance even for mobile.
Yes, there is an issue on that. https://github.com/PixelsCommander/HTML-GL/issues/11 I d like to fix it before going public but HTML5WEEKLY made announcement without notice, so everybody sees know that bug =( As for selection - it is selectable in it`s bounding box, but selection is not highlighted yet.
Is seems that rasterization happend before images loaded. Do not observe this on any devices accessible for me. Is there any specificity on system you tried it?
If you can get text selection (and copy/paste) working please post how you did it. I wrote Gate One which is a web-based terminal emulator. Right now I have to go through great pains to ensure that text being rendered to the screen happens in a performant way while still allowing for copy & paste functionality. It also has some features that will highlight text that you select with your mouse and other text formatting stuff.
I've experimented with canvas in the past to see if it could improve the performance of, say, a terminal that updates very quickly but it ultimately ended up being just as slow as the regular DOM because of all the extra code that had to be running in order to keep the hidden buffer in sync with what was displayed to the user.
I'd be curious to know how the performance of WebGL compares to the (accelerated) canvas.
Performance is much better if translation / scaling / opacity / rotation are used or shaders ofcourse. If image content redraws by CPU it sucks. Basically it is what GPU can do quickly. Selection is possible since elements are there but just invisible and when webgl textures were animated we can lazily update DOM positions without affecting performance a lot. Is there a repo for your terminal?
I wonder what exactly the use of this is, beyond Compiz for web pages. What's next, jiggly divs? :P
moron4hire made a good point about letting devs pick the layout engine, but I am not sure we want to ignore all the nice features browsers give us here.
Just chrome and Firefox canary I guess. But webgl is supports by all modern browsers. I can even see nicely rendered page on my windows phone with IE11 mobile installed.
It is already there https://www.chromeexperiments.com/webgl , but it starts to be even more exciting if you think about effects which could be applied to web sites and HTML content... Bumb mapping, reflections, motion blur, elastic elements when dragging. Whole bunch of shaders https://www.shadertoy.com/browse
Demoscene is not about flashy graphics. It's about doing something beautiful with code (this includes the way the code uses the machine to do things that go beyond, like mode 13h).
As far as I understand, this is roughly how iOS UI works (UIView - CALayer).
Obviously, this can be done at the browser engine level.
I have also tried to do this in my pet UI engine, but found out that it was tricky to figure out caching rules to avoid memory blowup and performance degradation for all possible cases.
This approach has proven itself very well in iOS, in fact, it was the absolute best mobile animation technology for a long time. Actually, the older versions of iOS used software rasterization with hardware accelerated bitmap caching and it worked smoothly on the original iPhone.
Haha yea, I am going to delve into your code and see if I can contribute. But nice work! Your site is probably getting hammered now with the traffic but I am going to come back and check it out.
It using html2canvas[0] to rasterize the browser rendering into a canvas, upload it to WebGL, apply effects and patch-back events(i.e. mouse clicks) back to the hidden browser dom elements. (On the other side, font hinting and such are not really a problem)
Implementing full UI framework over WebGL has it merits, and hopefully someone will make a useful(and not html+css based) one some day.
[0] https://github.com/niklasvh/html2canvas