I'm currently playing with applying simple filter effects to images on an HTML5 Canvas. Similar to the technique that is defined here:
HTML5 Canvas image contrast And in this HTML5 Rocks! blog post
However, this particular approach requires iterating over every pixel and applying a modifier before redrawing. For my particular use case, this requires 150ms+ to redraw my image (650px by 650px PNG)
Applying the same effect using CSS3's filter property (contrast or brightness) takes less than 10ms.
My questions are: How does CSS3's filter property work "under the hood"? And why is it significantly more performant? Are there ways to achieve similar performance in Canvas?
Canvas is exposed through an API. JavaScript is a runtime, and even though a great deal optimized these days, if you are going to manipulate a 2D grid with pixel-by-pixel access from JavaScript, you will be paying the penalty of said access from runtime all the way down to the buffer containing the image currently stored with the canvas, for each pixel. The machine is mostly bound to follow and implement your pattern of data access, so you can say you end up being the primary impediment to performance here, by "overdictating" the solution -- through expressing the filter as imperative pixel-by-pixel data assignment, for example, and because the interpreter is generally limited with regard to its ability to optimize "free form" code.
A CSS3 filter is a black box that can do the same transformation in bulk, meaning that a constrained GPU shader program, for instance, is running directly on the GPU, accessing image that is typically stored in the immediately addressable memory close to the GPU, and the latter benefiting from its SIMD-class instructions that are designed to process entire matrices of pixels. Also, native code accessing local memory -- pretty much as fast as it gets, without getting into details. The GPU utilizes a very long execution pipeline -- plainly speaking a long queue of operations that aren't restarted or checked, so that these can be executed as fast as possible. An analogy for the latter would be a heavy train passing a station it knows it isn't scheduled to stop at, so it can pass it at top speed as if it weren't there. This is one of the tricks a GPU uses to process data fast -- assume a restricted environment so that you don't have to think about "everything" and can optimize more.
Even when running on a CPU, without any GPU assistance whatsoever, we are talking native filter kernel directly manipulating the image in RAM, without any bytecode and more importantly without taking consideration for JavaScript at all. You declare your desired filter, user agent invokes the filter program on the canvas image, that's it. The CPU also has SIMD instructions to work on data vectors, which obviously helps significantly. The filter code is not even yours, you only reference it by name.
Now, if you could apply some kind of black box filter like one of those available in CSS, to a canvas pixel data directly, you'd probably achieve same speed as with CSS -- because the most important impediment to speed that you had -- pixel by pixel access expressed with JavaScript code -- is eliminated. So this is not just about JavaScript, this is about granularity of data access. In this case, applying a kernel to the data in bulk will always be faster than writing the kernel code yourself on a higher level. Simply speaking, which brings me to the last point below.
Now, if the JavaScript interpreter could understand your pixel-by-pixel-in-a-loop manipulations in the sense that it could convert it all into native code that utilizes SIMD and perhaps even GPU shaders, all from your freely-typed JavaScript filter code, that would bridge the performance gap. But you would be moving the complexity into JavaScript compiler/interpreter, and the problem of optimizing on such scale isn't a completely solved problem yet in computer science. Maybe artificial intelligence and machine learning will help, I won't speculate on that. Mind you, I am not talking JIT-compiling your JavaScript to equivalent native code, which has been the norm for years now, I am talking about recognizing freely-typed JavaScript code as something resembling a known or perhaps even arbitrary image kernel, much like a human would. Then replacing said code with a run-time counterpart that gives identical result but is written by the compiler to yield what it thinks will give optimal performance.
In practice, I think you can optimize Canvas-based JavaScript naive pixel-by-pixel filters, if you take a deeper look into the Canvas API. The image data can be accessed through a Uint8ClampedArray type which you can obtain from CanvasRenderingContext2D.getImageData() method call. This array type has some interesting "bulk" functions like filter, forEach, map, reduce etc. You need to think a bit like a "hacker" when browsing the Canvas API documentation, to assume a mindset of a demoscene person -- looking at the available methods and data types in terms of what they can give you. The rewards for doing this can be substantial.
If that wasn't enough, a canvas can be rendered with WebGL, an API which is a subset of OpenGL, typically implemented to run exclusively on the GPU. Shaders are guaranteed to be part of WebGL, meaning that you've got a free ticket to all kinds of advanced and superfast canvas filter programs with WebGL, programs that you write yourself but which typically execute on the GPU.