Description
Use case
This feature request is about the Canvas class in engine/painting.dart.
As extensively discussed in #31598, using regular canvas operations to draw pixel data is extremely slow.
This makes the only option we have decoding an image from our raw pixel data, e.g. using decodeImageFromPixels. This has another problem, pointed out by @andrewackerman in the mentioned issue, which is that creating an Image in Dart is an async operations.
This means that we will have to postpone drawing our raw pixel data to the next frame if we wish to use real-time operations.
The async Image problem
Images in Flutter currently have the inherent problem that rasterizing / decoding them is always an asynchronous operation. This is why using them for drawing pixel data does not work synchronously, but it also creates a whole nother class of problems:
Any other canvas operation in Flutter that involves Images is also asynchronous. So not only can you not draw pixel data synchronously, but you cannot rasterize any Picture synchronously, which is necessary for achieving various effects (e.g. this). Also Canvas.drawAtlas suffers from this problem.
drawPixels proposal
Here is an example proposal (written by @andrewackerman in #31598) for directly drawing pixel data.
Note that this might not cover all use cases because often we do need to have a decoded image (dart:ui.Image) available to us, e.g. when using Canvas.drawAtlas. It should, however, give an idea as to what we are looking for.
[...] it should have a similar signature to
decodImageFromPixels[sic], so something like:
class Canvas {
...
void drawPixels(
Offset c,
Uint8List bytes,
int width,
int height,
PixelFormat pixelFormat, {
int rowBytes,
int targetWidth,
int targetHeight,
}) {
assert(c != null && bytes != null && width != null && height != null && pixelFormat != null);
assert(bytes.length == width * height * 4);
if (rowBytes == null) rowBytes = width * 4;
if (targetWidth == null && targetHeight == null) {
targetWidth = width;
targetHeight = height;
} else if (targetWidth == null) {
targetWidth = (targetHeight * (width / height)).toInt();
} else if (targetHeight == null) {
targetHeight = (targetWidth * (height / width)).toInt();
}
_drawPixels(c, pixelData, width, height, pixelFormat, rowBytes, targetWidth, targetHeight);
}
}Click to expand details
cis the position that the image will be drawn at.
bytesis the pixel data that will be drawn to the canvas, represented as byte groups of 4 bytes per pixel. The color channels will be extrapolated based on the value ofpixelFormat.
widthandheightare the dimensions of the image represented by the pixel data. Ifwidth * height * 4is not equal topixelData.length, an error will be thrown.
rowBytesis the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the providedformat.The
targetWidth andtargetHeight` arguments specify the size of the output image, in image pixels. If they are not equal to the intrinsic dimensions of the image, then the image will be draw. If exactly one of these two arguments is specified, then the aspect ratio will be maintained while forcing the image to match the other given dimension. If neither is specified, then the image maintains its real size. (Debatable as to whether these parameters should be supported as they don't really fall into the whole "just draw these pixels" mentality of this method.)**
@andrewackerman assumed that the byte data will always have 32 bits per pixel, which makes sense because the only PixelFormat values that currently exist are bgra8888 and rgba8888, which use 32 bits per pixel.
The decodeImageFromPixels function is open to any PixelFormat that might be supported in the future:
rowBytesis the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the providedformat.
I think that this approach would make sense to also be used for the Canvas method (if it will use PixelFormat). Consequently, the assertion for width * height * 4 should probably be width * height * bytesPerPixel and similarly, bytes should be pixel data, represented as byte groups of bytesPerPixel bytes per pixel.