WebCL WEBCL_html_video Extension Draft Specification

Name

WEBCL_html_video

Contact

WebCL working group (public_webcl 'at' khronos.org)

Contributors

Tomi Aarnio, Nokia Research

Members of the WebCL working group

Version

Last modified date: January 17, 2014
Revision: 2

Number

WebCL extension #5

Dependencies

Written against the WebCL API 1.0 specification.

Overview

This extension makes it more convenient and potentially faster for applications to transfer image data from an HTMLVideoElement to a WebCLImage.

When this extension is enabled:

• The application may create a WebCLImage that is permanently bound to an HTMLVideoElement. The source video pixel format is treated as 32-bit RGBA (8 bits per component) with non-premultiplied alpha. The new WebCLImage will have `channelOrder` equal to `RGBA` and `channelType` equal to `UNORM_INT8`.

IDL

partial interface WebCLContext {

  WebCLImage createImage(CLenum memFlags,
                         HTMLVideoElement srcVideo);
};


partial interface WebCLCommandQueue {

  void enqueueWriteImage(
                    WebCLImage                            image,
                    CLboolean                             blockingWrite,
                    HTMLVideoElement                      srcVideo,
                    optional sequence<WebCLEvent>?        eventWaitList,
                    optional WebCLEvent?                  event);
};
  

New Functions

WebCLImage createImage(CLenum memFlags, HTMLVideoElement srcVideo)

Creates a new WebCLImage with the given pixel format, and binds the image permanently to the given HTMLVideoElement. The video element does not have to be fully loaded, but its dimensions must be known. The newly created WebCLImage will have width and height equal to the source video width and height.

Exceptions:

`WEBCL_EXTENSION_NOT_ENABLED` -- if the `WEBCL_html_video` extension has not been enabled

`INVALID_VALUE` -- if `memFlags` is not `MEM_READ_ONLY`

`INVALID_IMAGE_SIZE` -- if srcVideo.width > DEVICE_IMAGE2D_MAX_WIDTH

`INVALID_IMAGE_SIZE` -- if srcVideo.height > DEVICE_IMAGE2D_MAX_HEIGHT

`INVALID_HOST_PTR` -- if `srcVideo` has unknown dimensions, or is otherwise not valid

`INVALID_IMAGE_FORMAT_DESCRIPTOR` -- if `descriptor.channelOrder` or `descriptor.channelType` is not valid

`IMAGE_FORMAT_NOT_SUPPORTED` -- if the given combination of `channelOrder` and `channelType` is not supported for `srcVideo`

void enqueueWriteImage(WebCLImage image, CLboolean blockingWrite, HTMLVideoElement srcVideo, optional sequence<WebCLEvent>? eventWaitList, optional WebCLEvent? event)

Enqueues a command to write the current frame of `srcVideo` to the given WebCLImage. The WebCLImage must already be bound to `srcVideo`. If `blockingWrite` is `true`, the video frame is extracted immediately and then enqueued for writing. If `blockingWrite` is `false`, the frame is extracted later, when the enqueued write command is actually executed.

Exceptions:

`WEBCL_EXTENSION_NOT_ENABLED` -- if the `WEBCL_html_video` extension has not been enabled

`INVALID_CONTEXT` -- if this WebCLCommandQueue is not associated with the same WebCLContext as `image`

`INVALID_CONTEXT` -- if this WebCLCommandQueue is not associated with the same WebCLContext as all events in `eventWaitList`

`INVALID_MEM_OBJECT` -- if `image` is not a valid WebCLImage object

`INVALID_MEM_OBJECT` -- if `image` is not bound to `srcVideo`

`INVALID_IMAGE_SIZE` -- if the dimensions of `image` are not supported by this WebCLCommandQueue

`INVALID_EVENT_WAIT_LIST` -- if any event in `eventWaitList` is invalid

Sample Code

var ok = device.enableExtension("WEBCL_html_video");
if (ok) {
  var ctx = WebCL.createContext({ devices: [device] });
  var queue = ctx.createCommandQueue();
  var videoImage = ctx.createImage(WebCL.MEM_READ_ONLY, videoElement);
  var kernel = ctx.createProgram(kernelSource).createKernelsInProgram()[0];
  for (/* each frame */) {
    queue.enqueueWriteImage(videoImage, false, videoElement);
    queue.enqueueNDRangeKernel(/* process the frame with kernel */);
    queue.enqueueReadImage(videoImage, false, processedPixels);
    queue.finish();
    doSomething(processedPixels);
  }
  ctx.release();
}

Issues

1. Should we have both blocking and non-blocking forms of `enqueueWriteImage`?

TBD.

2. Should we have a way to get the timestamp of the extracted frame?

TBD.

3. Should we allow for extracting a sub-region of full frame?

TBD.

4. Should we specify at least one channelOrder/channelType combination that is guaranteed to work on any implementation?

RESOLVED: Yes, RGBA/UNORM_INT8.

5. Should we support channelOrder/channelType combinations other than RGBA/UNORM_INT8?

RESOLVED: No, not until browser implementations of HTMLVideoElement start supporting other formats.

Revision History

Revision 1, 2013/11/20

• Migrated from the WebCL Working Draft.

Revision 2, 2014/01/17

• Restricted the available video formats to RGBA/UNORM_INT8.