WebGL WEBGL_video_texture Extension Proposed Specification

DO NOT IMPLEMENT!!!

Name

WEBGL_video_texture

Contact

WebGL working group (public_webgl 'at' khronos.org)

Contributors

Byungseon Shin (sun.shin 'at' lge.com)

Andrey Volykhin (andrey.volykhin 'at' lge.com)

Members of the WebGL working group

Version

Last modified date: May 13, 2020
Revision: 7

Number

WebGL extension #NN

Dependencies

Written against the WebGL API 1.0 specification.

Overview

This extension exposes the OES_EGL_image_external functionality to WebGL.

The following WebGL-specific behavioral changes apply:

• Defines a new texture target TEXTURE_VIDEO_IMAGE_WEBGL and a new sampler type SAMPLER_VIDEO_IMAGE_WEBGL
• Provides a mechanism for binding HTMLVideoElement stream to video texture targets.
• Provides time of frame, texture width and height of HTMLVideoElement's texture binding.

Consult the above extension for documentation, issues and new functions and enumerants.

When this extension is enabled:

• Add support for WEBGL_video_texture binding of HTMLVideoElement.
• TEXTURE_VIDEO_IMAGE_WEBGL and TEXTURE_2D are distinct.
• Textures bound to TEXTURE_VIDEO_IMAGE_WEBGL only accept LINEAR and NEAREST for TEXTURE_MIN_FILTER and TEXTURE_MAG_FILTER.
• Textures bound to TEXTURE_VIDEO_IMAGE_WEBGL only accept CLAMP_TO_EDGE for TEXTURE_WRAP_S and TEXTURE_WRAP_T.
• The maximum mipmap level of textures that bind to TEXTURE_VIDEO_IMAGE_WEBGL is 1.
• shareVideoImageWEBGL will lock video frame and reject another video source until call releaseVideoImageWEBGL.
• If the application does not call releaseVideoImageWEBGL, then all of the WebGLTextures with which video frames were shared will be implicitly released when the current JavaScript code returns control to the user agent.
• When a fragment shader enables, requires, or warns WEBGL_video_texture with an #extension directive:
  • samplerVideoWEBGL is a built-in type.
  • vec4 textureVideoWEBGL(samplerVideoWEBGL sampler, vec2 coord) is a built-in function.
• The GLSL macro WEBGL_video_texture is defined as 1.

IDL

[Exposed=(Window,Worker), LegacyNoInterfaceObject]
interface WebGLVideoFrameInfo {
  readonly attribute double currentTime;
  readonly attribute unsigned long textureWidth;
  readonly attribute unsigned long textureHeight;
};

[Exposed=(Window,Worker), LegacyNoInterfaceObject]
interface WEBGL_video_texture {
    const GLenum TEXTURE_VIDEO_IMAGE_WEBGL             = 0x9248;
    const GLenum SAMPLER_VIDEO_IMAGE_WEBGL             = 0x9249;

    [RaisesException] WebGLVideoFrameInfo shareVideoImageWEBGL(
      GLenum target, HTMLVideoElement video);
    [RaiseException] undefined releaseVideoImageWEBGL(GLenum target);
};
  

Sample Code

This a fragment shader that samples a video texture.

    #extension GL_WEBGL_video_texture : require
    precision mediump float;
    varying vec2 v_texCoord;

    uniform samplerVideoWEBGL uSampler;

    void main(void) {
      gl_FragColor = textureVideoWEBGL(uSampler, v_texCoord);
    }
    

This shows application that renders video using proposed extension.

    var videoElement = document.getElementById("video");
    var videoTexture = gl.createTexture();

    function update() {
        var ext = gl.getExtension('WEBGL_video_texture');
        if(ext !=== null){
            gl.bindTexture(ext.TEXTURE_VIDEO_IMAGE_WEBGL, videoTexture);
            ext.shareVideoImageWEBGL(ext.TEXTURE_VIDEO_IMAGE_WEBGL, videoElement);
            ext.releaseVideoImageWEBGL(ext.TEXTURE_VIDEO_IMAGE_WEBGL);
            gl.bindTexture(ext.TEXTURE_VIDEO_IMAGE_WEBGL, null);
        }
    }

    function render() {
        gl.clearColor(0.0, 0.0, 1.0, 1.0);
        gl.clear(gl.COLOR_BUFFER_BIT);

        gl.bindBuffer(gl.ARRAY_BUFFER, squareVerticesBuffer);
        gl.vertexAttribPointer(vertexPositionAttribute, 3, gl.FLOAT, false, 0, 0);

        gl.activeTexture(gl.TEXTURE0);
        gl.bindTexture(ext.TEXTURE_VIDEO_IMAGE_WEBGL, videoTexture);
        gl.uniform1i(gl.getUniformLocation(shaderProgram, "uSampler"), 0);

        gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);

        ext.releaseVideoImageWEBGL(ext.TEXTURE_VIDEO_IMAGE_WEBGL);
    }
    

Application renders each video frames into WebGL canvas based on game-loop pattern.

    while (true) {
       update();
       processInput();
       render();
    }
    

Conformance Tests

Issues

Revision History

Revision 1, 2016/11/05

• Initial revision.

Revision 2, 2017/01/10

• Change EGLImageTargetTexture2DOES to be called at every WebGL rendering cycle.
• Add VideoFrameInfo interface.
• Change EGLImageTargetTexture2DOES to return VideoFrameInfo as a currently mapped frame.

Revision 3, 2017/08/03

• Change Extension name to WEBGL_video_texture for abstracion of OES_EGL_image_external extension.
• Define new sampler and texture type, TEXTURE_VIDEO_IMAGE and SAMPLER_VIDEO_IMAGE.
• Change EGLImageTargetTexture2DOES to VideoElementTargetVideoTexture.

Revision 4, 2019/04/10

• Add WEBGL suffix for extension enums.
• Change function name VideoElementTargetVideoTexture to shareVideoImageWEBGL
• Declare possible side effect of TEXTURE_VIDEO_IMAGE_WEBGL.
• Add limitations for TEXTURE_VIDEO_IMAGE_WEBGL
• Add releaseVideoImageWEBGL and implicit release rules

Revision 5, 2019/10/25

• Define textureVideoWEBGL, rather than using texture2D, in ESSL 1.00 shaders.

Revision 6, 2019/10/29

• Allocate enums for TEXTURE_VIDEO_IMAGE_WEBGL and SAMPLER_VIDEO_IMAGE_WEBGL out of WebGL's current assigned enum block in the OpenGL extension registry.

Revision 7, 2020/05/13

• Change return type of releaseVideoImageWEBGL from bool to void.