WebGL WEBGL_dynamic_texture Extension Proposed Specification

A dynamic texture is a texture whose image changes frequently. The source of the stream of images may be a producer outside the control of the WebGL application. The classic example is using a playing video to texture geometry. Texturing with video is currently achieved by using the TEXTURE2D target and passing an HTMLVideoElement to texImage2D. It is difficult, if not impossible to implement video texturing with zero-copy efficiency via this API and much of the behavior is underspecified.

This extension provides a mechanism for streaming image frames from an HTMLVideoElement, HTMLCanvasElement or HTMLImageElement (having multiple frames such those created from animated GIF, APNG and MNG files) into a WebGL texture. This is done via a new texture target, TEXTURE_EXTERNAL_OES which can only be specified as being the consumer of an image stream from a new WDTStream object which provides commands for connecting to a producer element.

There is no support for most of the functions that manipulate other texture targets (e.g. you cannot use *[Tt]ex*Image*() functions with TEXTURE_EXTERNAL_OES). Also, TEXTURE_EXTERNAL_OES targets never have more than a single level of detail. These restrictions enable dynamic texturing with maximum efficiency. They remove the need for a copy of the image data manipulable via the WebGL API and allow sources which have internal formats not otherwise supported by WebGL, such as planar or interleaved YUV data, to be WebGL texture target siblings.

The extension extends GLSL ES with a new samplerExternalOES type and matching sampling functions that provide a place for an implementation to inject code for sampling non-RGB data when necessary without degrading performance for other texture targets. Sampling a TEXTURE_EXTERNAL_OES via a sampler of type samplerExternalOES always returns RGBA data. This allows the implementation to decide the most efficient format to use whether it be RGB or YUV data. If the underlying format was exposed, the application would have to query the format in use and provide shaders to handle both cases.

WDTStream provides a command for latching an image frame into the consuming texture as its contents. This is equivalent to copying the image into the texture but, due to the restrictions outlined above a copy is not necessary. Most implementations will be able to avoid one so this can be much faster than using texImage2D. Latching can and should be implemented in a way that allows the producer to run independently of 3D rendering.

Terminology note: throughout this specification opaque black refers to the RGBA value (0,0,0,1).

For ease of reading, this specification briefly describes the new functions and enumerants of NV_EGL_stream_consumer_external. Consult that extension for detailed documentation of their meaning and behavior. Changes to the language of that extension are given later in this specification.

IDL

IP Status

New Functions

New Types

New Tokens

Additions to the WebGL Specification

In section 4.3 Supported GLSL Constructs, replace the paragraph beginning A WebGL implementation must ... with the following paragraph:

In section 5.14 The WebGL Context , add the following to the WebGLRenderingContext interface. Note that until such time as this extension enters core WebGL the tokens and commands mentioned below will be located on the WebGL_dynamic_texture extension interface shown above.

In section 5.14.3 Setting and getting state, add the following to the table under getParameter.

In section 5.14.8Texture objects, add the following to the table under getTexParameter.

In section 5.14.10 Uniforms and attributes, add the following to the table under getUniform.

Errors

New State

New Implementation-Dependent State

Sample Code

XXX IGNORE THIS SAMPLE CODE. IT HAS NOT YET BEEN UPDATED TO MATCH THE NEW SPEC TEXT. XXX

This a fragment shader that samples a video texture. Note that the surrounding <script> tag is not essential; it is merely one way to include shader text in an HTML file.

<script id="fshader" type="x-shader/x-fragment">
  #extension OES_EGL_image_external : enable 
  precision mediump float;

  uniform samplerExternalOES videoSampler;

  varying float v_Dot;
  varying vec2 v_texCoord;

  void main()
  {
    vec2 texCoord = vec2(v_texCoord.s, 1.0 - v_texCoord.t);
    vec4 color = texture2D(videoSampler, texCoord);
    color += vec4(0.1, 0.1, 0.1, 1);
    gl_FragColor = vec4(color.xyz * v_Dot, color.a);
  }
</script>

This shows fragments from an application that renders a spinning cube textured with a live video.

<html>
<script type="text/javascript">

  ///////////////////////////////////////////////////////////////////////
  // Create a video texture and bind a source to it.
  ///////////////////////////////////////////////////////////////////////

  // Array of files currently loading
  g_loadingFiles = [];

  // Clears all the files currently loading.
  // This is used to handle context lost events.
  function clearLoadingFiles() {
    for (var ii = 0; ii < g_loadingFiles.length; ++ii) {
      g_loadingFiles[ii].onload = undefined;
    }
    g_loadingFiles = [];
  }

  //
  // createVideoTexture
  //
  // Load video from the passed HTMLVideoElement id, bind it to a new WebGLTexture object
  // and return the WebGLTexture.
  //
  // Is there a constructor for an HTMLVideoElement so you can do like "new Image()?"
  //
  function createVideoTexture(ctx, videoId)
  {
    var texture = ctx.createTexture();
    var video = document.getElementById(videoId);
    g_loadingFiles.push(video);
    video.onload = function() { doBindVideo(ctx, video, texture) }
    return texture;
  }

  function doBindVideo(ctx, video, texture)
  {
    g_loadingFiles.splice(g_loadingFiles.indexOf(image), 1);
    ctx.bindTexture(ctx.TEXTURE_EXTERNAL_OES, texture);
    ctx.dynamicTextureSetSource(video);
    // These are the default values of these properties so the following
    // 4 lines are not necessary.
    ctx.texParameteri(ctx.TEXTURE_EXTERNAL_OES, ctx.TEXTURE_MAG_FILTER, ctx.LINEAR);
    ctx.texParameteri(ctx.TEXTURE_EXTERNAL_OES, ctx.TEXTURE_MIN_FILTER, ctx.LINEAR);
    ctx.texParameteri(ctx.TEXTURE_EXTERNAL_OES, ctx.TEXTURE_WRAP_S, ctx.CLAMP_TO_EDGE);
    ctx.texParameteri(ctx.TEXTURE_EXTERNAL_OES, ctx.TEXTURE_WRAP_T, ctx.CLAMP_TO_EDGE);
    ctx.bindTexture(ctx.TEXTURE_EXTERNAL_OES, null);
  }

  ///////////////////////////////////////////////////////////////////////
  // Initialize the application.
  ///////////////////////////////////////////////////////////////////////

  var g = {};
  var videoTexture;

  function init()
  {
    // Initialize
    var gl = initWebGL(
        // The id of the Canvas Element
        "example");
    if (!gl) {
      return;
    }
    var program = simpleSetup(
        gl,
        // The ids of the vertex and fragment shaders
        "vshader", "fshader",
        // The vertex attribute names used by the shaders.
        // The order they appear here corresponds to their index
        // used later.
        [ "vNormal", "vColor", "vPosition"],
        // The clear color and depth values
        [ 0, 0, 0.5, 1 ], 10000);

    // Set some uniform variables for the shaders
    gl.uniform3f(gl.getUniformLocation(program, "lightDir"), 0, 0, 1);
    // Use the default texture unit 0 for the video
    gl.uniform1i(gl.getUniformLocation(program, "samplerExternal"), 0);

    // Create a box. On return 'gl' contains a 'box' property with
    // the BufferObjects containing the arrays for vertices,
    // normals, texture coords, and indices.
    g.box = makeBox(gl);

    // Load an image to use. Returns a WebGLTexture object
    videoTexture = createVideoTexture(gl, "video");
    // Bind the video texture
    gl.bindTexture(gl.TEXTURE_EXTERNAL_OES, videoTexture);

    // Create some matrices to use later and save their locations in the shaders
    g.mvMatrix = new J3DIMatrix4();
    g.u_normalMatrixLoc = gl.getUniformLocation(program, "u_normalMatrix");
    g.normalMatrix = new J3DIMatrix4();
    g.u_modelViewProjMatrixLoc =
            gl.getUniformLocation(program, "u_modelViewProjMatrix");
    g.mvpMatrix = new J3DIMatrix4();

    // Enable all of the vertex attribute arrays.
    gl.enableVertexAttribArray(0);
    gl.enableVertexAttribArray(1);
    gl.enableVertexAttribArray(2);

    // Set up all the vertex attributes for vertices, normals and texCoords
    gl.bindBuffer(gl.ARRAY_BUFFER, g.box.vertexObject);
    gl.vertexAttribPointer(2, 3, gl.FLOAT, false, 0, 0);

    gl.bindBuffer(gl.ARRAY_BUFFER, g.box.normalObject);
    gl.vertexAttribPointer(0, 3, gl.FLOAT, false, 0, 0);

    gl.bindBuffer(gl.ARRAY_BUFFER, g.box.texCoordObject);
    gl.vertexAttribPointer(1, 2, gl.FLOAT, false, 0, 0);

    // Bind the index array
    gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, g.box.indexObject);

    return gl;
  }

  // ...

  ///////////////////////////////////////////////////////////////////////
  // Draw a frame
  ///////////////////////////////////////////////////////////////////////
  function draw(gl)
  {
    // Make sure the canvas is sized correctly.
    reshape(gl);

    // Clear the canvas
    gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

    // Make a model/view matrix.
    g.mvMatrix.makeIdentity();
    g.mvMatrix.rotate(20, 1,0,0);
    g.mvMatrix.rotate(currentAngle, 0,1,0);

    // Construct the normal matrix from the model-view matrix and pass it in
    g.normalMatrix.load(g.mvMatrix);
    g.normalMatrix.invert();
    g.normalMatrix.transpose();
    g.normalMatrix.setUniform(gl, g.u_normalMatrixLoc, false);

    // Construct the model-view * projection matrix and pass it in
    g.mvpMatrix.load(g.perspectiveMatrix);
    g.mvpMatrix.multiply(g.mvMatrix);
    g.mvpMatrix.setUniform(gl, g.u_modelViewProjMatrixLoc, false);

    // Acquire the latest video image
    gl.dynamicTextureAcquireImage();

    // Draw the cube
    gl.drawElements(gl.TRIANGLES, g.box.numIndices, gl.UNSIGNED_BYTE, 0);

    // Allow updates to the image again
    gl.dynamicTextureReleaseImage();

    // Show the framerate
    framerate.snapshot();

    currentAngle += incAngle;
    if (currentAngle > 360)
      currentAngle -= 360;
  }
</script>

<body onload="start()">
<video id="video" src="resources/video.ogv" autoplay="true" style="visibility: hidden">
</video>
<canvas id="example">
    If you're seeing this your web browser doesn't support the &lt;canvas&gt; element. Ouch!
</canvas>
<div id="framerate"></div>
</body>

</html>

Conformance Tests

Security Considerations

Statistical fingerprinting is a privacy concern where a malicious web site may determine whether a user has visited a third-party web site by measuring the timing of cache hits and misses of resources in the third-party web site. Though the ustnow method of this extension returns time data to a greater accuracy than before, it does not make this privacy concern significantly worse than it was already.

WebGL WEBGL_dynamic_texture Extension Proposed Specification

Name

Contact

Contributors

Version

Number

Dependencies

Overview

IDL

IP Status

New Functions

New Types

New Tokens

Additions to the WebGL Specification

5.14.8.1 External textures

5.16 Dynamic Textures

5.16.1 WDTStreamFrameInfo

5.16.1.1 Attributes

5.16.2 WDTStream

5.16.2.1 Attributes

5.16.2.2 commands

6.25 External Texture Support