Name KHR_stream_producer_eglsurface Name Strings EGL_KHR_stream_producer_eglsurface Contributors Acorn Pooley Jamie Gennis Marcus Lorentzon Contacts Acorn Pooley, NVIDIA (apooley 'at' nvidia.com) Notice Copyright (c) 2011-2013 The Khronos Group Inc. Copyright terms at http://www.khronos.org/registry/speccopyright.html Status Complete. Approved by the Khronos Board of Promoters on December 2, 2011. Version Version 11, June 18, 2012 Number EGL Extension #34 Dependencies Requires EGL 1.2. Requires OpenGL ES 1.1 or OpenGL ES 2.0. Requires the EGL_KHR_stream extension. Overview This extension allows an EGLSurface to be created as a producer of images to an EGLStream. Each call to eglSwapBuffers posts a new image frame into the EGLStream. New Procedures and Functions EGLSurface eglCreateStreamProducerSurfaceKHR( EGLDisplay dpy, EGLConfig config, EGLStreamKHR stream, const EGLint *attrib_list) New Tokens Bit that can appear in the EGL_SURFACE_TYPE of an EGLConfig: EGL_STREAM_BIT_KHR 0x0800 Add a row to "Table 3.2: Types of surfaces supported by an EGLConfig" in the EGL spec, right after the EGL_PBUFFER_BIT row: EGL Token Name Description -------------- -------------------------- EGL_STREAM_BIT_KHR EGLConfig supports streams In the second paragraph of section "Other EGLConfig Attribute Description" in the EGL spec, replace EGL_WINDOW_BIT | EGL_PIXMAP_BIT | EGL_PBUFFER_BIT with EGL_WINDOW_BIT | EGL_PIXMAP_BIT | EGL_PBUFFER_BIT | EGL_STREAM_BIT_KHR and replace "...cannot be used to create a pbuffer or pixmap." with "...cannot be used to create a pbuffer, pixmap, or stream." Replace section "3.10.3.1 No way to connect producer to EGLStream" in the EGL_KHR_stream extension with this: 3.10.3.1 Stream Surface Producer Call EGLSurface eglCreateStreamProducerSurfaceKHR( EGLDisplay dpy, EGLConfig config, EGLStreamKHR stream, const EGLint *attrib_list) to create an EGLSurface and connect it as the producer of . specifies a list of attributes for . The list has the same structure as described for eglChooseConfig. The attributes EGL_WIDTH and EGL_HEIGHT must both be specified in the . EGL_WIDTH and EGL_HEIGHT indicate the width and height (respectively) of the images that makes up the stream. The EGLSurface producer inserts an image frame into once for each time it is passed to eglSwapBuffers(). The image frame is inserted after the GL has finished previous rendering commands. Refer to section "3.10.5 EGLStream operation" in the EGL_KHR_stream extension specification for operation of the EGLStream when an image frame is inserted into it. If is not in the EGL_STREAM_STATE_EMPTY_KHR, EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR, or EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR when passed to eglSwapBuffers(), then eglSwapBuffers will return EGL_FALSE and generate an EGL_BAD_CURRENT_SURFACE error. If the application would like to have the results of rendering appear on the screen at a particular time then it must query the value of EGL_CONSUMER_LATENCY_USEC_KHR after calling eglCreateStreamProducerSurfaceKHR. This is the estimated time that will elapse between the time the image frame is inserted into the EGLStream and the time that the image frame will appear to the user. The image frame is not inserted into the EGLStream until the GL has finished rendering it. Therefore predicting exactly when the image frame will be inserted into the stream is nontrivial. If it is critical that this frame of data reach the screen at a particular point in time, then the application can - render the frame (using GL/GLES commands) - call glFinish (or use other synchronization techniques to ensure rendering has completed). - wait until the time that the frame should appear to the user MINUS the value of EGL_CONSUMER_LATENCY_USEC_KHR. - call eglSwapBuffers This will allow the image frame to be inserted into the EGLStream at the correct time ("Image Frame Intended Display Time" minus "Consumer Latency") so that it will be displayed ("Image Frame Actual Display Time" as close as possible to the desired time. However, this will cause the GPU to operate in lockstep with the CPU which can cause poor performance. In most cases it will be more important for the image frame to appear to the user "as soon as possible" rather than at a specific point in time. So in most cases the application can ignore the value of EGL_CONSUMER_LATENCY_USEC_KHR, not call glFinish, and not wait before calling eglSwapBuffers. On failure eglCreateStreamProducerSurfaceKHR returns EGL_NO_SURFACE and generates an error. - EGL_BAD_PARAMETER if EGL_WIDTH is not specified or is specified with a value less than 1. - EGL_BAD_PARAMETER if EGL_HEIGHT is not specified or is specified with a value less than 1. - EGL_BAD_STATE_KHR is generated if is not in state EGL_STREAM_STATE_CONNECTING_KHR. - EGL_BAD_MATCH is generated if does not have the EGL_STREAM_BIT_KHR set in EGL_SURFACE_TYPE. - EGL_BAD_MATCH is generated if the implementation is not able to convert color buffers described by into image frames that are acceptable by the consumer that is connected to . - EGL_BAD_STREAM_KHR is generated if is not a valid EGLStream created for . - EGL_BAD_DISPLAY is generated if is not a valid, initialized EGLDisplay. Add a section preceding "3.9.3 Posting Semantics" in the EGL specification: 3.9.x Posting to a Stream To post the color buffer to an EGLStream with an EGLSurface producer, call EGLBoolean eglSwapBuffers( EGLDisplay dpy, EGLSurface surface); If is the producer of an EGLStream then the contents of the color buffer are inserted as a new image frame into the EGLStream. When eglSwapBuffers returns the contents of the color buffer will have been inserted into the EGLStream as described in section "3.10.5 EGLStream operation" in the EGL_KHR_stream extension specification, and the EGL_PRODUCER_FRAME_KHR attribute and EGL_STREAM_STATE_KHR attribute values will reflect this. The contents of the color buffer and all ancillary buffers are always undefined after calling eglSwapBuffers. eglSwapBuffers is never synchronized to a video frame when is the producer for an EGLStream (it is as if the swapinterval (set by eglSwapInterval, see below section "3.9.3 Posting Semantics") is 0). It is implementation dependent whether eglSwapBuffers actually waits for rendering to the color buffer to complete before returning, but except for timing it must appear to the application that all rendering to the EGLSurface (e.g. all previous gl commands) completed before the image frame was inserted into the EGLStream and eglSwapBuffers returned (as described below in section "3.9.3 Posting Semantics"). Add to section "3.9.4 Posting Errors" in the EGL specification a new sentence as the 2nd to last sentence in the first paragraph: If eglSwapBuffers is called and the EGLStream associated with surface is no longer valid, an EGL_BAD_STREAM_KHR error is generated. Issues 1. How many image frame buffers should be used? DISCUSSION: - leave up to implementation? - leave up to producer? - need hints from consumer? - In practice 1, 2, and 3 buffers mean different semantics which are visible to both the producer and consumer. Each may be useful. I cannot think of a use for more than 3 buffers for EGL_KHR_stream_surface. (For a video producer more than 3 often does make sense, but that is a different extension.) One possibility: expose EGL_BUFFER_COUNT_KHR to application. It probably does not make sense to ever use more or less than 3 buffers. One that is the EGLSurface back buffer. One that is waiting for the consumer to acquire. And one that the consumer has acquired and is actively consuming. RESOLVED: remove the EGL_BUFFER_COUNT_KHR parameter and always use 3 buffers. This attribute can be added back with a layered extension later if needed. 2. How is the resolution (width/height) of image frames set? RESOLVED: The width and height are set with the required EGL_WIDTH and EGL_HEIGHT attributes. These do not change for the life of . 3. How is the image format, zbuffering, etc set? RESOLVED: These are all determined by the . These do not change for the life of . 4. How does eglSwapBuffers act if there are already image frames in the EGLStream when it is called. RESOLVED: Frames are inserted into the EGLStream as described in section "3.10.5 EGLStream operation" in the EGL_KHR_stream extension specification. In particular: If the value of EGL_STREAM_FIFO_LENGTH_KHR is 0 or if the EGL_KHR_stream_fifo extension is not supported then the new frame replaces any frames that already exist in the EGLStream. If the consumer is already consuming a frame then it continues to consume that same frame, but the next time the consumer begins to consume a frame (e.g. the next time eglStreamConsumerAcquireKHR() is called for a gltexture consumer) the newly rendered image frame will be consumed. (This is the standard behavior for ANY producer when EGL_STREAM_FIFO_LENGTH_KHR is 0, described as "mailbox mode"). If the EGL_KHR_stream_fifo extension is supported and the value of EGL_STREAM_FIFO_LENGTH_KHR is greater than 0 then the newly rendered frame will be inserted into the EGLStream. If the EGLStream is full (already contains EGL_STREAM_FIFO_LENGTH_KHR frames) then eglSwapBuffers will block until there is room in the fifo. Note that this can deadlock if the consumer is running in the same thread as the producer since the consumer will never be able to consume a frame if the thread is blocked waiting for room in the fifo. This fifo-related behavior is described in the EGL_KHR_stream_fifo specification (this behavior is not specific to this producer; it works the same for all producers and all consumers). All rendering commands must complete before the color buffer is inserted into the EGLStream, or at least this is how the behavior must appear to the application. To be precise: when eglSwapBuffers returns the rendering commands may or may not actually be complete, but the following must all be true: - The EGL_PRODUCER_FRAME_KHR value reflects the frame that was just swapped by eglSwapBuffers - The EGL_STREAM_STATE_KHR indicates that the image frame is available (i.e. its value is EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR) - In mailbox mode if the consumer consumes a new frame it will get this new frame (not an older frame). (For example, with a EGL_NV_stream_consumer_gltexture consumer, a call to eglStreamConsumerAcquireKHR() will latch this new frame.) - In fifo mode (see EGL_KHR_stream_fifo extension) if the consumer consumes a new frame and all previous frames have been consumed it will get this new frame (not an older frame). (For example, with a EGL_NV_stream_consumer_gltexture consumer, a call to eglStreamConsumerAcquireKHR() will latch this new frame.) - If a consumer consumes the swapped frame, all GL (and other API) commands called prior to eglSwapBuffers will take effect on the image frame before the consumer consumes it. In other words, the consumer will never consume a partially rendered frame. (For example, with EGL_NV_stream_consumer_gltexture consumer, if the app does this: eglSwapBuffers() // swap the producer EGLSurface eglStreamConsumerAcquireKHR() // acquire the swapped image glDrawArrays() // draw something using the texture then the texture used in the glDrawArrays() command will contain the image rendered by all gl (and/or other API) commands preceding the eglSwapBuffers call as if the app had called glFinish and/or eglWaitClient just before calling eglSwapBuffers (but note that this is implicit in eglSwapBuffers; the app does NOT need to actually call glFinish or any other synchronization functions in order to get this effect, and in fact explicitly calling glFinish and/or eglWaitClient there may significantly and negatively affect performance).) Revision History #11 (June 18. 2012) Acorn Pooley - Replace EGLStream with EGLStreamKHR in function prototypes. #10 (June 15, 2012) Acorn Pooley - Fix eglCreateStreamProducerSurfaceKHR name (was missing KHR) #9 (October 17, 2011) Acorn Pooley - Clarify issue 4 #8 (October 12, 2011) Acorn Pooley - remove interactions with EGL_KHR_stream_fifo extension (they are already decribed in that extension). #7 (October 11, 2011) Acorn Pooley - Add issue 4 - add changes to section 3.9 of the EGL spec to clarify eglSwapBuffer behavior #6 (October 4, 2011) Acorn Pooley - Convert from an NV extension to a KHR extension #5 (September 30, 2011) Acorn Pooley - Remove EGL_BUFFER_COUNT_NV (0x321D) attribute and resolve issue 1. #4 (September 27, 2011) Acorn Pooley - Assign enum values (bug 8064) #3 (July 6, 2011) Acorn Pooley - Rename EGL_KHR_image_stream to EGL_KHR_stream #2 (June 30, 2011) Acorn Pooley - remove dependence on EGLImage - clarify overview - remove glossary (it can be seen in EGL_KHR_stream ext) - Add EGL_STREAM_BIT - clarify description - describe attribute #1 (April 20, 2011) Acorn Pooley - initial draft # vim:ai:ts=4:sts=4:expandtab:textwidth=70