Name String cl_intel_va_api_media_sharing Contributors Ben Ashbaugh, Intel Jacek Danecki, Intel Krzysztof Laskowski, Intel Contact Ben Ashbaugh, Intel (ben.ashbaugh 'at' intel.com) Version Version 1, December 8, 2014 Number OpenCL Extension #36 Status Final Draft Dependencies OpenCL 1.2 is required and a VA API implementation supporting sharing of surfaces with OpenCL is required. This extension is written against revision 19 of the OpenCL 1.2 specification and against revision 19 of the OpenCL 1.2 extension specification. Overview The goal of this extension is to provide interoperability between OpenCL and the cross-platform Video Acceleration API (VA API). The extension specifically enables sharing of VA API surfaces for media. It uses similar mechanisms and provides similar surface sharing capabilities for VA API that are provided by the Khronos OpenCL 1.2 Media Surface Sharing extension for DX9. New Procedures and Functions cl_int clGetDeviceIDsFromVA_APIMediaAdapterINTEL ( cl_platform_id platform, cl_va_api_device_source_intel media_adapter_type , void *media_adapter, cl_va_api_device_set_intel media_adapter_set, cl_uint num_entries, cl_device_id *devices, cl_uint *num_devices) cl_mem clCreateFromVA_APIMediaSurfaceINTEL( cl_context context, cl_mem_flags flags, VASurfaceID *surface , cl_uint plane, cl_int *errcode_ret) cl_int clEnqueueAcquireVA_APIMediaSurfacesINTEL( cl_command_queue command_queue, cl_uint num_objects, const cl_mem *mem_objects, cl_uint num_events_in_wait_list, const cl_event *event_wait_list, cl_event *event) cl_int clEnqueueReleaseVA_APIMediaSurfacesINTEL( cl_command_queue command_queue, cl_uint num_objects, const cl_mem *mem_objects, cl_uint num_events_in_wait_list, const cl_event *event_wait_list, cl_event *event) New Tokens Accepted as a VA API device source in the parameter of clGetDeviceIDsFromVA_APIMediaAdapterINTEL: CL_VA_API_DISPLAY_INTEL 0x4094 Accepted as a set of VA API devices in the parameter of clGetDeviceIDsFromVA_APIMediaAdapterINTEL: CL_PREFERRED_DEVICES_FOR_VA_API_INTEL 0x4095 CL_ALL_DEVICES_FOR_VA_API_INTEL 0x4096 Accepted as a property name in the parameter of clCreateContext and clCreateContextFromType: CL_CONTEXT_VA_API_DISPLAY_INTEL 0x4097 Accepted as the property being queried in the parameter of clGetMemObjectInfo: CL_MEM_VA_API_MEDIA_SURFACE_INTEL 0x4098 Accepted as the property being queried in the parameter of clGetImageInfo: CL_IMAGE_VA_API_PLANE_INTEL 0x4099 Returned in the parameter of clGetEventInfo when is CL_EVENT_COMMAND_TYPE: CL_COMMAND_ACQUIRE_VA_API_MEDIA_SURFACES_INTEL 0x409A CL_COMMAND_RELEASE_VA_API_MEDIA_SURFACES_INTEL 0x409B Returned by clCreateContext and clCreateContextFromType if the VA API display specified for interoperability is not compatible with the devices against which the context is to be created: CL_INVALID_VA_API_MEDIA_ADAPTER_INTEL -1098 Returned by clCreateFromVA_APIMediaSurfaceINTEL when is not a VA API surface of the required type, by clGetMemObjectInfo when is CL_MEM_VA_API_MEDIA_SURFACE_INTEL when was not created from a VA API surface, and from clGetImageInfo when is CL_IMAGE_VA_API_PLANE_INTEL and was not created from a VA API surface: CL_INVALID_VA_API_MEDIA_SURFACE_INTEL -1099 Returned by clEnqueueAcquireVA_APIMediaSurfacesINTEL when any of are currently acquired by OpenCL: CL_VA_API_MEDIA_SURFACE_ALREADY_ACQUIRED_INTEL -1100 Returned by clEnqueueReleaseVA_APIMediaSurfacesINTEL when any of are not currently acquired by OpenCL: CL_VA_API_MEDIA_SURFACE_NOT_ACQUIRED_INTEL -1101 Additions to Chapter 4 of the OpenCL 1.2 Specification In section 4.4, replace the description of under clCreateContext with: " specifies a list of context property names and their corresponding values. Each property is followed immediately by the corresponding desired value. The list is terminated with zero. If a property is not specified in , then its default value (listed in table 4.4) is used (it is said to be specified implicitly). If is NULL or empty (points to a list whose first value is zero), all attributes take on their default values." Add the following to table 4.5: "-------------------------------------------------------------------------------------- cl_context_properties Value type Default value Description --------------------- ---------- ------------- ----------- CL_CONTEXT_VA_API_DISPLAY_INTEL VADisplay* NULL Specifies a VA display to use for VA API interop. --------------------------------------------------------------------------------------" Add to the list of errors for clCreateContext: "* CL_INVALID_VA_API_MEDIA_ADAPTER_INTEL if the value of the property CL_CONTEXT_VA_API_DISPLAY_INTEL is non-NULL and does not specify a valid VA API display with which the cl_device_ids against which this context is to be created may interoperate. * CL_INVALID_OPERATION if VA API interoperability is specified by setting CL_CONTEXT_VA_API_DISPLAY_INTEL to a non-NULL value, and interoperability with any other graphics API is also specified. " Add to the list of errors for clCreateContextFromType the same new errors described above for clCreateContext. Additions to Chapter 5 of the OpenCL 1.2 Specification 5.4.5 Memory Object Queries Add to the list of errors for clGetMemObjectInfo: "* CL_INVALID_VA_API_MEDIA_SURFACE_INTEL if is CL_MEM_VA_API_MEDIA_SURFACE_INTEL and was not created from a VA API surface using clCreateFromVA_APIMediaSurfaceINTEL." Extend table 5.11 to include the following entries: -------------------------------------------------------------------------- cl_mem_info Return type Info. returned in param_value ----------- --------------- ----------------------------- CL_MEM_VA_API_MEDIA_ VASurfaceID* Returns the argument SURFACE_INTEL value specified when was created. --------------------------------------------------------------------------- Add to the list of errors for clGetImageInfo: "* CL_INVALID_VA_API_MEDIA_SURFACE_INTEL if is CL_IMAGE_VA_API_PLANE_INTEL and was not created from a VA API surface using clCreateFromVA_APIMediaSurfaceINTEL." Extend table 5.9 to include the following entry: ------------------------------------------------------------------------ cl_image_info Return type Info. returned in param_value ------------- ----------- ----------------------------- CL_IMAGE_VA_API_PLANE_INTEL cl_uint Returns the argument value specified when was created. ------------------------------------------------------------------------ 5.9 Event Objects Add to Table 5.18 in the "Info returned in " column for cl_event_info CL_EVENT_COMMAND_TYPE: CL_COMMAND_ACQUIRE_VA_API_MEDIA_SURFACES_INTEL CL_COMMAND_RELEASE_VA_API_MEDIA_SURFACES_INTEL Add new section 9.X to the OpenCL 1.2 extension specification: 9.X Sharing VA API Media Surfaces with OpenCL This section discusses OpenCL functions that allow applications to use VA API media surfaces as OpenCL memory objects. This allows efficient sharing of data between OpenCL and VA API. The OpenCL API may be used to execute kernels that read and/or write memory objects that are also VA API surfaces. An OpenCL image object may be created from a VA API surface. OpenCL memory objects may be created from VA API surfaces if and only if the OpenCL context has been created from a VA API display. 9.X.1 Querying OpenCL Devices Corresponding to VA API Displays The OpenCL devices corresponding to a VA API display may be queried using the function cl_int clGetDeviceIDsFromVA_APIMediaAdapterINTEL( cl_platform_id platform, cl_va_api_device_source_intel media_adapter_type, void *media_adapter, cl_va_api_device_set_intel media_adapter_set, cl_uint num_entries, cl_device_id *devices, cl_uint *num_devices) refers to the platform ID returned by clGetPlatformIDs. specifies the type of , and must be one of the values shown in table 9.X.1.1. specifies the media adapter whose corresponding OpenCL devices are being queried. The type of must be as specified in table 9.X.1.1. specifies the set of devices to return and must be one of the values shown in table 9.X.1.2. is the number of cl_device_id entries that can be added to . If is not NULL then must be greater than zero. returns a list of OpenCL devices found. The cl_device_id values returned in can be used to identify a specific OpenCL device. If is NULL, this argument is ignored. The number of OpenCL devices returned is the minimum of the value specified by and the number of OpenCL devices corresponding to . returns the number of OpenCL devices available that correspond to . If is NULL, this argument is ignored. clGetDeviceIDsFromVA_APIMediaAdapterINTEL returns CL_SUCCESS if the function is executed successfully. Otherwise it may return: * CL_INVALID_PLATFORM if is not a valid platform, * CL_INVALID_VALUE if is not a valid value, is not a valid value, is equal to zero and is not NULL, or if both and are NULL. * CL_DEVICE_NOT_FOUND if no OpenCL devices that correspond to were found. -------------------------------------------------------------------- cl_va_api_device_source_intel Type of ----------------------------- ----------------------- CL_VA_API_DISPLAY_INTEL VADisplay* -------------------------------------------------------------------- Table 9.X.1.1: Types used to specify the object whose corresponding OpenCL devices are being queried by clGetDeviceIDsFromVA_APIMediaAdapterINTEL. -------------------------------------------------------------------- cl_va_api_device_set_intel Devices returned in -------------------------- ----------------------------- CL_PREFERRED_DEVICES_FOR_VA_API_INTEL The OpenCL devices associated with the specified media adapter. CL_ALL_DEVICES_FOR_VA_API_INTEL All OpenCL devices which may interoperate with the specified media adapter. Performance of sharing data on these devices may be considerably less than on the preferred devices. -------------------------------------------------------------------- Table 9.X.1.2: Types used to specify the set of devices returned from clGetDeviceIDsFromVA_APIMediaAdapterINTEL. 9.X.2 Lifetime of Shared Resources An OpenCL memory object created from a VA API surface remains valid as long as the corresponding VA API surface has not been deleted. If the VA API surface is deleted through the VA API, subsequent use of the OpenCL memory object will result in undefined behavior, including but not limited to possible OpenCL errors, data corruption, and program termination. 9.X.4 Sharing VA API Media Surfaces as OpenCL Image Objects The function cl_mem clCreateFromVA_APIMediaSurfaceINTEL( cl_context context, cl_mem_flags flags, VASurfaceID *surface, cl_uint plane, cl_int *errcode_ret) creates an OpenCL 2D image object from a VA API media surface or a plane of a VA API media surface. is a valid OpenCL context created from a VA API display. is a bit-field that is used to specify usage information. Refer to table 5.3 for a description of . Only CL_MEM_READ_ONLY, CL_MEM_WRITE_ONLY and CL_MEM_READ_WRITE values specified in table 5.3 can be used. is a pointer to the VA API surface to share. is the plane of to share, for planar formats. For non-planar formats, must be 0. will return an appropriate error code. If is NULL, no error code is returned. clCreateFromVA_APIMediaSurfaceINTEL returns a valid non-zero OpenCL 2D image object and is set to CL_SUCCESS if the OpenCL 2D image object is created successfully. Otherwise, it returns a NULL value with one of the following error values returned in : * CL_INVALID_CONTEXT if is not a valid context. * CL_INVALID_VALUE if values specified in are not valid or if is not a valid plane of . * CL_INVALID_VA_API_MEDIA_SURFACE_INTEL if is not a VA API surface, if a cl_mem from and has already been created, or if was not created against the same VA API display from which was created. * CL_INVALID_IMAGE_FORMAT_DESCRIPTOR if the format of is not listed in table 9.X.4.1. * CL_INVALID_OPERATION if there are no devices in that support VA API sharing. * CL_OUT_OF_RESOURCES if there is a failure to allocate resources required by the OpenCL implementation on the device. * CL_OUT_OF_HOST_MEMORY if there is a failure to allocate resources required by the OpenCL implementation on the host. The width and height of the returned OpenCL 2D image object are determined by the width and height of the of . The channel type and order of the returned OpenCL 2D image object is determined by the format and of in table 9.X.4.1. ------------------------------------------------------------------ VA_FOURCC cl_channel_order cl_channel_type ------------------------------ ---------------- --------------- VA_FOURCC_NV12,Plane0 CL_R CL_UNORM_INT8 VA_FOURCC_NV12,Plane1 CL_RG CL_UNORM_INT8 ------------------------------------------------------------------ Table 9.X.4.1: List of VA API FourCC codes and corresponding OpenCL Image Formats. For the NV12 surface format, plane 0 corresponds to the luminance (Y) channel data and plane 1 corresponds to the chrominance (UV) channel data. 9.X.5 Querying Properties of Memory Objects Created From VA API Surfaces Properties of media objects created from VA API surfaces may be queried using clGetMemObjectInfo and clGetImageInfo with CL_MEM_VA_API_MEDIA_SURFACE_INTEL and CL_IMAGE_VA_API_PLANE_INTEL as described in section 5.3.7 and 5.4.5. 9.X.6 Sharing Memory Objects Created From VA API Surfaces Between VA API and OpenCL The function cl_int clEnqueueAcquireVA_APIMediaSurfacesINTEL( cl_command_queue command_queue, cl_uint num_objects, const cl_mem *mem_objects, cl_uint num_events_in_wait_list, const cl_event *event_wait_list, cl_event *event) is used to acquire OpenCL memory objects that have been created from VA API surfaces. The VA API surfaces are acquired by the OpenCL context associated with and can therefore be used by all command- queues associated with the OpenCL context. OpenCL memory objects created from VA API surfaces must be acquired before they can be used by any OpenCL commands queued to a command-queue. If an OpenCL memory object created from a VA API surface is used while it is not acquired by OpenCL, the call attempting to use that OpenCL memory object will return CL_VA_API_MEDIA_SURFACE_NOT_ACQUIRED_INTEL. If CL_CONTEXT_INTEROP_USER_SYNC is not specified as CL_TRUE during context creation, clEnqueueAcquireVA_APIMediaSurfacesINTEL provides the synchronization guarantee that any VA API calls made before clEnqueueAcquireVA_APIMediaSurfacesINTEL is called will complete executing before reports completion and before the execution of any subsequent OpenCL work issued in begins. If the context was created with properties specifying CL_CONTEXT_INTEROP_USER_SYNC as CL_TRUE, the user is responsible for guaranteeing that any VA API calls involving the interop device(s) used in the OpenCL context made before clEnqueueAcquireVA_APIMediaSurfacesINTEL is called have completed before calling clEnqueueAcquireVA_APIMediaSurfacesINTEL. is a valid command-queue. is the number of memory objects to be acquired in . is a pointer to a list of OpenCL memory objects that were created from VA API surfaces. and specify events that need to complete before this particular command can be executed. If is NULL, then this particular command does not wait on any event to complete. If is NULL, must be 0. If is not NULL, the list of events pointed to by must be valid and must be greater than 0. The events specified in act as synchronization points. returns an event object that identifies this particular command and can be used to query or queue a wait for this particular command to complete. can be NULL in which case it will not be possible for the application to query the status of this command or queue a wait for this command to complete. clEnqueueAcquireVA_APIMediaSurfacesINTEL returns CL_SUCCESS if the function is executed successfully. If is 0 and is NULL then the function does nothing and returns CL_SUCCESS. Otherwise it returns one of the following errors: * CL_INVALID_VALUE if is zero and is not a NULL value or if > 0 and is NULL. * CL_INVALID_MEM_OBJECT if memory objects in are not valid OpenCL memory objects or if memory objects in have not been created from VA API surfaces. * CL_INVALID_COMMAND_QUEUE if is not a valid command-queue. * CL_INVALID_CONTEXT if the context associated with was not created from a VA API display. * CL_VA_API_MEDIA_SURFACE_ALREADY_ACQUIRED_INTEL if memory objects in have previously been acquired using clEnqueueAcquireVA_APIMediaSurfacesINTEL but have not been released using clEnqueueReleaseVA_APIMediaSurfacesINTEL. * CL_INVALID_EVENT_WAIT_LIST if is NULL and > 0, or is not NULL and is 0, or if event objects in are not valid events. * CL_OUT_OF_RESOURCES if there is a failure to allocate resources required by the OpenCL implementation on the device. * CL_OUT_OF_HOST_MEMORY if there is a failure to allocate resources required by the OpenCL implementation on the host. The function cl_int clEnqueueReleaseVA_APIMediaSurfacesINTEL( cl_command_queue command_queue, cl_uint num_objects, const cl_mem *mem_objects, cl_uint num_events_in_wait_list, const cl_event *event_wait_list, cl_event *event) is used to release OpenCL memory objects that have been created from VA API surfaces. The VA API surfaces are released by the OpenCL context associated with . OpenCL memory objects created from VA API surfaces which have been acquired by OpenCL must be released by OpenCL before they may be accessed by VA API. Accessing a VA API surface while its corresponding OpenCL memory object is acquired is in error and will result in undefined behavior, including but not limited to possible OpenCL errors, data corruption, and program termination. If CL_CONTEXT_INTEROP_USER_SYNC is not specified as CL_TRUE during context creation, clEnqueueReleaseVA_APIMediaSurfacesINTEL provides the synchronization guarantee that any VA API calls made after clEnqueueReleaseVA_APIMediaSurfacesINTEL is called will not start executing until after all events in are complete and all work already submitted to completes execution. If the context was created with properties specifying CL_CONTEXT_INTEROP_USER_SYNC specified as CL_TRUE, the user is responsible for guaranteeing that any VA API calls involving the interop device(s) used in the OpenCL context made after clEnqueueReleaseVA_APIMediaSurfacesINTEL is called will not start executing until after the event returned by clEnqueueReleaseVA_APIMediaSurfacesINTEL reports completion. is the number of memory objects to be released in . is a pointer to a list of OpenCL memory objects that were created from VA API surfaces. and specify events that need to complete before this particular command can be executed. If is NULL, then this particular command does not wait on any event to complete. If is NULL, must be 0. If is not NULL, the list of events pointed to by must be valid and must be greater than 0. The events specified in act as synchronization points. returns an event object that identifies this particular command and can be used to query or queue a wait for this particular command to complete. can be NULL in which case it will not be possible for the application to query the status of this command or queue a wait for this command to complete. clEnqueueReleaseVA_APIMediaSurfacesINTEL returns CL_SUCCESS if the function is executed successfully. If is 0 and is NULL the function does nothing and returns CL_SUCCESS. Otherwise it returns one of the following errors: * CL_INVALID_VALUE if is zero and is not a NULL value or if > 0 and is NULL. * CL_INVALID_MEM_OBJECT if memory objects in are not valid OpenCL memory objects or if memory objects in have not been created from VA API surfaces. * CL_INVALID_COMMAND_QUEUE if is not a valid command-queue. * CL_INVALID_CONTEXT if the context associated with was not created from a VA API display. * CL_VA_API_MEDIA_SURFACE_NOT_ACQUIRED_INTEL if memory objects in have not previously been acquired using clEnqueueAcquireVA_APIMediaSurfacesINTEL, or have already been released using clEnqueueReleaseVA_APIMediaSurfacesINTEL since the last time that they were acquired. * CL_INVALID_EVENT_WAIT_LIST if is NULL and > 0, or is not NULL and is 0, or if event objects in are not valid events. * CL_OUT_OF_RESOURCES if there is a failure to allocate resources required by the OpenCL implementation on the device. * CL_OUT_OF_HOST_MEMORY if there is a failure to allocate resources required by the OpenCL implementation on the host. Issues None Revision History Version 1 (2014/12/08): First public revision.