 |
OpenVX Neural Network Extension
7505566
|
This specification would not be possible without the contributions from this partial list of the following individuals from the Khronos Working Group and the companies that they represented at the time:
Deep Learning using Deep Convolutional Networks techniques is being increasingly used to perform vision classification and recognition tasks. Deep Convolutional Networks have significantly improved image recognition capabilities over previous technologies. The Neural Network extension for OpenVX is intended to enable the implementation of Deep Convolution Network in the OpenVX framework. It is well known that the Deep learning domain for vision, has two fundamental stages. At first the network topology is designed and trained given a collection of labelled data. The network topology is represented as a graph of several nodes comprising Deep Convolution Network building block. The trained data represents the problem to be addressed. During the training Phase, the parameters (also referred to as weights/biasses or coefficients) are determined for the given network topology. The network topology solution can then be deployed.
In Deployment the network topology as well as parameters are fixed which allow optimizing in hardware and software. In certain scenarios an additional intermediate step is performed to optimize the parameters to a certain target hardware. As an example, using fixed point calculations. When Deployed, the Deep Convolution Network is used for inferences on input data. The main objective of the Neural Network Extension for OpenVX is to enable the deployment phase (in other words inferences).
This section provides the definition of the basic terminology to be used across the document, in an attempt to address the various use and different naming in the academy as well as the industry. Those names refer to the same fundamental concept of Deep Convolutional Networks in the deep learning domain. We refer to the term Convolutional Network to the network topology of the deep learning network, that is composed of multiple layers in which one of the main layer is Convolution. Other names used in the academia and industry to refer to the same type of network topologies are CNN (Convolutional Neural Networks) and ConvNets. Throughout this document we will use the Convolution Network to refer to the Deep Convolution Network, CNN and ConvNet.
Weights - Will use the term Weights to refer to the parameters or coefficients that are the result of training the Convolution Network. Weights can be shared or non shared. Or have local connectivity.
Biasses - Will use the term Biasses to refer to the parameters or coefficients, per output only, that are the result of training the Convolution Network.
Convolution Layer - A type of layer in the Convolution Network that has local connectivity and shared weights, other naming are Locality connected with shared weights.
Fully Connected Layer - All inputs to the layer affect outputs of the layer , in other words connection from every element of input to every element of output.
Activation Layer - A layer that performs operations on every input data and is inspired by the neuron activation function approximated usually using non-Linear functions.
The documentation below uses the abbreviations IFM and OFM, which stand for “Input Feature Maps” and “Output Feature Maps,” respectively. Each feature map is a 2 dimensional image. A CNN input or output tensor will typically have 3 dimensions, where the first two are the width and height of the images, and the third is the number of feature maps. For inputs, the third dimension is the number of IFMs, and for outputs, the third dimension is the number of OFMs.
The Neural Networks extension enables execution and integration of Convolutional Networks in OpenVX processing graphs. The extension introduces the vx_tensor object, which is a multidimensional array with an arbitrary number of dimensions. The vx_tensor object can represent all varieties of data typically used in a Convolutional Network. It can represent 2-dimensional images, 3-dimensional sequences of images (usually the input and outputs of a Convolutional Network)and 4-dimensional weights.
OpenVX implementations compliant to this extension must support vx_tensor objects of at least 4 dimensions, although a vendor can choose to support more dimensions in his implementation. The maximum number of dimensions supported by a given implementation can be queried via the context attribute VX_CONTEXT_TENSOR_MAX_DIMENSION. Implementations must support tensors from one dimension (i.e., vectors) through VX_CONTEXT_TENSOR_MAX_DIMENSION, inclusive. The individual elements of the tensor object may be any numerical data type. A compliant implementations must support at least elements of type VX_TYPE_INT16. The VX_TYPE_INT16 elements can represent fractional values by assigning a non-zero radix point. Compliant implementations must support a radix point of 8, which corresponds to Q7.8 signed fixed-point in “Q” notation. A vendor may choose to support additional values for the radix point in his implementation.
Application can build an OpenVX graph that represents Convolution Network topologies where the layers are represented as OpenVX nodes (vx_node) and the vx_tensor as the data objects connecting the nodes (layers) of the OpenVX graph (Convolution Network). The application can as well build an OpenVX graph that is a mix of Convolution Network layers and Vision nodes. All graphs (including Convolution Networks) are treated as any OpenVX graph, and must comply with the graph concepts as specified in section 2.8 of OpenVX 1.1, especially but not limit to the graph formalisms in section 2.8.6. Additionally, this extension defines several auxiliary functions to create, release, and copy vx_tensor objects. Moreover, the extension introduces the concept of “view” for vx_tensor objects, which is similar to the ROI of a vx_image. The use of "view" enables splitting and merging vx_tensor objects, which are common operations in Convolutional Networks. The layers of the Convolutional Network (represented by vx_node objects) perform the computations on the tensor data objects and form a dataflow graph of computations. The extension defines the following layer types: convolution, activation, pooling, fully-connected, and soft-max.
It is assumed that the Convolutional Networks are trained in framework external to OpenVX and imported. This requires the application to allocate a memory area for the weights/biasses, read the weight values from a file into this memory area, and then use the vxCopyTensorPatch API to copy the weights/biasses from the memory area into the appropriate OpenVX Tensor object. The vxCopyTensorPatch function will convert the application memory to the implementation-specific format before putting it into the Tensor object. While effective, this method has the drawback that an intermediate memory area needs to be allocated and a copy and conversion needs to be done.
A separate “import/export” extension defines a vxImportBinary function that can be implemented more efficiently. Implementations of vxImportBinary could read a weight file or perhaps an entire graph description directly without the need for an intermediate copy. The format of this binary will be implementation-dependent. OpenVX implementations that support both the Neural Network extension and the binary import/export extension can use this more efficient method to set the Convolutional Networks weights/biasses. The vxImportBinary function will return a handle to an object that can be queried to get handles for the individual objects within it via the vxGetImportReferenceByName or vxGetImportReferenceByIndex functions. Further details and alternate usages of the vxImportBinary function are provided in the specification of the “import/export” extension.
1.8.6 
