Neural Network Exchange Format

For an exporter of NNEF, such as a deep learning framework, NNEF is a format to which its internal network representation can be converted, and afterwards all accelerator libraries that are able to consume NNEF will be able to execute the trained network (if the network matches its capabilities). The task of an exporter is to map its operations and data representation to the operations and data representation of NNEF, given that this mapping is possible.

NNEF also aims to be a distilled collection of deep learning operations that are widespread in successful neural architectures. It is the result of studying open-source deep learning frameworks such as Caffe, Torch, Theano, TensorFlow, CNTK, Chainer, and abstracting out the computations and data structures common to them. It mainly focuses on operations that are possibly efficiently implementable on various target hardware, such as massively parallelizable operations, especially 'local' ones that require a locally concentrated subset of the input data to compute outputs.

The importer of NNEF, such as a neural network accelerator library (and its underlying hardware), is able to import an NNEF document, and compile it to its internal representation ready for execution. This compilation may happen offline or online. During offline compilation, NNEF may be converted to an optimized, hardware specific representation format, which may be saved and later be quickly loaded for execution. During online compilation, the conversion and optimization may happen without saving it into a hardware specific format, but immediately executing the converted network.

NNEF collects operations into groups to indicate relatedness of operations. This may serve as a hint or guideline for hardware implementations, as related operations may require similar hardware capabilities.

For an application programmer, NNEF is a standardized way to store and transfer neural networks. Given a neural network in NNEF format, and a driver or library that is able to import it, the application programmer need not worry about where the network came from or what kind of underlying hardware will execute it, as long as it has the capabilities to do so. The application programmer may query the driver of the underlying hardware whether it is capable of executing the given network.

NNEF is not an API (Application Programming Interface). It does not define an execution model for neural networks, and hence it does not define what it means to correctly execute a neural network described in NNEF. Although it does define the semantics of operations supposing infinite arithmetics, defining correct execution of actual implementations would require finite arithmetics and underlying representations to be taken into account, which is out of the scope of NNEF.

Libraries that produce or consume NNEF may have various APIs. However, importantly for application programmers, an NNEF consumer that intends to execute a neural network will most probably have functionalities to import and compile a network described in NNEF, and feed that network with data afterwards. However, the exact nature of this API is out of the scope of NNEF. One such API is described by the OpenVX Khronos standard’s neural network extension, along with an execution model of neural networks.

The compilation of a computational graph starts from:

On a conceptual level, the compilation process may have the following steps:

An implementation is not restricted to compile a graph of primitive operations as listed in Operations, but instead it may implement operations such as those in the Compound Operations as atomic ones for improved efficiency. After building an executable graph, an implementation may optimize it for example by removing intermediate buffers and unnecessary operations or merging sequences of operations.

After the compilation process, the graph execution may have the following steps:

Note again that the exchange format does not define an execution model of the computational graph being described. The above is just an example of how it might be used.

A graph definition consists of a graph declaration and its body. The graph declaration has a list of parameters and results.

The graph definition itself consists of a list of assignments, where the left-hand-side of the assignment is an identifier expression (single identifier, tuple or array), and the right-hand-side is an operation invocation.

An invocation consists of an identifier and a list of arguments:

Expressions may be literals, identifiers, arrays and tuples. It is necessary to differentiate between left-value and right-value expressions.

Left-value expressions are allowed on the left-hand-side of assignments:

Right-value expressions are allowed on the right-hand-side of assignments (as argument values):

Invocations may have multiple results (if the operation defines multiple results). In this case, the returned expression is a tuple, and the left-hand-size expression must also be a tuple.

As an example, using the declarations above, we may define part of a graph as:

In the above example, external is an operation used to introduce tensors that receive their data from an external source (see Tensor Introducing Operations), and exemplary operations bar and foo are defined below.

The following syntax elements must be enabled by the extension KHR_enable_fragment_definitions.

A fragment is similar to the graph in that its body is defined by a list of assignments, but its declaration allows typed formal parameters and results.

An fragment declaration is introduced by the fragment keyword, has a name, a parameter list and a result list. Parameters are explicitly typed and may have default values. Results are introduced after the -> symbol.

Default values are literal expressions built from literals only:

A type specification may denote a primitive type, an array type or a tuple type.

For example, the following lines show some operation declarations:

The result types of fragments must be all tensors.

The following syntax elements must be enabled by the extension KHR_enable_operator_expressions.

The syntax enables more complex expressions to be used as right-value expressions. These expressions allow for compile-time arithmetic and argument composition.

Various arithmetic, comparison and logical operators can be used to build binary expressions:

A handful of unary operators are also available for arithmetic and logical expressions:

Operator expressions can then be built using unary and binary operators and parenthesizing:

The if-else expression implements branching by selecting one of two expressions depending on a condition.

The array comprehension expression implements a form of looping. It generates an array by iterating one or more others, optionally filtering the resulting items.

Subscripting expressions can reference a single entry in an array, or a range of entries, in which case the start (inclusive) and end (exclusive) of the range is separated by :. Both the start and the end are optional, in which case 0 or the length of the array is taken, respectively.

A few special keywords can be used for built-in functions.

Finally, extended right-value expressions are the union of all the above constructions (including the ones in the definition of basic right-value expressions and invocations). Assignments may contain any right-value expression, not only invocations:

The NNEF structure description consists of a version info, an optional list of extensions used, an optional list of operation definitions and a top-level graph definition. An NNEF document that does not contain operation definitions and extended expressions is said to be flat. A graph definition must be always present.

The version info is introduced by the version keyword, and is defined by a real-number numeric literal as major and minor versions separated by a dot:

Extensions can be specified after the extension keyword:

Here is a short example for illustration:

Note, that it is possible to build networks solely from predefined operations (see Operations), which need not be defined in an actual document description. Therefore, fragment definitions are usually unnecessary. Furthermore, the graph definition body can usually be written without using operator expressions; they are most useful for defining compound operations. Hence, networks without custom operations can usually be written using flat syntax only.

In the grammar defined in Syntax, formal parameters of operations are explicitly typed. Furthermore, literal constants also have an implicitly defined type. The types of expressions are derived from the types of their arguments.

Types can either be primitive or compound. Compound types are composed from primitives or from other compound types.

The expressions described here mainly use extended syntax (except for simple array and tuple construction). The purpose of such expressions is two-fold:

Thus, expressions are either of frequently used arithmetic, comparison or logical operators, or serve the purpose of manipulating a data structure, such as an array or a tuple to serve as arguments for operations. When manipulating data structures, the typical set of operations that are required are building a data structure from its components or parts, and dissecting a data structure to its components or parts.

The following subsections describe built-in operators and data-structure manipulator notations.

Certain tensors need to be referenced from outside the main description to be able to attach further information to the graph (such as parameter data or quantization information). There are two types of tensors that are possible to reference: variables and activation tensors.

Variables are declared with an explicit string label (see Variables), and this label must be globally unique (except for shared weights), hence it can be used for referencing a variable. This label is used for example to attach tensor data to variables that are serialized.

Activation tensors in the graph body also have a globally unique identifier (the left-hand side of assignments must have previously unused names), hence these can also be used to reference activation tensors, for example to attach quantization information to activations.

Note, that variables can also be part of the main graph description, and hence they may be referenced by two mechanisms (the string label of the variable definition, and the identifier name used in the assignment).

In the above example, the names 'variable42' and 'block1/conv1/weights' refer to the same tensor, albeit for different purposes.

An external data source is a tensor which must be fed to the graph from the outside.

Argument validity

Result semantics

The content of output is not defined by the operation. The tensor must be fed with input data before each execution of the graph. The content of the tensor is not expected to persist between subsequent invocations of the computational graph.

A constant is a tensor that has a fixed value.

Argument validity

Result semantics

Note, that a constant tensor of singular dimensions can be simply written as a numeric literal directly into expressions, so y = x + constant(shape = [1], value = [3.14]) is equivalent to y = x + 3.14, where x is a tensor of arbitrary shape. If the length of value is 1, that single value is repeated, so constant(shape = [1,3], value = [3.14]) is equivalent to constant(shape = [1,3], value = [3.14, 3.14, 3.14]).

A variable is a tensor that may be fed an initial value, may be updated by a computation, and its value persists between consecutive invocations of the computational graph. When a tensor is introduced as a variable, it is possible to update it later (see Variable Updates).

Argument validity

Result semantics

The content of output is not defined by the operation. The tensor may be filled with data from an external source and may be updated by update operations. The label argument is used to link the result to externally stored tensor data.

Unary operations have a single tensor argument and return a single result.

Arithmetic operations:

Logical operations:

Rounding operations:

Result semantics

For the operation neg, the output is defined as:

For the operation rcp, the output is defined as:

For the operation exp, the output is defined as:

For the operation log, the output is defined as:

For the operation sin, the output is defined as:

For the operation cos, the output is defined as:

For the operation abs, the output is defined as:

For the operation sign, the output is defined as:

For the operation floor, the output is defined as:

For the operation ceil, the output is defined as:

For the operation round, the output is defined as:

Binary operations take two tensor arguments and produce a single result. In the basic case of binary operations, by default, the shapes of both input tensors are the same, and the resulting output will also have the same shape. However, binary operations also support the case when the either operand has singleton shape in a dimension but the other operand is non-singleton; in this case the value of the singleton dimension is repeated (broadcast) along that dimension (for example adding a column vector to all columns of a matrix). An extreme case is a scalar operand repeated in all dimensions.

Arithmetic operations:

Comparison operations:

Logical operations:

Argument validity

Result semantics

The ternary operation select returns either of two values based on a condition (per element).

Argument validity

Result semantics

The selection condition is evaluated independently for each entry (if condition shape is not singular). The output equals true_value where condition is true and false_value otherwise. Arguments of singular shape are broadcast to the shape of output.

As a special case, the condition may evaluate to true or false for all items (in runtime), in which case the whole subgraph calculating true_value or false_value can be omitted (in runtime), which provides a chance for optimization (conditional execution). This is especially useful if the shape of condition is singular in all dimensions. For example:

In the above example, if condition evaluates to false, the whole calculate_more_outputs() invocation can be omitted, which may contain an arbitrary large sub-graph.

Some convenience operations are provided for often-used element-wise operations that can be expressed via other primitives.

The conv operation correlates a filter with an input, while the deconv operation performs the reverse, which is roughly equivalent to mathematical convolution. Up and down-scaling of tensor shapes happens only in the spatial dimensions. The batch dimension is the same for inputs and outputs, while the channel dimension of the output is derived from the batch dimension of the filters.

Argument validity

Result semantics

Operation conv maps from the up-sampled space to the down-sampled space, while deconv maps from the down-sampled space to the up-sampled space.

For the sake of explanation, suppose that the input and filter tensors are one dimensional (there is only one spatial dimension, batch and channel dimensions are singular). Furthermore, let bias be omitted (as if it was zero). Then for each index \(i \in [0,x)\), the output of conv is calculated as (mathematical correlation):

The deconv operation implements a calculation as if it was the reverse of correlation (i.e. mathematical convolution), taking stride and dilation into account. For each index \(i \in [0,X)\), the output of deconv is calculated as:

where the sum only includes terms for which \((i + p - j \cdot d) \mod s \ = \ 0\).

In general, the up-scaled space has dimensions \((B,C,X_1,X_2,...)\), the down-scaled space has shape \((B,c,x_1,x_2,...)\), and the filter has dimensions \((c,C,f_1,f_2,...)\). The following equations will suppose two spatial dimensions, but generalization to more dimensions is straightforward.

In case of the conv operation, for each batch index \(b \in [0..B)\) and for each \(k_2 \in [0..c)\), the output is calculated as:

and in case of deconv, for each \(k_1 \in [0..C)\):

When taking bias into account, the output is defined as if bias was added to the result of convolution without bias:

Box filtering performs summation in a local window.

Argument validity

Result semantics

Box filtering in spatial dimensions is equivalent to (depth-wise separable) convolution with a weight tensor of all 1s. Note however, that box filtering does not differentiate batch and channel dimensions, and it may also be applied to those dimensions.

For the sake of explanation, suppose that the input tensor is one dimensional. Then in the unnormalized case, for each index \(i \in [0,x)\), the output of box is calculated as:

The debox operation implements a calculation as if it was the reverse of box, taking stride and dilation into account. For each index \(i \in [0,X)\), the output of debox is calculated as:

where the sum only includes terms for which \((i + p - j \cdot d) \mod s \ = \ 0\).

The box filter is separable in the dimensions, so box filtering in multiple dimensions is equivalent to a sequence of 1-dimensional box filters.

The operation argmax_pool returns the maximum value positions in the local window.

Argument validity

Result semantics

For each position in the output space, argmax_pool chooses the maximum value in the kernel window, and outputs its index within the window. Indices correspond to row-major ordering of positions within the window. For the sake of explanation, suppose that the input tensor is one dimensional. Then for each index \(i \in [0,x)\), the output of argmax_pool is calculated as:

If there are multiple equal maximal values, only the first index is stored in index. Note, that the argmax operation is not separable.

Two related operations are sample and desample. The operation sample performs sampling given the indices, obtained from the argmax_pool operation. That is, it fetches the actual values indicated by the indices. The operation desample performs the reverse, populating the up-scaled space from the values in the down-scaled space using the sampling indices, summing values that are mapped to the same up-scaled position.

Argument validity

Result semantics

For the sake of explanation, suppose that the input and index tensors are one dimensional. Then for each index \(i \in [0,x)\), the output of sample is calculated as:

For each index \(i \in [0,X)\), the output of desample is calculated as:

where the sum only includes terms for which \((i + p - j \cdot d) \mod s \ = \ 0\). Note, that any border value other than 'constant' (zeros) loses its utility because the condition \(\textrm{index}[(i + p - j \cdot d) \ / \ s] \ = \ j\) on the sampling indices is never met outside the valid region.

Tensors may be up and down-sampled along the spatial dimensions. For down-sampling, two methods are given: area interpolation and nearest neighbor interpolation. For up-sampling, also two methods are given: multi-linear interpolation and nearest neighbor interpolation. Only integer scaling factors are allowed, such that these methods are consistent with other sliding window operations. Some of these operations are readily expressible via other primitives.

The two down-sampling methods are:

The two up-sampling methods are:

Argument validity

Result semantics

Multi-linear up-sampling weighs input values to form an output value. In a single dimension, the weighting happens according to one of various schemes. Let \(f\) be the sampling factor, and let us assume that the input is one dimensional. In the basic scheme, for output index \(i\), let \(j(i) = \textrm{floor} \big ( \frac{i}{f} \big )\) and let \(u(i) = \textrm{frac} \big ( \frac{i}{f} \big )\). Then the output is calculated as:

When \(j(i)+1\) equals the input extent \(x\), the index \(j(i)+1\) is clamped to \(x - 1\), so index \(j(i)\) is used, which results in \(\textrm{output}[i] = \textrm{input}[j(i)]\) for multiple values of \(i\) on the higher end of the output range. For this reason, this method is called 'asymmetric'.

To correct this asymmetric edge effect, various methods exist. One is to change the weighting such that the indices need not be clamped, by aligning the edges of the input and output intervals. Define \(g(x) = \frac{x \cdot f - 1}{x - 1}\), and define \(j(i) = \textrm{floor} \big ( \frac{i}{g(x)} \big )\) and \(u(i) = \textrm{frac} \big ( \frac{i}{g(x)} \big )\). This way, only the last point of the output interval maps to the last point in the input interval, hence the method is called 'aligned'. A potential disadvantage of this method is that the weighting depends on the input extent \(x\).

An alternative solution is to make the edge effect symmetric, by shifting the index calculation, resulting in the method called 'symmetric'. Let \(f_0 = \frac{f - 1}{2}\), and define \(j(i) = \textrm{floor} \big ( \frac{i - f_0}{f} \big )\) and \(u(i) = \textrm{frac} \big ( \frac{i - f_0}{f} \big )\). When \(j = -1\) or \(j + 1 = x\), the index is clamped to \(0\) or \(x - 1\) respectively.

Clamping along the edges corresponds to border = 'replicate'. Alternatively, the values outside the input range may be taken as zeros, which corresponds to border = 'constant'.

When method = 'symmetric' or method = 'asymmetric', multilinear upsampling can be described as a (depth-wise separable) deconvolution with constant weights that only depend on the upsampling factor, and are uniform in the whole input range. As special cases, here are the equivalents for the linear and bilinear case when the sampling factor is uniformly 2. Note, that these definitions do not form part of the standard, they are only shown here for convenience.

Argument validity

Result semantics

The reshape operation considers the content of the tensor to be laid out linearly in row-major order. Reshaping does not affect the content of the tensor, instead it only provides a new indexing to the same data. For this reason, the amount of data indexed must be the same under the two (original and new) indexing schemes.

An often used case of reshaping is when singleton dimensions are removed or inserted, named squeeze and unsqueeze respectively. Specialized fragments for these cases avoid mentioning the explicit shape of tensors:

The axes parameter denotes the list of dimension indices to be removed in case of squeeze and in case of unsqueeze, axes denote the dimension indices as they will appear in the output after insertion.

Argument validity

Result semantics

Transposing a tensor reorganizes its data. It is the n dimensional generalization of matrix transpose, allowing a permutation of the dimensions, and hence changing the ordering of the data.

Let \(k \in [0..n)\) and let \(x_k\) denote the extent of the input in dimension \(k\). In \(n\)-dimensional indexing notation, where index \(i_k \in [0..x_k)\), the output can be described as:

Splitting and concatenation convert between a single tensor and sections of it, separated along one of the dimensions. That is, they convert between a (conceptually) single chunk of data and many chunks of data.

Argument validity

Result semantics

Let \(n\) be the number of items in ratios, \(r_i\) equal ratios[i], and \(R = \sum_{i=0}^n r_i\). Furthermore, let \(S\) equal the shape of value along dimension axis, \(\mu = S / R\), and let \(s_i = \mu \cdot r_i\)

Argument validity

Result semantics

To precisely define the relation between the split values and the concatenated value for both split and concat operations, let \(\sigma_0 = 0\) and for \(i \in [1..n]\), let \(\sigma_i = \sum_{j=0}^{i-1} s_j\). For the sake of argument, suppose that value and values[i] are of rank 3 and axis = 1. Then, the values in values[i] are equal to the values in the section of value that runs from \(\sigma_{i}\) to \(\sigma_{i+1}\) along dimension axis:

where \(\sigma_i:\sigma_{i+1}\) means the section of the tensor from index \(\sigma_i\) (inclusive) to index \(\sigma_{i+1}\) (exclusive) along a given dimension, and the sole \(:\) means the whole section of the tensor along the given dimension.

A special case of concatenation and splitting is when an array of tensors are concatenated/split along a new (singular) dimension. These operations are called stacking and unstacking:

Stacking is conceptually equivalent to inserting a singleton dimension to each tensor at position given by axis, and then concatenating them along the axis. Unstacking is its reverse, splitting along the given axis to singleton slices, and then removing the singleton dimension.

Slicing takes a section of a tensor along some specified dimensions.

Argument validity

Result semantics

For the sake of explanation, let the input be 3 dimensional, and let axes = [1]. Supposing positive values in begin and end, the output is calculated as:

Padding adds extra border to the tensor contents, filling it according to the specified scheme.

Argument Validity

Result semantics

The value of output is the same as that of input, except on the borders of size (\(p_k,q_k\)), where the value depends on the value of border as defined in Border Modes. If border = 'constant', the value of the constant \(c\) in the formula is set by the parameter value.

Tiling repeats the contents of the tensor along certain dimensions.

Argument Validity

Result semantics

The value of output is defined as follows:

RoI pooling generates a fixed size output by pooling regions of variable size.

Argument validity

Let \(N\) be the number of RoIs.

Result semantics

Let the shape of input be \((B,C,H,W)\).

RoI coordinates are first rounded to the nearest integer values. Then, each RoI is cropped out from the feature map of the batch item indicated by the corresponding batch_index.

Let a row in the rois tensor be denoted by a 4-tuple \((y_0,x_0,y_1,x_1)\). Let the height of the RoI be denoted as \(r_H = y_1 - y_0\), and the width of a RoI be denoted as \(r_W = x_1 - x_0\).

Each RoI is divided into \(p_H \times p_W\) cells. Cells have height \(c_H = ceil(\frac{r_H}{p_H})\) and width \(c_W = ceil(\frac{r_W}{p_W})\), and are strided with strides \(s_H = floor(\frac{r_H}{p_H})\) and \(s_W = floor(\frac{r_W}{p_W})\); cell \((i,j)\) starts at position \((c_H + i * s_H, c_W + j * s_W)\), where \(i\) and \(j\) are zero-based indices.

For each cell, the average or max values are calculated the same way as for the avg_pool and max_pool operations.

Enhanced versions of RoI pooling aim to eliminate the quantization effect of rounding RoI coordinates to integer values and the way cells are generated. Instead, each RoI is directly mapped to a fixed intermediate size by resampling. Afterwards, max or average pooling can be applied to arrive to a final size.

The roi_resample operation resamples a RoI to a fixed size using multilinear interpolation.

Argument validity

Let \(N\) be the number of RoIs.

Result semantics

Let the shape of input be \((B,C,H,W)\).

The interpolation methods are the same as described in Up and Down-Sampling.

Using operation roi_resample, it is possible to describe roi_align operations as compounds: