Optical Flow Pyramid (LK)

Detailed Description

Computes the optical flow using the Lucas-Kanade method between two pyramid images.

The function is an implementation of the algorithm described in [1]. The function inputs are two vx_pyramid objects, old and new, along with a vx_array of vx_keypoint_t structs to track from the old vx_pyramid. The function outputs a vx_array of vx_keypoint_t structs that were tracked from the old vx_pyramid to the new vx_pyramid. Each element in the vx_array of vx_keypoint_t structs in the new array may be valid or not. The implementation shall return the same number of vx_keypoint_t structs in the new vx_array that were in the older vx_array.

In more detail: The Lucas-Kanade method finds the affine motion vector \( V \) for each point in the old image tracking points array, using the following equation:

\[ \begin{bmatrix} V_x \\ V_y \end{bmatrix} = \begin{bmatrix} \sum_{i}{I_x}^2 & \sum_{i}{I_x*I_y} \\ \sum_{i}{I_x*I_y} & \sum_{i}{I_y}^2 \end{bmatrix}^{-1} \begin{bmatrix} -\sum_{i}{I_x*I_t} \\ -\sum_{i}{I_y*I_t} \end{bmatrix} \]

Where \( I_x \) and \( I_y \) are obtained using the Scharr gradients on the input image:

\[ G_x = \begin{bmatrix} +3 & 0 & -3 \\ +10 & 0 & -10 \\ +3 & 0 & -3 \end{bmatrix} \]

\[ G_y = \begin{bmatrix} +3 & +10 & +3 \\ 0 & 0 & 0 \\ -3 & -10 & -3 \end{bmatrix} \]

\( I_t \) is obtained by a simple difference between the same pixel in both images. \( i \) is defined as the adjacent pixels to the point \( p(x,y) \) under consideration. With a given window size of \( M \), \( i \) is \( M^2 \) points. The pixel \( p(x,y) \) is centered in the window. In practice, to get an accurate solution, it is necessary to iterate multiple times on this scheme (in a Newton-Raphson fashion) until:

• the residual of the affine motion vector is smaller than a threshold
• And/or maximum number of iteration achieved. Each iteration, the estimation of the previous iteration is used by changing \( I_t \) to be the difference between the old image and the pixel with the estimated coordinates in the new image. Each iteration the function checks if the pixel to track was lost. The criteria for lost tracking is that the matrix above is invertible. (The determinant of the matrix is less than a threshold : \( 10^{-7} \) .) Or the minimum eigenvalue of the matrix is smaller then a threshold ( \( 10^{-4} \) ). Also lost tracking happens when the point tracked coordinate is outside the image coordinates. When vx_true_e is given as the input to use_initial_estimates, the algorithm starts by calculating \( I_t \) as the difference between the old image and the pixel with the initial estimated coordinates in the new image. The input vx_array of vx_keypoint_t structs with tracking_status set to zero (lost) are copied to the new vx_array.

Clients are responsible for editing the output vx_array of vx_keypoint_t structs array before applying it as the input vx_array of vx_keypoint_t structs for the next frame. For example, vx_keypoint_t structs with tracking_status set to zero may be removed by a client for efficiency.

This function changes just the x, y, and tracking_status members of the vx_keypoint_t structure and behaves as if it copied the rest from the old tracking vx_keypoint_t to new image vx_keypoint_t.

Functions
vx_node	vxOpticalFlowPyrLKNode (vx_graph graph, vx_pyramid old_images, vx_pyramid new_images, vx_array old_points, vx_array new_points_estimates, vx_array new_points, vx_enum termination, vx_scalar epsilon, vx_scalar num_iterations, vx_scalar use_initial_estimate, vx_size window_dimension)
	[Graph] Creates a Lucas Kanade Tracking Node. More...
vx_status	vxuOpticalFlowPyrLK (vx_context context, vx_pyramid old_images, vx_pyramid new_images, vx_array old_points, vx_array new_points_estimates, vx_array new_points, vx_enum termination, vx_scalar epsilon, vx_scalar num_iterations, vx_scalar use_initial_estimate, vx_size window_dimension)
	[Immediate] Computes an optical flow on two images. More...

Function Documentation

vx_node vxOpticalFlowPyrLKNode	(	vx_graph	graph,
		vx_pyramid	old_images,
		vx_pyramid	new_images,
		vx_array	old_points,
		vx_array	new_points_estimates,
		vx_array	new_points,
		vx_enum	termination,
		vx_scalar	epsilon,
		vx_scalar	num_iterations,
		vx_scalar	use_initial_estimate,
		vx_size	window_dimension
	)

[Graph] Creates a Lucas Kanade Tracking Node.

Parameters

[in]	graph	The reference to the graph.
[in]	old_images	Input of first (old) image pyramid in VX_DF_IMAGE_U8.
[in]	new_images	Input of destination (new) image pyramid VX_DF_IMAGE_U8.
[in]	old_points	An array of key points in a vx_array of VX_TYPE_KEYPOINT; those key points are defined at the old_images high resolution pyramid.
[in]	new_points_estimates	An array of estimation on what is the output key points in a vx_array of VX_TYPE_KEYPOINT; those keypoints are defined at the new_images high resolution pyramid.
[out]	new_points	An output array of key points in a vx_array of VX_TYPE_KEYPOINT; those key points are defined at the new_images high resolution pyramid.
[in]	termination	The termination can be VX_TERM_CRITERIA_ITERATIONS or VX_TERM_CRITERIA_EPSILON or VX_TERM_CRITERIA_BOTH.
[in]	epsilon	The vx_float32 error for terminating the algorithm.
[in]	num_iterations	The number of iterations. Use a VX_TYPE_UINT32 scalar.
[in]	use_initial_estimate	Use a VX_TYPE_BOOL scalar.
[in]	window_dimension	The size of the window on which to perform the algorithm. See VX_CONTEXT_ATTRIBUTE_OPTICAL_FLOW_WINDOW_MAXIMUM_DIMENSION

Returns

vx_node.

Return values

0	Node could not be created.
*	Node handle.

vx_status vxuOpticalFlowPyrLK	(	vx_context	context,
		vx_pyramid	old_images,
		vx_pyramid	new_images,
		vx_array	old_points,
		vx_array	new_points_estimates,
		vx_array	new_points,
		vx_enum	termination,
		vx_scalar	epsilon,
		vx_scalar	num_iterations,
		vx_scalar	use_initial_estimate,
		vx_size	window_dimension
	)

[Immediate] Computes an optical flow on two images.

Parameters

[in]	context	The reference to the overall context.
[in]	old_images	Input of first (old) image pyramid
[in]	new_images	Input of destination (new) image pyramid
[in]	old_points	an array of key points in a vx_array of VX_TYPE_KEYPOINT those key points are defined at the old_images high resolution pyramid
[in]	new_points_estimates	an array of estimation on what is the output key points in a vx_array of VX_TYPE_KEYPOINT those keypoints are defined at the new_images high resolution pyramid
[out]	new_points	an output array of key points in a vx_array of VX_TYPE_KEYPOINT those key points are defined at the new_images high resolution pyramid
[in]	termination	termination can be VX_TERM_CRITERIA_ITERATIONS or VX_TERM_CRITERIA_EPSILON or VX_TERM_CRITERIA_BOTH
[in]	epsilon	is the vx_float32 error for terminating the algorithm
[in]	num_iterations	is the number of iterations
[in]	use_initial_estimate	Can be set to either vx_false_e or vx_true_e.
[in]	window_dimension	The size of the window on which to perform the algorithm. See VX_CONTEXT_ATTRIBUTE_OPTICAL_FLOW_WINDOW_MAXIMUM_DIMENSION

Returns

A vx_status_e enumeration.

Return values

VX_SUCCESS	Success
*	An error occurred. See vx_status_e.