Phase correlation is one of the classic methods for sparse motion or displacement estimation. It is renowned in the literature for high precision and insensitivity against illumination variations. We propose several important enhancements to the phase correlation (PhC) method which render it more robust against those situations where a motion measurement is not possible (low structure, too much noise, too different image content in the corresponding measurement windows). This allows the method to perform self-diagnosis in adverse situations. Furthermore, we extend the PhC method by a robust scheme for detecting and classifying the presence of multiple motions and estimating their uncertainties. Experimental results on the Middlebury Stereo Dataset and on the KITTI Optical Flow Dataset show the potential offered by the enhanced method in contrast to the PhC implementation of OpenCV.

An open issue in multiple view geometry and structure from motion, applied to real life scenarios, is the sparsity of the matched key-points and of the reconstructed point cloud. We present an approach that can significantly improve the density of measured displacement vectors in a sparse matching or tracking setting, exploiting the partial information of the motion field provided by linear oriented image patches (edgels). Our approach assumes that the epipolar geometry of an image pair already has been computed, either in an earlier feature-based matching step, or by a robustified differential tracker. We exploit key-points of a lower order, edgels, which cannot provide a unique 2D matching, but can be employed if a constraint on the motion is already given. We present a method to extract edgels, which can be effectively tracked given a known camera motion scenario, and show how a constrained version of the Lucas-Kanade tracking procedure can efficiently exploit epipolar geometry to reduce the classical KLT optimization to a 1D search problem. The potential of the proposed methods is shown by experiments performed on real driving sequences.

Reliable detection of obstacles at long range is crucial for the timely response to hazards by fast-moving safety-critical platforms like autonomous cars. We present a novel method for the joint detection and localization of distant obstacles using a stereo vision system on a moving platform. The approach is applicable to both static and moving obstacles and pushes the limits of detection performance as well as localization accuracy. The proposed detection algorithm is based on sound statistical tests using local geometric criteria which implicitly consider non-flat ground surfaces. To achieve maximum performance, it operates directly on image data instead of precomputed stereo disparity maps. A careful experimental evaluation on several datasets shows excellent detection performance and localization accuracy up to very large distances, even for small obstacles. We demonstrate a parallel implementation of the proposed system on a GPU that executes at real-time speeds.

Tracking keypoints through a video sequence is a crucial first step in the processing chain of many visual SLAM approaches. This paper presents a robust initialization method to provide the initial match for a keypoint tracker, from the 1st frame where a keypoint is detected to the 2nd frame, that is: when no depth information is available. We deal explicitly with the case of long displacements. The starting position is obtained through an optimization that employs a distribution of motion priors based on pyramidal phase correlation, and epipolar geometry constraints. Experiments on the KITTI dataset demonstrate the significant impact of applying a motion prior to the matching. We provide detailed comparisons to the state-of-the-art methods.

Visual odometry is one of the most active topics in computer vision. The automotive industry is particularly interested in this field due to the appeal of achieving a high degree of accuracy with inexpensive sensors such as cameras. The best results on this task are currently achieved by systems based on a calibrated stereo camera rig, whereas monocular systems are generally lagging behind in terms of performance. We hypothesise that this is due to stereo visual odometry being an inherently easier problem, rather than than due to higher quality of the state of the art stereo based algorithms. Under this hypothesis, techniques developed for monocular visual odometry systems would be, in general, more refined and robust since they have to deal with an intrinsically more difficult problem. In this work we present a novel stereo visual odometry system for automotive applications based on advanced monocular techniques. We show that the generalization of these techniques to the stereo case result in a significant improvement of the robustness and accuracy of stereo based visual odometry. We support our claims by the system results on the well known KITTI benchmark, achieving the top rank for visual only systems∗ .

The motion of a driving car is highly constrained and we claim that powerful predictors can be built that learn the typical egomotion statistics, and support the typical tasks of feature matching, tracking, and egomotion estimation. We analyze the statistics of the ground truth data given in the KITTI odometry benchmark sequences and confirm that a coordinated turn motion model, overlaid by moderate vibrations, is a very realistic model. We develop a predictor that is able to significantly reduce the uncertainty about the relative motion when a new image frame comes in. Such predictors can be used to steer the matching process from frame n to frame n + 1. We show that they can also be employed to detect outliers in the temporal sequence of egomotion parameters.

The present paper analyzes some previously unexplored aspects of motion estimation that are fundamental both for discrete block matching as well as for differential optical flow approaches a la Lucas-Kanade. It aims at providing a complete estimation-theoretic approach that makes the assumptions about noisy observations of samples from a continuous signal of a certain class explicit. It turns out that motion estimation is a combination of simultaneously estimating the true underlying continuous signal and optimizing the displacement between two hypothetical copies of this unknown signal. Practical schemes such as the current variants of Lucas-Kanade are just approximations to the fundamental estimation problem identified in the present paper. Derivatives appear as derivatives to the continuous signal representation kernels, not as ad hoc discrete derivative masks. The formulation via an explicit signal space defined by kernels is a precondition for analyzing e.g. the convergence range of iterative displacement estimation procedures, and for systematically chosing preconditioning filters. The paper sets the stage for further in-depth analysis of some fundamental issues that have so far been overlooked or ignored in motion analysis.

We discuss matching measures (scores and residuals) for comparing image patches under unknown affine photometric (=intensity) transformations. In contrast to existing methods, we derive a fully symmetric matching measure which reflects the fact that both copies of the signal are affected by measurement errors (noise), not only one. As it turns out, this evolves into an eigensystem problem; however a simple direct solution for all entities of interest can be given. We strongly advocate for constraining the estimated gain ratio and the estimated mean value offset to realistic ranges, thus preventing the matching scheme from locking into unrealistic correspondences.

The development of vehicles that perceive their environment, in particular those using computer vision, indispensably requires large databases of sensor recordings obtained from real cars driven in realistic traffic situations. These datasets should be time shaped for enabling synchronization of sensor data from different sources. Furthermore, full surround environment perception requires high frame rates of synchronized omnidirectional video data to prevent information loss at any speeds.

This paper describes an experimental setup and software environment for recording such synchronized multi-sensor data streams and storing them in a new open source format. The dataset consists of sequences recorded in various environments from a car equipped with an omnidirectional multi-camera, height sensors, an IMU, a velocity sensor, and a GPS. The software environment for reading these data sets will be provided to the public, together with a collection of long multi-sensor and multi-camera data streams stored in the developed format.

We propose and evaluate a versatile scheme for image pre-segmentation that generates a partition of the image into a selectable number of patches (’superpixels’), under the constraint of obtaining maximum homogeneity of the ’texture’ inside of each patch, and maximum accordance of the contours with both the image content as well as a Gibbs-Markov random field model. In contrast to current state-of-the art approaches to superpixel segmentation, ’homogeneity’ does not limit itself to smooth region-internal signals and high feature value similarity between neighboring pixels, but is applicable also to highly textured scenes. The energy functional that is to be maximized for this purpose has only a very small number of design parameters, depending on the particular statistical model used for the images.

The capability of the resulting partitions to deform according to the image content can be controlled by a single parameter. We show by means of an extensive comparative experimental evaluation that the compactness-controlled contour-relaxed superpixels method outperforms the state-of-the art superpixel algorithms with respect to boundary recall and undersegmentation error while being faster or on a par with respect to runtime.