This paper explores the use of LH Interval Calculus, a novel qualitative spatial reasoning formalism, to create a human-readable representation of environments observed by UAVs. The system simplifies data from multiple UAVs collaborating on environment mapping. Real UAV-captured data was used for evaluation. In tests involving two UAVs mapping an outdoor area, LH Calculus proved effective in generating a cohesive high-level description of the environment, contingent on consistent input data.

This paper considers fusion of dimension-reduced estimates in a decentralized sensor network. The benefits of a decentralized sensor network include modularity, robustness and flexibility. Moreover, since preprocessed data is exchanged between the agents it allows for reduced communication. Nevertheless, in certain applications the communication load is required to be reduced even further. One way to decrease the communication load is to exchange dimension-reduced estimates instead of full estimates. Previous work on this topic assumes global availability of covariance matrices, an assumption which is not realistic in decentralized applications. Hence, in this paper we consider the problem of deriving dimension-reduced estimates using only local information. The proposed solution is based on an estimate of the information common to the network. This common information estimate is computed locally at each agent by fusion of all information that is either received or transmitted by that agent. It is shown how the common information estimate is utilized for fusion of dimension-reduced estimates using two well-known fusion methods: the Kalman fuser which is optimal under the assumption of uncorrelated estimates, and covariance intersection. One main theoretical result is that the common information estimate allows for a decorrelation procedure such that uncorrelated estimates can be maintained. This property is crucial to be able to use the Kalman fuser without double counting of information. A numerical comparison suggests that the performance degradation of using the common information estimate, compared to having local access to the actual covariance matrices computed by other agents, is relatively small.

Accurate localization is a part of most autonomous systems. GNSS is today the go to solution for localization but is unreliable due to jamming and is not available indoors. Inertial navigation aided by visual measurements, e.g., optical flow, offers an alternative. Traditional feature-based optical flow is limited to scenes with good features, current development of deep neural network derived dense optical flow is an interesting alternative. This paper proposes a method to evaluate the result of dense optical flow on real image sequences using traditional feature-based optical flow and uses this to compare six different dense optical flow methods. The results of the dense methods are promising, and no clear winner amongst the methods can be determined. The results are discussed in the context of how they can be used to support localization.

This report aims to describe the latest research and method developmentof image-based multi sensor fusion navigation and summarizes open aerialdatasets which can support the latest research related to this project. Itsupports the initial setting of the direction of the algorithm development inthe early stage of the project.The Multi Sensor Image-based Navigation project aims to study and developthe methods focusing on image-based multisensor navigation in orderto acquire a precise localization of the aircraft. GNSS-based localizationand navigation systems are sensitive to disturbances and jamming, hencethe capability to provide reliable position accuracy without GNSS is a keyelement to develop the navigation systems.The output of this project can be utilized in a wide range of applications,such as aircraft operation in GNSS denied environments or urban air mobilitycontext.

Network-centric multitarget tracking under communication constraints is considered, where dimension-reduced track estimates are exchanged. Previous work on target tracking in this subfield has focused on fusion aspects only and derived optimal ways of reducing dimensionality based on fusion performance. In this work we propose a novel problem formalization where estimates are reduced based on association performance. The problem is analyzed theoretically and problem properties are derived. The theoretical analysis leads to an optimization strategy that can be used to partly preserve association quality when reducing the dimensionality of communicated estimates. The applicability of the suggested optimization strategy is demonstrated numerically in a multitarget scenario.

Mean square error optimal estimation requires the full correlation structure to be available. Unfortunately, it is not always possible to maintain full knowledge about the correlations. One example is decentralized data fusion where the cross-correlations between estimates are unknown, partly due to information sharing. To avoid underestimating the covariance of an estimate in such situations, conservative estimation is one option. In this paper the conservative linear unbiased estimator is formalized including optimality criteria. Fundamental bounds of the optimal conservative linear unbiased estimator are derived. A main contribution is a general approach for computing the proposed estimator based on robust optimization. Furthermore, it is shown that several existing estimation algorithms are special cases of the optimal conservative linear unbiased estimator. An evaluation verifies the theoretical considerations and shows that the optimization based approach performs better than existing conservative estimation methods in certain cases.

In this work, estimators for platform pose and landmark maps for visual-inertial data are analysed. It is shown that the full, non-linear, visual-inertial problem has a conditionally linear substructure in the 2D case which can be exploited for efficient solutions, e.g., Block Coordinate Descent (BCD). It is also shown that the measurement noise from the non-linear model becomes parameter dependent resulting in biased estimates if that fact is ignored. However, the bias can be accounted for using the Iteratively Reweighted Least Squares (IRLS) method. In the 3D case the conditionally linear substructure is not separable. However, it can be shown that the Jacobian of the non-linear substructure can be calculated recursively resulting in an efficient solution. A simulated 2D visual-inertial example is used to illustrate the theoretical results.

Data fusion in a communication constrained sensor network is considered. The problem is to reduce the dimensionality of the joint state estimate without significantly decreasing the estimation performance. A method based on scalar subspace projections is derived for this purpose. We consider the cases where the estimates to be fused are: (i) uncorrelated, and (ii) correlated. It is shown how the subspaces can be derived using eigenvalue optimization. In the uncorrelated case guarantees on mean square error optimality are provided. In the correlated case an iterative algorithm based on alternating minimization is proposed. The methods are analyzed using parametrized examples. A simulation evaluation shows that the proposed method performs well both for uncorrelated and correlated estimates.

Reinforcement learning techniques have successfully been applied to solve challenging problems. Among the more famous examples are playing games such as Go and real-time computer games such as StarCraft II. In addition, reinforcement learning has successfully been deployed in cyber-physical systems such as robots playing a curling-based game. These are all important and significant achievements indicating that the techniques can be of value for the aerospace industry. However, to use these techniques in the aerospace industry, very high requirements on verification and validation must be met. In this position paper, we outline four key problems for verification and validation of reinforcement learning techniques. Solving these are an important step towards enabling reinforcement learning techniques to be used in safety critical domains such as the aerospace industry.

We consider a decentralized sensor network of multiple nodes with limited communication capability where the cross-correlations between local estimates are unknown. To reduce the bandwidth the individual nodes determine which subset of local information is the most valuable from a global perspective. Three information selection methods (ISM) are derived. The proposed ISM require no other information than the communicated estimates. The simulation evaluation shows that by using the proposed ISM it is possible to determine which subset of local information is globally most valuable such that both reduced bandwidth and high performance are achieved.

This paper addresses the problem of retrieving consistentestimates in a distributed network where the communication between the nodes is constrained such that only the diagonal elements of the covariance matrix are allowed to be exchanged. Several methods are developed for preserving and/or recovering consistency under the constraints imposed by the communication protocol. The proposed methods are used in conjunction with the covariance intersection method and the estimation performance is evaluated based on information usage and consistency. The results show that among the proposed methods, consistency can be preserved equally well at the transmitting node as at the receiving node.

An initial work has been performed to implement a sequential Monte Carlo method to solve the data association problem. The main motivation is to overcome the incorrect association when the state estimates are inaccurate. The solution is based on modeling the data association as a stochastic variable and estimated with a bootstrap particle filter. Two variants of the proposal function are evaluated, one with the uniform distribution over possible associations, and the other one with the distribution depending on the measurements and state estimates. The performance of both proposals is evaluated on the small simulation example, and compared to a purely deterministic approach, Nearest-Neighbour, as well. The obtained initial results are quite promising, and more evaluation and expansion to more examples and real data sets is suggested for the future work.

The general Simultaneous Localisation and Mapping (SLAM) problem aims at estimating the state of a moving platform simultaneously with building a map of the local environment. There are essentially three classes of algorithms. EKF- SLAM and FastSLAM solve the problem on-line, while Nonlinear Least Squares (NLS) is a batch method. All of them scales badly with either the state dimension, the map dimension or the batch length. We investigate the EM algorithm for solving a generalized version of the NLS problem. This EM-SLAM algorithm solves two simpler problems iteratively, hence it scales much better with dimensions. The iterations switch between state estimation, where we propose an Extended Rauch-Tung-Striebel smoother, and map estimation, where a quasi-Newton method is suggested. The proposed method is evaluated in real experiments and also in simulations on a platform with a monocular camera attached to an inertial measurement unit. It is demonstrated to produce lower RMSE than with a standard Levenberg-Marquardt solver of NLS problem, at a computational cost that increases considerably slower. 

This paper presents three iterative methods for orientation estimation. The first two are based on iterated Extended Kalman filter (IEKF) formulations with different state representations. The first is using the well-known unit quaternion as state (q-IEKF) while the other is using orientation deviation which we call IMEKF. The third method is based on nonlinear least squares (NLS) estimation of the angular velocity which is used to parametrise the orientation. The results are obtained using Monte Carlo simulations and the comparison is done with the non-iterative EKF and multiplicative EKF (MEKF) as baseline. The result clearly shows that the IMEKF and the NLS-based method are superior to q-IEKF and all three outperform the non-iterative methods.

This paper presents a batch estimation method for Simultaneous Localization and Mapping (SLAM) using the Prediction Error Method (PEM). The estimation problem considers landmarks as parameter while treating dynamics using state space models. The gradient needed for parameter estimation is computed recursively using an Extended Kalman Filter (EKF). Results using simulations with a monocular camera and inertial sensors are presented and compared to a Nonlinear Least- Squares (NLS) estimator. The presented method produce both lower RMSE’s and scale better to the batch length. 

A method for fusing Synthetic Aperture Radar (SAR) images with opticalaerial images is presented. This is done in a navigation framework, where the absolute position and orientation of the flying platform, as computed from the inertial navigation system, is corrected based on the aerial image coordinates taken as ground truth. The method is suitable for new low-price SAR systems for small unmanned vehicles. The primary application is remote sensing, where the SAR image provides one further "colour" channel revealing reflectivity to radio waves. The method is based on first applying an edge detection algorithm to the images and then optimising the most important navigation states by matching the two binary images. To get a measure of the estimation uncertainty, we embed the optimisation in a least squares framework, where an explicit method to estimate the (relative) size of the errors is presented. The performance is demonstrated on real SAR and aerial images, leading to an error of only a few pixels.

This paper presents a method for matching spotlight Synthetic Aperture Radar (SAR) images with a georeferenced 3D-map as means for navigational aid. A hypothesis of the flying platform's absolute position, velocity and direction - which later can be used to correct the inertial navigation system - is attained by image matching and optimization. A projective model with 6 DoF is used to create a simulated SAR image from a 3D map. The parameters of the projective model represents the most important of the platform's navigation state, and these are adjusted by Chamfer matching the captured SAR image to simulated ones. The performance is demonstrated on real spotlight SAR images and 3D-map, and the error is shown to be only a few pixels in average, which in our case is about 3 meters.

Synthetic aperture radar (SAR) equipment is a radar imaging system that can be used to create high-resolution images of a scene by utilizing the movement of a flying platform. Knowledge of the platforms trajectory is essential to get good and focused images. An emerging application field is real-time SAR imaging using small and cheap platforms where estimation errors in navigation systems imply unfocused images. This contribution investigates a joint estimation of the trajectory and SAR image. Starting with a nominal trajectory, we successively improve the image by optimizing a focus measure and updating the trajectory accordingly. The method is illustrated using simulations using typical navigation performance of an unmanned aerial vehicle. One real data set is used to show feasibility, where the result indicates that, in particular, the azimuth position error is decreased as the image focus is iteratively improved.

The dependence of radio signal propagation on the environment is  well known, and both statistical and deterministic methods have been presented in the literature. Such methods are either based on randomised or actual reflectors of radio signals. In this work, we instead aim at estimating the location of the reflectors based on geo-localised radio channel impulse reponse measurements and using methods from synthetic aperture radar (SAR). Radio channel data measurements from 3GPP E-UTRAN have been used to verify the usefulness of the proposed approach. The obtained images show that  the estimated reflectors are well correlated with the aerial map of the environment. Also, which part of the trajectory contributed to different reflectors have been estimated with promising results.

Simultaneous Localisation and Mapping (SLAM) denotes the problem of jointly localizing a moving platform and mapping the environment. This work studies the SLAM problem using a combination of inertial sensors, measuring the platform's accelerations and angular velocities, and a monocular camera observing the environment. We formulate the SLAM problem on a nonlinear least squares (NLS) batch form, whose solution provides a smoothed estimate of the motion and map. The NLS problem is highly nonconvex in practice, so a good initial estimate is required. We propose a multi-stage iterative procedure, that utilises the fact that the SLAM problem is linear if the platform's rotations are known. The map is initialised with camera feature detections only, by utilising feature tracking and clustering of  feature tracks. In this way, loop closures are automatically detected. The initialization method and subsequent NLS refinement is demonstrated on both simulated and real data.

Small and medium sized Unmanned Aerial Vehicles (UAV) are today used in military missions, and will in the future find many new application areas such as surveillance for exploration and security. To enable all these foreseen applications, the UAV's have to be cheap and of low weight, which restrict the sensors that can be used for navigation and surveillance. This thesis investigates several aspects of how fusion of navigation and imaging sensors can improve both tasks at a level that would require much more expensive sensors with the traditional approach of separating the navigation system from the applications. The core idea is that vision sensors can support the navigation system by providing odometric information of the motion, while the navigation system can support the vision algorithms, used to map the surrounding environment, to be more efficient. The unified framework for this kind of approach is called  Simultaneous Localisation and Mapping (SLAM) and it will be applied here to inertial sensors, radar and optical camera.

Synthetic Aperture Radar (SAR) uses a radar and the motion of the UAV to provide an image of the microwave reflectivity of the ground. SAR images are a good complement to optical images, giving an all-weather surveillance capability, but they require an accurate navigation system to be focused which is not the case with typical UAV sensors. However, by using the inertial sensors, measuring UAV's motion, and information from the SAR images, measuring how image quality depends on the UAV's motion, both higher navigation accuracy and, consequently, more focused images can be obtained. The fusion of these sensors can be performed in both batch and sequential form. For the first approach, we propose an optimisation formulation of the navigation and focusing problem while the second one results  in a filtering approach. For the optimisation method the measurement of the focus in processed SAR images is performed with the image entropy and with an image matching approach, where SAR images are matched to the map of the area. In the proposed filtering method the motion information is estimated from the raw radar data and it corresponds to the time derivative of the range between UAV and the imaged scene, which can be related to the motion of the UAV.

Another imaging sensor that has been exploited in this framework is  an ordinary optical camera. Similar to the SAR case, camera images and inertial sensors can also be used to support the navigation estimate and simultaneously build a three-dimensional map of the observed environment, so called inertial/visual SLAM. Also here, the problem is posed in optimisation framework leading to batch Maximum Likelihood (ML) estimate of the navigation parameters and the map. The ML problem is solved in both the straight-forward way,  resulting in nonlinear least squares where both map and navigation parameters are considered as parameters, and with the Expectation-Maximisation (EM) approach. In the EM approach, all unknown variables are split into two sets, hidden variables and actual parameters, and in this case the map is considered as parameters and the navigation states are seen as hidden  variables. This split enables the total problem to be solved computationally cheaper then the original ML formulation. Both optimisation problems mentioned above are nonlinear and non-convex requiring good initial solution in order to obtain good parameter estimate. For this purpose a method for initialisation of inertial/visual SLAM is devised where the conditional linear structure of the problem is used to obtain the initial estimate of the parameters. The benefits and performance improvements of the methods are illustrated on both simulated and real data.

Synthetic Aperture Radar (SAR) equipment is a radar imaging system that can be used to create high resolution images of a scene by utilising the movement of a flying platform. Knowledge of the platform's trajectory is essential to get good and focused images. An emerging application field is real-time SAR imaging using small and cheap platforms with poorer navigation systems implying unfocused images. This contribution investigatesa joint estimation of the trajectory and SAR image.

In this paper we present a solution to the simultaneous localisation and mapping (SLAM) problem using a camera and inertial sensors. Our approach is based on the maximum a posteriori (MAP) estimate of the complete SLAM problem. The resulting problem is posed in a nonlinear least-squares framework which we solve with the Gauss-Newton method. The proposed algorithm is evaluated on experimental data using a sensor platform mounted on an industrial robot. In this way, accurate ground truth is available, and the results are encouraging.

Synthetic Aperture Radar (SAR) equipment is an all-weather radar imaging system that can be used to create high resolution images of the scene by utilising the movement of the flying platform. It is therefore essential to accurately estimate the platform's trajectory in order to get good and focused images. Recently, both real time applications and smaller and cheaper platforms have been considered. This, in turn, leads to unfocused images since cheaper platforms, in general, have navigation systems with poorer performance. At the same time the radar data contain information about the platform's motion that can be used to estimate the trajectory andget more focused images. Here, a method of utilising the phase gradient of the SAR data in a sensor fusion framework is presented. The method is illustrated on a simulated example with promising results. At the end a discussion about the obtained results and future work is covered.

Since its discovery, in the 1940's, radar (Radio Detection and Ranging) has become an important ranging sensor in many areas of technology and science. Most of the military and many civilian applications are unimaginable today without radar. With technology development, radar application areas have become larger and more available. One of these applications is Synthetic Aperture Radar (SAR), where an airborne radar is used to create high resolution images of the imaged scene. Although known since the 1950's, the SAR methods have been continuously developed and improved and new algorithms enabling real-time applications have emerged lately. Together with making the hardware components smaller and lighter, SAR has become an interesting sensor to be mounted on smaller unmanned aerial vehicles (UAV's). One important thing needed in the SAR algorithms is the estimate of the platform's motion, like position and velocity. Since this estimate is always corrupted with errors, particularly if lower grade navigation system, common in UAV applications, is used, the SAR images will be distorted. One of the most frequently appearing distortions caused by the unknown platform's motion is the image defocus. The process of correcting the image focus is called auto-focusing in SAR terminology. Traditionally, this problem was solved by methods that discard the platform's motion information, mostly due to the off-line processing approach, i.e. the images were created after the flight. Since the image (de)focus and the motion of the platform are related to each other, it is possible to utilise the information from the SAR images as a sensor and improve the estimate of the platform's motion. The auto-focusing problem can be cast as a sensor fusion problem. Sensor fusion is the process of fusing information from different sensors, in order to obtain best possible estimate of the states. Here, the information from sensors measuring platform's motion, mainly accelerometers, will be fused together with the information from the SAR images to estimate the motion of the flying platform. Two different methods based on this approach are tested on the simulated SAR data and the results are evaluated. One method is based on an optimisation based formulation of the sensor fusion problem, leading to batch processing, while the other method is based on the sequential processing of the radar data, leading to a filtering approach. The obtained results are promising for both methods and the obtained performance is comparable with the performance of a high precision navigation aid, such as Global Positioning System (GPS).

Synthetic Aperture Radar (SAR) equipment is an all-weather radar imaging system that can create high resolution images by means of utilising the movement of the flying platform. Accurate knowledge of the flown trajectory is essential in order to get focused images. Recently SAR systems are becoming more used on smaller and cheaper flying platforms like Unmanned Aerial Vehicles (UAV). Since UAVs in general have navigation systems with poorer performance than manned aircraft, the resulting images will inevitably be unfocused. At the same time, the unfocused images carry the information about the platforms trajectory that can be utilised. Here a way of using SAR images and their focus measure in a sensor fusion framework in order to simultaneously obtain both improved images and trajectory estimate is presented. The method is illustrated on a simple simulated example with promising results. Finally a discussion about the results and future work is given.

Synthetic Aperture Radar (SAR) equipment is an all-weather radar imaging system that can create high resolution images by means of utilising the movement of the flying platform. Accurate knowledge of the flown trajectory is essential in order to get focused images. Recently SAR systems are becoming more used on smaller and cheaper flying platforms like Unmanned Aerial Vehicles (UAV). Since UAVs in general have navigation systems with poorer performance than manned aircraft, the resulting images will inevitably be unfocused. At the same time, the unfocused images carry the information about the platforms trajectory that can be utilised. Here a way of using SAR images and their focus measure in a sensor fusion framework in order to simultaneously obtain both improved images and trajectory estimate is presented. The method is illustrated on a simple simulated example with promising results. Finally a discussion about the results and future work is given.

In this paper we present a solution to the simultaneous localization and mapping (SLAM) problem for unmanned aerial vehicles (UAV) using a camera and inertial sensors. A good SLAM solution is an important enabler for autonomous robots. Our approach is based on an optimization based formulation of the problem, which results in a smoother, rather than a filter. The proposed algorithm is evaluated on experimental data and the resultsare compared with accurate ground truth data. The results from this comparisons are encouraging.

Control system design for advanced, highly agile fighter aircraft, with unstable nonlinear aerodynamic characteristics, rely heavily on flight mechanical simulations. This makes the accuracy of the aerodynamic model in the simulators very important. Here, two methods for estimating parameters of nonlinear unstable systems where the control system is unknown are presented. Both approaches are direct prediction-error methods, either with a directly parametrized observer or with an Extended Kalman Filter as a predictor. These methods have been validated on simulated data, as well as on real flight test data and all approaches show promising results.