For virtually every area of science and engineering, state estimation is ubiquitous. Accurate state estimation requires a moderately precise mathematical model of the system, typically based on domain expertise. These models exist for a plethora of applications and available state estimators can generally produce accurate estimates. However, the models usually ignore hard-to-model phenomena, either due to the cost or the difficulty of modeling these characteristics. Further, the most widely used state estimator for nonlinear systems is still the extended Kalman filter (EKF), which may suffer from divergence for complex models, which essentially restricts the complexity of the usable models. Generally speaking, this thesis investigates ways of improving state estimation. Firstly, existing state-space models (SSMs) for target tracking are augmented with a Gaussian process (GP) in order to learn hard-to-model system characteristics online. Secondly, improved linearization-based state estimators are proposed that exhibit favorable robustness properties to the parameters of the noise processes driving the SSM.

The first part of the thesis explores joint state estimation and model learning in partially unknown SSMs, where some a priori domain expertise is available, but parts of the model need to be learned online. Paper A combines a linear, a priori identified, SSM with an approximate GP. An EKF is applied to this GP-augmented SSM in order to jointly estimate the state of the system and learn the, a priori, unknown dynamics. This empirically works well and substantially reduces the prediction error of the dynamical model as compared to a non-augmented SSM. Paper B explores ways of reducing the computational complexity of the method of Paper A. Crucially, it uses a compact kernel in the GP, which admits an equivalent basis function (BF) representation where only a few BFs are non-zero at any given system state. This enables a method that is essentially computationally invariant to the number of parameters, where the computational complexity can be tuned by hyperparameters of the BFs.

The second part explores iterated filters as a means to increase robustness to improper noise parameter choices. As the nonlinearities in the model are mainly contained in the dynamics, standard iterated filters such as the iterated extended Kalman filter (IEKF) can not be used. Papers C and D develop dynamically iterated filters (DIFs), which is a unified framework for linearization-based iterated filters that deal with nonlinearities in both the dynamics as well as the measurement model. The DIFs are shown to be robust toward improper noise parameter tuning and improve the mean square error (MSE) as compared to their corresponding non-iterated baselines.

The third and final part of the thesis considers an alternative bf representation of the GP model, the Hilbert-space Gaussian process (HGP), which is essentially a sinusoidal representation on a compact domain. Paper E identifies previously unutilized Hankel-Toeplitz structure in the HGP, which enables a time complexity for learning that is linear in the number of BFs, without further approximation. Lastly, Paper F improves the computational complexity of prediction in the HGP, by adaptively choosing the most important BFs for prediction in a certain region of the input.