Recent studies of selective auditory attention have demonstrated that neural responses recorded with electroencephalogram (EEG) can be decoded to classify the attended talker in everyday multitalker cocktail-party environments. This is generally referred to as the auditory attention decoding (AAD) and could lead to a breakthrough for the next-generation of hearing aids (HAs) to have the ability to be cognitively controlled. The aim of this paper is to investigate whether cepstral analysis can be used as a more robust mapping between speech and EEG. Our preliminary analysis revealed an average AAD accuracy of 96%. Moreover, we observed a significant increase in auditory attention classification accuracies with our approach over the use of traditional AAD methods (7% absolute increase). Overall, our exploratory study could open a new avenue for developing new AAD methods to further advance hearing technology. We recognize that additional research is needed to elucidate the full potential of cepstral analysis for AAD.

This presentation details and evaluates a method for estimating the attended speaker during a two-person conversation by means of in-ear electro-oculography (EOG). Twenty-five hearing-impaired participants were fitted with molds equipped with EOG electrodes (in-ear EOG) and wore eye-tracking glasses while watching a video of two life-size people in a dialog solving a Diapix task. The dialogue was directionally presented and together with background noise in the frontal hemisphere at 60 dB SPL. During three conditions of steering (none, in-ear EOG, conventional eye-tracking), participants comprehension was periodically measured using multiple-choice questions. Based on eye movement detection by in-ear EOG or conventional eye-tracking, the estimated attended speaker was amplified by 6 dB. In the in-ear EOG condition, the estimate was based on one selected channel pair of electrodes out of 36 possible electrodes. A novel calibration procedure introducing three different metrics was used to select the measurement channel. The in-ear EOG attended speaker estimates were compared to those of the eye-tracker. Across participants, the mean accuracy of in-ear EOG estimation of the attended speaker was 68%, ranging from 50 to 89%. Based on offline simulation, it was established that higher scoring metrics obtained for a channel with the calibration procedure were significantly associated with better data quality. Results showed a statistically significant improvement in comprehension of about 10% in both steering conditions relative to the no-steering condition. Comprehension in the two steering conditions was not significantly different. Further, better comprehension obtained under the in-ear EOG condition was significantly correlated with more accurate estimation of the attended speaker. In conclusion, this study shows promising results in the use of in-ear EOG for visual attention estimation with potential for applicability in hearing assistive devices.

In this work, estimators for platform pose and landmark maps for visual-inertial data are analysed. It is shown that the full, non-linear, visual-inertial problem has a conditionally linear substructure in the 2D case which can be exploited for efficient solutions, e.g., Block Coordinate Descent (BCD). It is also shown that the measurement noise from the non-linear model becomes parameter dependent resulting in biased estimates if that fact is ignored. However, the bias can be accounted for using the Iteratively Reweighted Least Squares (IRLS) method. In the 3D case the conditionally linear substructure is not separable. However, it can be shown that the Jacobian of the non-linear substructure can be calculated recursively resulting in an efficient solution. A simulated 2D visual-inertial example is used to illustrate the theoretical results.

Linearized Direction of Arrival (LinDoA) is a method for sound source localization that is designed for use with wearable microphone arrays. The method uses a Taylor series expansion of the sound source signal in the time domain to beamform and estimate the direction of arrival. The original method is limited to spatial sampling, but is here generalized to also consider temporal sampling for improved performance and usability. The proposed generalization allows for time-domain formulations of the Delay-and-Sum and Minimum-Variance Distortionless Response beamformers in addition to the original formulation by implementing interpolation and estimating the noise covariance. A number of variants of the method are described and the design choices are discussed. The methods are evaluated on data gathered by a head-worn array in real and simulated experiments and are compared to conventional methods. They are shown to perform on par with conventional methods at a reduced computational cost.

Introduction: By means of adding more sensor technology, modern hearing aids (HAs) strive to become better, more personalized, and self-adaptive devices that can handle environmental changes and cope with the day-to-day fitness of the users. The latest HA technology available in the market already combines sound analysis with motion activity classification based on accelerometers to adjust settings. While there is a lot of research in activity tracking using accelerometers in sports applications and consumer electronics, there is not yet much in hearing research. Objective: This study investigates the feasibility of activity tracking with ear-level accelerometers and how it compares to waist-mounted accelerometers, which is a more common measurement location. Method: The activity classification methods in this study are based on supervised learning. The experimental set up consisted of 21 subjects, equipped with two XSens MTw Awinda at ear-level and one at waist-level, performing nine different activities. Results: The highest accuracy on our experimental data as obtained with the combination of Bagging and Classification tree techniques. The total accuracy over all activities and users was 84% (ear-level), 90% (waist-level), and 91% (ear-level + waist-level). Most prominently, the classes, namely, standing, jogging, laying (on one side), laying (face-down), and walking all have an accuracy of above 90%. Furthermore, estimated ear-level step-detection accuracy was 95% in walking and 90% in jogging. Conclusion: It is demonstrated that several activities can be classified, using ear-level accelerometers, with an accuracy that is on par with waist-level. It is indicated that step-detection accuracy is comparable to a high-performance wrist device. These findings are encouraging for the development of activity applications in hearing healthcare.

The ability to understand speech in complex listening environments reflects an interaction of cognitive and sensory capacities that are difficult to capture with behavioural tests. The study of natural listening behaviours may lead to the development of new metrics that better reflect real-life communication abilities. To this end, we investigated the relationship between speech comprehension and eye-movements among hearing-impaired people in a challenging listening situation. While previous research has investigated the effect of background noise on listeners’ gaze patterns with single talkers, the effect of noise in multitalker conversations remains unknown. We tracked eye-movements of seven aided hearing-impaired adults while they viewed video recordings of two life-sized talkers engaged in an unscripted dialogue. Hearing loss ranged from moderate to severe. We used multiple-choice questions to measure participants’ comprehension of the conversation in multitalker babble noise at three different signal-to-noise ratios. All participants made saccades between the two talkers more frequently than the talkers’ conversational turns. This measure tended to correlate positively with participants’ comprehension scores, but the effect was significant in only one signal-to-noise ratio condition. Post-hoc investigation suggests that intertalker saccade rate is driven by an interaction of hearing ability and conversational turn-taking events, which will be further discussed.

A wearable microphone array platform is used tolocalize stationary sound sources and amplify the sound inthe desired directions using several beamforming methods. Theplatform is equipped with inertial sensors and a magnetometerallowing predictions of source locations during orientationchanges and compensation for the displacement in the arrayconfiguration. The platform is modular, open and 3D printedto allow for easy reconfiguration of the array and for reuse inother applications, e.g., mobile robotics. The software componentsare based on open source. A new method for source localizationand signal reconstruction using Taylor expansion of the signals isproposed. This and various standard and non-standard Directionof Arrival (DOA) methods are evaluated in simulation andexperiments with the platform to track and reconstruct multipleand single sources. Results show that sound sources can belocalized and tracked robustly and accurately while rotating theplatform and that the proposed method outperforms standardmethods at reconstructing the signals.

People with hearing impairment typically have difficulties following conversations in multi-talker situations. Previous studies have shown that utilizing eye gaze to steer audio through beamformers could be a solution for those situations. Recent studies have shown that in-ear electrodes that capture electrooculography in the ear (EarEOG) can estimate the eye-gaze relative to the head, when the head was fixed. The head movement can be estimated using motion sensors around the ear to create an estimate of the absolute eye-gaze in the room. In this study, an experiment was designed to mimic a multi-talker situation in order to study and model the EarEOG signal when participants attempted to follow a conversation. Eleven hearing impaired participants were presented speech from the DAT speech corpus (Bo Nielsen et al., 2014), with three targets positioned at -30 degrees, 0 degrees and +30 degrees azimuth. The experiment was run in two setups: one where the participants had their head fixed in a chinrest, and the other where they were free to move their head. The participants task was to focus their visual attention on an LED-indicated target that changed regularly. A model was developed for the relative eye-gaze estimation, taking saccades, fixations, head movement and drift from the electrode-skin half-cell into account. This model explained 90.5% of the variance of the EarEOG when the head was fixed, and 82.6% when the head was free. The absolute eye-gaze was also estimated utilizing that model. When the head was fixed, the estimation of the absolute eye-gaze was reliable. However, due to hardware issues, the estimation of the absolute eye-gaze when the head was free had a variance that was too large to reliably estimate the attended target. Overall, this study demonstrated the potential of estimating absolute eye-gaze using EarEOG and motion sensors around the ear.

The unscented Kalman filter (UKF) is a very popular solution for estimation of the state in nonlinear systems. Similar to the extended Kalman filter (EKF) and contrary to the Kalman filter (KF) for linear systems, the UKF provides no guarantees that the filter updates will improve the filtered state estimate. In the past, the iterated EKF (IEKF) has been suggested as a way to online monitor the filter performance and try to improve it using optimization techniques. In this paper we do the same for the UKF, deriving six iterated UKF (IUKF) variations based on two cost functions and three optimization algorithms. The methods are evaluated and compared to IEKF versions and to two versions of the iterative posterior linearization filter (IPLF) in three benchmark simulation studies. The results show that IUKF algorithms can be used as a derivative free alternative to IEKF, and provide insights about the different design choices available in IUKF algorithms.

The general Simultaneous Localisation and Mapping (SLAM) problem aims at estimating the state of a moving platform simultaneously with building a map of the local environment. There are essentially three classes of algorithms. EKF- SLAM and FastSLAM solve the problem on-line, while Nonlinear Least Squares (NLS) is a batch method. All of them scales badly with either the state dimension, the map dimension or the batch length. We investigate the EM algorithm for solving a generalized version of the NLS problem. This EM-SLAM algorithm solves two simpler problems iteratively, hence it scales much better with dimensions. The iterations switch between state estimation, where we propose an Extended Rauch-Tung-Striebel smoother, and map estimation, where a quasi-Newton method is suggested. The proposed method is evaluated in real experiments and also in simulations on a platform with a monocular camera attached to an inertial measurement unit. It is demonstrated to produce lower RMSE than with a standard Levenberg-Marquardt solver of NLS problem, at a computational cost that increases considerably slower. 

This paper presents three iterative methods for orientation estimation. The first two are based on iterated Extended Kalman filter (IEKF) formulations with different state representations. The first is using the well-known unit quaternion as state (q-IEKF) while the other is using orientation deviation which we call IMEKF. The third method is based on nonlinear least squares (NLS) estimation of the angular velocity which is used to parametrise the orientation. The results are obtained using Monte Carlo simulations and the comparison is done with the non-iterative EKF and multiplicative EKF (MEKF) as baseline. The result clearly shows that the IMEKF and the NLS-based method are superior to q-IEKF and all three outperform the non-iterative methods.

This paper investigates the usefulness of multi-frequency received signal strength (RSS) for indoor localization. Acollected set of data from four sites containing 7 frequencies fromdual receivers and a high accuracy reference positioning systemis presented. The collected data is also made publicly availablethrough ResearchGate. The data is analyzed with respect tospatial variations using Gaussian processes ( GP ). The resultsshow that there are more rapid signal variations across corridorsthan along them. The uniqueness of RSS fingerprints is analyzedsuggesting that sequences of measurements in smoothing, orsmoothing-like, algorithms that can handle temporary positionambiguities are likely the best choice for localization applications.

This paper presents a batch estimation method for Simultaneous Localization and Mapping (SLAM) using the Prediction Error Method (PEM). The estimation problem considers landmarks as parameter while treating dynamics using state space models. The gradient needed for parameter estimation is computed recursively using an Extended Kalman Filter (EKF). Results using simulations with a monocular camera and inertial sensors are presented and compared to a Nonlinear Least- Squares (NLS) estimator. The presented method produce both lower RMSE’s and scale better to the batch length. 

The extended Kalman filter (EKF) has been animportant tool for state estimation of nonlinear systems sinceits introduction. However, the EKF does not possess the same optimality properties as the Kalman filter, and may perform poorly. By viewing the EKF as an optimization problem it is possible to, in many cases, improve its performance and robustness. The paper derives three variations of the EKF by applying different optimisation algorithms to the EKF costfunction and relate these to the iterated EKF. The derived filters are evaluated in two simulation studies which exemplify the presented filters.

This paper presents a method for global pose estimation using inertial sensors, monocular vision, and ultra wide band (UWB) sensors. It is demonstrated that the complementary characteristics of these sensors can be exploited to provide improved global pose estimates, without requiring the introduction of any visible infrastructure, such as fiducial markers. Instead, natural landmarks are jointly estimated with the pose of the platform using a simultaneous localization and mapping framework, supported by a small number of easy-to-hide UWB beacons with known positions. The method is evaluated with data from a controlled indoor experiment with high precision ground truth. The results show the benefit of the suggested sensor combination and suggest directions for further work.

This paper presents a system which combines a zero-velocity-update-(ZUPT-)aided inertial navigation system (INS), using a foot-mounted inertial measurement unit (IMU), with opportunistic use of multi-frequency received signal strength (RSS) measurements. The system does not rely on maps or pre-collected data from surveys of the radio-frequency (RF) environment. Instead it builds its own database of collected RSS measurements during the course of the operation. New RSS measurements are continuously compared with the stored values in the database, and when the user returns to a previously visited area this can thus be detected. This enables loop-closures to be detected online and used for error drift correction. The system utilises a distributed particle simultaneous localization and mapping (DP-SLAM) algorithm which provides a flexible 2D navigation platform that can be extended with more sensors. The experimental results presented in this paper indicates that the developed RSS SLAM algorithm can, in many cases, significantly improve the positioning performance of a foot-mounted INS. 

Navigation and mapping in unknown environments is an important building block for increased autonomy of unmanned vehicles, since external positioning systems can be susceptible to interference or simply being inaccessible. Navigation and mapping require signal processing of vehicle sensor data to estimate motion relative to the surrounding environment and to simultaneously estimate various properties of the surrounding environment. Physical models of sensors, vehicle motion and external influences are used in conjunction with statistically motivated methods to solve these problems. This thesis mainly addresses three navigation and mapping problems which are described below.

We study how a vessel with known magnetic signature and a sensor network with magnetometers can be used to determine the sensor positions and simultaneously determine the vessel's route in an extended Kalman filter (EKF). This is a so-called simultaneous localisation and mapping (SLAM) problem with a reversed measurement relationship.

Previously determined hydrodynamic models for a remotely operated vehicle (ROV) are used together with the vessel's sensors to improve the navigation performance using an EKF. Data from sea trials is used to evaluate the system and the results show that especially the linear velocity relative to the water can be accurately determined.

The third problem addressed is SLAM with inertial sensors, accelerometers and gyroscopes, and an optical camera contained in a single sensor unit. This problem spans over three publications.

We study how a SLAM estimate, consisting of a point cloud map, the sensor unit's three dimensional trajectory and speed as well as its orientation, can be improved by solving a nonlinear least-squares (NLS) problem. NLS minimisation of the predicted motion error and the predicted point cloud coordinates given all camera measurements is initialised using EKF-SLAM.

We show how NLS-SLAM can be initialised as a sequence of almost uncoupled problems with simple and often linear solutions. It also scales much better to larger data sets than EKF-SLAM. The results obtained using NLS-SLAM are significantly better using the proposed initialisation method than if started from arbitrary points. A SLAM formulation using the expectation maximisation (EM) algorithm is proposed. EM splits the original problem into two simpler problems and solves them iteratively. Here the platform motion is one problem and the landmark map is the other. The first problem is solved using an extended Rauch-Tung-Striebel smoother while the second problem is solved with a quasi-Newton method. The results using EM-SLAM are better than NLS-SLAM both in terms of accuracy and complexity.

Simultaneous Localisation and Mapping (SLAM) denotes the problem of jointly localizing a moving platform and mapping the environment. This work studies the SLAM problem using a combination of inertial sensors, measuring the platform's accelerations and angular velocities, and a monocular camera observing the environment. We formulate the SLAM problem on a nonlinear least squares (NLS) batch form, whose solution provides a smoothed estimate of the motion and map. The NLS problem is highly nonconvex in practice, so a good initial estimate is required. We propose a multi-stage iterative procedure, that utilises the fact that the SLAM problem is linear if the platform's rotations are known. The map is initialised with camera feature detections only, by utilising feature tracking and clustering of  feature tracks. In this way, loop closures are automatically detected. The initialization method and subsequent NLS refinement is demonstrated on both simulated and real data.

Courses at the Master’s level in automatic control and signal processing cover mathematical theories and algorithms for control, estimation, and filtering. However, giving students practical experience in how to use these algorithms is also an important part of these courses. A goal is that the students should not only be able to understand and derive these algorithms, but also be able to apply them to real-life technical problems. The latter is achieved by assigning more time to the laboratory tutorials and designing them in such a way that the exercises are open for interpretation; an example of this would be giving the students more freedom to decide how to acquire the data needed to solve the given exercises.The students are asked to hand in a laboratory report in which they describe how they solved the exercises. This paper presents a double-blind peer-review process for laboratory reports, introduced at the Department of Electrical Engineering, Linköping University, Sweden. A survey was administered to students, and the results are summarized in this paper. Also discussed are the teachers’ experiences of peer review and of how students perform later in their education in writing their Master’s theses.

We compare dead-reckoning of underwater vehicles based on inertial sensors and kinematic models on one hand, and control inputs and hydrodynamic model on the other hand. Both can be used in an inertial navigation system to provide relative motion and absolute orientation of the vehicle. The combination of them is particularly useful for robust navigation in the case of missing data from the crucial doppler log speedometer. As a concrete result, we demonstrate that the performance critical doppler log can be replaced with longitudinal dynamics in the case of missing data, based on field test data of a remotely operated vehicle.

In this paper we present a solution to the simultaneous localisation and mapping (SLAM) problem using a camera and inertial sensors. Our approach is based on the maximum a posteriori (MAP) estimate of the complete SLAM problem. The resulting problem is posed in a nonlinear least-squares framework which we solve with the Gauss-Newton method. The proposed algorithm is evaluated on experimental data using a sensor platform mounted on an industrial robot. In this way, accurate ground truth is available, and the results are encouraging.

This paper studies the accuracy of the estimated eyepoint of an Optical See-Through Head-Mounted Display (OST HMD) calibrated using the Single Point Active Alignment Method (SPAAM). Quantitative evaluation of calibration procedures for OST HMDs is complicated as it is currently not possible to share the subject’s view. Temporarily replacing the subject’s eye with a camera during the calibration or evaluation stage has been proposed, but the uncertainty of a correct eyepoint estimation remains. In the experiment reported in this paper, subjects were used for all stages of calibration and the results were verified with a 3D measurement device. The nine participants constructed 25 visual alignments per calibration after which the estimated pinhole camera model was decomposed into its intrinsic and extrinsic parameters using two common methods. Unique to this experiment, compared to previous evaluations, is the measurement device used to cup the subject’s eyeball. It measures the eyepoint location relative to the head tracker, thereby establishing the calibration accuracy of the estimated eyepoint location. As the results on accuracy are expressed as individual pinhole camera parameters, rather than a compounded registration error, this paper complements  previously published work on parameter variance as the former denotes bias and the latter represents noise. Results indicate that the calibrated eyepoint is on average 5 cm away from its measured location and exhibits a vertical bias which potentially causes dipvergence for stereoscopic vision for objects located further away than 5.6 m. Lastly, this paper closes with a discussion on the suitability of the traditional pinhole camera model for OST HMD calibration.

The parameter estimation variance of the Single Point Active Alignment Method (SPAAM) is studied through an experiment where 11 subjects are instructed to create alignments using an Optical See-Through Head Mounted Display (OSTHMD) such that three separate correspondence point distributions are acquired. Modeling the OSTHMD and the subject's dominant eye as a pinhole camera, findings show that a correspondence point distribution well distributed along the user's line of sight yields less variant parameter estimates. The estimated eye point location is studied in particular detail. Thefindings of the experiment are complemented with simulated datawhich show that image plane orientation is sensitive to the numberof correspondence points. The simulated data also illustrates someinteresting properties on the numerical stability of the calibrationproblem as a function of alignment noise, number of correspondencepoints, and correspondence point distribution.

Processing and interpretation of visual content is essential to many systems and applications. This requires knowledge of how the content is sensed and also what is sensed. Such knowledge is captured in models which, depending on the application, can be very advanced or simple. An application example is scene reconstruction using a camera; if a suitable model of the camera is known, then a model of the scene can be estimated from images acquired at different, unknown, locations, yet, the quality of the scene model depends on the quality of the camera model. The opposite is to estimate the camera model and the unknown locations using a known scene model. In this work, two such problems are treated in two rather different applications.

There is an increasing need for navigation solutions less dependent on external navigation systems such as the Global Positioning System (GPS). Simultaneous Localisation and Mapping (slam) provides a solution to this by estimating both navigation states and some properties of the environment without considering any external navigation systems.

The first problem considers visual inertial navigation and mapping using a monocular camera and inertial measurements which is a slam problem. Our aim is to provide improved estimates of the navigation states and a landmark map, given a slam solution. To do this, the measurements are fused in an Extended Kalman Filter (ekf) and then the filtered estimates are used as a starting solution in a nonlinear least-squares problem which is solved using the Gauss-Newton method. This approach is evaluated on experimental data with accurate ground truth for reference.

In Augmented Reality (ar), additional information is superimposed onto the surrounding environment in real time to reinforce our impressions. For this to be a pleasant experience it is necessary to have a good models of the ar system and the environment.

The second problem considers calibration of an Optical See-Through Head Mounted Display system (osthmd), which is a wearable ar system. We show and motivate how the pinhole camera model can be used to represent the osthmd and the user’s eye position. The pinhole camera model is estimated using the Direct Linear Transformation algorithm. Results are evaluated in experiments which also compare different data acquisition methods.

The correct spatial registration between virtual and real objects in optical see-through augmented reality implies accurate estimates of the user’s eyepoint relative to the location and orientation of the display surface. A common approach is to estimate the display parameters through a calibration procedure involving a subjective alignment exercise. Human postural sway and targeting precision contribute to imprecise alignments, which in turn adversely affect the display parameter estimation resulting in registration errors between virtual and real objects. The technique commonly used has its origin incomputer vision, and calibrates stationary cameras using hundreds of correspondence points collected instantaneously in one video frame where precision is limited only by pixel quantization and image blur. Subsequently the input noise level is several order of magnitudes greater when a human operator manually collects correspondence points one by one. This paper investigates the effect of human alignment noise on view parameter estimation in an optical see-through head mounted display to determine how well astandard camera calibration method performs at greater noise levels than documented in computer vision literature. Through Monte-Carlo simulations we show that it is particularly difficult to estimate the user’s eyepoint in depth, but that a greater distribution of correspondence points in depth help mitigate the effects of human alignment noise.

The correct spatial registration between virtual and real objects in optical see-through augmented reality implies accurate estimates of the user’s eyepoint relative to the location and orientation of the display surface. A common approach is to estimate the display parameters through a calibration procedure involving a subjective alignment exercise. Human postural sway and targeting precision contribute to imprecise alignments, which in turn adversely affect the display parameter estimation resulting in registration errors between virtual and real objects. The technique commonly used has its origin incomputer vision, and calibrates stationary cameras using hundreds of correspondence points collected instantaneously in one video frame where precision is limited only by pixel quantization and image blur. Subsequently the input noise level is several order of magnitudes greater when a human operator manually collects correspondence points one by one. This paper investigates the effect of human alignment noise on view parameter estimation in an optical see-through head mounted display to determine how well astandard camera calibration method performs at greater noise levels than documented in computer vision literature. Through Monte-Carlo simulations we show that it is particularly difficult to estimate the user’s eyepoint in depth, but that a greater distribution of correspondence points in depth help mitigate the effects of human alignment noise.

Sensor localization is a central problem for sensor networks. If the sensor positions are uncertain, the target tracking ability of the sensor network is reduced. Sensor localization in underwater environments is traditionally addressed using acoustic range measurements involving known anchor or surface nodes. We explore the usage of triaxial magnetometers and a friendly vessel with known magnetic dipole to silently localize the sensors. The ferromagnetic field created by the dipole is measured by the magnetometers and is used to localize the sensors. The trajectory of the vessel and the sensor positions are estimated simultaneously using an Extended Kalman Filter (EKF). Simulations show that the sensors can be accurately positioned using magnetometers.