The paper is concerned with methods for computing the best low multilinear rank approximation of large and sparse tensors. Krylov-type methods have been used for this problem; here block versions are introduced. For the computation of partial eigenvalue and singular value decompositions of matrices the Krylov-Schur (restarted Arnoldi) method is used. A generalization of this method to tensors is described, for computing the best low multilinear rank approximation of large and sparse tensors. In analogy to the matrix case, the large tensor is only accessed in multiplications between the tensor and blocks of vectors, thus avoiding excessive memory usage. It is proved that if the starting approximation is good enough, then the tensor Krylov-Schur method is convergent. Numerical examples are given for synthetic tensors and sparse tensors from applications, which demonstrate that for most large problems the Krylov-Schur method converges faster and more robustly than higher order orthogonal iteration.

The problem of partitioning a large and sparse tensor is considered, where the tensor consists of a sequence of adjacency matrices. Theory is developed that is a generalization of spectral graph partitioning. A best rank-(2,2,lambda) approximation is computed for lambda=1,2,3, and the partitioning is computed from the orthogonal matrices and the core tensor of the approximation. It is shown that if the tensor has a certain reducibility structure, then the solution of the best approximation problem exhibits the reducibility structure of the tensor. Further, if the tensor is close to being reducible, then still the solution of the exhibits the structure of the tensor. Numerical examples with synthetic data corroborate the theoretical results. Experiments with tensors from applications show that the method can be used to extract relevant information from large, sparse, and noisy data.

It is well known that the classical exploratory factor analysis (EFA) of data with more observations than variables has several types of indeterminacy. We study the factor indeterminacy and show some new aspects of this problem by considering EFA as a specific data matrix decomposition. We adopt a new approach to the EFA estimation and achieve a new characterization of the factor indeterminacy problem. A new alternative model is proposed, which gives determinate factors and can be seen as a semi-sparse principal component analysis (PCA). An alternating algorithm is developed, where in each step a Procrustes problem is solved. It is demonstrated that the new model/algorithm can act as a specific sparse PCA and as a low-rank-plus-sparse matrix decomposition. Numerical examples with several large data sets illustrate the versatility of the new model, and the performance and behaviour of its algorithmic implementation.

Bilinear tensor least squares problems occur in applications such as Hammerstein system identification and social network analysis. A linearly constrained problem of medium size is considered, and nonlinear least squares solvers of Gauss-Newton-type are applied to numerically solve it. The problem is separable, and the variable projection method can be used. Perturbation theory is presented and used to motivate the choice of constraint. Numerical experiments with Hammerstein models and random tensors are performed, comparing the different methods and showing that a variable projection method performs best.

n/a

n/a

n/a

Axel Ruhe passed away April 4, 2015. He was cross-country-skiing with friends in the Swedish mountains when after 21 km he suddenly died. He is survived by his wife Gunlaug and three children from his first marriage....

Partial least squares is a common technique for multivariate regression. The pro- cedure is recursive and in each step basis vectors are computed for the explaining variables and the solution vectors. A linear model is fitted by projection onto the span of the basis vectors. The procedure is mathematically equivalent to Golub-Kahan bidiagonalization, which is a Krylov method, and which is equiv- alent to a pair of matrix factorizations. The vectors of regression coefficients and prediction are non-linear functions of the right hand side. An algorithm for computing the Frechet derivatives of these functions is derived, based on perturbation theory for the matrix factorizations. From the Frechet derivative of the prediction vector one can compute the number of degrees of freedom, which can be used as a stopping criterion for the recursion. A few numerical examples are given.

n/a

In this paper, we propose a method to find the best Kronecker product approximationof the blurring operator which arises in three dimensional image restoration problems. We show thatthis problem can be reduced to a well known rank-1 approximation of the scaled three dimensionalpoint spread function (PSF) array, which is much smaller. This approximation can be used as apreconditioner in solving image restoration problems with iterative methods. The comparison ofthe approximation by the new scaled PSF array and approximation by the original PSF array that is used in [J. G. Nagy and M. E. Kilmer, IEEE Trans. Image Process., 15 (2006), pp. 604–613],confirms the performance of the new proposed approximation.

An ill-posed Cauchy problem for a 3D elliptic partial differential equation with variable coefficients is considered. A well-posed quasi-boundary-value (QBV) problem is given to approximate it. Some stability estimates are given. For the numerical implementation, a large sparse system is obtained from discretizing the QBV problem using the finite difference method. A left-preconditioned generalized minimum residual method is used to solve the large system effectively. For the preconditioned system, a fast solver using the fast Fourier transform is given. Numerical results show that the method works well.

The sideways parabolic equation (SPE) is a model of the problem of determiningthe temperature on the surface of a body from the interior measurements. Mathematically it can beformulated as a noncharacteristic Cauchy problem for a parabolic partial differential equation. Thisproblem is severely ill-posed in an L2 setting. We use a preconditioned generalized minimum residualmethod (GMRES) to solve a two-dimensional SPE with variable coefficients. The preconditioner issingular and chosen in a way that allows efficient implementation using the FFT. The preconditioneris a stabilized solver for a nearby problem with constant coefficients, and it reduces the numberof iterations in the GMRES algorithm significantly. Numerical experiments are performed thatdemonstrate the performance of the proposed method.

Using the technique of semantic mirroring a graph is obtained that represents words and their translationsfrom a parallel corpus or a bilingual lexicon. The connectedness of the graph holds information about the semanticrelations of words that occur in the translations. Spectral graph theory is used to partition the graph, which leadsto a grouping of the words in different clusters. We illustrate the method using a small sample of seed words froma lexicon of Swedish and English adjectives and discuss its application to computational lexical semantics andlexicography.

Several Krylov-type procedures are introduced that generalize matrix Krylov methods for tensor computations. They are denoted minimal Krylov recursion, maximal Krylov recursion, and contracted tensor product Krylov recursion. It is proved that, for a given tensor A with multilinear rank-(p; q; r), the minimal Krylov recursion extracts the correct subspaces associated to the tensor in p+q+r number of tensor-vector-vector multiplications. An optimized minimal Krylov procedure is described that, for a given multilinear rank of an approximation, produces a better approximation than the standard minimal recursion. We further generalize the matrix Krylov decomposition to a tensor Krylov decomposition. The tensor Krylov methods are intended for the computation of low multilinear rank approximations of large and sparse tensors, but they are also useful for certain dense and structured tensors for computing their higher order singular value decompositions or obtaining starting points for the best low-rank computations of tensors. A set of numerical experiments, using real-world and synthetic data sets, illustrate some of the properties of the tensor Krylov methods.

Almost singular linear systems arise in discrete ill-posed problems. Either because ofthe intrinsic structure of the problem or because of preconditioning, the spectrum of the coefficientmatrix is often characterized by a sizable gap between a large group of numerically zero eigenvaluesand the rest of the spectrum. Correspondingly, the right-hand side has leading eigencomponentsassociated with the eigenvalues away from zero. In this paper the effect of this setting in theconvergence of the generalized minimal residual (GMRES) method is considered. It is shown thatin the initial phase of the iterative algorithm, the residual components corresponding to the largeeigenvalues are reduced in norm, and these can be monitored without extra computation. Theanalysis is supported by numerical experiments. In particular, ill-posed Cauchy problems for partialdifferential equations with variable coefficients are considered, where the preconditioner is a fast,low-rank solver for the corresponding problem with constant coefficients.

The concepts of tensors with diagonal and circulant structure are defined and aframework is developed for the analysis of such tensors. It is shown a tensor of arbitraryorder, which is circulant with respect to two particular modes, can be diagonalized inthose modes by discrete Fourier transforms. This property can be used in the efficientsolution of linear systems involving contractive products of tensors with circulantstructure. Tensors with circulant structure occur in models for image blurring withperiodic boundary conditions. It is shown that the new framework can be applied tosuch problems.

The problem of computing the best rank-(p,q,r) approximation of a third order tensor is considered. First the problem is reformulated as a maximization problem on a product of three Grassmann manifolds. Then expressions for the gradient and the Hessian are derived in a local coordinate system at a stationary point, and conditions for a local maximum are given. A first order perturbation analysis is performed using the Grassmann manifold framework. The analysis is illustrated in a few examples, and it is shown that the perturbation theory for the singular value decomposition is a special case of the tensor theory.

The sideways Heat equation (SHE) is a model of the problem of determining the temperature on the surface of a body from the interior measurements. Mathematically it can be formulated as a non-characteristic Cauchy problem for a  parabolic partial differential equation. This problem is severely ill-posed: the solution does not depend continuously on the data. We use a preconditioned Generalized Minimum Residuals Method (GMRES) to solve a 1D SHE. Regularization is used in the preconditioner as well as truncating the GMRES algorithm. Numerical experiments demonstrate that the proposed method works well.

The sideways parabolic equation (SPE) is a model of the problem of determining the temperature on the surface of a body from the interior measurements. Mathematically it can be formulated as a non-characteristic Cauchy problem for a parabolic partial differential equation. This problem is severely ill-posed: the solution does not depend continuously on the data. We consider both one and two-dimensional SPE with both constant and variable coefficients. We apply the preconditioned Generalized Minimum Residuals Method (GMRES) on these problems. Preconditioners are chosen in ways that allow efficient implementation using the Fast Fourier Transform (FFT). Regularization is used in the preconditioner as well as truncating the GMRES algorithm. Numerical experiments demonstrate that the proposed method works well.

A Cauchy problem for elliptic equations with nonhomogeneous Neumann datain a cylindrical domain is investigated in this paper. For the theoretical aspect the a-prioriand a-posteriori parameter choice rules are suggested and the corresponding error estimatesare obtained. About the numerical aspect, for a simple case results given by twomethods based on the discrete Sine transform and the finite difference method are presented;an idea of left-preconditioned GMRES (Generalized Minimum Residual) methodis proposed to deal with the high dimensional case to save the time; a view of dealingwith a general domain is suggested. Some ill-posed problems regularized by the quasiboundary-value method are listed and some rules of this method are suggested.

We consider a Cauchy problem for a parabolic equation as a mathematical model of the temperature development inside a suction pyrometer. Such devices are often used to calibrate the temperature sensor in a combustion chamber. Mathematically the problem is severely ill-posed and needs to be regularized. The model is simplified to make it one-dimensional in space. The temperature measurements are done in two steps. First, the heat transfer coefficient is approximated via a calibration experiment. Then the gas temperature in the combustion chamber is computed using a convection boundary condition. In both steps one computes the surface temperature and heat flux based on interior measurements in the thermocouple. Numerical examples are presented to test the proposed approach.

Using the technique of ”semantic mirroring”a graph is obtained that representswords and their translations from a parallelcorpus or a bilingual lexicon. The connectednessof the graph holds informationabout the different meanings of words thatoccur in the translations. Spectral graphtheory is used to partition the graph, whichleads to a grouping of the words accordingto different senses. We also report resultsfrom an evaluation using a small sample ofseed words from a lexicon of Swedish andEnglish adjectives.

n/a

The problem of approximating a matrix by another matrix of lower rank, when a modest portion of its elements are missing, is considered. The solution is obtained using Newtons algorithm to find a zero of a vector field on a product manifold. As a preliminary the algorithm is formulated for the well-known case with no missing elements where also a rederivation of the correction equation in a block Jacobi-Davidson method is included. Numerical examples show that the Newton algorithm grows more efficient than an alternating least squares procedure as the amount of missing values increases.

We consider a backward parabolic partial differential equation (BPPDE) withvariable coefficient a.x; t / in time. A new modification is used on the logarithmic convexitymethod to obtain a conditional stability estimate. Based on a formal solution, wereveal the essence of the ill-posedness and propose a simple regularization method. Moreover,we apply the regularization method to two representative cases. The results of boththeoretical and numerical performance show the validity of our method.

Published in honor of his 70th birthday, this volume explores and celebrates the work of G.W. (Pete) Stewart, a world-renowned expert in computational linear algebra. This volume includes: forty-four of Stewart's most influential research papers in two subject areas: matrix algorithms, and rounding and perturbation theory; a biography of Stewart; a complete list of his publications, students, and honors; selected photographs; and commentaries on his works in collaboration with leading experts in the field. G.W. Stewart: Selected Works with Commentaries will appeal to graduate students, practitioners, and researchers in computational linear algebra and the history of mathematics.

We derive a Newton method for computing the best rank-$(r_1,r_2,r_3)$ approximation of a given $J\times K\times L$ tensor $\mathcal{A}$. The problem is formulated as an approximation problem on a product of Grassmann manifolds. Incorporating the manifold structure into Newton's method ensures that all iterates generated by the algorithm are points on the Grassmann manifolds. We also introduce a consistent notation for matricizing a tensor, for contracted tensor products and some tensor-algebraic manipulations, which simplify the derivation of the Newton equations and enable straightforward algorithmic implementation. Experiments show a quadratic convergence rate for the Newton–Grassmann algorithm.

We study the numerical solution of a Cauchy problem for a self-adjoint elliptic partial differential equation u(zz) - L-u = 0 in three space dimensions (x, y, z), where the domain is cylindrical in z. Cauchy data are given on the lower boundary and the boundary values on the upper boundary are sought. The problem is severely ill-posed. The formal solution is written as a hyperbolic cosine function in terms of the two-dimensional elliptic operator L (via its eigenfunction expansion), and it is shown that the solution is stabilized (regularized) if the large eigenvalues are cut off. We suggest a numerical procedure based on the rational Krylov method, where the solution is projected onto a subspace generated using the operator L-1. This means that in each Krylov step, a well-posed two-dimensional elliptic problem involving L is solved. Furthermore, the hyperbolic cosine is evaluated explicitly only for a small symmetric matrix. A stopping criterion for the Krylov recursion is suggested based on the relative change of an approximate residual, which can be computed very cheaply. Two numerical examples are given that demonstrate the accuracy of the method and the efficiency of the stopping criterion.

The properties of materials which are present in a scene determine how geometry reflects and distributes light in the scene. This text presents work-in-progress on numerical analysis of bidirectional reflection distribution functions (BRDF) corresponding to various materials, with a focus on inverse rendering. An analysis of these functions is vital for the understanding of the behaviour of reflected light under different lighting conditions, and in the application of inverse rendering, it is important in order to determine what quality one can expect from recovered data. We discuss the singular value decompositions of a few materials, their effect on the ill-posedness of the inverse problem related to the reflectance equation and how regularization affects the solution of the problem.

In a recent paper, claims were made that most current implementations of PLS provide wrong and misleading residuals [1]. In this paper the relation between PLS and Lanczos bidiagonalization is described and it is shown that there is a good rationale behind current implementations of PLS. Most importantly, the residuals determined in current implementations of PLS are independent of the scores used for predicting the dependent variable(s). Oppositely, in the newly suggested approach, the residuals are correlated to the scores and hence may be high due to variation that is actually used for predicting. It is concluded that the current practice of calculating residuals be maintained.

In this paper we present two algorithms for handwritten digit classification based on the higher order singular value decomposition (HOSVD). The first algorithm uses HOSVD for construction of the class models and achieves classification results with error rate lower than 6%. The second algorithm uses the HOSVD for tensor approximation simultaneously in two modes. Classification results for the second algorithm are almost down at 5% even though the approximation reduces the original training data with more than 98% before the construction of the class models. The actual classification in the test phase for both algorithms is conducted by solving a series least squares problems. Considering computational amount for the test presented the second algorithm is twice as efficient as the first one.

The mathematical and numerical properties of an ill-posed Cauchy problem for a convection - diffusion equation are investigated in this study. The problem is reformulated as a Volterra integral equation of the first kind with a smooth kernel. The rate of decay of the singular values of the integral operator determines the degree of ill-posedness. The purpose of this article is to study how the convection term influences the degree of ill-posedness by computing numerically the singular values. It is also shown that the sign of the coefficient in the convection term determines the rate of decay of the singular values. Some numerical examples are also given to illustrate the theory.

Numerical techniques for data analysis and feature extraction are discussed using the framework of matrix rank reduction. The singular value decomposition (SVD) and its properties are reviewed, and the relation to Latent Semantic Indexing (LSI) and Principal Component Analysis (PCA) is described. Methods that approximate the SVD are reviewed. A few basic methods for linear regression, in particular the Partial Least Squares (PLS) method, arepresented, and analyzed as rank reduction methods. Methods for feature extraction, based on centroids and the classical Linear Discriminant Analysis (LDA), as well as an improved LDA based on the generalized singular value decomposition (LDA/GSVD) are described. The effectiveness of these methods are illustrated using examples from information retrieval, and 2 dimensional representation of clustered data.

Ideas and algorithms from numerical linear algebra are important in several areas of data mining. We give an overview of linear algebra methods in text mining (information retrieval), pattern recognition (classification of handwritten digits), and Page Rank computations for web search engines. The emphasis is on rank reduction as a method of extracting information from a data matrix, low-rank approximation of matrices using the singular value decomposition and clustering, and on eigenvalue methods for network analysis. © Cambridge University Press, 2006.

A two-dimensional inverse steady state heat conduction problem in the unit square is considered. Cauchy data are given for y ≤ 0, and boundary data are for x ≤ 0 and x ≤ 1. The elliptic operator is self-adjoint with non-constant, smooth coefficients. The solution for y ≤ 1 is sought. This Cauchy problem is ill-posed in an L2-setting. A stability functional is defined, for which a differential inequality is derived. Using this inequality a stability result of Hölder type is proved. It is demonstrated explicitly how the stability depends on the smoothness of the coefficients. The results can also be used for rectangle-like regions that can be mapped conformally onto a rectangle. © 2005 IOP Publishing Ltd.

The minimization of linear functionals defined on the solutions of discrete ill-posed problems arises, e.g., in the computation of confidence intervals for these solutions. In 1990, Eldén proposed an algorithm for this minimization problem based on a parametric programming reformulation involving the solution of a sequence of trust-region problems, and using matrix factorizations. In this paper, we describe MLFIP, a large-scale version of this algorithm where a limited-memory trust-region solver is used on the subproblems. We illustrate the use of our algorithm in connection with an inverse heat conduction problem.

We consider two dimensional inverse steady state heat conductionproblems in complex geometries. The coefficients of the elliptic equation are assumed to be non-constant. Cauchy data are given on onepart of the boundary and we want to find the solution in the wholedomain. The problem is ill--posed in the sense that the solution doesnot depend continuously on the data.

Using an orthogonal coordinate transformation the domain is mappedonto a rectangle. The Cauchy problem can then be solved by replacing one derivative by a bounded approximation. The resulting well--posed problem can then be solved by a method of lines. A bounded approximation of the derivative can be obtained by differentiating a cubic spline, that approximate the function in theleast squares sense. This particular approximation of the derivativeis computationally efficient and flexible in the sense that its easy to handle different kinds of boundary conditions.This inverse problem arises in iron production, where the walls of amelting furnace are subject to physical and chemical wear. Temperature and heat--flux data are collected by several thermocouples locatedinside the walls. The shape of the interface between the molten ironand the walls can then be determined by solving an inverse heatconduction problem.  In our work we make extensive use of Femlab for creating testproblems. By using FEMLAB we solve relatively complex model problems for the purpose of creating numerical test data used for validating our methods. For the types of problems we are intressted in numerical artefacts appear, near corners in the domain, in the gradients that Femlab calculates. We demonstrate why this happen and also how we deal with the problem.

We consider two dimensional inverse steady state heat conduction problems in complex geometries. The coefficients of the elliptic equation are assumed to be non-constant. Cauchy data are given on one part of the boundary and we want to find the solution in the whole domain. The problem is ill--posed in the sense that the solution does not depend continuously on the data. Using an orthogonal coordinate transformation the domain is mapped onto a rectangle. The Cauchy problem can then be solved by replacing one derivative by a bounded approximation. The resulting well--posed problem can then be solved by a method of lines. A bounded approximation of the derivative can be obtained by differentiating a cubic spline, that approximate the function in the least squares sense. This particular approximation of the derivative is computationally efficient and flexible in the sense that its easy to handle different kinds of boundary conditions. This inverse problem arises in iron production, where the walls of a melting furnace are subject to physical and chemical wear. Temperature and heat--flux data are collected by several thermocouples located inside the walls. The shape of the interface between the molten iron and the walls can then be determined by solving an inverse heat conduction problem. In our work we make extensive use of Femlab for creating test problems. By using Femlab we solve relatively complex model problems for the purpose of creating numerical test data used for validating our methods. For the types of problems we are intressted in numerical artefacts appear, near corners in the domain, in the gradients that Femlab calculates. We demonstrate why this happen and also how we deal with the problem.

Gas sensors are used in many different application areas. As many gas sensor components are battery heated, one major limit of the operation time is the power dissipated as heat. The aim of this work has been to simulate the heat transfer on a hydrogen gas sensor component. Modelling and simulations have been performed in FEMLAB. The partial differential equation with boundary conditions was solved and the solution was validated against experimental data. Convection increases with the increase of hydrogen concentration. A great effort was made to find a model for the convection. When the simulations were compared to experiments, it turned out that the theoretical convection model was insufficient to describe this small system involving hydrogen, which was an unexpected but interesting result.

In previous work by Stoica and Viberg the reduced-rank regression problem is solved in a maximum likelihood sense. The present paper proposes an alternative numerical procedure. The solution is written in terms of the principal angles between subspaces spanned by the data matrices. It is demonstrated that the solution is meaningful also in the case when the maximum likelihood criterion is not valid. A numerical example is given. Copyright (c) 2005 John Wiley & Sons, Ltd.

Multiple linear regression is considered and the partial least-squares method (PLS) for computing a projection onto a lower-dimensional subspace is analyzed. The equivalence of PLS to Lanczos bidiagonalization is a basic part of the analysis. Singular value analysis, Krylov subspaces, and shrinkage factors are used to explain why, in many cases, PLS gives a faster reduction of the residual than standard principal components regression. It is also shown why in some cases the dimension of the subspace, given by PLS, is not as small as desired.