liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Regularization for Sparseness and Smoothness: Applications in System Identification and Signal Processing
Linköping University, Department of Electrical Engineering, Automatic Control. Linköping University, The Institute of Technology.
2010 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

In system identification, the Akaike Information Criterion (AIC) is a well known method to balance the model fit against model complexity. Regularization here acts as a price on model complexity. In statistics and machine learning, regularization has gained popularity due to modeling methods such as Support Vector Machines (SVM), ridge regression and lasso. But also when using a Bayesian approach to modeling, regularization often implicitly shows up and can be associated with the prior knowledge. Regularization has also had a great impact on many applications, and very much so in clinical imaging. In e.g., breast cancer imaging, the number of sensors is physically restricted which leads to long scantimes. Regularization and sparsity can be used to reduce that. In Magnetic Resonance Imaging (MRI), the number of scans is physically limited and to obtain high resolution images, regularization plays an important role.

Regularization shows-up in a variety of different situations and is a well known technique to handle ill-posed problems and to control for overfit. We focus on the use of regularization to obtain sparseness and smoothness and discuss novel developments relevant to system identification and signal processing.

In regularization for sparsity a quantity is forced to contain elements equal to zero, or to be sparse. The quantity could e.g., be the regression parameter vectorof a linear regression model and regularization would then result in a tool for variable selection. Sparsity has had a huge impact on neighboring disciplines, such as machine learning and signal processing, but rather limited effect on system identification. One of the major contributions of this thesis is therefore the new developments in system identification using sparsity. In particular, a novel method for the estimation of segmented ARX models using regularization for sparsity is presented. A technique for piecewise-affine system identification is also elaborated on as well as several novel applications in signal processing. Another property that regularization can be used to impose is smoothness. To require the relation between regressors and predictions to be a smooth function is a way to control for overfit. We are here particularly interested in regression problems with regressors constrained to limited regions in the regressor-space e.g., a manifold. For this type of systems we develop a new regression technique, Weight Determination by Manifold Regularization (WDMR). WDMR is inspired byapplications in biology and developments in manifold learning and uses regularization for smoothness to obtain smooth estimates. The use of regularization for smoothness in linear system identification is also discussed.

The thesis also presents a real-time functional Magnetic Resonance Imaging (fMRI) bio-feedback setup. The setup has served as proof of concept and been the foundation for several real-time fMRI studies.

Place, publisher, year, edition, pages
Linköping: Linköping University Electronic Press , 2010. , 89 p.
Series
Linköping Studies in Science and Technology. Dissertations, ISSN 0345-7524 ; 1351
Keyword [en]
Regularization, sparsity, smothness, lasso, l1, fMRI, bio-feedback
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:liu:diva-60531ISBN: 978-91-7393-287-5 (print)OAI: oai:DiVA.org:liu-60531DiVA: diva2:360033
Public defence
2010-11-26, I101, Hus I, Campus Valla, Linköping University, Linköping, 10:15 (Swedish)
Opponent
Supervisors
Available from: 2010-11-03 Created: 2010-10-16 Last updated: 2010-11-03Bibliographically approved
List of papers
1. Segmentation of ARX-Models using Sum-of-Norms Regularization
Open this publication in new window or tab >>Segmentation of ARX-Models using Sum-of-Norms Regularization
2010 (English)In: Automatica, ISSN 0005-1098, E-ISSN 1873-2836, Vol. 46, no 6, 1107-1111 p.Article in journal (Refereed) Published
Abstract [en]

Segmentation of time-varying systems and signals into models whose parameters are piecewise constant in time is an important and well studied problem. Here it is formulated as a least-squares problem with sum-of-norms regularization over the state parameter jumps. a generalization of L1-regularization. A nice property of the suggested formulation is that it only has one tuning parameter, the regularization constant which is used to trade-off fit and the number of segments.

Place, publisher, year, edition, pages
Elsevier, 2010
Keyword
Segmentation, Regularization, ARX-models
National Category
Control Engineering
Identifiers
urn:nbn:se:liu:diva-58384 (URN)10.1016/j.automatica.2010.03.013 (DOI)000278675500020 ()
Projects
CADICS
Available from: 2010-08-13 Created: 2010-08-11 Last updated: 2017-12-12
2. Identification of Piecewise Affine Systems Using Sum-of-Norms Regularization
Open this publication in new window or tab >>Identification of Piecewise Affine Systems Using Sum-of-Norms Regularization
2011 (English)In: Proceedings of the 18th IFAC World Congress, 2011, 6640-6645 p.Conference paper, Published paper (Refereed)
Abstract [en]

Systems today often consist of logic switches working besides continuous physical systems. The demand for novel hybrid system identification algorithms is therefore of growing interest and essential for the development of control algorithms for this type of systems. An important type of hybrid systems is piecewise affine systems. The identification of piecewise affine systems is here tackled by overparametrizing and assigning a regressor-parameter to each of the observations. The regressor parameters are forced to be the same if that not causes a major increase in the fit term. The formulation takes the shape of a least-squares problem with sum-of-norms regularization over regressor parameter differences, a generalization of l1-regularization. The regularization constant is used to trade off fit and the number of partitions of the model.

Keyword
Hybrid systems modeling and control, Nonlinear system identification, Nonparametric methods
National Category
Control Engineering
Identifiers
urn:nbn:se:liu:diva-60984 (URN)10.3182/20110828-6-IT-1002.00611 (DOI)978-3-902661-93-7 (ISBN)
Conference
18th IFAC World Congress, Milano, Italy, 28 August-2 September, 2011
Projects
CADICS
Funder
Swedish Foundation for Strategic Research Swedish Research Council
Available from: 2013-04-04 Created: 2010-11-01 Last updated: 2013-07-10Bibliographically approved
3. Smoothed State Estimates under Abrupt Changes using Sum-of-Norms Regularization
Open this publication in new window or tab >>Smoothed State Estimates under Abrupt Changes using Sum-of-Norms Regularization
2012 (English)In: Automatica, ISSN 0005-1098, E-ISSN 1873-2836, Vol. 48, no 4, 595-605 p.Article in journal (Refereed) Published
Abstract [en]

The presence of abrupt changes, such as impulsive and load disturbances, commonly occur in applications, but make the state estimation problem considerably more difficult than in the standard setting with Gaussian process disturbance. Abrupt changes often introduce a jump in the state, and the problem is therefore readily and often treated by change detection techniques. In this paper, we take a different approach. The state smoothing problem for linear state space models is here formulated as a constrained least-squares problem with sum-of-norms regularization, a generalization of l1-regularization. This novel formulation can be seen as a convex relaxation of the well known generalized likelihood ratio method by Willsky and Jones. Another nice property of the suggested formulation is that it only has one tuning parameter, the regularization constant which is used to trade off fit and the number of jumps. Good practical choices of this parameter along with an extension to nonlinear state space models are given. 

Place, publisher, year, edition, pages
Elsevier, 2012
Keyword
State estimation, Impulsive disturbance, Load disturbance, Smoothing, Sparsity, Regularization, Change detection
National Category
Control Engineering
Identifiers
urn:nbn:se:liu:diva-60985 (URN)10.1016/j.automatica.2011.08.063 (DOI)000302766400002 ()
Projects
CADICS
Funder
Swedish Research Council
Available from: 2010-11-01 Created: 2010-11-01 Last updated: 2017-12-12
4. Trajectory Generation Using Sum-of-Norms Regularization
Open this publication in new window or tab >>Trajectory Generation Using Sum-of-Norms Regularization
2010 (English)In: Proceedings of the 49th IEEE Conference on Decision and Control, 2010, 540-545 p.Conference paper, Published paper (Refereed)
Abstract [en]

Many tracking problems are split into two sub-problems, first a smooth reference trajectory is generated that meet the control design objectives, and then a closed loop control system is designed to follow this reference trajectory as well as possible. Applications of this kind include (autonomous) vehicle navigation systems and robotics. Typically, a spline model is used for trajectory generation and another physical and dynamical model is used for the control design. Here we propose a direct approach where the dynamical model is used to generate a control signal that takes the state trajectory through the waypoints specified in the design goals. The strength of the proposed formulation is the methodology to obtain a control signal with compact representation and that changes only when needed, something often wanted in tracking. The formulation takes the shape of a constrained least-squares problem with sum-of-norms regularization, a generalization of the ℓ1-regularization. The formulation also gives a tool to, e.g. in model predictive control, prevent chatter in the input signal, and also select the most suitable instances for applying the control inputs.

Keyword
Closed loop systems, Control system synthesis, Least squares approximations, Predictive control
National Category
Control Engineering
Identifiers
urn:nbn:se:liu:diva-60983 (URN)10.1109/CDC.2010.5717368 (DOI)978-1-4244-7745-6 (ISBN)
Conference
49th IEEE Conference on Decision and Control, Atlanta, GA, USA, 15-17 December, 2010
Projects
CADICS
Available from: 2010-11-01 Created: 2010-11-01 Last updated: 2013-07-09
5. Weight Determination by Manifold Regularization
Open this publication in new window or tab >>Weight Determination by Manifold Regularization
2010 (English)In: Distributed Decision-Making and Control / [ed] Rolf Johansson and Anders Rantzer, Springer London, 2010, 195-214 p.Chapter in book (Refereed)
Abstract [en]

A new type of linear kernel smoother is derived and studied. The smoother, referred to as weight determination by manifold regularization, is the solution to a regularized least squares problem. The regularization avoids overfitting and can be used to express prior knowledge of an underlying smooth function. An interesting property ofthe kernel smoother is that it is well suited for systems govern by the semi-supervised smoothness assumption. Several examples are given to illustrate this property. We also discuss why these types of techniques can have a potential interest for the system identification community.

Place, publisher, year, edition, pages
Springer London, 2010
Series
Lecture Notes in Control and Information Sciences, ISSN 0170-8643 ; 417
Keyword
Regularization, Weight determination
National Category
Control Engineering
Identifiers
urn:nbn:se:liu:diva-60987 (URN)10.1007/978-1-4471-2265-4_9 (DOI)978-1-4471-2264-7 (ISBN)978-1-4471-2265-4 (ISBN)
Available from: 2010-11-01 Created: 2010-11-01 Last updated: 2013-09-29
6. On the Estimation of Transfer Functions, Regularizations and Gaussian Processes – Revisited
Open this publication in new window or tab >>On the Estimation of Transfer Functions, Regularizations and Gaussian Processes – Revisited
2010 (English)In: Proceedings of the 18th IFAC World Congress, 2010, 2303-2308 p.Conference paper, Published paper (Refereed)
Abstract [en]

Intrigued by some recent results on impulse response estimation by kernel and nonparametric techniques, we revisit the old problem of transfer function estimation from input-output measurements.We formulate a classical regularization approach, focused on finite impulse response (FIR) models, and find that regularization is necessary to cope with the high variance problem. This basic, regularized least squares approach is then a focal point for interpreting other techniques, like Bayesian inference and Gaussian process regression.

Keyword
System identification, Transfer function estimation, Reqularization
National Category
Control Engineering
Identifiers
urn:nbn:se:liu:diva-60986 (URN)10.3182/20110828-6-IT-1002.00573 (DOI)978-3-902661-93-7 (ISBN)
Conference
18th IFAC World Congress, Milano, Italy, 28 August-2 September, 2011
Funder
Swedish Foundation for Strategic Research Swedish Research Council
Note

Submitted

Available from: 2010-11-01 Created: 2010-11-01 Last updated: 2016-01-11
7. Enabling Bio-Feedback using Real-Time fMRI
Open this publication in new window or tab >>Enabling Bio-Feedback using Real-Time fMRI
Show others...
2008 (English)In: 47th IEEE Conference on Decision and Control, 2008, CDC 2008, IEEE , 2008, 3336-3341 p.Conference paper, Published paper (Refereed)
Abstract [en]

Despite the enormous complexity of the human mind, fMRI techniques are able to partially observe the state of a brain in action. In this paper we describe an experimental setup for real-time fMRI in a bio-feedback loop. One of the main challenges in the project is to reach a detection speed, accuracy and spatial resolution necessary to attain sufficient bandwidth of communication to close the bio-feedback loop. To this end we have banked on our previous work on real-time filtering for fMRI and system identification, which has been tailored for use in the experiment setup. In the experiments presented the system is trained to estimate where a person in the MRI scanner is looking from signals derived from the visual cortex only. We have been able to demonstrate that the user can induce an action and perform simple tasks with her mind sensed using real-time fMRI. The technique may have several clinical applications, for instance to allow paralyzed and "locked in" people to communicate with the outside world. In the meanwhile, the need for improved fMRI performance and brain state detection poses a challenge to the signal processing community. We also expect that the setup will serve as an invaluable tool for neuro science research in general.

Place, publisher, year, edition, pages
IEEE, 2008
Series
IEEE Conference on Decision and Control. Proceedings, ISSN 0191-2216
Keyword
fMRI, System identification, Bio-feedback
National Category
Engineering and Technology Control Engineering
Identifiers
urn:nbn:se:liu:diva-44641 (URN)10.1109/CDC.2008.4738759 (DOI)000307311603077 ()77222 (Local ID)978-1-4244-3123-6 (ISBN)e-978-1-4244-3124-3 (ISBN)77222 (Archive number)77222 (OAI)
Conference
47th IEEE Conference on Decision and Control, Cancun, Mexico, December, 2008
Note

©2010 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. Henrik Ohlsson, Joakim Rydell, Anders Brun, Jacob Roll, Mats Andersson, Anders Ynnerman and Hans Knutsson, Enabling Bio-Feedback Using Real-Time fMRI, 2008, Proceedings of the 47th IEEE Conference on Decision and Control, 2008, 3336.

Available from: 2009-10-10 Created: 2009-10-10 Last updated: 2015-10-08Bibliographically approved

Open Access in DiVA

Regularization for Sparseness and Smoothness : Applications in System Identification and Signal Processing(773 kB)1595 downloads
File information
File name FULLTEXT02.pdfFile size 773 kBChecksum SHA-512
c0d76c6c12b0212d27d002e683cac1335a9fe7bb6bee15c3459f941c37865e31149293fbe176b47e10f031f4d85afad902145872d52027b82bc9e32952f96f1b
Type fulltextMimetype application/pdf
Cover(36 kB)29 downloads
File information
File name COVER02.pdfFile size 36 kBChecksum SHA-512
259162ef5a76bd0847da54a7c6236268f3f977acd99d1789e783392b47a1461802b4a00557c896b71a8b0b8c9e4f34638710f5f3b3d19a2ea44e24a5040fc767
Type coverMimetype application/pdf

Authority records BETA

Ohlsson, Henrik

Search in DiVA

By author/editor
Ohlsson, Henrik
By organisation
Automatic ControlThe Institute of Technology
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 1595 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1977 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf