liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Learning via nonlinear conjugate gradients and depth-varying neural ODEs
Linköping University, Department of Science and Technology, Physics, Electronics and Mathematics. Linköping University, Faculty of Science & Engineering.
Linköping University, Department of Science and Technology, Media and Information Technology. Linköping University, Faculty of Science & Engineering.ORCID iD: 0000-0002-9217-9997
Linköping University, Department of Science and Technology, Physics, Electronics and Mathematics. Linköping University, Faculty of Science & Engineering.
Linköping University, Department of Science and Technology, Physics, Electronics and Mathematics. Linköping University, Faculty of Science & Engineering.ORCID iD: 0000-0001-9066-7922
Show others and affiliations
(English)Manuscript (preprint) (Other academic)
Abstract [en]

The inverse problem of supervised reconstruction of depth-variable (time-dependent) parameters in a neural ordinary differential equation (NODE) is considered, that means finding the weights of a residual network with time continuous layers. The NODE is treated as an isolated entity describing the full network as opposed to earlier research, which embedded it between pre- and post-appended layers trained by conventional methods. The proposed parameter reconstruction is done for a general first order differential equation by minimizing a cost functional covering a variety of loss functions and penalty terms. A nonlinear conjugate gradient method (NCG) is derived for the minimization. Mathematical properties are stated for the differential equation and the cost functional. The adjoint problem needed is derived together with a sensitivity problem. The sensitivity problem can estimate changes in the network output under perturbation of the trained parameters. To preserve smoothness during the iterations the Sobolev gradient is calculated and incorporated. As a proof-of-concept, numerical results are included for a NODE and two synthetic datasets, and compared with standard gradient approaches (not based on NODEs). The results show that the proposed method works well for deep learning with infinite numbers of layers, and has built-in stability and smoothness. 

Keywords [en]
Artificial neural networks, conjugate gradient method, deep learning, inverse problems, neural ordinary differential equations, Sobolev gradient
National Category
Mathematics Computer and Information Sciences
Identifiers
URN: urn:nbn:se:liu:diva-182907OAI: oai:DiVA.org:liu-182907DiVA, id: diva2:1637363
Available from: 2022-02-14 Created: 2022-02-14 Last updated: 2023-04-03
In thesis
1. Inverse Problems for Tumour Growth Models and Neural ODEs
Open this publication in new window or tab >>Inverse Problems for Tumour Growth Models and Neural ODEs
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

This thesis concerns the application of methods and techniques from the theory of inverse problems and differential equations to study models arising in the areas of mathematical oncology and deep learning. 

The first problem studied is to develop methods to perform numerical simulations with full 3-dimensional brain imaging data of reaction-diffusion models for tumour growth forwards as well as backwards in time with the goal of enabling the numerical reconstruction of the source of the tumour given an image (or similar data) at a later stage in time of the tumour. This inverse ill-posed problem is solved as a sequence of well-posed forward problems using the nonlinear Landweber regularization method. Such models and method allow to generate realistic synthetic medical images that can be used for data augmentation. Mathematical analysis of the problems solved as well as establishing uniqueness of the source are presented. 

The second problem includes a novel method allowing training self-contained neural ordinary differential equation networks (termed standalone NODEs) via a nonlinear conjugate gradient method, where the Sobolev gradient can be incorporated to improve smoothness of model weights. Relevant functions spaces are introduced, the adjoint problems with the needed gradients are calculated and the robustness is studied. The developed framework has many advantages in that it can incorporate relevant dynamics from physical models as well as help to understand more on how neural networks actually work and how sensitive they are to natural and adversarial perturbations. 

Combination of the two main problems will allow for example the training of neural networks to identify tumours in real imaging data. 

Abstract [sv]

Syftet med avhandlingen är att undersöka matematiska modeller och metoder för tillväxt av hjärntumörer samt för artificiella neurala nätverk inom maskininlärning.

I första delen studeras en matematisk modell för tillväxt av hjärntumörer. Den anger hur tumörcellerna ändras över tid. Data innehåller bland annat information om värden på parametrar och cellernas fördelning vid en viss tidpunkt. Modellen har den bra egenskapen att fel i data inte direkt stör beräknad tillväxt. Ibland behöver parametrar skattas eller så behöver man stega bakåt i tiden för att ta reda på hur tumören såg ut tidigare. Dessa uppgifter är exempel på inversa och illaställda problem. Sådana problem är svårare för att fel i data inte kan kontrolleras utan kan förstöra beräkningar. I avhandlingen föreslås metoder som kallas regulariserande eftersom dessa på ett stabilt sätt, d.v.s. utan att felen förstör, beräknar tumörens form bakåt i tiden. Egenskaper hos metoderna undersöks och 3-dimensionella datorberäkningar utförs av tumörens förändring.

Den andra och sista delen studerar en gren av artificiell intelligens och maskininlärning, så kallade djupa neurala nätverk. Dessa är inspirerade av hjärnan med dess många nervceller kopplade i form av lager. Sådana nätverk tränas att klara en uppgift som att finna tumörer i medicinska bilder. I de här neurala nätverken finns lager med okända parametrar, vilka för en sådan uppgift bestäms från en stor mängd data. En matematisk modell som kan hantera oändligt många lager i ett neuralt nätverk föreslås och det inversa problemet att beräkna de okända parametrarna i nätverket undersöks. Stabila metoder som i första delen framtas. Med hjälp av matematik och modellens struktur kan viktiga frågor om stabilitet och känslighet hos djupinlärning besvaras.

I framtiden kan de två delarna, matematisk modellering och maskininlärning, kopplas samman för att på ett förfinat och stabilt sätt bestämma tillväxten hos tumörer både framåt och bakåt i tiden.  

Place, publisher, year, edition, pages
Linköping: Linköping University Electronic Press, 2023. p. 29
Series
Linköping Studies in Science and Technology. Dissertations, ISSN 0345-7524 ; 2276
National Category
Mathematics
Identifiers
urn:nbn:se:liu:diva-191799 (URN)10.3384/9789179295905 (DOI)9789179295899 (ISBN)9789179295905 (ISBN)
Public defence
2023-03-17, TP2, Täppan, Campus Norrköping, Norrköping, 09:15 (English)
Opponent
Supervisors
Available from: 2023-02-15 Created: 2023-02-15 Last updated: 2023-04-03Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

arXiv:2202.05766

Authority records

Baravdish, GeorgeEilertsen, GabrielJaroudi, RymJohansson, TomasMalý, LukášUnger, Jonas

Search in DiVA

By author/editor
Baravdish, GeorgeEilertsen, GabrielJaroudi, RymJohansson, TomasMalý, LukášUnger, Jonas
By organisation
Physics, Electronics and MathematicsFaculty of Science & EngineeringMedia and Information Technology
MathematicsComputer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 448 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf