The development of artificial intelligence, and in particular machine learning, has seen tremendous success in recent years. The use of machine learning, however, extends to vast application areas outside of those that we encounter in our day-to-day life. One such area is within the natural sciences, where research has shown promising results in using machine learning for modeling of systems of atoms. This is also the type of application for which the methods developed in this thesis are motivated. The thesis investigates and develops both predictive models that can predict properties and simulate these types of systems, and generative models that can propose new potential materials or molecules.

Particular emphasis is put on methods that model data as graphs, and the thesis starts with investigations of graph neural networks (GNNs) designed for predicting material and molecular properties. The performance of these models in the context of high-throughput screenings are put under scrutiny. The performance of GNNs when predicting properties of materials which are only hypothetical and structures which are not completely relaxed is investigated, and the insights are used to suggest a workflow that combines machine learning and conventional high-throughput methods. Additionally, an investigation of so-called knowledge distillation in the context of GNNs for systems of atoms has been performed. This study proposes some simple techniques for improving the performance of this type of GNNs, without sacrificing speed.

The generative modeling techniques developed in the thesis are both more generally applicable and specifically targeting the materials science domain. Among the general methods, the thesis investigates a type of generative autoregressive models where the generation order is a random variable, and develops discriminator guidance for such models. Additionally, a new sequential Monte Carlo algorithm, DDSMC, is developed for general Bayesian inverse problems. A dedicated materials science model, WyckoffDiff, is developed, utilizing a description of materials that explicitly encode information of their symmetries, with the aim of facilitating generation of materials with strict symmetrical properties.

While predictive and generative models can be useful on their own, the study on WyckoffDiff also highlights how they can be used together as parts of a materials discovery pipeline, with predictive models predicting the properties of the materials generated by a generative model like WyckoffDiff.

A recent line of research has exploited pre-trained generative diffusion models as priors for solving Bayesian inverse problems. We contribute to this research direction by designing a sequential Monte Carlo method for linear-Gaussian inverse problems which builds on “decoupled diffusion", where the generative process is designed such that larger updates to the sample are possible. The method is asymptotically exact and we demonstrate the effectiveness of our Decoupled Diffusion Sequential Monte Carlo (DDSMC) algorithm on both synthetic as well as protein and image data. Further, we demonstrate how the approach can be extended to discrete data.

Crystalline materials often exhibit a high level of symmetry. However, most generative models do not account for symmetry, but rather model each atom without any constraints on its position or element. We propose a generative model, Wyckoff Diffusion (WyckoffDiff), which generates symmetry-based descriptions of crystals. This is enabled by considering a crystal structure representation that encodes all symmetry, and we design a novel neural network architecture which enables using this representation inside a discrete generative model framework. In addition to respecting symmetry by construction, the discrete nature of our model enables fast generation. We additionally present a new metric, Fréchet Wrenformer Distance, which captures the symmetry aspects of the materials generated, and we benchmark WyckoffDiff against recently proposed generative models for crystal generation. As a proof-of-concept study, we use WyckoffDiff to find new materials below the convex hull of thermodynamical stability.

We introduce discriminator guidance in the setting of Autoregressive Diffusion Models. The use of a discriminator to guide a diffusion process has previously been used for continuous diffusion models, and in this work we derive ways of using a discriminator together with a pretrained generative model in the discrete case. First, we show that using an optimal discriminator will correct the pretrained model and enable exact sampling from the underlying data distribution. Second, to account for the realistic scenario of using a sub-optimal discriminator, we derive a sequential Monte Carlo algorithm which iteratively takes the predictions from the discriminator into account during the generation process. We test these approaches on the task of generating molecular graphs and show how the discriminator improves the generative performance over using only the pretrained model.

Recent advances in graph neural networks (GNNs) have enabled more comprehensive modeling of molecules and molecular systems, thereby enhancing the precision of molecular property prediction and molecular simulations. Nonetheless, as the field has been progressing to bigger and more complex architectures, state-of-the-art GNNs have become largely prohibitive for many large-scale applications. In this paper, we explore the utility of knowledge distillation (KD) for accelerating molecular GNNs. To this end, we devise KD strategies that facilitate the distillation of hidden representations in directional and equivariant GNNs, and evaluate their performance on the regression task of energy and force prediction. We validate our protocols across different teacher-student configurations and datasets, and demonstrate that they can consistently boost the predictive accuracy of student models without any modifications to their architecture. Moreover, we conduct comprehensive optimization of various components of our framework, and investigate the potential of data augmentation to further enhance performance. All in all, we manage to close the gap in predictive accuracy between teacher and student models by as much as 96.7% and 62.5% for energy and force prediction respectively, while fully preserving the inference throughput of the more lightweight models.