The three-dimensional structure of proteins plays a crucial role in determining their function. Protein structure prediction methods, like AlphaFold, offer rapid access to a protein’s structure. However, large protein complexes cannot be reliably predicted, and proteins are dynamic, making it important to resolve their full conformational distribution. Single-particle cryo-electron microscopy (cryo-EM) is a powerful tool for determining the structures of large protein complexes. Importantly, the numerous images of a given protein contain underutilized information about conformational heterogeneity. These images are very noisy projections of the protein, and traditional methods for cryo-EM reconstruction are limited to recovering only one or a few consensus conformations.In this paper, we introduce cryoSPHERE, which is a deep learning method that uses a nominal protein structure (e.g., from AlphaFold) as input, learns how to divide it into segments, and moves these segments as approximately rigid bodies to fit the different conformations present in the cryo-EM dataset. This approach provides enough constraints to enable meaningful reconstructions of single protein structural ensembles. We demonstrate this with two synthetic datasets featuring varying levels of noise, as well as two real dataset. We show that cryoSPHERE is very resilient to the high levels of noise typically encountered in experiments, where we see consistent improvements over the current state-of-the-art for heterogeneous reconstruction.

We introduce discriminator guidance in the setting of Autoregressive Diffusion Models. The use of a discriminator to guide a diffusion process has previously been used for continuous diffusion models, and in this work we derive ways of using a discriminator together with a pretrained generative model in the discrete case. First, we show that using an optimal discriminator will correct the pretrained model and enable exact sampling from the underlying data distribution. Second, to account for the realistic scenario of using a sub-optimal discriminator, we derive a sequential Monte Carlo algorithm which iteratively takes the predictions from the discriminator into account during the generation process. We test these approaches on the task of generating molecular graphs and show how the discriminator improves the generative performance over using only the pretrained model.

Detecting microsecond structural perturbations in biomolecules has wide relevance in biology, chemistry and medicine. Here we show how MHz repetition rates at X-ray free-electron lasers can be used to produce microsecond time-series of protein scattering with exceptionally low noise levels of 0.001%. We demonstrate the approach by examining J alpha helix unfolding of a light-oxygen-voltage photosensory domain. This time-resolved acquisition strategy is easy to implement and widely applicable for direct observation of structural dynamics of many biochemical processes. The MHz repetition rates available at second-generation X-ray free-electron lasers enable the collection of microsecond time-resolved X-ray scattering data with exceptionally low noise, providing insights into protein structural dynamics.

A prominent self-supervised learning paradigm is to model the representations as clusters, or more generally as a mixture model. Learning to map the data samples to compact representations and fitting the mixture model simultaneously leads to the representation collapse problem. Regularizing the distribution of data points over the clusters is the prevalent strategy to avoid this issue. While this is sufficient to prevent full representation collapse, we show that a partial prototype collapse problem still exists in the DINO family of methods, that leads to significant redundancies in the prototypes. Such prototype redundancies serve as shortcuts for the method to achieve a marginal latent class distribution that matches the prescribed prior. We show that by encouraging the model to use diverse prototypes, the partial prototype collapse can be mitigated. We study the downstream impact of effective utilization of the prototypes during pre-training. We show that it enables the methods to learn more fine-grained clusters, encouraging more informative representations. We demonstrate that this is especially beneficial when pre-training on a long-tailed fine-grained dataset.

Noise-contrastive estimation (NCE) is a popular method for estimating unnormalised probabilistic models, such as energy-based models, which are effective for modelling complex data distributions. Unlike classical maximum likelihood (ML) estimation that relies on importance sampling (resulting in ML-IS) or MCMC (resulting in contrastive divergence, CD), NCE uses a proxy criterion to avoid the need for evaluating an often intractable normalisation constant. Despite apparent conceptual differences, we show that two NCE criteria, ranking NCE (RNCE) and conditional NCE (CNCE), can be viewed as ML estimation methods. Specifically, RNCE is equivalent to ML estimation combined with conditional importance sampling, and both RNCE and CNCE are special cases of CD. These findings bridge the gap between the two method classes and allow us to apply techniques from the ML-IS and CD literature to NCE, offering several advantageous extensions.

In recent years, machine learning has established itself as a powerful tool forhigh-resolution weather forecasting. While most current machine learning modelsfocus on deterministic forecasts, accurately capturing the uncertainty in thechaotic weather system calls for probabilistic modeling. We propose a probabilisticweather forecasting model called Graph-EFM, combining a flexible latent-variableformulation with the successful graph-based forecasting framework. The use of ahierarchical graph construction allows for efficient sampling of spatially coherentforecasts. Requiring only a single forward pass per time step, Graph-EFM allowsfor fast generation of arbitrarily large ensembles. We experiment with the modelon both global and limited area forecasting. Ensemble forecasts from Graph-EFMachieve equivalent or lower errors than comparable deterministic models, with theadded benefit of accurately capturing forecast uncertainty.

Logic-based Benders decomposition is a technique to solve optimization problems to optimality. It works by splitting the problem into a master problem, which neglects some aspects of the problem, and a subproblem, which is used to iteratively produce cuts for the master problem to account for those aspects. It is critical for the computational performance that these cuts are strengthened, but the strengthening of cuts comes at the cost of solving additional subproblems. In this work we apply a graph neural network in an autoregressive fashion to approximate the compilation of an irreducible cut, which then only requires few postprocessing steps to ensure its validity. We test the approach on a job scheduling problem with a single machine and multiple time windows per job and compare to approaches from the literature. Results show that our approach is capable of considerably reducing the number of subproblems that need to be solved and hence the total computational effort.

Detecting novelties given unlabeled examples of normal data is a challenging task in machine learning, particularly when the novel and normal categories are semantically close. Large deep models pretrained on massive datasets can provide a rich representation space in which the simple k-nearest neighbor distance works as a novelty measure. However, as we show in this paper, the basic k-NN method might be insufficient in this context due to ignoring the 'local geometry' of the distribution over representations as well as the impact of irrelevant 'background features'. To address this, we propose a fully unsupervised novelty detection approach that integrates the flexibility of k-NN with a locally adapted scaling of dimensions based on the 'neighbors of nearest neighbor' and computing a 'likelihood ratio' in pretrained (self-supervised) representation spaces. Our experiments with image data show the advantage of this method when off-the-shelf vision transformers (e.g., pretrained by DINO) are used as the feature extractor without any fine-tuning.

Generative flow networks (GFNs) are a class of probabilistic models for sequential samplingof composite objects, proportional to a target distribution that is defined in terms of anenergy function or a reward. GFNs are typically trained using a flow matching or trajectorybalance objective, which matches forward and backward transition models over trajectories.In this work we introduce a variational objective for training GFNs, which is a convexcombination of the reverse- and forward KL divergences, and compare it to the trajectorybalance objective when sampling from the forward- and backward model, respectively. Weshow that, in certain settings, variational inference for GFNs is equivalent to minimizing thetrajectory balance objective, in the sense that both methods compute the same score-functiongradient. This insight suggests that in these settings, control variates, which are commonlyused to reduce the variance of score-function gradient estimates, can also be used with thetrajectory balance objective. We evaluate our findings and the performance of the proposedvariational objective numerically by comparing it to the trajectory balance objective on twosynthetic tasks.

Annotating data for supervised learning can be costly. When the annotation budget is limited, active learning can be used to select and annotate those observations that are likely to give the most gain in model performance. We propose an active learning algorithm that, in addition to selecting which observation to annotate, selects the precision of the annotation that is acquired. Assuming that annotations with low precision are cheaper to obtain, this allows the model to explore a larger part of the input space, with the same annotation budget. We build our acquisition function on the previously proposed BALD objective for Gaussian Processes, and empirically demonstrate the gains of being able to adjust the annotation precision in the active learning loop.