Background Cardiovascular disease (CVD) is a major global health issue, primarily caused by atherosclerosis. Psychological factors may play a role in the development and progression of CVD. However, the relationship between psychological factors and atherosclerosis is complex and poorly understood. This study, therefore, aimed to examine the association of psychological factors with (i) coronary and carotid atherosclerosis and (ii) cardiovascular health according to Life's Essential 8, in a large Swedish cohort. Methods This study utilized data from the Swedish CArdioPulmonary bioImage Study (SCAPIS), a large population-based project including individuals aged 50 to 65 years. Several psychological factors were analysed: general stress, stress at work, financial stress, major adverse life events, locus of control, feeling depressed, and depression. Coronary atherosclerosis was assessed as the degree of stenosis by coronary computed tomography angiography (CCTA) and coronary artery calcification (CAC) scores. Carotid atherosclerosis was examined using ultrasound. In addition, cardiovascular health was examined using the Life's Essential 8 concept created by the American Heart Association, which includes four health behaviors and four health factors. Associations were examined through binomial logistic regression (atherosclerosis variables) and linear regression (Life's Essential 8). Results A total of 25,658 participants were included in the study. The presence of financial stress, higher locus of control, and depression was weakly associated with increased odds of CCTA stenosis, CAC >= 1 and the presence of carotid plaques (all odds ratios: 1.10-1.21, 95% CI: 1.02-1.32) after adjusting for sex, age, and study site. However, these associations were attenuated and not statistically significant after additional adjustments for socioeconomic factors and health behaviors. Conversely, we observed inverse associations between the worst category for all psychological factors and cardiovascular health according to Life's Essential 8 score (all standardized beta-Coefficient <=-0.033, p < 0.001). Conclusion While there were no strong and consistent associations between psychological factors and atherosclerosis, the consistent associations of psychological factors with cardiovascular health by Life's Essential 8 may have relevance for future CVD risk. However, further studies are needed to elucidate the long-term effects of psychological factors on atherosclerosis development and cardiovascular health.