Cardiovascular diseases (CVDs) are a leading cause of morbidity and mortality, and physical activity (PA) is central to secondary prevention. However, factors associated with PA among individuals with CVDs remain poorly understood, with few studies using accelerometer-based methods. This cross-sectional study examined the associations between accelerometer-measured PA and demographic, socioeconomic, lifestyle factors, and comorbidity in a large Swedish cohort of individuals with CVDs. Participants aged between 50 and 64 years with a confirmed diagnosis of angina pectoris, myocardial infarction, atrial fibrillation and flutter, heart failure, or stroke were included. Moderate-to-vigorous physical activity was measured using hip-worn triaxial accelerometers over seven consecutive days and categorized into tertiles of low, medium and high PA level. Demographic, socioeconomic, lifestyle factors, and comorbidities were assessed through questionnaires and registry data. Associations with PA levels were analyzed using multiple ordinal logistic regression. A total of 1,484 participants (32% women; median age 60.2 years) were included. Being in the oldest age group was significantly associated with lower odds of having a higher PA level (OR = 0.59; 95% CI [0.44, 0.77]), as well as female sex (OR = 0.73; 95% CI [0.59, 0.91]), regular/occasional smoking (OR = 0.37; 95% CI [0.26, 0.52]), and having one (OR = 0.61; 95% CI [0.48, 0.77]) or more comorbidities (OR = 0.45; 95% CI [0.30, 0.68]). A healthy diet was significantly associated with higher PA levels (OR = 1.84; 95% CI [1.46, 2.31]). Financial strain, level of education and alcohol consumption had no association with PA. Using data from a representative sample of middle-aged adults, the findings indicate that engagement in PA is primarily associated with lifestyle factors, demographic characteristics, and comorbidity. These identified characteristics may inform future research aimed at CVD populations with low PA. Further research should investigate mediating and moderating pathways to clarify how these factors influence PA.