Background The association between body composition and survival in rectal cancer patients is still unclear. Therefore, we aimed to evaluate the impact of computed tomography (CT)-measured body composition on survival in rectal cancer patients, stratifying our analyses by sex, tumour location, tumour stage and radiotherapy. Methods This retrospective cohort study included 173 patients with rectal adenocarcinoma. CT colonography scans at the time of diagnosis were used to assess the skeletal muscle index (SMI) and the visceral adipose tissue area (VAT). The patients were divided into a low or high SMI group and a low or high VAT group according to previously defined cutoff values. Endpoints included cancer-specific survival (CSS) and overall survival (OS). Results In all patients, low SMI was associated with worse CSS (HR, 2.63; 95% CI, 1.35-5.12; P = 0.004) and OS (HR, 3.57; 95% CI, 2.01-6.34; P < 0.001) compared to high SMI. The differences remained significant after adjusting for potential confounders (CSS: adjusted HR, 2.28; 95% CI, 1.13-4.58; P = 0.021; OS: adjusted HR, 3.17; 95% CI, 1.73-5.82; P < 0.001). Low SMI was still related to a poor prognosis after stratifying by sex, tumour location, stage and radiotherapy (P < 0.05). High VAT was associated with better CSS (HR, 0.31; 95% CI, 0.11-0.84; P = 0.022) and OS (HR, 0.40; 95% CI, 0.17-0.97; P = 0.044) compared to low VAT among men with rectal cancer <= 10 cm from the anal verge. High VAT was associated with worse CSS (HR, 4.15; 95% CI, 1.10-15.66; P = 0.036) in women with rectal cancer <= 10 cm from the anal verge. Conclusions Low SMI was associated with worse survival. High VAT predicted better survival in men but worse survival in women. The results suggest that CT-measured body composition is a useful tool for evaluating the prognosis of rectal cancer patients and demonstrate the need to include the sex and the tumour location in the analyses.