Rectal or colorectal cancer is one of the leading causes of cancer-related death. With the advancement in surgical techniques, the survival rate has been improved. Predicting the survival rate is an important factor for enabling optimal treatments to prolong rectal-cancer patients lives. Methods of artificial intelligence and machine learning have been applied for assisting physicians in cancer research. In this study, we investigated the use of pretrained convolutional neural networks and support vector machines for predicting the survival rate of a cohort of rectal-cancer patients using metastatic immunohistochemistry samples staining for protein RhoB. The combination of convolutional neural networks and support vector machines achieved better classification results than using individual pretrained deep networks in most cases, and where manual pathological analysis is encountered with great difficulty. In particular, the combination of ResNet-101 and SVM produced an average accuracy of 86% for non-radiotherapy, and Inception-v3 and SVM resulted in an average accuracy of 85% for radiotherapy.