Electroencephalography (EEG) is another method for performing person identification (PI). Due to the nature of the EEG signals, EEG-based PI is typically done while a person is performing a mental task such as motor control. However, few studies used EEG-based PI while the person is in different mental states (affective EEG). The aim of this paper is to improve the performance of affective EEG-based PI using a deep learning (DL) approach. We proposed a cascade of DL using a combination of convolutional neural networks (CNNs) and recurrent neural networks (RNNs). CNNs are used to handle the spatial information from the EEG while RNNs extract the temporal information. We evaluated two types of RNNs, namely long short-term memory (LSTM) and gated recurrent unit (GRU). The proposed method is evaluated on the state-of-the-art affective data set DEAP. The results indicate that CNN-GRU and CNN-LSTM can perform PI from different affective states and reach up to 99.90%-100% mean correct recognition rate. This significantly outperformed a support vector machine baseline system that used power spectral density features. Notably, the 100% mean CRR came from 32 subjects in DEAP data set. Even after the reduction of the number of EEG electrodes from 32 to 5 for more practical applications, the model could still maintain an optimal result obtained from the frontal region, reaching up to 99.17%. Amongst the two DL models, we found that CNN-GRU and CNN-LSTM performed similarly while CNN-GRU expended faster training time. In conclusion, the studied DL approaches overcame the influence of affective states in EEG-Based PI reported in the previous works.