The fuel economy of hybrid electric vehicles (HEVs) deteriorates due to high cabin heat demands and engine efficiency decay caused by temperature drops in low-temperature environments. This article proposes a novel energy management strategy based on deep reinforcement learning (RL) for plug-in HEVs (PHEVs) under cold environments. A series HEV model is first presented, and the coupling relationship between the engine-cabin-coupled thermal management system (CTMS) and the energy management system is analyzed. Considering the influence of the engine coolant temperature on the fuel consumption rate and cabin heat demand, an energy management optimal control problem is developed. An efficient RL framework based on a double-deep Q-learning (DDQL) algorithm is designed to solve the problem. The method does not rely on precise models and future traffic information. Dynamic programming (DP) and model predictive control (MPC) methods are also employed and compared with the proposed method under the training driving cycle and driving cycles randomly generated by the Markov-driver model. Experimental results show that the proposed method has efficient performance in maintaining battery state of charge (SOC) stability, real-time performance, fuel economy, and adaptability. The fuel economy can reach the 97.5% level of the DP-based strategy, which is approximately 5.9% higher than that of the MPC-based strategy.