Low open-circuit voltage (V-oc) induced by energy loss in organic solar cells is considered to be one of the most influencing factors limiting device performance, in which morphology of the active layer plays a crucial role in determining energy loss. By employing a bilayer structure of the P3HT:N2200 all-polymer system, we have identified the isolated impact of a molecular packing structure on device V-oc with analysis of energy loss processes. Thermal annealing and various solvents were used to control molecular orientation in P3HT:N2200 bilayer devices, in which different V-oc spanning from 0.45 to 0.54 V could be obtained. It was found that energy of charge-transfer state (E-ct) differed in these bilayer devices. Besides, increased charge recombination could be observed in bilayer devices when N2200 layers exhibited face-on orientation, which caused an additional energy loss and decreased V-oc. Our results suggest that rational control of polymer molecular orientation is essential to reduce the energy loss and ultimately achieve high V-oc in all-polymer solar cells.