The considerable improvement on the power conversion efficiency (PCE) for emerging nonfullerene polymer solar cells is still limited by considerable voltage losses that have become one of the most significant obstacles in further boosting desired photovoltaic performance. Here, a comprehensive study is reported to understand the impacts of charge transport, energetic disorder, and charge transfer states (CTS) on the losses in open-circuit voltage (V-oc) based on three high performing bulk heterojunction solar cells with the best PCE exceeding 11%. It is found that the champion poly[(2,6-(4,8-bis(5-(2-ethylhexyl)thiophen-2-yl)benzo[1,2-b:4,5-b]dithiophene)-co-(1,3-di(5-thiophene-2-yl)-5,7-bis(2-ethylhexyl)-benzo[1,2-c:4,5-c]dithiophene-4,8-dione))] (PBDB-T):IT-M solar cell (PCE = 11.5%) is associated with the least disorder. The determined energetic disorder in part reconciles the difference in V-oc between the solar cells. A reduction is observed in the nonradiative losses (V-nonrad) coupled with the increase of energy of CTS for the PBDB-T:IT-M device, which may be related to the improved balance in carrier mobilities, and partially can explain the gain in V-oc. The determined radiative limit for V-oc combined with the V-nonrad generates an excellent agreement for the V-oc with the experimental values. The results suggest that minimizing the energetic disorder related to transport and CTS is critical for the mitigation of V-oc losses and improvements on the device performance.