We demonstrate that voltage losses due to both radiative and non-radiative recombination of charge carriers are strongly dependent on D/A phase separation. By processing the active layer with various solvent additives, we create distinct morphologies that lead to significantly different device open-circuit voltages (V-OC), even though the charge transfer state energy (E-CT) of the D/A blend remains rather constant. We find that radiative recombination losses are significantly increased for a finely intermixed morphology, due to the large D/A interface area. This leads to a total recombination loss of E-CT - qV(OC) approximate to 0.7 eV. However, considerably smaller losses (0.5 eV), due to suppressed non-radiative recombination, are possible in solar cells where the D/A materials are organized to only allow for selective charge carrier extraction. Using a drift diffusion model, we show that the origin of the reduced non-radiative recombination losses is related to an effect which has not been considered for optimized solar cells - the suppression of minority carrier diffusion to the wrong contact. Our results suggest that the built-in field is not sufficiently strong even in optimized organic solar cells and that selective carrier extraction is critical for further improvements in V-OC.