We examine the thermodynamic stability of compounds and alloys in the ternary B-As-P system theoretically using first-principles calculations. We demonstrate that the icosahedral B12As2 is the only stable compound in the binary B-As system, while the zinc-blende BAs is thermodynamically unstable with respect to B12As2 and the pure arsenic phase at 0 K, and increasingly so at higher temperature, suggesting that BAs may merely exist as a metastable phase. On the contrary, in the binary B-P system, both zinc-blende BP and icosahedral B12P2 are predicted to be stable. As for the binary As-P system, As1-xPx disordered alloys are predicted at elevated temperature-for example, a disordered solid solution of up to similar to 75 at.% As in black phosphorus as well as a small solubility of similar to 1 at.% P in gray arsenic at T = 750 K, together with the presence of miscibility gaps. The calculated large solubility of As in black phosphorus explains the experimental syntheses of black-phosphorus-type As1-xPx alloys with tunable compositions, recently reported in the literature. We investigate the phase stabilities in the ternary B-As-P system and demonstrate a high tendency for a formation of alloys in the icosahedral B-12(As1-xPx)(2) structure by intermixing of As and P atoms at the diatomic chain sites. The phase diagram displays noticeable mutual solubility of the icosahedral subpnictides in each other even at room temperature as well as a closure of a pseudobinary miscibility gap around 900 K. As for pseudobinary BAs1-xPx alloys, only a tiny amount of BAs is predicted to be able to dissolve in BP to form the BAs1-xPx disordered alloys at elevated temperature. For example, less than 5% of BAs can dissolve in BP at T = 1000 K. The small solubility limit of BAs in BP is attributed to the thermodynamic instability of BAs with respect to B12As2 and As.