Raman measurements in backscattering configuration are employed to characterize the effect of nitrogen on the structural properties of a GaNxP1-x alloy with x<=3%. The following effects of N incorporation on the vibrational spectra of GaNP are observed. First, frequencies of GaP-like and GaN-like longitudinal optical phonons exhibit strong compositional dependence, due to a combined effect of alloying and biaxial strain. Second, a dramatic quenching of two-phonon Raman scattering and an emergence of zone-edge GaP-like vibrations are observed. These effects are tentatively attributed to a local distortion of the GaNP lattice and/or compositional disorder in the alloy.