We demonstrate improved signal and idler-beam quality of a 3-mm-aperture quasi-phase-matched RbTiOAsO4 optical parametric oscillator through use of a confocal unstable resonator as compared with a plane-parallel resonator. Both oscillators were singly resonant, and the periodically poled RbTiOAsO4 crystal generated a signal at 1.56 mum and an idler at 3.33 mum when pumped at 1.064 mum. We compared the beam quality produced by the 1.2-magnification confocal unstable resonator with the beam quality produced by the plane-parallel resonator by measuring the signal and the idler beam M-2 value. We also investigated the effect of pump-beam intensity distribution by comparing the result of a Gaussian and a top-hat intensity profile pump beam. We generated a signal beam of M-2 approximate to 7 and an idler beam of M-2 approximate to 2.5 through use of an unstable resonator and a Gaussian intensity profile pump beam. This corresponds to an increase of a factor of approximately 2 in beam quality for the signal and a factor of 3 for the idler, compared with the beam quality of the plane-parallel resonator optical parametric oscillator. (C) 2001 Optical Society of America.