Sexual harassment and abuse (SHA) can have a profound negative impact, but research on SHA in sport is scarce and studies of SHA in Swedish sport are absent. This study explores (a) self-reported prevalence of SHA perpetrated by coaches among male and female Swedish athletes, and (b) descriptive statistics for coach–athlete relationship factors and the association between these relationship factors and reported SHA. Current and former Swedish club sport athletes (n = 477) aged 25 participated in the survey. Athletes reported 5.5% prevalence of coach SHA, of which inappropriate, unpleasant, or offensive physical contact were most common. No significant differences of SHA frequency were displayed across gender, sport performance levels, or individual/team sports. A majority of athletes (55–95%) reported trust, closeness, substantial coach influence over sport performance, and instructional physical contact as main coach–athlete relationship factors. A minority (13–39%) reported dependency, substantial coach influence over personal-life, non-instructional physical contact, sexualized comments and jokes, and flirting. Prevalence of coach–athlete friendships, athlete attraction to coaches, and coaches’ instructional physical contact differed significantly between male and female athletes. Closeness and athlete attraction to coaches were negatively related, and coaches’ non-instructional physical contact and flirting were positively related to reported SHA. Multi-causality and ambiguity of coach–athlete relationship factors are discussed.