Many of the deep learning models currently driving the advancements in computer vision, expected to transform our society, require extensive training data. However, privacy regulations require explicit consent or anonymization of personal data, and traditional anonymization methods degrade data quality, thus hindering model performance. To address this challenge, we introduce the Realistic Anonymization using Diffusion (RAD) framework, which uses Stable Diffusion and ControlNet to produce high-quality synthetic images. RAD's three-step pipeline maintains contextual integrity and data utility, achieving superior image quality compared to previous GAN-based methods. We evaluated RAD's privacy preservation and data utility through face recognition accuracy, a segmentation task, and human assessment. RAD anonymized faces in 95.5% of cases, with high photo-realism ratings from human evaluators. Segmentation tasks on both original and anonymized images showed minimal performance drop, confirming RAD's high utility. Our analysis also identifies the strengths and weaknesses of using Stable Diffusion for full-body anonymization in various conditions. In summary, our work advances the understanding of high-utility anonymized data generation, and demonstrates that RAD can effectively balance privacy and utility.