We explore the interplay between automation and human involvement in data science. Emerging from in-depth discussions at a Dagstuhl seminar, we synthesize perspectives from Automated Data Science (AutoDS) and Interactive Data Visualization (VIS) – two fields that traditionally represent opposing ends of the human-machine spectrum. While AutoDS seeks to enhance efficiency through increasing automation, VIS underscores the critical value of human involvement in providing nuanced understanding, creativity, innovation, and contextual relevance. We explore these dichotomies through an online survey and advocate for a balanced approach that harmonizes the speed and consistency of effective automation with the indispensable insights of human expertise and thought. Ultimately, we confront the essential question: what aspects of data science should we automate?