In March 2017, the Swedish government decided to introduce digital competence - including programming - in primary school. As a consequence, every math and technology teacher in grades 1-9 in Sweden is expected to integrate programming in their teaching. Furthermore, the Swedish school law requires that teaching is based on scientific evidence and proven experience. In addition to professional development for teachers, it is hence crucial to also conduct research on different aspects of programming in the classroom. In this paper, we describe the process of developing a scientifically grounded instrument for assessing students' programming skills, as part of a longitudinal research project investigating how students in primary school learn programming. We also present the main findings related to the suitability of the instrument based on a pilot study conducted in spring 2019, collecting data from 310 students.