The transition towards a circular economy is being increasingly envisaged in Europe. Increased utilisation of secondary construction aggregates poses a great opportunity in this regard given its potential to avoid waste disposal, natural resource extraction, and reduce costs. Nevertheless, the Swedish construction aggregates market is dominantly characterised by primary aggregates utilisation. This study employs a qualitative approach to analyse the Swedish construction aggregates market from the user perspective in order to identify the challenges of increased utilisation of secondary aggregates. The dominant and systematic utilisation of primary aggregates is chiefly driven by market processes allowing economic advantages and convenience for the users in doing so, whereas the situational and non-systematic utilisation of secondary aggregates is opportunistically driven by different actors in order to seize opportunities such as avoided costs and improved environmental performance. The realisation of increased secondary aggregates utilisation needs to be first facilitated via changes to the institutional structure that makes such materials a priority issue for the supply chain actors, which necessitates national-level policy interventions and supply chain actor collaborations. Second, the competitiveness of secondary aggregates needs to be lifted to a comparable state to that of primary aggregates, which necessitates the development of specific sectoral application standards and vertical integration of secondary aggregates supply by construction corporations.