The knowledge of lattice thermal conductivity of materials under realistic conditions is vitally important since many modern technologies require either high or low thermal conductivity. Here, we propose a theoretical model for determining lattice thermal conductivity, which takes into account the effect of microstructure. It is based on ab initio description that includes the temperature dependence of the interatomic force constants and treats anharmonic lattice vibrations. We choose ScN as a model system, comparing the computational predictions to the experimental data by time-domain thermoreflectance. Our experimental results show a trend of reduction in lattice thermal conductivity with decreasing domain size predicted by the theoretical model. These results suggest a possibility to control thermal conductivity by microstructural tailoring and provide a predictive tool for the effect of the microstructure on the lattice thermal conductivity of materials based on ab initio calculations.