Deaf signers and hearing non-signers have previously been shown to recruit partially different brain regions during simple arithmetic. In light of the triple code model, the differences were interpreted as relating to stronger recruitment of the verbal system of numerical processing, that is, left angular and inferior frontal gyrus, in hearing non-signers, and of the quantity system of numerical processing, that is, right horizontal intraparietal sulcus, for deaf signers. The main aim of the present study was to better understand similarities and differences in the neural correlates supporting arithmetic in deaf compared to hearing individuals. Twenty-nine adult deaf signers and 29 hearing non-signers were enrolled in an functional magnetic resonance imaging study of simple and difficult subtraction and multiplication. Brain imaging data were analyzed using whole-brain analysis, region of interest analysis, and functional connectivity analysis. Although the groups were matched on age, gender, and nonverbal intelligence, the deaf group performed generally poorer than the hearing group in arithmetic. Nevertheless, we found generally similar networks to be involved for both groups, the only exception being the involvement of the left inferior frontal gyrus. This region was activated significantly stronger for the hearing compared to the deaf group but showed stronger functional connectivity with the left superior temporal gyrus in the deaf, compared to the hearing, group. These results lend no support to increased recruitment of the quantity system in deaf signers. Perhaps the reason for performance differences is to be found in other brain regions not included in the original triple code model.