The working memory model for Ease of Language Understanding (ELU) predicts that processing differences between language modalities emerge when cognitive demands are explicit. Previous behavioural and neurocognitive work has shown that cognitive processing differences may be related to the different spatial and temporal processing demands involved in sign language and speech. In a set of working memory experiments with participants who were Deaf Signers (DS), Hearing Signers (HS) or Hearing Nonsigners (HN), we manipulated level of explicit processing required as well as temporal and spatial demands. Easily nameable pictures were used as stimuli to avoid confounds relating to sensory modality. When explicit processing demands were low, performance was largely similar for DS, HS and HN. However, when explicit and temporal processing demands were high, DS did not perform as well as HN. This effect was compounded by oral education. These findings suggest that temporal organization is not as prominent in working memory for sign language as it is in working memory for speech. A general effect of semantic similarity was also found. These findings are discussed in relation to the ELU model.