People with epilepsy often display cognitive decline. Language function in epilepsy has been most thoroughly studied in temporal lobe epilepsy, but the impact of language deficits in epilepsy is not fully understood. The aim of this study was to evaluate the effect of epilepsy on language function with functional magnetic resonance imaging of brain activation, with behavioral methods and to relate language performance to demographic data, self-esteem and Quality of life. We specifically aimed to investigate if variation in epilepsy origin would relate to differences in language performance and if these differences could be associated with specific language activation patterns in the brain. We recruited people with epilepsy (29 in total), with focal onset seizures in either the left or right hemispheres or with generalized epilepsy; and 27 matching healthy controls. The participants’ language skills were measured with a phonemic word fluency test and a broader test measuring higher language functions. Functional magnetic resonance images of the brain were obtained during a word fluency and a sentence reading paradigm. Questionnaires on self-esteem and quality of life were collected. People with epilepsy of both focal and generalized origin had impaired function in semantic and verbal fluency tasks compared to the controls. The causes of language impairment were multifactorial; the most important determinants were education and onset age of epilepsy. Impaired language function was correlated to low self-esteem for participants with focal onset seizures; however Quality of life did not seem to be affected by language impairment. The functional magnetic resonance imaging investigation demonstrated altered functional activity during language tasks for participants with epilepsy compared to healthy controls. In epilepsy with focal seizures originating in the left hemisphere we found increased bilateral activation of supporting areas in the anterior mid-cingulate cortex and the left anterior ventral insula, indicating a compensational functional reorganization. In generalized epilepsy, the functional language network showed an imbalance expressed as an inadequate suppression of activation in the left anterior temporal lobe during semantic processing. Our study shows not only that reduced language functioning is present in people with epilepsy other than in the temporal lobe, but also that frequency of convulsive seizures correlates to language impairment. For patients with focalized seizures, low self esteem correlated also to language impairment. Our results highlight the importance of addressing the negative consequences of language decline in people with epilepsy of both focal and generalized origin.