Humans use touch to maintain their social relationships, and the emotional qualities of touch depend on who touches whom. However, it is not known how affective and social dimensions of touch are processed in the brain. We measured haemodynamic brain activity with functional magnetic resonance imaging (fMRI) from 19 subjects (10 males), while they were touched on their upper thigh by either their romantic partner, or an unfamiliar female or male confederate or saw the hand of one of these individuals near their upper thigh but were not touched. We used multi-voxel pattern analysis on pre-defined regions of interest to reveal areas that encode social touch in a relationship-specific manner. The accuracy of the machine learning classifier to identify actor for both feeling touch and seeing hand exceeded the chance level in the primary somatosensory cortex, while in the insular cortex accuracy was above chance level only for the touch condition. When classifying the relationship (partner or stranger), while keeping the toucher sex fixed, amygdala (AMYG), orbitofrontal cortex (OFC), and primary and secondary somatosensory cortices were able to discriminate toucher significantly above chance level. These results suggest that information on the social relationship of the toucher is processed consistently across several regions. More complex information about toucher identity is processed in the primary somatosensory and insular cortices, both of which can be considered early sensory areas. This article is part of a Special Issue entitled: The Neurobiology of Social and Affective Touch. (c) 2020 The Authors. Published by Elsevier Ltd on behalf of IBRO. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/ 4.0/).