Currently the most popular approach to facial expression analysis uses categorical representations of expressions based on labels like sad, happy and angry. Subtle expression variations require however a quantitative and continuous representation. Besides, todays subjective expression spaces, built by semantic differential level scores and reduced to low dimensional continuous spaces using MDS or PCA have no direct correspondence with the physical stimuli or the expression images. On the other hand, the spaces used in engineering are based on purely physical stimuli or images which can hardly be called expression spaces. Even in models incorporating spacial structure, the geometry of the expression space received little attention and is usually assumed to be Euclidean. The aim of this paper is to build an expression space which is directly connected with the physical stimuli or the expression images. At the same time, it has to incorporate the subjective characteristics of expression perception. We use methods from psychophysics to build an expression space based on the physical stimuli or expression image space equipped with JND or discrimination threshold data. The construction follows the approach used in color science where the MacAdam ellipsoids provide for every color a metric tensor in a Riemannian space. We show that the discrimination thresholds indicate that the space is not Non- Euclidean. We will also illustrate the intrinsic geometrical structure of the expression spaces for several observers obtained from two large image databases of face expressions.