We present a polarization-sensitive hyperspectral imaging system (SkinSpect) that employs a spectrally-programmable light source in the visible and NIR domains. Multiple tissue-mimicking phantoms were fabricated to mimic the optical properties of normal skin as well as pigmented light and dark moles. The phantoms consist of titanium dioxide and a mixture of coffee, red food dye, and naphthol green as the scattering and the three absorptive agents in a polydimethylsiloxane (PDMS) base. Phantoms were produced with both smooth and rough textured surfaces and tested using Spatial Frequency Domain Imaging (SFDI) and Spatially Modulated Quantitative Spectroscopy (SMoQS) for homogeneity as well as determining absorption and scattering variance, respectively. The reflectance spectral images were also recorded using the SkinSpect research prototype; the spectral signatures of the phantoms were calculated using a two-flux single-layer Kubelka-Munk model and non-negative least square fitting routine was applied to extract the relative concentrations of the individual phantom components.