A polarization-sensitive hyperspectral imaging system (SkinSpect) has been built and evaluated using two-layer tissue phantoms, fabricated to mimic the optical properties of melanin in different epidermal thickness and hemoglobin in dermal layers. Multiple tissue-mimicking phantoms with varying top layer thicknesses were measured for optical system calibration and performance testing. Phantom properties were characterized and validated using SkinSpect. The resulting analysis shows that the proposed system is capable of distinguishing and differentiating the layer-dependent absorption spectra and the depths at which this absorption occurs.