Approximately 20-30% of all road fatalities are related to driver sleepiness. A long-lasting goal in driver state research has therefore been to develop a robust sleepiness detection system. Since the alertness level is reflected in autonomous nervous system activity, it has been suggested that various heart rate variability (HRV) metrics can be used as features for driver sleepiness classification. Since the heart rate is modulated by many different factors, and not just by sleepiness, it is relevant to question the high driver sleepiness classification accuracies that have occasionally been presented in the literature. The main objective of this paper is thus to test how well a sleepiness classification system based on HRV features really is. A unique data set with 86 drivers, obtained while driving on real roads in real traffic, both in alert and sleep deprived conditions, was used to train and test a support vector machine (SVM) classifier. Subjective ratings based on the Karolinska sleepiness scale (KSS) was used as ground truth to divide the data into three classes (alert, somewhat sleepy and severely sleepy). Even though nearly all the 24 investigated HRV metrics showed significant differences between sleepiness levels, the SVM results only reached a mean accuracy of 61 %, with the worst results originating from the severely sleepy cases. In summary, the high classification performance that may arise in studies with high experimental control could not be replicated under realistic driving conditions. Future works should focus on how various confounding factors should be accounted for when using HRV based metrics as input to a driver sleepiness detection system.