A variance-based measure of inter-rater agreement in medical databases
2002 (English)In: Journal of Biomedical Informatics, ISSN 1532-0464, E-ISSN 1532-0480, Vol. 35, no 5-6, 331-342 p.Article in journal (Refereed) Published
The increasing use of encoded medical data requires flexible tools for data quality assessment. Existing methods are not always adequate, and this paper proposes a new metric for inter-rater agreement of aggregated diagnostic data. The metric, which is applicable in prospective as well as retrospective coding studies, quantifies the variability in the coding scheme, and the variation can be differentiated in categories and in coders. Five alternative definitions were compared in a set of simulated coding situations and in the context of mortality statistics. Two of them were more effective, and the choice between them must be made according to the situation. The metric is more powerful for larger numbers of coded cases, and Type I errors are frequent when coding situations include different numbers of cases. We also show that it is difficult to interpret the meaning of variation when the structures of the compared coding schemes differ.
Place, publisher, year, edition, pages
2002. Vol. 35, no 5-6, 331-342 p.
Medical and Health Sciences
IdentifiersURN: urn:nbn:se:liu:diva-24560DOI: 10.1016/S1532-0464(03)00036-4Local ID: 6722OAI: oai:DiVA.org:liu-24560DiVA: diva2:244881