liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Calibration tests in multi-class classification: A unifying framework
Uppsala Univ, Sweden.
Linköping University, Department of Computer and Information Science, The Division of Statistics and Machine Learning. Linköping University, Faculty of Science & Engineering.
Uppsala Univ, Sweden.
2019 (English)In: ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), NEURAL INFORMATION PROCESSING SYSTEMS (NIPS) , 2019, Vol. 32Conference paper, Published paper (Refereed)
Abstract [en]

In safety-critical applications a probabilistic model is usually required to be calibrated, i.e., to capture the uncertainty of its predictions accurately. In multi-class classification, calibration of the most confident predictions only is often not sufficient. We propose and study calibration measures for multi-class classification that generalize existing measures such as the expected calibration error, the maximum calibration error, and the maximum mean calibration error. We propose and evaluate empirically different consistent and unbiased estimators for a specific class of measures based on matrix-valued kernels. Importantly, these estimators can be interpreted as test statistics associated with well-defined bounds and approximations of the p-value under the null hypothesis that the model is calibrated, significantly improving the interpretability of calibration measures, which otherwise lack any meaningful unit or scale.

Place, publisher, year, edition, pages
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS) , 2019. Vol. 32
Series
Advances in Neural Information Processing Systems, ISSN 1049-5258
National Category
Probability Theory and Statistics
Identifiers
URN: urn:nbn:se:liu:diva-167713ISI: 000535866903083OAI: oai:DiVA.org:liu-167713DiVA, id: diva2:1454553
Conference
33rd Conference on Neural Information Processing Systems (NeurIPS)
Note

Funding Agencies|Swedish Research Council via the project Learning of Large-Scale Probabilistic Dynamical Models [2016-04278]; Swedish Research Council via the project Counterfactual Prediction Methods for Heterogeneous Populations [2018-05040]; Swedish Foundation for Strategic Research via the project Probabilistic Modeling and Inference for Machine Learning [ICA16-0015]; Wallenberg Al, Autonomous Systems and Software Program (WASP) - Knut and Alice Wallenberg Foundation

Available from: 2020-07-17 Created: 2020-07-17 Last updated: 2020-07-17

Open Access in DiVA

No full text in DiVA

Search in DiVA

By author/editor
Lindsten, Fredrik
By organisation
The Division of Statistics and Machine LearningFaculty of Science & Engineering
Probability Theory and Statistics

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 21 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf