Moments and root-mean-square error of the Bayesian MMSE estimator of classification error in the Gaussian model

Amin Zollanvari, Edward R. Dougherty

Research output: Contribution to journalArticle

14 Citations (Scopus)

Abstract

The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been designed. Use of prior knowledge, in the form of a prior distribution on an uncertainty class of feature-label distributions to which the true, but unknown, feature-distribution belongs, can facilitate accurate error estimation (in the mean-square sense) in circumstances where accurate completely model-free error estimation is impossible. This paper provides analytic asymptotically exact finite-sample approximations for various performance metrics of the resulting Bayesian Minimum Mean-Square-Error (MMSE) error estimator in the case of linear discriminant analysis (LDA) in the multivariate Gaussian model. These performance metrics include the first, second, and cross moments of the Bayesian MMSE error estimator with the true error of LDA, and therefore, the root-mean-square (RMS) error of the estimator. We lay down the theoretical groundwork for Kolmogorov double-asymptotics in a Bayesian setting, which enables us to derive asymptotic expressions of the desired performance metrics. From these we produce analytic finite-sample approximations and demonstrate their accuracy via numerical examples. Various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired RMS. The Supplementary Material contains derivations for some equations and added figures.

Original languageEnglish
Pages (from-to)2178-2192
Number of pages15
JournalPattern Recognition
Volume47
Issue number6
DOIs
Publication statusPublished - Jun 2014
Externally publishedYes

Fingerprint

Mean square error
Error analysis
Classifiers
Discriminant analysis
Labels

Keywords

  • Bayesian minimum mean-square error estimator
  • Double asymptotics
  • Kolmogorov asymptotics
  • Linear discriminant analysis
  • Performance metrics
  • RMS

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Signal Processing

Cite this

Moments and root-mean-square error of the Bayesian MMSE estimator of classification error in the Gaussian model. / Zollanvari, Amin; Dougherty, Edward R.

In: Pattern Recognition, Vol. 47, No. 6, 06.2014, p. 2178-2192.

Research output: Contribution to journalArticle

@article{009ea79dfa9249729fd4f43b82f078d2,
title = "Moments and root-mean-square error of the Bayesian MMSE estimator of classification error in the Gaussian model",
abstract = "The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been designed. Use of prior knowledge, in the form of a prior distribution on an uncertainty class of feature-label distributions to which the true, but unknown, feature-distribution belongs, can facilitate accurate error estimation (in the mean-square sense) in circumstances where accurate completely model-free error estimation is impossible. This paper provides analytic asymptotically exact finite-sample approximations for various performance metrics of the resulting Bayesian Minimum Mean-Square-Error (MMSE) error estimator in the case of linear discriminant analysis (LDA) in the multivariate Gaussian model. These performance metrics include the first, second, and cross moments of the Bayesian MMSE error estimator with the true error of LDA, and therefore, the root-mean-square (RMS) error of the estimator. We lay down the theoretical groundwork for Kolmogorov double-asymptotics in a Bayesian setting, which enables us to derive asymptotic expressions of the desired performance metrics. From these we produce analytic finite-sample approximations and demonstrate their accuracy via numerical examples. Various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired RMS. The Supplementary Material contains derivations for some equations and added figures.",
keywords = "Bayesian minimum mean-square error estimator, Double asymptotics, Kolmogorov asymptotics, Linear discriminant analysis, Performance metrics, RMS",
author = "Amin Zollanvari and Dougherty, {Edward R.}",
year = "2014",
month = "6",
doi = "10.1016/j.patcog.2013.11.022",
language = "English",
volume = "47",
pages = "2178--2192",
journal = "Pattern Recognition",
issn = "0031-3203",
publisher = "Elsevier",
number = "6",

}

TY - JOUR

T1 - Moments and root-mean-square error of the Bayesian MMSE estimator of classification error in the Gaussian model

AU - Zollanvari, Amin

AU - Dougherty, Edward R.

PY - 2014/6

Y1 - 2014/6

N2 - The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been designed. Use of prior knowledge, in the form of a prior distribution on an uncertainty class of feature-label distributions to which the true, but unknown, feature-distribution belongs, can facilitate accurate error estimation (in the mean-square sense) in circumstances where accurate completely model-free error estimation is impossible. This paper provides analytic asymptotically exact finite-sample approximations for various performance metrics of the resulting Bayesian Minimum Mean-Square-Error (MMSE) error estimator in the case of linear discriminant analysis (LDA) in the multivariate Gaussian model. These performance metrics include the first, second, and cross moments of the Bayesian MMSE error estimator with the true error of LDA, and therefore, the root-mean-square (RMS) error of the estimator. We lay down the theoretical groundwork for Kolmogorov double-asymptotics in a Bayesian setting, which enables us to derive asymptotic expressions of the desired performance metrics. From these we produce analytic finite-sample approximations and demonstrate their accuracy via numerical examples. Various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired RMS. The Supplementary Material contains derivations for some equations and added figures.

AB - The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been designed. Use of prior knowledge, in the form of a prior distribution on an uncertainty class of feature-label distributions to which the true, but unknown, feature-distribution belongs, can facilitate accurate error estimation (in the mean-square sense) in circumstances where accurate completely model-free error estimation is impossible. This paper provides analytic asymptotically exact finite-sample approximations for various performance metrics of the resulting Bayesian Minimum Mean-Square-Error (MMSE) error estimator in the case of linear discriminant analysis (LDA) in the multivariate Gaussian model. These performance metrics include the first, second, and cross moments of the Bayesian MMSE error estimator with the true error of LDA, and therefore, the root-mean-square (RMS) error of the estimator. We lay down the theoretical groundwork for Kolmogorov double-asymptotics in a Bayesian setting, which enables us to derive asymptotic expressions of the desired performance metrics. From these we produce analytic finite-sample approximations and demonstrate their accuracy via numerical examples. Various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired RMS. The Supplementary Material contains derivations for some equations and added figures.

KW - Bayesian minimum mean-square error estimator

KW - Double asymptotics

KW - Kolmogorov asymptotics

KW - Linear discriminant analysis

KW - Performance metrics

KW - RMS

UR - http://www.scopus.com/inward/record.url?scp=84894366246&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84894366246&partnerID=8YFLogxK

U2 - 10.1016/j.patcog.2013.11.022

DO - 10.1016/j.patcog.2013.11.022

M3 - Article

AN - SCOPUS:84894366246

VL - 47

SP - 2178

EP - 2192

JO - Pattern Recognition

JF - Pattern Recognition

SN - 0031-3203

IS - 6

ER -