Cross-validation under separate sampling: Strong bias and how to correct it

Ulisses M. Braga-Neto, Amin Zollanvari, Edward R. Dougherty

Research output: Contribution to journalArticlepeer-review

19 Citations (Scopus)


Motivation: It is commonly assumed in pattern recognition that cross-validation error estimation is 'almost unbiased' as long as the number of folds is not too small. While this is true for random sampling, it is not true with separate sampling, where the populations are independently sampled, which is a common situation in bioinformatics. Results: We demonstrate, via analytical and numerical methods, that classical cross-validation can have strong bias under separate sampling, depending on the difference between the sampling ratios and the true population probabilities. We propose a new separate-sampling cross-validation error estimator, and prove that it satisfies an 'almost unbiased' theorem similar to that of random-sampling cross-validation. We present two case studies with previously published data, which show that the results can change drastically if the correct form of cross-validation is used.

Original languageEnglish
Pages (from-to)3349-3355
Number of pages7
Issue number23
Publication statusPublished - Dec 1 2014

ASJC Scopus subject areas

  • Statistics and Probability
  • Biochemistry
  • Molecular Biology
  • Computer Science Applications
  • Computational Theory and Mathematics
  • Computational Mathematics

Fingerprint Dive into the research topics of 'Cross-validation under separate sampling: Strong bias and how to correct it'. Together they form a unique fingerprint.

Cite this