Statistical Hypothesis Testing in Positive Unlabelled Data

Abstract

We propose a set of novel methodologies which enable valid statistical hypothesis testing when we have only positive and unlabelled (PU) examples. This type of problem, a special case of semi-supervised data, is common in text mining, bioinformatics, and computer vision. Focusing on a generalised likelihood ratio test, we have 3 key contributions: (1) a proof that assuming all unlabelled examples are negative cases is sufficient for independence testing, but not for power analysis activities; (2) a new methodology that compensates this and enables power analysis, allowing sample size determination for observing an effect with a desired power; and finally, (3) a new capability, supervision determination, which can determine a-priori the number of labelled examples the user must collect before being able to observe a desired statistical effect. Beyond general hypothesis testing, we suggest the tools will additionally be useful for information theoretic feature selection, and Bayesian Network structure learning.

Publication
European Conference on Machine Learning and Principles and Practice of Knowledge Discovery (ECML/PKDD). Acceptance rate 115/483 (23.8%).
AWARD Best Student Paper Award in ECML/PKDD 2014 (sponsored by Springer).
AWARD Runner up Best Paper Prize of the School of Computer Science at the University of Manchester 2014 (sponsored by IBM)

Related