Information theoretic learning with K nearest neighbors : a new clustering algorithm

Abstract

The machine learning field based on information theory has received a lot of attention in recent years. Through kernel estimation of the probability density functions, methods developed with information theoretic measures are able to use all the statistical information available in the data, not just a finite number of moments. However, by using kernel estimation, the methods are dependent on choosing a suitable bandwidth parameter and have trouble dealing with data which vary on different scales.

In this thesis, the field of information theoretic learning has been explored using k-nearest neighbor estimates for the probability density functions instead. The developed estimators of the information theoretic measures was used in a clustering routine and compared with the traditional kernel estimators.Performing clustering on a range of datasets and comparing the performance, the new method proved to provide superior results without the need of tuning any parameters. The performance difference was found to be especially large when clustering datasets where groups were on different scales.