Flexibly modelling conditional distributions in regression
Seminar Room 1, Newton Institute
A general methodology for nonparametric regression modelling is proposed based on a mixture-of-experts model extended along two important dimensions. First, the experts are allowed to be heteroscedastic. The standard model with homoscedastic experts is shown to give a poor fit to heteroscedastic data in finite samples, especially when the number of covariates is large. Moreover, with heteroscedastic experts we typically need a lot fewer of them, which is beneficial for interpretation and the efficiency of the inference algorithm. The second main extension is the introduction of variable selection among the covariates in the mean, variance, and in the set of covariates that control the mixture probabilities. The variable selection acts as a self-adjusting mechanism which is a very effective guard against overfitting, and makes fitting of high-dimensional nonparametric models feasible. We also point out a certain type of identification problem that arises with nonparametric experts, and we design the variable selection prior to solve this problem.