Abstract

Mixture-of-experts (MoE) provide a flexible statistical model to capture unobserved heterogeneity in data. In modern applications of MoEs, the dimension of the feature space is large compared to a typical sample size of a training dataset, and hence statistical inference becomes intrinsically challenging. In this work, we propose and study, for the first time, penalized likelihood estimation and feature selection methods in ultrahigh-dimensional spaces for sparse MoEs, where the experts belong to generalized linear regression models (GLM). We refer to this model as sparse MoE-GLM. Under general conditions, we establish consistency in estimation and feature selection for the proposed methods. We assess the empirical performance of the methods through extensive simulations and illustrate their application in a real data analysis. Our work offers a comprehensive and detailed analysis of regularization methods for sparse MoE-GLM models in an ultrahigh-dimensional setting. This work is based on a chapter from the PhD thesis of my student, Pengqi Liu, at McGill University.