Full-Span Log-Linear Model and Fast Learning Algorithm

Abstract

The full-span log-linear (FSLL) model introduced in this letter is considered an nth order Boltzmann machine, where n is the number of all variables in the target system. Let X=(X0,…,Xn-1) be finite discrete random variables that can take |X|=|X0|…|Xn-1| different values. The FSLL model has |X|-1 parameters and can represent arbitrary positive distributions of X. The FSLL model is a highest-order Boltzmann machine; nevertheless, we can compute the dual parameter of the model distribution, which plays important roles in exponential families in O(|X|log|X|) time. Furthermore, using properties of the dual parameters of the FSLL model, we can construct an efficient learning algorithm. The FSLL model is limited to small probabilistic models up to |X|≈225; however, in this problem domain, the FSLL model flexibly fits various true distributions underlying the training data without any hyperparameter tuning. The experiments showed that the FSLL successfully learned six training data sets such that |X|=220 within 1 minute with a laptop PC.