Hyper basis function network

The typical HyperBF network structure consists of a real input vector ${\displaystyle x\in \mathbb {R} ^{n}}$, a hidden layer of activation functions and a linear output layer. The output of the network is a scalar function of the input vector, ${\displaystyle \phi$ :\mathbb {R} ^{n}\to \mathbb {R} } , is given by

where ${\displaystyle N}$ is a number of neurons in the hidden layer, ${\displaystyle \mu _{j}}$ and ${\displaystyle a_{j}}$ are the center and weight of neuron ${\displaystyle j}$. The activation function ${\displaystyle \rho _{j}(||x-\mu _{j}||)}$ at the HyperBF network takes the following form

where ${\displaystyle R_{j}}$ is a positive definite ${\displaystyle d\times d}$ matrix. Depending on the application, the following types of matrices ${\displaystyle R_{j}}$ are usually considered^[3]

• ${\displaystyle R_{j}={\frac {1}{2\sigma ^{2}}}\mathbb {I} _{d\times d}}$, where ${\displaystyle \sigma >0}$. This case corresponds to the regular RBF network.
• ${\displaystyle R_{j}={\frac {1}{2\sigma _{j}^{2}}}\mathbb {I} _{d\times d}}$, where ${\displaystyle \sigma _{j}>0}$. In this case, the basis functions are radially symmetric, but are scaled with different width.
• ${\displaystyle R_{j}=diag\left({\frac {1}{2\sigma _{j1}^{2}}},...,{\frac {1}{2\sigma _{jz}^{2}}}\right)\mathbb {I} _{d\times d}}$, where ${\displaystyle \sigma _{ji}>0}$. Every neuron has an elliptic shape with a varying size.
• Positive definite matrix, but not diagonal.

Training HyperBF networks involves estimation of weights ${\displaystyle a_{j}}$, shape and centers of neurons ${\displaystyle R_{j}}$ and ${\displaystyle \mu _{j}}$. Poggio and Girosi (1990) describe the training method with moving centers and adaptable neuron shapes. The outline of the method is provided below.

Consider the quadratic loss of the network ${\displaystyle H[\phi ^{*}]=\sum _{i=1}^{N}(y_{i}-\phi ^{*}(x_{i}))^{2}}$. The following conditions must be satisfied at the optimum:

where ${\displaystyle R_{j}=W^{T}W}$. Then in the gradient descent method the values of ${\displaystyle a_{j},\mu _{j},W}$ that minimize ${\displaystyle H[\phi ^{*}]}$ can be found as a stable fixed point of the following dynamic system:

where ${\displaystyle \omega }$ determines the rate of convergence.

Overall, training HyperBF networks can be computationally challenging. Moreover, the high degree of freedom of HyperBF leads to overfitting and poor generalization. However, HyperBF networks have an important advantage that a small number of neurons is enough for learning complex functions.^[2]