ALOPEX

ALOPEX, in its simplest form is defined by an updating equation:

${\displaystyle \Delta \ W_{ij}(n)=\gamma \ \Delta \ W_{ij}(n-1)\Delta \ R(n)+r_{i}(n)}$

where:

• ${\displaystyle n\geq 0}$ is the iteration or time-step.
• ${\displaystyle \Delta \ W_{ij}(n)}$ is the difference between the current and previous value of system variable ${\displaystyle \ W_{ij}}$ at iteration ${\displaystyle n}$.
• ${\displaystyle \Delta \ R(n)}$ is the difference between the current and previous value of the response function ${\displaystyle \ R,}$ at iteration ${\displaystyle n}$.
• ${\displaystyle \gamma }$ is the learning rate parameter ${\displaystyle (\gamma \ <0}$ minimizes ${\displaystyle R,}$ and ${\displaystyle \gamma \ >0}$ maximizes ${\displaystyle R\ )}$
• ${\displaystyle r_{i}(n)\sim \ N(0,\sigma \ ^{2})}$

Essentially, ALOPEX changes each system variable ${\displaystyle W_{ij}(n)}$ based on a product of: the previous change in the variable ${\displaystyle \Delta }$${\displaystyle W_{ij}(n-1)}$, the resulting change in the cost function ${\displaystyle \Delta }$${\displaystyle R(n)}$, and the learning rate parameter ${\displaystyle \gamma }$. Further, to find the absolute minimum (or maximum), the stochastic process ${\displaystyle r_{ij}(n)}$ (Gaussian or other) is added to stochastically "push" the algorithm out of any local minima.

• Harth, E.; Tzanakou, E. (December 1974). "ALOPEX: A stochastic method for determining visual receptive fields". Vision Research. 14 (12): 1475–1482. doi:10.1016/0042-6989(74)90024-8. PMID 4446379.