When considering memory structures in neural software systems, we are primarily dealing with the art of orderly forgetting. A neural system gains hardly any or no information at all from a simple storing of data. The question is not so much: What must we remember? Rather it is: What must we forget when and how?
Normally, neural memory structures are not exact storage devices. On this point, they resemble our human memory. If we are asked, for example, what the air temperature was last Monday, we are most probably unable to give a precise answer. But perhaps we can remember, whether it was warmer or colder than it is today.
The techniques used in the implementation of neural memory structures are very familiar to many followers of technical analysis and technical trading. For many of them use so-called moving averages in the analysis of time series and charts. Few users, however, are aware that a moving average – regardless of how it is calculated – represents a memory structure. Usually, this mathematical tool is used merely as a filter in order to generalize a large amount of information in a meaningful way. For example, if our task is to describe the set of natural numbers between 1 and 10 as precisely as possible using only a single number, the average of these numbers would lend itself for this purpose: