In 1982, John Hopfield constructed the spin glass system and recurrent artificial neural network known as the Hopfield network, based on the Ising model created by Wilhelm Lenz and Ernst Ising.

Hopfield networks can be used as content-addressable memory systems with binary threshold nodes or continuous variable memory systems. Hopfield networks can also be used to represent human memory.

Network structure

These networks were designed to help people remember things and find specific patterns later on. These networks can also be used for auto-association and job optimization. Each node in this network has infinite connections to all other nodes. These nodes can be activated (state 1) or deactivated (state 0). The information obtained from other nodes can be used to restore these states. In contrast to other neural networks, the Hopfield network produces a finite result. In addition, these networks require uniformity in the sizes of inputs and outputs. 

Associative memory is what makes up the Hopfield network. This storage space permits data recovery from memory by employing only a partial copy. Hopfield networks are promising for pattern recognition because of this property. The relation between the input vector and the desired output vector is established in associative memory, a type of content-addressable memory. Data in memory can be reshuffled according to how well it matches the input vector.

Optimization structure

In 1985, Hopfield and Tank demonstrated the Hopfield network's application to solve the classic travelling salesman problem. The Hopfield network has since seen extensive use in the optimization community. The Hopfield network is a simple concept to apply to optimization issues: If the constrained/unconstrained cost function can be expressed as the Hopfield energy function E, then the constrained/unconstrained optimization issue has a Hopfield network whose equilibrium points are the optimal solutions.  

The Hopfield energy function minimizes the goal function and meets the limitations since the restrictions are "embedded" in the network's synaptic weights. The Hopfield energy function has been applied to many complex optimization problems with constraints, including associative memory systems.

Running

Hopfield networks are initialized by programming the units with their initial values. Hopfield demonstrated that the attractors of this nonlinear dynamical system are stable rather than periodic or chaotic, ensuring that the system will eventually converge. Accordingly, in the context of Hopfield networks, an attractor pattern is a final stable state, a pattern in which no value within it may change during updating.

Training

The energy of states that the net should "remember" is reduced during training with a Hopfield net. In other words, the network can be used as a content-addressable memory system, which will converge to a "remembered" state if only a subset of the state is provided. The net can revert to the learned state that is most comparable to the input after the input has been distorted. Associative memory is a type of memory retrieval that uses similarities to identify and recall past experiences. 

A Hopfield net with five units can be trained so that it converges to the minimum-energy state (1, 1, 1, 1, 1) when given the input state (1, 1, 1, 1). When the energy of states the network should remember is a local minimum, the network has been adequately trained. It's important to remember that, unlike Perceptron training, the neurons' thresholds are never changed.

Image source: Unsplash