Researchers believe a sophisticated AI model known as a transformer might be implemented using the brain's neuron and astrocyte cell networks.

The findings could help scientists understand how the brain works and why transformers efficiently perform machine-learning tasks.

Scientists found a new type of more powerful neural network model called a transformer about six years ago. These models can reach exceptional results, such as synthesizing text from prompts with near-human precision. A transformer is at the heart of AI systems like ChatGPT and Bard. While highly effective, transformers are also mysterious: Unlike other brain-inspired neural network models, how to construct them using biological components has yet to be established.

Biological materials

Researchers have now developed a concept that may explain how a transformer could be constructed using biological materials found in the brain. They hypothesize that a biological network of neurons and other brain cells known as astrocytes may do the same essential computing as a transformer.

A recent study has revealed that astrocytes, non-neuronal cells common in the brain, communicate with neurons and play a role in several physiological processes, such as blood flow regulation. However, scientists still need to figure out what these cells perform computationally.

Astrocytes

The current study looked at the role of astrocytes in the brain from a computational standpoint and created a mathematical model that shows how they could be employed with neurons to make a biologically plausible transformer.

Their concept offers insights that may drive future neuroscience research into how the human brain functions. Simultaneously, it could assist machine-learning researchers in explaining why transformers are so successful across various challenging tasks.

Transformers

Transformers are not like other neural network models. For example, a recurrent neural network trained for natural language processing would compare each word in a sentence to an internal state set by the words before it. On the other hand, a transformer compares all of the words in the sentence simultaneously to develop a prediction, a process known as self-attention.

A presynaptic neuron sends chemicals called neurotransmitters through the synapse that connects it to a postsynaptic neuron when two neurons communicate. An astrocyte is also sometimes attached; it wraps a long, thin tentacle around the synapse, forming a tripartite (three-part) synapse. A single astrocyte can form millions of tripartite synapses.

Neurotransmitters

The astrocyte collects some neurotransmitters as they pass past the synaptic connection. The astrocyte can eventually send a signal back to the neurons. Because astrocytes operate on a much longer time scale than neurons — they generate signals by gradually increasing and lowering their calcium response — these cells can hold and integrate information conveyed by neurons.

With this knowledge, the researchers developed their idea that astrocytes could influence how transformers compute. They then created a mathematical model of a neuron-astrocyte network that would function similarly to a transformer. Based on a deep dive into the literature and help from neurologist collaborators, they took the essential mathematics that makes a transformer and constructed simple biophysical models of what astrocytes and neurons do when they communicate in the brain. The models were then integrated into various ways until they arrived at an equation of a neuron-astrocyte network that describes the self-attention of a transformer.

Conclusion

Through their investigation, the researchers demonstrated that their biophysical neuron-astrocyte network theoretically resembles a transformer. Furthermore, they ran numerical simulations by feeding images and text paragraphs to transformer models and comparing the results to their simulated neuron-astrocyte network. Both responded similarly to the prompts, supporting their theoretical hypothesis. 

The researchers' next step will be to move from theory to practice. They seek to utilize this knowledge to refine or reject their hypothesis by comparing the model's predictions to those seen in biological investigations. Furthermore, one result of their research is that astrocytes are engaged in long-term memory because the network needs to store information to act on it in the future.

Image source: Unsplash