Can Transformer Substitute Graph Neural Networks?

Author(s): Salvatore Raieli

Originally published on Towards AI.

Are transformers able to do graph reasoning and to which extent?
image generated by the author using AI

Mathematical reasoning may be regarded rather schematically as the exercise of a combination of two facilities, which we may call intuition and ingenuity — Alan Turing

The transformer when it was released revolutionized machine translation. Although originally designed for a specific task, this revolutionary architecture showed that it was easily adaptable to different tasks. The transformer itself then became a standard even for data other than what it was originally designed for (images and any other sequential data).

Will be the transformer the model leading us to artificial general intelligence? Or will be replaced?

towardsdatascience.com

Later, however, people also began to look for an alternative, especially in order to be able to reduce its computational cost (derived from self-attention and its quadratic cost). In recent times there have been discussions about which architecture is superior in terms of computational cost, but this is only one of the points of disagreement. In fact, what has made the transformer successful is that by scaling it, the model is able to show reasoning ability.

The Hyena model shows how convolution could be faster than self-attention

levelup.gitconnected.com

Researchers have massively improved LSTM, but what does it mean for the future?

levelup.gitconnected.com

How can we analyze the reasoning… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI