Perforated Backpropagation: Improved Models with Minutes of Coding

Author(s): Dr. Rorry Brenner

Originally published on Towards AI.

From LLMs to robotics to medical AI, modern deep learning has been incredibly effective in a wide array of applications across nearly all industries. As more time and funding is spent on research this trend will continue with better architectures, bigger datasets, and powerful new hardware. However, the foundational building block, the artificial neuron, remains the same since its original mathematical instantiation in 1943. Additionally, the foundational algorithm, backpropagation with gradient descent, has had only updates to its optimization technique since first being applied to neural networks in the 1980s. It’s time for a long overdue update to the components we’re building with and how we train them.

While what can be done with neural networks is complex, the artificial neuron is simple. Neurons in a network are linear classifiers. They compute a weighted sum of their input and pass that value through a nonlinear function for their output. With backpropagation they are able to learn to be feature detectors, identifying when an input pattern represents the feature they have learned to code for. But as linear classifiers, it is unavoidable that there will be false positives and false negatives within the decisions they learn to make. Can you imagine the additional intelligence a neural network could have if each individual neuron was able to know when a pattern was going to be incorrectly classified? If it had an additional learnable weight from a oracle node, outside of the network, that identified when it was going to make a mistake?

This is exactly what is happening with perforated backpropagation, but it’s not an oracle. The process is actually very simple to understand, and the most interesting part is that it’s inspired by real neuroscience we have discovered in the eight decades since artificial neurons were first created.

Dendrite Learning

When training a deep learning system, input data is propagated forward through the network, an error value is calculated, and that error is then propagated backwards through the network. During this process each neuron will have determined an error term representing its “correctness” on the decision it made. This is the value used to adjust its weights. But this value is also a quantitative identifier for when a particular input pattern is landing on the wrong side of a neuron’s decision boundary. This value is what is used to train the new “Dendrite” nodes.

There are two significant differences that make Dendrites distinct from neurons. The first comes into play when training their input weights. When Dendrites are created they have no output connection, but form weighted input connections to the same inputs as their associated neuron. However, rather than using standard gradient descent to calculate their error term, they use a covariance based loss function. This function aims to maximize a covariance value between the Dendrite’s output and the neuron’s calculated error. In other words, Dendrites adjust their weights such that their output is high for input patterns where the neuron’s error is high. This is what allows them to learn how to identify patterns that are outliers for the neuron’s classification decision. It is theoretically similar to learning residual error within a ResNet, but applied to each individual neuron.

Neuron with Dendrite Learning

Once these weights have been learned, the Dendrite is connected to the network by adding a single output weight from the Dendrite to its associated neuron. This weight is then adjusted based on the neuron’s error in the same way as the rest of its weights. However, the second difference between neurons and Dendrites is that backpropagation of error does not continue through this connection. This means that neurons only factor the error of other neurons into their weight changes. By placing Dendrites “outside” of the network each neuron continues to code for the same feature as it was originally, while also being empowered to make better decisions about that feature. It follows that empowering each neuron also empowers the network as a whole.

Example Use Case

We recently ran a hackathon to get more users to try this system out. The winner was the CTO of an AI consulting company specializing in providing custom AI solutions, many of which leverage various BERT models for NLP. The first graph in this article showed the results as our winner consistently improved accuracy results across all model architectures tested, ranging from 17% to 3% improved scores. Additionally, the introduction of Dendrites allows neurons so much more intelligent that the new modules can even be used to create compressed architectures without a loss in accuracy. This is done by starting with reduced models that remain smaller even with the extra parameters from adding Dendrites. The graph below shows Deep-Summing-Network based models catching up and even beating a BERT-tiny model with only 11% as many parameters.

What’s Next

Perforated Backpropagation is implemented with a simple pip install as an add on to PyTorch. In can be integrated into any PyTorch based system for experimentation and it is currently free during beta testing. Perforated Backpropagation can be used to increase accuracy (Stock Forecasting, PEFT with LoRA, Computer Vision) and compress models (NLP, Biotech Classification, Time Series Prediction). Stay tuned for more articles about background, use cases, and coding examples of this groundbreaking new technique. In the meantime check out the paper or our website. If you’d like to achieve up to 40% increased accuracy or 90% model compression with only minutes of coding, get started with our API here.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI