Month in 4 Papers (June 2023)

Author(s): Ala Falaki, PhD

Originally published on Towards AI.

Advancing Language Models through Efficient Training and Alignment Techniques.

This series of posts is designed to bring you the newest findings and developments in the NLP field. I’ll delve into four significant research papers each month, offering a comprehensive summary. Be sure to visit my blog regularly or subscribe to my newsletter for monthly updates. Let’s dive in!

📝 Better & Faster Large Language Models via Multi-token Prediction [paper]

This paper proposes an approach where multiple tokens are predicted using multiple heads, shifting from the conventional method of predicting only the next token. The method uses a shared model (called trunk) containing 13 billion parameters. During training, tokens are processed individually, with their losses computed and aggregated before the backward pass and weight updates are done. This ensures that memory usage will not grow.

During the inference phase, the model can generate output tokens sequentially as previously done or leverage the proposed method to accelerate the inference process by a factor of three.

This method proved most effective on coding benchmarks like HumanEval and MBPP. Their thorough analysis indicates that the effectiveness of this method becomes more apparent as the scale increases. Moreover, experimenting with various numbers of heads revealed that predicting four tokens in… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI