Month in 4 Papers (January 2025)

Author(s): Ala Falaki, PhD

Originally published on Towards AI.

This member-only story is on us. Upgrade to access all of Medium.

How Language Models Learn to Think, Judge, and Scale: From Code Evaluation to Memory-Efficient Reasoning.

This series of posts is designed to bring you the newest findings and developments in the NLP field. I’ll delve into four significant research papers each month, offering a comprehensive summary. Be sure to visit my blog regularly or subscribe to my newsletter for monthly updates. Let’s dive in!

📝 CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? [paper] [code]

The paper introduces a new coding benchmark (CJ-Eval) focusing on the model’s ability to understand the written code instead of the code generation task. The idea behind the benchmark is inspired by educational theory, which says that if someone can correctly evaluate other candidates’ solutions, they will likely fully understand the given task. Means there is a difference in being able to generate code and understanding it.

They used the same concept of LLM-as-a-judge to use a group of proprietary and open-source models to judge whether a provided code is correct. The output could be (AC=Accepted), or different errors like WA (Wrong Answer) or RE (Runtime Error), to name a few. Their finding shows that… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI