TAI #141: Claude 3.7 Sonnet; Software Dev Focus in Anthropic’s First Thinking Model

Author(s): Towards AI Editorial Team

Originally published on Towards AI.

What happened this week in AI by Louie

Anthropic Claude 3.7 Sonnet reasoning model stole the show this week. This is partly due to how quickly you can test and see the model’s coding talent with native code rendering in Claude’s Artefacts feature. This was also a positive week for open-source reasoning LLMs with Alibaba’s QwQ-Max-Preview now available and set for an imminent open-weights release. Meanwhile, Prime Intellect released SYNTHETIC-1 — the largest open reasoning dataset yet, comprising 2 million verified math, coding, and science reasoning traces from DeepSeek-R1 — alongside a 7B model fine-tuned on this data. This dataset will be valuable for fine-tuning reasoning models and customizing them to specific domains. Meanwhile, OpenAI disclosed that ChatGPT has hit 400 million weekly active users, which we calculate now covers 7.2% of global internet users!

Anthropic Claude 3.7 Sonnet’s headline feature is its “extended thinking” mode, where the model now explicitly shows multi-step reasoning before finalizing answers. Anthropic noted that it focuses its reinforcement learning training on real-world code problems relative to math problems and competition code (a slight dig at OpenAI’s o3 Codeforces focus here). This focus shows very impressive SWE-bench agentic coding capabilities while Maths benchmarks lag other leading reasoning models. Claude 3.7 Sonnet scores 62.3% without thinking mode on SWE (70.3% with a scaffold on a subset of problems), significantly ahead of OpenAI o1 at 48.9%, o3-mini high at 49.3% and in the ballpark of OpenAI o3’s reported 71.7% score. On AIME MATH 2024, Claude 3.7 Sonnet scores 61.3% pass@1 (80.0% with parallel scaling), a big jump from Sonnet 3.5 New’s 16.0% but behind Grok-3’s 83.9% (93.3% with parallel scaling) and OpenAI o3 at 96.7% score with parallel scaling.

Looking at GPQA Diamond, Claude 3.7 Sonnet with Extended thinking achieves 78.2% pass @1 (or 84.8% with up to 64k tokens, including parallel scaling of multiple samples). This outperforms OpenAI o1 at 75.7% (78.3% with parallel scaling) and OpenAI o3-mini at 79.7%. However, Grok-3 thinking mode wins here with pass @1 80.2% (84.6% with parallel scaling), and OpenAI’s unreleased o3 still leads at 87.7% with parallel scaling. In non-thinking modes, Claude 3.7 Sonnet scores 68.0% — below Grok-3’s non-thinking score of 75%, but outperforming Gemini-Pro 2.0 at 65% and OpenAI 4o at 51%.

Claude 3.7 Sonnet keeps the same price as 3.5 even in thinking mode: $3 per million input tokens and $15 per million output tokens. This is a positive surprise given both OpenAI and Deepseek charge a large premium for their reasoning models — which we think is justified by higher compute and memory usage and lower inference batch sizes possible when the average output generation length is longer (even with the same architecture). Via the API, Claude 3.7 Sonnet users also get extra flexibility to control the budget for thinking: you can tell Claude to think for no more than N tokens (up to 128k) — allowing you to decide your own trade-offs between latency, cost, and capability.

Anthropic recently revealed that 37% of Claude Chatbot queries are software-related — and this doesn’t even count the heavy use of Claude APIs for coding copilots and agents such as Cursor. Perhaps Anthropic is beginning to see this as a point of differentiation against OpenAI, Google Gemini, and xAI — in any case — beyond Claude 3.7’s focused improvement on software-related benchmarks — Anthropic also released its own coding agent. Claude Code is now in beta as a research preview. Designed to operate directly in the terminal, it integrates directly with developers’ workflows without additional servers or complex setups. Claude Code can edit files, fix bugs, execute tests, search through git history, resolve merge conflicts, and create commits — all through natural language commands.

Why should you care?

As we noted last week — reasoning models with test time compute scaling capability have become the new battleground for SOTA in LLMs. Now, Anthropic has also joined the party (following OpenAI, Deepseek, Alibaba, Google Deepmind, and xAI). Potentially in part due to this competition, Anthropic has also come with a more user-friendly approach — both in its subscription chatbot, where 3.7’s Thinking mode can simply be toggled (similar to Grok-3) and in its API — where it provides the most flexibility yet for directly controlling thinking token usage and offers flat pricing vs. the base model. It will take some practice to test how well this thinking token control works and how much benefit you really get from pushing to the highest settings.

On our side, the early testing of 3.7 shows some impressive capabilities, particularly in front-end code with mini apps and websites. We think this will only accelerate the momentum in “LLM Native” coding or what Andrej Karpathy calls “vibes coding,” where even experienced developers can quickly learn new software skills and languages top-down — starting with natural language and an LLM-generated project. More on this soon, with the imminent release of our “LLM native” Python course for coding novices!

— Louie Peters — Towards AI Co-founder and CEO

Hottest News

1. Anthropic Introduced Claude 3.7 Sonnet, Their Hybrid Reasoning Model

Anthropic announced Claude 3.7 Sonnet, its most intelligent and first hybrid reasoning model. It can produce near-instant responses or extended, step-by-step thinking that is visible to the user. API users also have fine-grained control over how long the model can think. Anthropic also introduced a limited research preview of Claude Code, a command-line tool for agentic coding.

2. Prime Intellect Released SYNTHETIC-1, the Largest Open Reasoning Dataset

Prime Intellect has introduced SYNTHETIC-1, an open-source dataset designed to provide verified reasoning traces in math, coding, and science. Built with the support of DeepSeek-R1, this dataset consists of 1.4 million structured tasks and verifiers. The objective of SYNTHETIC-1 is to improve reasoning models by supplying them with well-organized, reliable data, addressing the shortcomings of existing resources.

3. Arc Institute Developed Evo 2, the Largest AI Model for Biology

Arc Institute developed Evo 2, trained on the DNA of over 100,000 species across the entire tree of life. The model can write whole chromosomes and small genomes from scratch. It can also make sense of existing DNA, including hard-to-interpret ‘non-coding’ gene variants linked to disease.

4. Alibaba Unveils QwQ-Max-Preview

Alibaba launched QwQ-Max-Preview, a new reasoning model in the Qwen family of AI models. The model is built on the Qwen 2.5 Max and specializes in mathematics and coding-based tasks. The model is in its preview stage, and the company is expected to announce its full version soon.

5. Open AI Introduced the SWE-Lancer Benchmark

SWE-Lancer introduces a benchmark with over 1,400 freelance software engineering tasks from Upwork, valued at $1 million. It evaluates model performance on tasks, ranging from $50 bug fixes to $32,000 feature implementations, through tests verified by engineers. Frontier models struggle with most tasks.

6. Perplexity AI Open-Sourcing a Post-Trained DeepSeek-R1 With Censorship Removed

Perplexity AI introduced the R1 1776 model, a post-trained DeepSeek-R1 designed to eliminate Chinese Communist Party censorship while maintaining high reasoning abilities. Rigorous multilingual evaluations using human annotators and LLM judges confirmed that decensoring did not impact the model’s core reasoning capabilities. The model performed on par with the base R1 model across various sensitive topics.

Five 5-minute reads/videos to keep you learning

1. Andrej Karpathy’s Early Access Review of Grok 3

In this article, Andrej Karpathy evaluated Grok 3, noting its strong performance in thinking tasks, comparable to OpenAI’s models. Despite issues with humor and ethical sensitivity, it surpassed DeepSeek-R1 and Gemini 2.0 in some areas.

2. DeepSeek vs. ChatGPT — A Detailed Architectural and Functional Breakdown

This article compares two leading LLMs, ChatGPT and DeepSeek. It focuses on architectural design, training methodologies, performance, and limitations. While ChatGPT is more generic and can handle a broader variety of tasks, DeepSeek becomes a more feasible alternative for tightly focused applications.

3. Build Your LLM Engineer Portfolio (Part 2): A 3-Month Roadmap

This step-by-step guide is designed to help you build, refine, and showcase a RAG portfolio to kickstart your career. It provides a comprehensive plan for your three-month journey toward creating an impressive RAG portfolio. You’ll find essential preparation steps to establish a solid foundation for your projects, seven impactful projects that will enhance your expertise and help you stand out, and strategies for deploying and presenting your portfolio to achieve maximum exposure.

4. 1 Billion Classifications

This blog explains how to calculate cost and latency for large-scale classification and embedding. It analyzes different model architectures, benchmarks costs across hardware choices and provides a clear framework for optimizing your own setup.

5. Insights on Crosscoder Model Diffing

Crosscoder-based model diffing is a promising method for isolating differences between two models with a single SAE training run. In this note, Anthropic discusses a few unexpected observations when applying this technique to real models. This will be helpful for researchers working actively in this space.

Repositories & Tools

1. Aibrix is an open-source initiative that provides building blocks to construct scalable GenAI inference infrastructure.
2. FlashMLA is an MLA decoding kernel for Hopper GPUs optimized for variable-length sequence serving.
3. Mastra is a Typescript framework that helps you build AI applications and features.

Top Papers of The Week

1. Accelerating Scientific Breakthroughs With an AI Co-Scientist

This paper introduces AI co-scientist, a multi-agent AI system built with Gemini 2.0. The agent acts as a virtual scientific collaborator to help scientists generate novel hypotheses and research proposals. It can potentially help accelerate the clock speed of scientific and biomedical discoveries.

2. Qwen2.5-VL Technical Report

Qwen2.5-VL advances visual recognition with enhanced object localization, robust document parsing, and long-video comprehension. It accurately extracts structured data and analyzes charts. Featuring dynamic resolution processing and Window Attention, Qwen2.5-VL reduces computational overhead.

3. MoBA: Mixture of Block Attention for Long-Context LLMs

This paper introduces Mixture of Block Attention (MoBA), an innovative approach that applies the principles of Mixture of Experts (MoE) to the attention mechanism. This architecture demonstrates superior performance on long-context tasks while seamlessly transitioning between full and sparse attention, enhancing efficiency without compromising performance.

4. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

NSA, a Natively trainable Sparse Attention mechanism, improves efficiency in long-context modeling by integrating algorithmic innovations with hardware-aligned optimizations. It achieves substantial speedups and maintains model performance across benchmarks. NSA’s dynamic hierarchical strategy combines coarse-grained token compression with fine-grained selection, excelling over Full Attention in 64k-length sequences during decoding, forward propagation, and backward propagation.

5. CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

CodeI/O enhances reasoning in language models by transforming code into input-output prediction formats, exposing models to reasoning patterns like logic flow planning and decision tree traversal. This method improves performance across multiple reasoning tasks.

Quick Links

1. OpenAI’s COO recently shared some key updates in a post on X. ChatGPT has surpassed 400 million weekly active users. He also hinted at the upcoming GPT-4.5 and GPT-5 releases for both the chat interface and the API. Additionally, free users will have unlimited access to GPT-5 and enhanced agent capabilities.

2. Meta announces LlamaCon, its first generative AI dev conference focusing on open-source AI developments. Despite competitive pressures from DeepSeek, Meta plans to release new Llama models with reasoning and multimodal capabilities.

3. Together AI secured $305 million in Series B funding, led by General Catalyst and joined by notable investors like NVIDIA and Salesforce Ventures. This investment accelerates the company’s AI Acceleration Cloud for open-source and enterprise AI application development.

Who’s Hiring in AI

Gen AI Developer @Syncreon Consulting (Toronto, Canada)

Junior Data & AI Engineer — Emirati @Ghobash Group (Dubai, UAE)

AI Software Engineer (Mid/Senior) @SmartDev (Hanoi, Vietnam)

Global Data Office Intern — Summer 2025 @Visa (CA, USA)

PDM Engineering Data Administrator @Bosch Group (Burnsville, MN, USA)

Software Engineer — NLU @Cognigy (Remote)

Interested in sharing a job opportunity here? Contact [email protected].

Think a friend would enjoy this too? Share the newsletter and let them join the conversation.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI