Inside DeepSeek-R1: The Amazing Model that Matches GPT-o1 on Reasoning at a Fraction of the Cost

Author(s): Jesus Rodriguez

Originally published on Towards AI.

I recently started an AI-focused educational newsletter, that already has over 175,000 subscribers. TheSequence is a no-BS (meaning no hype, no news, etc) ML-oriented newsletter that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers, and concepts. Please give it a try by subscribing below:

Quite often we see releases in generative AI that truly challenges people’s imagination. This is the DeepSeek-R1, the newest model by the famous Chinese eval lab that dabbles into reasoning. One of the dominat reasoning thesis in the market is that it’s an emerging property of the scaling laws. In other words, you need big models to get reasoning. DeepSeek-R1 challenges that thesis achieving reasoning by leveraging a very clever post-training process. The model is able to match the performance of GPT-o1 at a fraction of the compute cost. Quite amazing.

Let’s dive in:

Introduction to DeepSeek-R1 and its Motivation

The field of Large Language Models (LLMs) has seen remarkable progress, yet achieving robust reasoning capabilities remains a significant challenge. Many models rely on extensive supervised fine-tuning (SFT), which can be computationally expensive and may not fully unlock a model’s potential for self-improvement. DeepSeek-R1 and its precursor, DeepSeek-R1-Zero, represent a departure from this paradigm, exploring the power of reinforcement learning (RL) to develop and enhance reasoning capabilities in LLMs. This essay will delve into the technical details of the DeepSeek-R1 architecture and training process, highlighting key innovations and contributions.

DeepSeek-R1’s development was driven by the goal of exploring the potential of LLMs to develop reasoning skills without relying on a foundation of supervised data. The research began with the idea of pure RL to allow the model to self-evolve. This approach resulted in DeepSeek-R1-Zero, a model that demonstrated the possibility of incentivizing reasoning capabilities purely through RL. DeepSeek-R1 was created to address the issues of poor readability and language mixing observed in DeepSeek-R1-Zero, while further improving reasoning performance. DeepSeek-R1 incorporates multi-stage training and a cold-start data approach before RL. The goal of the DeepSeek project is to create better models and share them with the research community.

DeepSeek-R1-Zero: A Pure Reinforcement Learning Approach

DeepSeek-R1-Zero stands out as a model trained via large-scale reinforcement learning (RL) without any prior supervised fine-tuning (SFT). This approach aimed to explore the model’s capacity for self-evolution in reasoning.

• Reinforcement Learning Algorithm: DeepSeek-R1-Zero utilizes Group Relative Policy Optimization (GRPO). GRPO is a cost-effective RL method that omits the use of a critic model and instead estimates the baseline from group scores. Given a question q, GRPO samples a group of outputs from the old policy and optimizes the policy by maximizing a defined objective function. The objective function includes an advantage term, calculated using the rewards of the group’s output and a Kullback-Leibler (KL) divergence term which ensures that the policy updates are not too large.
• Reward Modeling: The reward system for DeepSeek-R1-Zero focuses on accuracy and format.
• Accuracy rewards evaluate the correctness of responses. For example, math problems require a precise answer in a specific format, which allows for rule-based verification.
• Format rewards enforce the inclusion of the model’s thinking process within <think> and </think> tags.
• Notably, DeepSeek-R1-Zero does not use process-based or outcome neural reward models.
• Training Template: A simple template guides the model to produce a reasoning process followed by the final answer. The template is designed to be free of content-specific biases to observe the model’s natural progression during RL.

DeepSeek-R1-Zero showed significant improvements on the AIME 2024 benchmark during training, going from 15.6% to 71.0% pass@1, which is comparable to OpenAI’s o1–0912. With majority voting, its score further improved to 86.7%. The model also demonstrated self-evolution by increasing its thinking time (response length) as training progressed, enabling more sophisticated problem-solving strategies such as reflection and exploration of alternative approaches. The model also exhibited an “aha moment”, where it learned to rethink its initial approach by allocating more thinking time.

DeepSeek-R1: Incorporating Cold Start Data and Multi-Stage Training

While DeepSeek-R1-Zero demonstrated the potential of pure RL, it suffered from issues such as poor readability and language mixing. DeepSeek-R1 was developed to address these issues and to further enhance performance through a multi-stage training pipeline that incorporates a small amount of “cold-start” data.

• Cold Start Data: DeepSeek-R1 is fine-tuned on thousands of long Chain-of-Thought (CoT) examples before RL training, which acts as the “cold start”. These examples are collected using methods such as few-shot prompting with long CoTs, directly prompting models for detailed answers with reflection and verification, refining DeepSeek-R1-Zero’s outputs and post-processing by human annotators. This cold start data helps address readability by using a readable output format that includes a summary at the end of each response and filters out responses that are not user-friendly.
• The output format is defined as |special_token|<reasoning_process>|special_token|, with the reasoning process being the CoT for the query and the summary summarizing the reasoning results.
• Reasoning-Oriented Reinforcement Learning: After fine-tuning on cold-start data, DeepSeek-R1 undergoes the same large-scale RL training as DeepSeek-R1-Zero. This phase focuses on enhancing reasoning capabilities for coding, math, science, and logic reasoning tasks. A language consistency reward was introduced to mitigate language mixing during RL training, though ablation experiments show that the reward results in a small performance degradation.
• Rejection Sampling and Supervised Fine-Tuning: Upon reaching convergence in the reasoning-oriented RL process, SFT data is generated via rejection sampling using the RL checkpoint, combined with supervised data from DeepSeek-V3 in areas such as writing and factual QA. Data is expanded beyond rule-based reward evaluation by incorporating a generative reward model using DeepSeek-V3 to judge ground-truth and model predictions. Non-reasoning data was also included from DeepSeek-V3 to enhance the model’s general capabilities.
• Reinforcement Learning for All Scenarios: A second RL stage aligns the model with human preferences, focusing on helpfulness and harmlessness. Rule-based rewards are used for reasoning data, while reward models capture preferences in general data.

Distillation and Evaluation

DeepSeek-R1’s reasoning capabilities were also transferred to smaller, more efficient models through distillation.

• Distillation Process: Open-source models like Qwen and Llama were directly fine-tuned using the 800k samples from DeepSeek-R1. This approach is effective in improving the reasoning abilities of smaller models. The base models used include Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5–14B, Qwen2.5–32B, Llama-3.1–8B, and Llama-3.3–70B-Instruct. Only SFT is applied to the distilled models, with no RL stage.
• Evaluation Metrics and Benchmarks: Models are evaluated on a range of benchmarks including MMLU, MMLU-Redux, MMLU-Pro, C-Eval, CMMLU, IFEval, FRAMES, GPQA Diamond, SimpleQA, SWE-Bench Verified, Aider, LiveCodeBench, Codeforces, Chinese National High School Mathematics Olympiad (CNMO 2024) and American Invitational Mathematics Examination 2024 (AIME 2024). Additionally, open-ended generation tasks are judged using LLMs, specifically AlpacaEval 2.0 and Arena-Hard. Evaluation prompts follow the setup in DeepSeek-V3, using the simple-evals framework, or their original protocols.
• Key Findings: DeepSeek-R1 achieves performance comparable to OpenAI-o1–1217 on a range of tasks. It shows superior performance in STEM-related questions compared to DeepSeek-V3, demonstrating the effectiveness of large-scale reinforcement learning. DeepSeek-R1 also shows strong document analysis capabilities as well as fact-based query abilities. The model also excels in writing tasks and open-domain question answering. On math tasks, DeepSeek-R1 is comparable to OpenAI-o1–1217. The distilled models show significant improvements, with DeepSeek-R1–7B outperforming GPT-4o-0513. Furthermore, DeepSeek-R1–14B surpassed QwQ-32B-Preview on all metrics. The distilled 32B and 70B models significantly outperformed o1-mini on most benchmarks, highlighting the effectiveness of distillation.

Key Contributions, Discussion, and Future Directions

DeepSeek-R1’s development highlights several key contributions:

• Pure RL for Reasoning: It validates that reasoning capabilities in LLMs can be incentivized purely through RL, without the need for SFT.
• Effective Multi-Stage RL Training Pipeline: The approach combines two RL and two SFT stages to improve reasoning patterns and align with human preferences.
• Distillation of Reasoning: DeepSeek-R1 demonstrates that reasoning patterns from larger models can be distilled into smaller ones, yielding improved performance.

The R1 paper also discusses some unsuccessful attempts including Process Reward Model (PRM) and Monte Carlo Tree Search (MCTS).

• Process Reward Model (PRM) was found to have limitations, such as difficulty in defining fine-grained steps in general reasoning, challenging evaluation of intermediate steps, and reward hacking.
• Monte Carlo Tree Search (MCTS) encountered difficulties due to the exponentially large search space and the challenge of training a fine-grained value model for token generation.

Future research directions include:

• General Capability Enhancement: Expanding DeepSeek-R1’s abilities in function calling, multi-turn interactions, complex role-playing, and JSON output.
• Language Mixing Mitigation: Addressing language mixing issues when handling queries in languages other than English and Chinese.
• Prompt Engineering: Improving the model’s robustness to variations in prompts, moving beyond its sensitivity to few-shot prompting.
• Software Engineering Tasks: Expanding RL to software engineering tasks by implementing rejection sampling or asynchronous evaluations to improve efficiency.

Conclusion

DeepSeek-R1 represents a significant advancement in the development of LLMs with enhanced reasoning capabilities. By employing innovative reinforcement learning techniques, a multi-stage training pipeline, and effective distillation methods, DeepSeek-R1 not only achieves impressive performance but also offers valuable insights into the potential for self-evolution and knowledge transfer in AI. The open-sourcing of DeepSeek-R1 and its distilled models will significantly contribute to the research community, enabling further advancements in this field.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI