u0012\n\nmin \n\n\u0012 𝜋 𝜃 (𝑜 𝑖 |𝑞 )\n\n𝜋 𝜃 𝑜𝑙𝑑 (𝑜 𝑖 |𝑞 ) 𝐴 𝑖 , clip \n\n\u0012 𝜋 𝜃 (𝑜 𝑖 |𝑞 )\n\n𝜋 𝜃 𝑜𝑙𝑑 (𝑜 𝑖 |𝑞 ) , 1 − 𝜀 , 1 + 𝜀 \n\n\u0013\n\n𝐴 𝑖 \n\n\u0013\n\n− 𝛽 D𝐾𝐿 \n\n\u0000𝜋 𝜃 || 𝜋 𝑟𝑒 𝑓 \n\n\u0001\u0013\n\n, (1) \n\nD𝐾𝐿 \n\n\u0000𝜋 𝜃 || 𝜋 𝑟𝑒 𝑓 \n\n\u0001 = 𝜋 𝑟𝑒 𝑓 (𝑜 𝑖 |𝑞 )\n\n𝜋 𝜃 (𝑜 𝑖 |𝑞 ) − log 𝜋 𝑟𝑒 𝑓 (𝑜 𝑖 |𝑞 )\n\n𝜋 𝜃 (𝑜 𝑖 |𝑞 ) − 1, (2) where 𝜀 and 𝛽 are hyper-parameters, and 𝐴 𝑖 is the advantage, computed using a group of rewards {𝑟 1, 𝑟 2, . . . , 𝑟 𝐺 } corresponding to the outputs within each group: \n\n𝐴 𝑖 = 𝑟 𝑖 − m𝑒𝑎𝑛 ({ 𝑟 1, 𝑟 2, · · · , 𝑟 𝐺 }) \n\ns𝑡𝑑 ({ 𝑟 1, 𝑟 2, · · · , 𝑟 𝐺 }) . (3) 5A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within and tags, respectively, i.e., reasoning process here answer here . User: prompt. Assistant: \n\nTable 1 | Template for DeepSeek-R1-Zero. prompt will be replaced with the specific reasoning question during training. \n\n2.2.2. Reward Modeling \n\nThe reward is the source of the training signal, which decides the optimization direction of RL. To train DeepSeek-R1-Zero, we adopt a rule-based reward system that mainly consists of two types of rewards: • Accuracy rewards : The accuracy reward model evaluates whether the response is correct. For example, in the case of math problems with deterministic results, the model is required to provide the final answer in a specified format (e.g., within a box), enabling reliable rule-based verification of correctness. Similarly, for LeetCode problems, a compiler can be used to generate feedback based on predefined test cases. • Format rewards : In addition to the accuracy reward model, we employ a format reward model that enforces the model to put its thinking process between ‘’ and ‘’ tags. We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero, because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process, and retraining the reward model needs additional training resources and it complicates the whole training pipeline. \n\n2.2.3. Training Template \n\nTo train DeepSeek-R1-Zero, we begin by designing a straightforward template that guides the base model to adhere to our specified instructions. As depicted in Table 1, this template requires DeepSeek-R1-Zero to first produce a reasoning process, followed by the final answer. We intentionally limit our constraints to this structural format, avoiding any content-specific biases—such as mandating reflective reasoning or promoting particular problem-solving strate-gies—to ensure that we can accurately observe the model’s natural progression during the RL process. \n\n2.2.4. Performance, Self-evolution Process and Aha Moment of DeepSeek-R1-Zero \n\nPerformance of DeepSeek-R1-Zero Figure 2 depicts the performance trajectory of DeepSeek-R1-Zero on the AIME 2024 benchmark throughout the RL training process. As illustrated, DeepSeek-R1-Zero demonstrates a steady and consistent enhancement in performance as the RL training advances. Notably, the average pass@1 score on AIME 2024 shows a significant increase, jumping from an initial 15.6% to an impressive 71.0%, reaching performance levels comparable to OpenAI-o1-0912. This significant improvement highlights the efficacy of our RL algorithm in optimizing the model’s performance over time. Table 2 provides a comparative analysis between DeepSeek-R1-Zero and OpenAI’s o1-0912 models across a variety of reasoning-related benchmarks. The

Text

Podcast Information

Roles

Scripts