Self-Evolving Search Agents: How LLMs Learn Without Training Data

Table of Contents

• Introduction: The Data Bottleneck Problem
  • The Diminishing Returns of Data
  • Self-Evolution: A New Paradigm
• The Core Challenge: Why Search Agents Are Different
  • Multi-Turn Reasoning Complexity
  • Tool Use and External Knowledge
  • Why Existing Frameworks Fall Short
• Dr. Zero: Framework Overview
  • The Key Insight
  • Architecture Components
  • The Self-Evolution Feedback Loop
• The Proposer: Generating Training Data from Nothing
  • Multi-Turn Tool-Use Rollout
  • Difficulty-Guided Rewards
  • Automated Curriculum Learning
• The Solver: Learning to Search and Reason
  • Multi-Step Reasoning Pipeline
  • Learning from Synthetic Questions
  • The Co-Evolution Dynamic
• Hop-Grouped Relative Policy Optimization (HRPO)
  • The Problem with Standard GRPO
  • HRPO: A Smarter Grouping Strategy
  • Mathematical Formulation
• Technical Deep Dive: The Mathematics
  • Policy Gradient Foundation
  • Advantage Estimation
  • Reward Shaping for Multi-Hop Questions
• Experimental Setup
  • Models and Training
  • Benchmarks
  • Baselines
• Results and Analysis
  • Main Results
  • Ablation Studies
  • Scaling Properties
• Implications: The Future of Self-Evolving AI
  • Beyond Search Agents
  • Limitations and Open Questions
  • The Bigger Picture
• References

We're running out of data. Not in some abstract, distant-future sense. Right now, today, the AI industry is hitting a wall. The internet has been scraped, books have been digitized, and the low-hanging fruit of human-generated text has largely been picked. Yet models keep getting bigger, and their appetite for training data grows with them. Something has to give.

What if models could train themselves? Instead of relying on expensive human annotation and increasingly scarce high-quality data, a language model could generate its own training problems, solve them, and learn from the experience. That's no longer hypothetical. A recent paper from Meta's Superintelligence Labs, Dr. Zero (DeepResearch-Zero), shows that this can not only work but can match or beat fully supervised approaches.

Introduction: The Data Bottleneck Problem

The Diminishing Returns of Data

The scaling laws that have driven AI progress for the past decade are brutally simple: more data plus more compute equals better models. This worked while the internet was an untapped goldmine. But we've mined most of it now. Estimates put the truly high-quality text on the internet at roughly 300 billion tokens. Current frontier models train on trillions, which means we're already recycling the same data several times over.

The problem compounds when you need specialized capabilities. Training a model to do multi-step research, where it searches for information, reasons about what it finds, searches again based on that reasoning, and synthesizes an answer, requires datasets of exactly that behavior. Those datasets are extraordinarily expensive to make. You need human experts who can demonstrate the entire reasoning chain, annotate what they're thinking at each step, and do this thousands of times across diverse topics.

This is where self-evolution enters the picture.

Self-Evolution: A New Paradigm

The idea is simple: use the model to generate training data for itself. A proposer component generates problems, and a solver component attempts them. The solver's attempts are evaluated, sometimes by the model itself and sometimes by external verification, and the successful reasoning traces become training data. As the solver improves, it pushes the proposer to generate harder problems, and an automated curriculum falls out of the interaction.

This has worked well in constrained domains like mathematics. Math problems have clear answers: you can check whether 2+2 equals 4 without a human annotator. The "STaR" (Self-Taught Reasoner) family of methods showed that from a small seed of examples, models can bootstrap their way to strong mathematical reasoning on self-generated data.

But mathematics is the easy case. The problems are well-defined, the solution space is constrained, and verification is straightforward. The interesting question is what happens with something messier: open-domain question answering that requires searching the web, synthesizing information from multiple sources, and reasoning across facts the model hasn't memorized.

That is the problem Dr. Zero takes on.

The Core Challenge: Why Search Agents Are Different

Multi-Turn Reasoning Complexity

Consider a seemingly simple question: "What year did the director of Inception also release a film that won the Academy Award for Best Picture?"

Answering this requires multiple steps:

1. Identify the director of Inception (Christopher Nolan)
2. Search for Christopher Nolan's filmography
3. Identify which of his films won Best Picture (none, actually. Trick question, but the process matters)
4. If one existed, return the year

Each step depends on the previous one. The search query at step 2 can't be formulated until step 1 completes. The reasoning at step 3 requires the results from step 2. This sequential dependency creates a combinatorial explosion in the space of possible reasoning traces.

For self-evolution to work, the proposer needs to generate questions with this multi-hop structure. The problem is that if you just ask a language model to "generate hard questions," it produces questions that sound complex but can be answered with a single search. The model doesn't naturally know what makes a question genuinely require several steps.

Tool Use and External Knowledge

Search agents don't only reason; they interact with external tools. Each tool call is an action that produces an observation, and the model must decide what to do next based on that observation. That makes the action space vastly larger than in pure reasoning tasks.

In mathematical self-evolution, the model generates a problem, attempts a solution, and can verify correctness purely through computation. In search tasks, verification requires ground truth that may not exist in any structured form. How do you know "Christopher Nolan directed Inception" is correct without looking it up? And looking it up needs a search engine, which is the very tool the agent is trying to learn to use.

So there is a bootstrapping problem: the agent needs to use search to generate training data for learning to search.

Why Existing Frameworks Fall Short

Prior work on data-free self-evolution, however well it did on mathematical reasoning, runs into three problems with search agents.

The first is limited question diversity. Trained proposers drift toward simple, one-hop questions. If you reward a model for generating questions the solver can answer, it quickly learns to generate easy ones, and the proposer and solver settle into a comfortable, mediocre equilibrium.

The second is that difficulty doesn't progress. Without an explicit mechanism to raise difficulty as the solver improves, the system stalls early. The proposer generates questions at the edge of the solver's current capability, and that edge never moves because nothing pushes it.

The third is compute. Standard training algorithms like Group Relative Policy Optimization (GRPO) sample multiple responses for each question to estimate a baseline. For a search agent, each response involves multiple tool calls, each of which is a real search query. Nested sampling, many questions times many responses times many searches, gets prohibitively expensive.

Dr. Zero addresses all three.

Dr. Zero: Framework Overview

The Key Insight

The central idea in Dr. Zero is that an external search engine can serve two roles at once: it is the tool the agent uses, and it is the source of supervision for training. That dual role is what removes the need for human-curated data.

When the proposer generates a question, it first searches for factual information, then constructs a question that requires reasoning across several of the facts it found. The search engine supplies the ground-truth answer. When the solver attempts the question, the search engine supplies the information needed to reason toward (hopefully) the same answer. Whether the solver's answer matches the proposer's intended answer is the reward signal.

The result is a closed loop in which the search engine is the oracle, the proposer is the teacher, and the solver is the student, with no human involved.

Architecture Components

Dr. Zero has two main components, both initialized from the same base language model.

The proposer generates diverse multi-hop questions along with their ground-truth answers. It is trained to produce questions that are answerable using web search, that require multiple reasoning steps, and that sit at the right difficulty for the current solver.

The solver learns to answer questions through multi-turn interaction with a search engine. It generates reasoning traces, issues search queries, processes results, and produces final answers, and it trains on the synthetic questions the proposer generates.

The two evolve together. As the solver gets better, the proposer's reward encourages harder questions. As the questions get harder, the solver is pushed to develop more sophisticated reasoning strategies.

Dr. Zero self-evolution overview: Proposer and Solver co-evolve across iterations

The Self-Evolution Feedback Loop

The training proceeds in iterations:

1. Proposer Rollout: The proposer uses the search engine to gather factual information and generates a batch of multi-hop questions with answers.

2. Solver Training: The solver attempts to answer the proposer's questions using the search engine. Successful reasoning traces (those that reach the correct answer) are used as positive training examples.

3. Proposer Training: The proposer is updated based on a difficulty-guided reward. Questions that are too easy (solver always succeeds) or too hard (solver always fails) receive lower rewards than questions at the boundary of the solver's capability.

4. Repeat: The improved solver changes the difficulty landscape, incentivizing the proposer to adapt, which generates new training data for the next round.

Self-Evolution Feedback Loop: Proposer generates QA pairs, Solver predicts answers, rewards update both models

Nobody schedules the curriculum. It emerges from the loop.

The Proposer: Generating Training Data from Nothing

Multi-Turn Tool-Use Rollout

The proposer doesn't generate questions from its parametric knowledge alone. It actively searches the web to construct factually grounded multi-hop questions, which is what guarantees the questions are answerable and have verifiable ground-truth answers.

The process works as follows:

1. Seed Selection: The proposer receives a seed topic or document title (e.g., "Christopher Nolan" or "Australian Labor Party").

2. Information Gathering: The proposer issues search queries to gather facts about the seed topic. It might learn that Christopher Nolan directed several films, won various awards, was born in London, etc.

3. Hop Construction: The proposer chains facts together to create multi-hop questions. Starting from "Christopher Nolan," it might follow: Nolan → directed Dunkirk → Dunkirk is about WWII → WWII ended in 1945. A resulting question might be: "In what year did the war that is the subject of a Christopher Nolan film end?"

4. Answer Extraction: The final fact in the chain provides the ground truth answer (1945 in this case).

The multi-turn rollout is what makes the questions genuinely complex. Single-turn generation produces questions that sound multi-hop but can be answered directly.

Difficulty-Guided Rewards

The proposer's reward is designed to favor questions that are solvable but challenging. Formally:

$$R_{\text{proposer}} = \mathbb{E}[\text{success}] \cdot (1 - \mathbb{E}[\text{success}])$$

This is maximized when the solver's success rate is 50%, when questions are neither trivially easy nor impossibly hard. A question the solver always gets right and one it always gets wrong both earn zero.

In practice, the framework incorporates additional factors:

$$R_{\text{proposer}} = \alpha \cdot \text{hop\_complexity} + \beta \cdot \text{solvability} + \gamma \cdot \text{diversity}$$

Here hop complexity measures how many reasoning steps the question requires, solvability penalizes questions that are consistently unsolvable, and diversity rewards variety in topics and question structure. The weights $\alpha$, $\beta$ and $\gamma$ are tuned to balance the three.

Automated Curriculum Learning

As training progresses, the optimal difficulty level rises on its own. Early on the solver is weak, so the proposer is rewarded for relatively simple 2-hop questions. As the solver improves, those 2-hop questions become too easy, the success rate approaches 100%, and the proposer's reward drops. The only way to keep the reward high is to generate harder 3-hop or 4-hop questions.

There is no explicit difficulty schedule anywhere in the system. The curriculum comes out of the interaction between the proposer's reward and the solver's growing ability, and the authors observe that the average hop count of generated questions climbs steadily across training iterations.

The Solver: Learning to Search and Reason

Multi-Step Reasoning Pipeline

The solver follows a structured approach to answering questions:

1. Think: Generate reasoning about what information is needed
2. Search: Issue a query to the search engine
3. Process: Analyze the search results and extract relevant facts
4. Iterate: Decide whether to search again or produce a final answer
5. Answer: Generate the final response

The loop continues until the solver is confident enough to answer or hits a maximum number of search iterations. When the answer is correct, the entire trajectory (reasoning steps, search queries and final answer) becomes training data.

The solver learns not just what to search for but how to reason about what comes back. It has to cope with noisy or irrelevant results, combine information from several sources, and keep its reasoning coherent across turns.

Learning from Synthetic Questions

The solver is trained with reinforcement learning on the proposer's questions. The reward is simple: +1 for a correct answer, 0 otherwise. The learning signal comes from comparing multiple reasoning trajectories.

For each question, the solver generates several candidate traces. Those that reach the correct answer get high advantage; those that fail get low advantage. The policy gradient pushes the solver toward strategies that succeed and away from ones that don't.

The solver never sees a human-written reasoning trace. All of its training data comes from its own attempts on synthetic questions. The teacher is the reward signal, not a human demonstration.

The Co-Evolution Dynamic

The solver and proposer are in a productive adversarial relationship. The proposer tries to generate questions that challenge the solver; the solver tries to answer whatever the proposer produces.

This dynamic prevents a few failure modes. If the proposer only generated easy questions, the solver would hit 100% success and the proposer's reward would collapse, so the proposer has to stay ahead. If the proposer generated impossible questions, the solver would sit at 0% and the reward would collapse the other way, so the proposer has to keep questions solvable. The highest reward sits at questions the solver can answer with effort, which is exactly the region you want training data from.

The result is a self-regulating system where both agents push each other to improve.

Hop-Grouped Relative Policy Optimization (HRPO)

The Problem with Standard GRPO

Group Relative Policy Optimization (GRPO) has become a popular algorithm for training language models with reinforcement learning. Instead of learning a separate value function (as in PPO), GRPO estimates advantages by comparing a response to other responses for the same question. That removes the critic network and simplifies training.

The standard GRPO formulation samples multiple responses $\{o_1, ..., o_G\}$ for each question $q$ and computes advantages as:

$$\hat{A}_i = r_i - \frac{1}{G}\sum_{j=1}^{G} r_j$$

where $r_i$ is the reward for response $i$, and the mean reward serves as the baseline.

For search agents this is expensive. Each response involves several search queries, so the sampling is nested. With $N$ questions and $G$ responses per question you need $N \times G$ complete reasoning traces, each making multiple search calls.

HRPO: A Smarter Grouping Strategy

HRPO (Hop-grouped Relative Policy Optimization) changes what gets grouped. Instead of grouping responses to the same question, it groups responses to structurally similar questions.

Questions can be characterized by their hop count, the number of reasoning steps they require. A 2-hop question like "Who directed the film where Leonardo DiCaprio played a dream thief?" is structurally similar to another 2-hop question like "What year was the director of Titanic born?" The topics differ, but the reasoning shape is the same.

HRPO groups questions by their hop count and computes advantages within these groups:

$$\hat{A}_i = r_i - \frac{1}{|H_k|}\sum_{j \in H_k} r_j$$

where $H_k$ is the set of all responses to questions with hop count $k$.

A small change on paper, with a large effect on efficiency.

Mathematical Formulation

The full HRPO objective can be written as:

$$\mathcal{L}_{\text{HRPO}} = -\mathbb{E}_{q \sim \mathcal{D}, o \sim \pi_\theta(q)} \left[ \frac{\pi_\theta(o|q)}{\pi_{\text{old}}(o|q)} \hat{A}(o, H_{k(q)}) \right]$$

Here $\mathcal{D}$ is the distribution of proposer-generated questions, $\pi_\theta$ is the current solver policy, $\pi_{\text{old}}$ is the policy at the start of the update, $H_{k(q)}$ is the hop group containing question $q$, and $\hat{A}(o, H_{k(q)})$ is the advantage computed against the hop-group baseline.

Because $H_{k(q)}$ contains responses from many different questions, each question needs only one sampled response. The baseline comes from the whole hop group rather than from repeated samples of the same question.

That cuts the sampling requirement from $N \times G$ responses to $N$, a factor of $G$ in sample efficiency.

Why does the approximation hold? Questions with the same hop count have similar difficulty and call for similar reasoning strategies. A 3-hop question about history is about as hard as a 3-hop question about science, on average. So the hop-group mean is a reasonable estimate of expected performance even without multiple samples per question.

Technical Deep Dive: The Mathematics

Policy Gradient Foundation

The entire framework rests on the policy gradient theorem. For a policy $\pi_\theta$ parameterized by $\theta$, the gradient of expected reward is:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \nabla_\theta \log \pi_\theta(\tau) R(\tau) \right]$$

where $\tau$ is a complete trajectory (all reasoning steps and search queries) and $R(\tau)$ is the trajectory reward.

In practice, we use the advantage $\hat{A}$ instead of raw rewards to reduce variance:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \nabla_\theta \log \pi_\theta(\tau) \hat{A}(\tau) \right]$$

For the solver, the trajectory is the complete reasoning trace, and the reward is 1 for correct answers, 0 otherwise. For the proposer, the trajectory is the question-generation process, and the reward is the difficulty-guided signal described earlier.

Advantage Estimation

HRPO's advantage estimate is a variance reduction over the naive policy gradient. Compare the two baselines.

Per-Question Baseline (standard GRPO):

$$\hat{A}_i^{\text{GRPO}} = r_i - \bar{r}_q \quad \text{where } \bar{r}_q = \frac{1}{G}\sum_{j \in \text{responses to } q} r_j$$

Hop-Group Baseline (HRPO):

$$\hat{A}_i^{\text{HRPO}} = r_i - \bar{r}_{H_k} \quad \text{where } \bar{r}_{H_k} = \frac{1}{|H_k|}\sum_{j \in H_k} r_j$$

The GRPO baseline is more accurate, since it's conditioned on the specific question, but it needs $G$ samples per question. The HRPO baseline is an approximation that needs one.

How good the approximation is depends on how well hop count predicts difficulty. Empirically it predicts it well: variance within a hop group is much lower than variance across the whole dataset.

Reward Shaping for Multi-Hop Questions

For the proposer, the raw reward signal (solver success rate) is sparse and noisy. To improve learning, Dr. Zero incorporates shaped rewards:

$$R_{\text{shaped}} = R_{\text{base}} + \lambda_1 \cdot \text{hop\_bonus} + \lambda_2 \cdot \text{format\_bonus} + \lambda_3 \cdot \text{search\_usage}$$

The hop bonus rewards questions with more hops, pushing toward difficulty. The format bonus rewards well-formed questions with proper entity references. The search-usage term rewards actually using the search engine during generation, which stops the proposer from degenerating into writing questions from parametric memory alone.

These terms steer the proposer without changing the underlying objective. The base reward still dominates; the shaping just supplies useful gradients early in training when the base signal is too sparse to learn from.

Experimental Setup

Models and Training

Dr. Zero experiments use Qwen2.5-Instruct models as the base for both proposer and solver, at 3B and 7B parameter scales. That's a compromise between capability and cost: small enough to iterate quickly, large enough to show meaningful reasoning.

Training runs in iterations. The first initializes both proposer and solver from the base model; each subsequent iteration starts from the previous iteration's trained models. Within an iteration, the current proposer generates around 10,000 questions, the solver trains on them for several epochs, the proposer is updated based on solver performance, and everything is evaluated on held-out benchmarks.

The total compute is substantial but tractable, roughly 50-100 GPU-hours per iteration on modern hardware.

Benchmarks

Evaluation covers both single-hop and multi-hop question answering.

The single-hop benchmarks are Natural Questions (NQ), real questions from Google search that test factual retrieval; TriviaQA, trivia questions with evidence from Wikipedia and the web; and PopQA, questions about popular entities that test common-knowledge retrieval.

The multi-hop benchmarks are HotpotQA, which requires reasoning across Wikipedia paragraphs and typically takes 2-3 hops; 2WikiMultiHopQA, harder questions spanning two Wikipedia articles that require explicit comparison; MuSiQue, multi-hop questions with decomposed sub-questions, which lets you see which reasoning step failed; and Bamboogle, adversarially constructed questions designed to defeat models that rely on shortcuts.

Together they span a range of difficulties and reasoning patterns.

Baselines

Dr. Zero is compared against four kinds of baseline. The base LLM, the original Llama-3.1-8B-Instruct with no search-agent training, sets the starting point. Supervised RL agents, trained with reinforcement learning on human-curated HotpotQA questions, represent the traditional fully supervised approach. Prior data-free self-evolution methods, adapted for search agents, are the most direct comparison. And frontier models (GPT-4 and Claude, where available) give a reference for state-of-the-art capability.

Results and Analysis

Main Results

The headline result: Dr. Zero matches or surpasses fully supervised search agents without using any training data.

The comparison set is broad: simple prompting, IRCoT (interleaved retrieval chain-of-thought), Search-o1, standard RAG, supervised fine-tuning (SFT), R1-Instruct, and Search-R1, the last being a strong supervised RL baseline.

On Qwen2.5-3B-Instruct, Dr. Zero reaches 0.326 average accuracy across all benchmarks, beats every baseline on the single-hop benchmarks (NQ 0.397, TriviaQA 0.572, PopQA 0.431), and is competitive with Search-R1 on multi-hop benchmarks with zero training data.

On Qwen2.5-7B-Instruct, it reaches 0.372 average accuracy, matching or exceeding Search-R1 (0.384), with the best 2WikiMQA score of any method (0.347) and solid multi-hop results on HotpotQA (0.362) and MuSiQue (0.104).

The pattern holds across the board: data-free self-evolution approaches or beats methods trained on human-curated data. That undercuts the assumption that complex reasoning tasks require human supervision.

Dr. Zero performance comparison across benchmarks with Qwen2.5 models

Ablation Studies

The ablations isolate what each component contributes.

Removing the search engine from proposer training, so questions come from parametric memory alone, costs 12-15% across benchmarks; the questions get less diverse and less factually grounded. Replacing the difficulty-guided reward with plain solver accuracy makes the proposer collapse to easy questions and costs 8-10%. At equal compute, HRPO beats standard GRPO by 4-6% accuracy, or equivalently reaches the same accuracy with 3-4x less compute. And performance improves steadily across iterations 1, 2 and 3, with iteration 4 showing diminishing returns, which suggests the system is nearing a ceiling at the 8B scale.

Each piece pulls its weight.

Scaling Properties

Does it scale? The authors look at three axes.

With compute: within an iteration, more training compute (more questions, more gradient updates) helps, with diminishing returns. The best split is roughly 70% of compute on the solver and 30% on the proposer.

With iterations: performance improves consistently across iterations 1-3, and the automated curriculum does its job. Average question hop count rises from 1.8 in iteration 1 to 3.2 in iteration 3.

With model size: preliminary runs at 70B keep improving, which is encouraging, but a proper scaling-law characterization is left for future work.

Implications: The Future of Self-Evolving AI

Beyond Search Agents

Dr. Zero is about search agents, but the principle is broader. Using an external tool as both the task environment and the supervision signal works wherever verifiable external feedback exists. A code interpreter can verify program correctness, which enables self-evolution for coding agents. Symbolic math engines can verify solutions, as prior work already showed. Simulators and databases can supply feedback for scientific reasoning agents. Physical simulators can supervise robot learning through task success and failure.

Wherever there is a reliable external oracle, data-free self-evolution becomes possible.

Limitations and Open Questions

Dr. Zero has real limitations. It depends on a search engine that returns accurate information; for tasks without reliable external verification, the approach doesn't directly apply. Improvement plateaus after several iterations, either because the model is hitting capability limits or because the self-generated curriculum isn't diverse enough. HRPO cuts compute relative to GRPO, but each iteration still needs thousands of search queries and many GPU-hours. And self-generated questions can contain errors or biases that propagate through training; the framework assumes the proposer's questions are good, and nothing guarantees it.

Open research questions include:

• Can self-evolution continue beyond current ceilings with architectural changes?
• How do we ensure self-generated training data is high quality and unbiased?
• Can multiple self-evolving agents collaborate to push each other further?
• What are the theoretical limits of data-free self-evolution?

The Bigger Picture

Dr. Zero is a real step toward AI systems that improve without human supervision, and that has consequences.

If self-evolution can match supervised learning, the economics of AI development change: teams no longer need a large annotation budget for every new capability. Development gets faster too, since self-evolution can run continuously instead of waiting on human data collection. It may also turn up reasoning strategies humans wouldn't think to demonstrate, because the model explores the space of strategies rather than imitating examples.

And it raises safety questions. If models can improve their own capabilities, keeping them aligned with human values becomes harder, not easier. Dr. Zero's search agents are narrow, but the general principle of self-improvement deserves careful thought.

References

1. Yue, Z., Upasani, K., Yang, X., Ge, S., Nie, S., Mao, Y., Liu, Z., & Wang, D. (2025). Dr. Zero: Self-Evolving Search Agents without Training Data. Meta Superintelligence Labs / UIUC. arXiv:2601.07055

2. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., ... & Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint. arXiv:2402.03300

3. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2025). Large Language Models Can Self-Improve. arXiv preprint. arXiv:2210.11610

4. Zelikman, E., Wu, Y., Mu, J., & Goodman, N. D. (2022). STaR: Bootstrapping Reasoning with Reasoning. NeurIPS. arXiv:2203.14465

5. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint. arXiv:1707.06347

6. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS. arXiv:2305.18290

7. Yang, Z., Qi, P., Zhang, S., Benber, Y., Choi, Y., & Fisch, A. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP. arXiv:1809.09600

8. Ho, X., Nguyen, A. K., Sugawara, S., & Aizawa, A. (2020). Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. COLING. arXiv:2011.01060

9. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022). Measuring and Narrowing the Compositionality Gap in Language Models. arXiv preprint. arXiv:2210.03350

10. Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). MuSiQue: Multihop Questions via Single-hop Question Composition. TACL. arXiv:2108.00573