Skip to content

fix(azure): answer file-level questions with their diff - #5109

Open
tanishqbaweja wants to merge 1 commit into
The-PR-Agent:mainfrom
tanishqbaweja:fix/azure-devops-file-questions
Open

tanishqbaweja wants to merge 1 commit into
The-PR-Agent:mainfrom
tanishqbaweja:fix/azure-devops-file-questions

Conversation

@tanishqbaweja

@tanishqbaweja tanishqbaweja commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

User description

#4887 and #5062 made the GitHub App and the GitLab webhook answer /ask on a whole file from that file's diff, in the file comment's thread, as the GitHub Action does since #4656. Both descriptions name the Azure DevOps webhook as still answering such a question about the whole PR. This is that change.

On Azure DevOps, reviewers start a thread on a whole file with Add comment in the file's options menu (Microsoft Learn, Review and comment on pull requests, "Provide feedback in comments"). Such a thread, on a whole file rather than on lines, has a threadContext with the file's filePath and none of the leftFileStart, leftFileEnd, rightFileStart and rightFileEnd positions. I took that shape from Azure DevOps' REST reference, where a CommentThreadContext is a filePath and the positions of the first and last character of the thread's span in the left and right file, and from this repo's code: test_addressed_comment_without_line_range_becomes_threaded_ask feeds handle_line_comment such a context, and AzureDevopsProvider.publish_inline_comments builds one with only filePath for a comment that is not on a line (azuredevops_provider.py:1435), although no command calls that path on main. For /ask, /ask_question or a comment that mentions PR-Agent in such a thread, handle_line_comment reads the thread's context (azuredevops_server_webhook.py:152 on fe67e4b), finds no line range, logs "No line range found in thread context" and returns the question it builds for a thread on the PR, /ask --comment_id=<thread> --origin_comment_id=<comment> <question> (azuredevops_server_webhook.py:169-171). The file's path is dropped there. PRQuestions then has no file question in the request context, so _get_question_diff packs the diff of every file (pr_questions.py:68) and nothing in the prompt says which file the thread is on. The answer is still posted in the thread (_publish_answer replies to --comment_id, pr_questions.py:220), with the thread's earlier comments as history, so it reads like an answer about the file while the model was asked about the whole PR. On a PR whose diff does not fit the model's input budget, get_pr_diff prunes it to the patches that fit and at most the names of the other files (pr_processing.py:302), so the file the thread is on can be missing from the prompt altogether.

Before / after

azuredevops_server_webhook.handle_request_azure() with a "Pull request commented on" event (resource version 2.0) for /ask in a thread on a whole file, as a reply to another reviewer's comment in that thread; the real AzureDevopsProvider with its Azure DevOps client answered locally, and the model's answer canned (script below):

# main @ fe67e4b09
files in the model's prompt: ['/src/billing/rounding.py', '/src/billing/invoice.py', '/docs/billing.md']
earlier comment in the model's prompt: True
Azure DevOps calls:
  GET pullRequests/7/threads/5
  POST pullRequests/7/threads/5/comments: 'On it! ⏳\n\n<!-- pr-agent-response -->\n<!-- pr-agent-progress -->'...
  GET pullRequests/7/threads/5
  POST pullRequests/7/threads: 'Preparing answer...'...
  POST pullRequests/7/threads/5/comments: '### **Ask** ❓\nWhy does this file round before converting?\n\n### **Answer:**\nRounding first keeps the converted '...
  DELETE pullRequests/7/threads/9/comments/1
  DELETE pullRequests/7/threads/5/comments/3

# this branch
files in the model's prompt: ['/src/billing/rounding.py']
earlier comment in the model's prompt: True
Azure DevOps calls:
  GET pullRequests/7/threads/5
  POST pullRequests/7/threads/5/comments: 'On it! ⏳\n\n<!-- pr-agent-response -->\n<!-- pr-agent-progress -->'...
  GET pullRequests/7/threads/5
  POST pullRequests/7/threads: 'Preparing answer...'...
  POST pullRequests/7/threads/5/comments: '### **Ask** ❓\n[View question](https://dev.azure.com/acme/billing/_git/billing/pullrequest/7?discussionId=5)\n\n#'...
  DELETE pullRequests/7/threads/9/comments/1
  DELETE pullRequests/7/threads/5/comments/3

repro.py ask_question and repro.py mention (the question addressed to PR-Agent's Azure DevOps identity instead of a slash command) print the same as repro.py ask on both.

repro.py
# LITELLM_LOCAL_MODEL_COST_MAP=True PYTHONPATH=. uv run python repro.py [ask|ask_question|mention]
# (LiteLLM's bundled model map, as tests/unittest/conftest.py uses, so nothing is fetched.)
# A question in an Azure DevOps thread on a whole file, delivered to the Azure DevOps webhook's event handler.
# The provider is the real AzureDevopsProvider with its Azure DevOps client answered locally, and the model's
# answer is canned.
import asyncio
import copy
import sys
from types import SimpleNamespace
from unittest.mock import patch

from azure.devops.released.git import (
    Comment,
    CommentThreadContext,
    GitPullRequest,
    GitPullRequestCommentThread,
    IdentityRef,
)
from starlette_context import context, request_cycle_context

from pr_agent.agent import pr_agent as pr_agent_module
from pr_agent.agent.pr_agent import PRAgent
from pr_agent.config_loader import global_settings
from pr_agent.git_providers.azuredevops_provider import AzureDevopsProvider
from pr_agent.log import get_logger
from pr_agent.servers import azuredevops_server_webhook as webhook

get_logger().remove()  # print only the summary below

kind = sys.argv[1] if len(sys.argv) > 1 else "ask"
QUESTION = "Why does this file round before converting?"
body = f"@<pr-agent-guid> {QUESTION}" if kind == "mention" else f"/{kind} {QUESTION}"
REPO_URL = "https://dev.azure.com/acme/billing/_git/billing"
PR_API = "pullRequests/7"
THREAD_ID = 5
# The comment as the "Pull request commented on" service hook (resource version 2.0) sends it.
payload = {
    "eventType": "ms.vss-code.git-pullrequest-comment-event",
    "resourceVersion": "2.0",
    "resource": {
        "comment": {
            "id": 2, "parentCommentId": 1, "content": body, "commentType": "text",
            "_links": {"threads": {"href": f"https://dev.azure.com/acme/_apis/git/repositories/billing/"
                                           f"{PR_API}/threads/{THREAD_ID}"}},
        },
        "pullRequest": {"pullRequestId": 7, "repository": {"webUrl": REPO_URL}},
    },
}
FILES = {  # path: (base content, head content)
    "/src/billing/rounding.py": ("def to_cents(amount, rate):\n    return int(amount * rate * 100)\n",
                                 "def to_cents(amount, rate):\n    rounded = round(amount, 2)\n"
                                 "    return int(rounded * rate * 100)\n"),
    "/src/billing/invoice.py": ("class Invoice:\n    def close(self, lines):\n        self.total = sum(lines)\n",
                                "class Invoice:\n    def close(self, lines):\n"
                                "        self.total = sum(line.amount for line in lines)\n"),
    "/docs/billing.md": ("# Billing\n", "# Billing\nAmounts are rounded to cents before conversion.\n"),
}
# The thread the question was posted in: a thread on the whole file, so its context has the file's path and no
# line range. Another reviewer started it; the question is a reply.
THREAD = GitPullRequestCommentThread(
    id=THREAD_ID,
    thread_context=CommentThreadContext(file_path="/src/billing/rounding.py"),
    comments=[
        Comment(id=1, content="Rounding to cents here drops the fraction we bill for metered plans.",
                author=IdentityRef(display_name="Alice")),
        Comment(id=2, content=body, author=IdentityRef(display_name="Reviewer")),
    ],
)
calls = []
comment_ids = iter(range(3, 100))

class AzureDevOpsClient:
    """The GitClient calls the run makes, answered locally."""

    def get_pull_request_by_id(self, pull_request_id, project=None):
        return GitPullRequest(
            pull_request_id=7, title="Round amounts before converting currencies", description="",
            source_ref_name="refs/heads/rounding", target_ref_name="refs/heads/main",
            last_merge_commit=SimpleNamespace(commit_id="head"),
            last_merge_target_commit=SimpleNamespace(commit_id="base"),
        )

    def get_pull_request_thread(self, repository_id, pull_request_id, thread_id, project=None):
        calls.append(f"GET {PR_API}/threads/{thread_id}")
        return THREAD

    def get_pull_request_iterations(self, repository_id, pull_request_id, project=None):
        return [SimpleNamespace(id=1)]

    def get_pull_request_iteration_changes(self, repository_id, pull_request_id, iteration_id, project=None,
                                           top=None, skip=None, compare_to=None):
        return SimpleNamespace(change_entries=[{"item": {"path": path}, "changeType": "edit"} for path in FILES],
                               next_skip=0, next_top=0)

    def get_item(self, repository_id, path, project=None, version_descriptor=None, download=None,
                 include_content=None):
        base, head = FILES[path]
        return SimpleNamespace(content=head if version_descriptor.version == "head" else base)

    def get_items(self, **_kwargs):
        return [SimpleNamespace(path=path, git_object_type="blob") for path in FILES]

    def create_comment(self, comment, repository_id, pull_request_id, thread_id, project=None):
        calls.append(f"POST {PR_API}/threads/{thread_id}/comments: {comment.content[:110]!r}...")
        return Comment(id=next(comment_ids), content=comment.content)

    def create_thread(self, comment_thread, project=None, repository_id=None, pull_request_id=None):
        calls.append(f"POST {PR_API}/threads: {comment_thread.comments[0].content[:110]!r}...")
        return GitPullRequestCommentThread(id=9, comments=[Comment(id=1)])

    def delete_comment(self, repository_id, pull_request_id, thread_id, comment_id, project=None):
        calls.append(f"DELETE {PR_API}/threads/{thread_id}/comments/{comment_id}")

class Model:
    prompts = []

    async def chat_completion(self, model, system, user, temperature=0.2, img_path=None):
        self.prompts.append(user)
        return "Rounding first keeps the converted amount from carrying sub-cent noise.", "stop"

async def main():
    with request_cycle_context({}):
        # As the webhook route prepares each delivery.
        context["settings"] = copy.deepcopy(global_settings)
        context["settings"].set("config.git_provider", "azure")
        context["settings"].set("azure_devops.org", "https://dev.azure.com/acme")
        context["settings"].set("azure_devops_server.agent_identity", "pr-agent-guid")
        context["settings"].set("config.repo_context_files", [])  # no AGENTS.md read
        with (patch.object(AzureDevopsProvider, "_get_azure_devops_client",
                           lambda self: (AzureDevOpsClient(), None)),
              patch.object(webhook, "PRAgent", lambda: PRAgent(ai_handler=Model)),
              patch.object(pr_agent_module, "apply_repo_settings", lambda _url: None),
              patch.object(pr_agent_module, "enforce_request_policy", lambda _url: True)):
            await webhook.handle_request_azure(payload, {})

asyncio.run(main())
print("files in the model's prompt:",
      [path for path in FILES for prompt in Model.prompts if f"## File: '{path}'" in prompt])
print("earlier comment in the model's prompt:",
      any("Alice: Rounding to cents" in prompt for prompt in Model.prompts))
print("Azure DevOps calls:")
print("\n".join(f"  {call}" for call in calls))

Fix

  • In handle_line_comment, a question in a thread whose context has a file path but no line range now records the file, the thread, the comment and a link to the thread (?discussionId=<thread>, the link AzureDevopsProvider.get_comment_url builds) in the request context: the keys the GitHub App and the GitLab webhook fill since fix(github-app): answer file-level questions with their diff #4887 and fix(gitlab): answer file-level questions with their diff #5062. The path is the context's filePath, the value the webhook already passes to /ask_line for a thread on lines. The request itself is unchanged: /ask --comment_id=<thread> --origin_comment_id=<comment>, any leading options of a slash /ask, and the question.
  • PRQuestions then gives the question fix(github-action): answer file-level questions with their diff #4656's handling, with no change to it or to the provider: that file's complete diff within the file filters and input budget, a note instead of a model call when the file has no analyzable diff or does not fit, the thread's earlier comments with pr_questions.use_conversation_history (file questions read them when supports_line_question_history() is true, as it is for Azure DevOps), and the answer in the thread with a link to the question.
  • A question in a thread on the PR, in a thread whose context has no file path, or on lines is routed as before.
  • ask.md describes the Azure DevOps webhook's handling of these threads after the paragraph on the GitHub App, GitHub Actions and GitLab webhook.

Tests

  • test_azure_devops_webhook_comments.py, through handle_request_azure with a "Pull request commented on" event, recording the request context when the request reaches PRAgent.handle_request: /ask, /ask_question and a mention of the agent in a thread whose context has a file path and no line range reach /ask --comment_id=22 --origin_comment_id=31 with the question, and the file, thread, comment and thread link in the request context. A question in a thread without a context, in a thread whose context has a missing or blank file path, and on lines (/ask_line) records no file question. On main the three file cases fail with nothing in the context; the four others pass on both.
  • The file cases replace test_addressed_comment_without_line_range_becomes_threaded_ask, which asserted the old routing by calling handle_line_comment outside a request context, where recording the file question raises ContextDoesNotExistError; they assert the same request together with the file question.
  • python -m pytest tests/unittest/test_azure*.py tests/unittest/test_pr_questions_*.py tests/unittest/test_run_details_ask_commands.py tests/unittest/test_comment_command_gating.py tests/unittest/test_disable_auto_feedback_servers.py tests/unittest/test_webhook_body_size_limit.py tests/unittest/test_request_policy.py (the Azure DevOps, question and webhook tests, 19 files): 822 passed on main; with the new tests on main's webhook, 3 failed (the three file cases) and 825 passed; 828 passed with the change.
  • python -m pytest tests/unittest, run before rebasing from db4c4ad onto fe67e4b (the commits between them change none of this PR's files, nor pr_questions.py or pr_processing.py): 18619 passed, 81 skipped and 83 failed or errored on main; 18628 passed, 81 skipped and 80 failed or errored with the change. By test id, 79 of them are the same on both, none in the 19 files above, and they come from my environment (Windows, Python 3.12, PYTHONUTF8=1: symlinks and hard links, os.getuid in the Prometheus multiprocess setup, the release workflow's bash steps, CRLF line endings from git, file names Windows cannot create, git settings from my shell, home-directory lookups, POSIX file modes, a subprocess started without SystemRoot, LiteLLM 1.109.1 in my virtualenv where pyproject.toml pins 1.110.1, and a cleanup-deadline timing check in test_litellm_callback_drain.py). The other five differ between the two runs, not between the trees, and none of them imports the webhook; they are timing-dependent: test_cli_exit_status.py::test_process_entrypoints_map_request_status[false-false-1] and [false-true-1], test_cli_exit_status.py::test_process_entrypoints_preserve_argparse_status and test_github_common.py::test_import_action_runner_without_github_app hit their 30- and 60-second subprocess timeouts during the main run on a busy machine and passed in reruns (one of them also timed out once in a rerun with the change); test_litellm_callback_drain.py::test_batch_cancellation_uses_one_cleanup_deadline_for_wrapper_and_upload failed only in the run with the change, and in three reruns on each tree it failed once and passed twice on both.
  • ruff check (0.17.5) on the changed files: clean.

Not tested against an Azure DevOps organization: the provider's client calls and the model's answer are faked in the script and the tests.

Sibling servers

Not changed

  • Leading --key=value options of a slash /ask or /ask_question in such a thread still apply as settings, as they do in a thread on the PR or on lines (_split_leading_cli_args); in a GitHub or GitLab file question, text like that is part of the question. A mention is still never parsed for options.
  • /ask_line typed in a thread on a whole file is passed on as written: Azure DevOps sends no line for the thread, so there are no lines to ask about.
  • Command names stay case-sensitive on Azure DevOps, as ask.md says.
  • The temporary "Preparing answer..." comment is still posted in a new thread and removed, as for every question.

PR Type

Bug fix, Tests, Documentation


Description

  • Scope Azure DevOps file questions to file diffs.

  • Preserve thread replies, links, and conversation history.

  • Test question aliases and unchanged fallback routing.

  • Document Azure DevOps whole-file question support.


Diagram Walkthrough

flowchart LR
  A["Azure DevOps comment question"]
  B["Whole-file thread context"]
  C["Request-scoped file and thread metadata"]
  D["PRQuestions: file diff and threaded answer"]
  A --> B
  B --> C
  C --> D
Loading

File Walkthrough

Relevant files
Bug fix
azuredevops_server_webhook.py
Route Azure whole-file questions through file context       

pr_agent/servers/azuredevops_server_webhook.py

  • Detect whole-file threads with a nonblank file path and no line range.
  • Record the file path, thread ID, originating comment ID, and
    discussion URL in request context.
  • Enable existing file-question handling while preserving threaded
    command routing.
+7/-0     
Tests
test_azure_devops_webhook_comments.py
Cover whole-file question dispatch and fallback routing   

tests/unittest/test_azure_devops_webhook_comments.py

  • Add a webhook-event helper that captures dispatched commands and
    request-scoped metadata.
  • Cover whole-file questions using /ask, /ask_question, and agent
    mentions.
  • Verify PR-level, missing-path, blank-path, and line-level questions
    retain existing routing without file-question metadata.
+87/-13 
Documentation
ask.md
Document Azure DevOps whole-file question behavior             

docs/docs/commands/ask.md

  • Document Azure DevOps whole-file questions through commands and agent
    mentions.
  • Explain file-diff scope, threaded answers, question links, and
    conversation history.
+4/-0     


⚙️ Agent run details
  • Model: openai/gpt-6.1-sol
  • Tokens: 6,882 in / 435 out / 7,317 total
  • Time cost: 8.6s
  • AI calls: 1

Azure DevOps keeps a comment on a whole file in a thread whose context
has the file's path and no line range. handle_line_comment finds no
line range in it and passes /ask, /ask_question or a mention on as a
question about the whole PR, so the model got the diff of every file
and was not told which file the thread is on.

Record the file, the thread, the comment and a link to the thread in
the request context, as the GitHub App and the GitLab webhook do since
The-PR-Agent#4887 and The-PR-Agent#5062. PRQuestions then answers from that file's diff, in
the thread, with the thread's earlier comments as history. Questions in
a thread on the PR or on lines are routed as before.
@the-pr-agent-bot

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
⚠️ Risk level: Low
✅ Merge recommendation: No concerns found
🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected

⚙️ Agent run details
  • Model: openai/gpt-6.1-sol
  • Tokens: 18,672 in / 122 out / 18,794 total
  • Time cost: 4.2s
  • AI calls: 1

@the-pr-agent-bot

Copy link
Copy Markdown

PR Code Suggestions ✨

No code suggestions found for the PR.


⚙️ Agent run details
  • Model: openai/gpt-6.1-sol
  • Tokens: 6,696 in / 90 out / 6,786 total
  • Time cost: 4.7s
  • AI calls: 1

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant