Repository navigation
fix(azure): answer file-level questions with their diff - #5109
Open
tanishqbaweja wants to merge 1 commit into
Open
tanishqbaweja wants to merge 1 commit into
tanishqbaweja wants to merge 1 commit into
Conversation
Azure DevOps keeps a comment on a whole file in a thread whose context has the file's path and no line range. handle_line_comment finds no line range in it and passes /ask, /ask_question or a mention on as a question about the whole PR, so the model got the diff of every file and was not told which file the thread is on. Record the file, the thread, the comment and a link to the thread in the request context, as the GitHub App and the GitLab webhook do since The-PR-Agent#4887 and The-PR-Agent#5062. PRQuestions then answers from that file's diff, in the thread, with the thread's earlier comments as history. Questions in a thread on the PR or on lines are routed as before.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
⚙️ Agent run details
|
PR Code Suggestions ✨No code suggestions found for the PR. ⚙️ Agent run details
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
#4887 and #5062 made the GitHub App and the GitLab webhook answer
/askon a whole file from that file's diff, in the file comment's thread, as the GitHub Action does since #4656. Both descriptions name the Azure DevOps webhook as still answering such a question about the whole PR. This is that change.On Azure DevOps, reviewers start a thread on a whole file with Add comment in the file's options menu (Microsoft Learn, Review and comment on pull requests, "Provide feedback in comments"). Such a thread, on a whole file rather than on lines, has a
threadContextwith the file'sfilePathand none of theleftFileStart,leftFileEnd,rightFileStartandrightFileEndpositions. I took that shape from Azure DevOps' REST reference, where aCommentThreadContextis afilePathand the positions of the first and last character of the thread's span in the left and right file, and from this repo's code:test_addressed_comment_without_line_range_becomes_threaded_askfeedshandle_line_commentsuch a context, andAzureDevopsProvider.publish_inline_commentsbuilds one with onlyfilePathfor a comment that is not on a line (azuredevops_provider.py:1435), although no command calls that path onmain. For/ask,/ask_questionor a comment that mentions PR-Agent in such a thread,handle_line_commentreads the thread's context (azuredevops_server_webhook.py:152on fe67e4b), finds no line range, logs "No line range found in thread context" and returns the question it builds for a thread on the PR,/ask --comment_id=<thread> --origin_comment_id=<comment> <question>(azuredevops_server_webhook.py:169-171). The file's path is dropped there.PRQuestionsthen has no file question in the request context, so_get_question_diffpacks the diff of every file (pr_questions.py:68) and nothing in the prompt says which file the thread is on. The answer is still posted in the thread (_publish_answerreplies to--comment_id,pr_questions.py:220), with the thread's earlier comments as history, so it reads like an answer about the file while the model was asked about the whole PR. On a PR whose diff does not fit the model's input budget,get_pr_diffprunes it to the patches that fit and at most the names of the other files (pr_processing.py:302), so the file the thread is on can be missing from the prompt altogether.Before / after
azuredevops_server_webhook.handle_request_azure()with a "Pull request commented on" event (resource version 2.0) for/askin a thread on a whole file, as a reply to another reviewer's comment in that thread; the realAzureDevopsProviderwith its Azure DevOps client answered locally, and the model's answer canned (script below):repro.py ask_questionandrepro.py mention(the question addressed to PR-Agent's Azure DevOps identity instead of a slash command) print the same asrepro.py askon both.repro.py
Fix
handle_line_comment, a question in a thread whose context has a file path but no line range now records the file, the thread, the comment and a link to the thread (?discussionId=<thread>, the linkAzureDevopsProvider.get_comment_urlbuilds) in the request context: the keys the GitHub App and the GitLab webhook fill since fix(github-app): answer file-level questions with their diff #4887 and fix(gitlab): answer file-level questions with their diff #5062. The path is the context'sfilePath, the value the webhook already passes to/ask_linefor a thread on lines. The request itself is unchanged:/ask --comment_id=<thread> --origin_comment_id=<comment>, any leading options of a slash/ask, and the question.PRQuestionsthen gives the question fix(github-action): answer file-level questions with their diff #4656's handling, with no change to it or to the provider: that file's complete diff within the file filters and input budget, a note instead of a model call when the file has no analyzable diff or does not fit, the thread's earlier comments withpr_questions.use_conversation_history(file questions read them whensupports_line_question_history()is true, as it is for Azure DevOps), and the answer in the thread with a link to the question.ask.mddescribes the Azure DevOps webhook's handling of these threads after the paragraph on the GitHub App, GitHub Actions and GitLab webhook.Tests
test_azure_devops_webhook_comments.py, throughhandle_request_azurewith a "Pull request commented on" event, recording the request context when the request reachesPRAgent.handle_request:/ask,/ask_questionand a mention of the agent in a thread whose context has a file path and no line range reach/ask --comment_id=22 --origin_comment_id=31with the question, and the file, thread, comment and thread link in the request context. A question in a thread without a context, in a thread whose context has a missing or blank file path, and on lines (/ask_line) records no file question. Onmainthe three file cases fail with nothing in the context; the four others pass on both.test_addressed_comment_without_line_range_becomes_threaded_ask, which asserted the old routing by callinghandle_line_commentoutside a request context, where recording the file question raisesContextDoesNotExistError; they assert the same request together with the file question.python -m pytest tests/unittest/test_azure*.py tests/unittest/test_pr_questions_*.py tests/unittest/test_run_details_ask_commands.py tests/unittest/test_comment_command_gating.py tests/unittest/test_disable_auto_feedback_servers.py tests/unittest/test_webhook_body_size_limit.py tests/unittest/test_request_policy.py(the Azure DevOps, question and webhook tests, 19 files): 822 passed onmain; with the new tests onmain's webhook, 3 failed (the three file cases) and 825 passed; 828 passed with the change.python -m pytest tests/unittest, run before rebasing from db4c4ad onto fe67e4b (the commits between them change none of this PR's files, norpr_questions.pyorpr_processing.py): 18619 passed, 81 skipped and 83 failed or errored onmain; 18628 passed, 81 skipped and 80 failed or errored with the change. By test id, 79 of them are the same on both, none in the 19 files above, and they come from my environment (Windows, Python 3.12,PYTHONUTF8=1: symlinks and hard links,os.getuidin the Prometheus multiprocess setup, the release workflow'sbashsteps, CRLF line endings from git, file names Windows cannot create, git settings from my shell, home-directory lookups, POSIX file modes, a subprocess started withoutSystemRoot, LiteLLM 1.109.1 in my virtualenv wherepyproject.tomlpins 1.110.1, and a cleanup-deadline timing check intest_litellm_callback_drain.py). The other five differ between the two runs, not between the trees, and none of them imports the webhook; they are timing-dependent:test_cli_exit_status.py::test_process_entrypoints_map_request_status[false-false-1]and[false-true-1],test_cli_exit_status.py::test_process_entrypoints_preserve_argparse_statusandtest_github_common.py::test_import_action_runner_without_github_apphit their 30- and 60-second subprocess timeouts during themainrun on a busy machine and passed in reruns (one of them also timed out once in a rerun with the change);test_litellm_callback_drain.py::test_batch_cancellation_uses_one_cleanup_deadline_for_wrapper_and_uploadfailed only in the run with the change, and in three reruns on each tree it failed once and passed twice on both.ruff check(0.17.5) on the changed files: clean.Not tested against an Azure DevOps organization: the provider's client calls and the model's answer are faked in the script and the tests.
Sibling servers
Not changed
--key=valueoptions of a slash/askor/ask_questionin such a thread still apply as settings, as they do in a thread on the PR or on lines (_split_leading_cli_args); in a GitHub or GitLab file question, text like that is part of the question. A mention is still never parsed for options./ask_linetyped in a thread on a whole file is passed on as written: Azure DevOps sends no line for the thread, so there are no lines to ask about.ask.mdsays.PR Type
Bug fix, Tests, Documentation
Description
Scope Azure DevOps file questions to file diffs.
Preserve thread replies, links, and conversation history.
Test question aliases and unchanged fallback routing.
Document Azure DevOps whole-file question support.
Diagram Walkthrough
File Walkthrough
azuredevops_server_webhook.py
Route Azure whole-file questions through file contextpr_agent/servers/azuredevops_server_webhook.py
discussion URL in request context.
command routing.
test_azure_devops_webhook_comments.py
Cover whole-file question dispatch and fallback routingtests/unittest/test_azure_devops_webhook_comments.py
request-scoped metadata.
/ask,/ask_question, and agentmentions.
retain existing routing without file-question metadata.
ask.md
Document Azure DevOps whole-file question behaviordocs/docs/commands/ask.md
mentions.
conversation history.
⚙️ Agent run details