GitHub agent UI and cloud usage

[raineorshine]

This thread will be used by em admin to discuss our overall agent strategy on the project, including preferred models, local coding assistant workflows, custom Copilot agents, and CI integration.

Converted thread to GitHub agent UI discussion. Create new threads for other topics.

[raineorshine]

Looks like creating a custom agent on GitHub is pretty straightforward:

Then specific agents can be assigned to issues:

[raineorshine]

Quick note: UX to assign an issue to a custom agent is slightly different between the GitHub iOS app and Mobile Safari.

iOS:

Mobile Safari:

[raineorshine]

You can also choose the model to use when @ing Copilot in a comment:

[fbmcipher]

Nice! I know that a lot of people think the latest version of GPT-5.4 is superior to Claude Opus 4.6. I personally find Claude is still the better all-round model for building apps, but I felt GPT-5.4 had a superior level of technical correctness which might serve us well, especially for our code review agent where that is key.

That said, with how fast things are moving in the industry, it's all kind of neck-and-neck and the best model for a given task is always changing.

When building agents (like our code review agent) we could test run and benchmark across multiple models to make sure we get the best results and cost efficiency, even as new models drop.

[raineorshine]

The setting to auto-approve Cl runs from Copilot PRs just landed last week, which is a small change but very helpful when queuing up multiple issue-solving agents in parallel.

https://github.blog/changelog/2026-03-13-optionally-skip-approval-for-copilot-coding-agent-actions-workflows/?utm_source=chatgpt.com

[fbmcipher]

Great to see! Skipping approval on AI edits can be helpful – I personally let my local Claude Code instances write code without approval.

Here, though, I think we'd want to make sure that any agents that have the ability to write without approval are very confident about their changes. Otherwise, they could quickly end up creating more problems than they solve.

[raineorshine]

This setting is just related to approval of CI runs. It doesn't affect write access or PR approval.

[raineorshine]

@fbmcipher Something we didn't discuss was analyzing Copilot/Claude sessions, but I think this will be critical for us to improve agent performance.

1. See where the agent goes off the rails.
2. Identify what context was missing.
3. Provide the context in copilot-instructions.md.
4. Observe and repeat.

I will assign you Copilot PRs that run into blocks so that you can investigate. Hopefully this effort will have big payoffs as we improve our agent usage across the project.

[fbmcipher]

Thank you! I see the issue, but I can't actually see the "View Session" button:

Screenshot

I am guessing that I need access to a GitHub Copilot seat in the org to view the session.

[raineorshine]

Any luck now?

[fbmcipher]

Can confirm I can access the session now – thank you! I will investigate and update you in a new GitHub Discussion post or directly in the issues you assign me to – whichever feels easier/more efficient.

[raineorshine]

Thanks!

Maybe discuss specifics directly on the Copilot-created PR, and summarize findings and discuss generalize principles in a Discussion.

[fbmcipher]

Experiment

• We have 9 years of PR review data.
• Let's see if it's possible to use that data to produce a rulebook codifying em's engineering principles.
• Then, see if it's viable for an AI agent to use that rulebook, along with an incoming PR diff, to produce a first-pass code review that surfaces these kinds of mechanical/repetitive issues.

Approach

Here's the general process I followed:

1. Scrape: Pull all 4,074 comments you've left on em PRs via the GitHub CLI.
2. Seed: Take a randomized sample of ~200 comments, along with the code hunk from the PR they're associated with. For each, extract the underlying coding principle being enforced (e.g. HYGIENE-001: remove debug logs before merging)
   • Prefix each rule with an ID and a domain-specific prefixes (like REDUX-, PANDACSS-, TEST-, REACT-, ARCH-, etc.)
   • This is useful for storage, and allows humans and AIs to quickly refer to a specific rule.
3. Classify: After a human review phase to ensure the initial results from the seed phase look good, run all 4,074 comments against the codebook. For each comment, classify it as: matching an existing rule, proposing a new rule, too specific to generalize, or not a rule (i.e. excerpts of conversation). New rules discovered during classification get added to the codebook.
4. Consolidate: Deduplication phase. Ensure that each rule in the codebook is a single, atomic, distinct rule.
5. Verify: In a separate thread, run the codebook against the full corpus of 4,074 comments to work out how many of them are represented in the final codebook.

From that JSON, we can deterministically produce a .md file to pass as custom instructions to an agent, along with a simple prompt:

You are a code review agent for a project called em. Your task is to catch mechanical, rule-based issues so the lead reviewer can focus on higher-level judgment call feedback.

Read the attached codebook, which contains coding derived from 9 years of review history on this project. Each rule has an ID (e.g. HYGIENE-001), a label, and a description of the violation pattern.

Then, review the attached PR diff. For each violation you find:
1. Cite the rule ID and label.
2. Quote the specific line(s) in the diff.
3. Suggest a fix – a code snippet if straightforward, or a high-level description if the change is more involved.

**IMPORTANT:** Only flag violations of rules explicitly listed in the codebook. Do not invent rules or identify issues that aren't listed in the codebook.

Result

• This process yielded a codebook with 210 distinct rules.
• Of the 4074 comments, 67.4% match to rules in the codebook.
  • 25.7% of the comments were conversational – praise, questions, design discussions, replies or mentoring.
  • The remaining 6.8% of the comments were one-off corrections that the AI couldn't generalize into a rule.
  • This indicates a high level of repetition within the corpus and validates the initial premise. In other words, nearly 7 out of 10 review comments are repetitive or codifiable enough to be covered by the codebook.
• The process identified these high-level categories for rules:
  • Architecture
  • Testing
  • App-specific (em domain rules)
  • Code style
  • Performance
  • React
  • Code strictness
  • Redux
  • TypeScript
  • Documentation
  • PandaCSS
  • Hygiene
  • Naming conventions

The next step is to validate the codebook against some PRs. Initial testing against my PR #3936 flagged 9 legitimate issues, 4 of which you hadn't actually caught – good start! – but I want to do some more thorough testing over the following week.

For the most accurate testing, we'd probably want to run against the next PR that comes through – something that's completely unrepresented in our dataset.

I've attached a few files to this comment in case you want to review:

• codebook-v2-final.json: raw JSON rules
• CODEBOOK.md: a Markdown version of the codebook we can pass to an AI agent as custom instructions
• initial-result.md: an initial code review produced by Claude Code given the prompt and codebook against my PR [Liminal UI] Sidebar header with animations #3936

[raineorshine]

Thank you! I created a new discussion here: #4012.

Thanks for your patience as I figure out the best use of GitHub Discussions. Adding multiple disparate ideas to the same thread didn't work so well, as there is no way to collapse or resolve them, and it's hard to see new replies unless I directly follow a notification link.

Hiding is manual and does not affect replies, so it's not a good replacement of an archive feature.

[raineorshine]

Thank you for pointing this out and sorry for the delay!

I will start by inviting you to the org: https://github.com/cybersemics

[fbmcipher]

Thank you, and no worries at all!

While accepting the invite, I had the option to ask for a Copilot seat, so I ticked it.

[raineorshine]

Enabled!