test: end-to-end test against a personal HTCondor pool

[kratsg]

Adds an end-to-end test that runs af-credmon inside a real HTCondor pool, as suggested by @bbockelm when looking at the README.

What it does

tests/test_e2e_htcondor.py starts condor_master from a private CONDOR_CONFIG (free collector port, no shared port) with collector, negotiator, schedd, startd, credd and af-credmon as CREDMON_OAUTH, then:

1. Stores af_servicex and af_krb5 top tokens with condor_store_cred add-oauth -u <user>@<domain> as a separate account listed in CRED_SUPER_USERS, the way the broker stores them on users' behalf.
2. Checks af-credmon redeems both at a local broker stand-in and writes root-owned, 0400 .use files.
3. Submits a job with use_oauth_services = af_servicex, af_krb5 and checks its stdout shows both credentials in $_CONDOR_CREDS.

Only the broker is a stand-in (a local HTTP server serving POST /v1/credentials/<kind>/redeem); everything HTCondor does is real.

The questions it answers

• How credd finds the credmon: credd SIGHUPs whatever pid is in <SEC_CREDENTIAL_DIRECTORY_OAUTH>/pid. The test runs af-credmon with --interval 3600, so the redeems (which land about 0.1s after credd writes each .top) can only come from that SIGHUP. It also checks the pid in that file is af-credmon.
• Which credmon variant: nothing is declared to credd. The OAuth-family credmons share the OAuth directory and are told apart by which files they claim (af-credmon only handles af_*). The test doesn't cover running alongside another credmon.

Running it

It needs an installed HTCondor and root (credd and the shadow check that .use files are root-owned), and submits as the user who invoked sudo:

sudo env "PATH=$(pixi run -e py313 printenv PATH)" pytest tests/test_e2e_htcondor.py

It skips without condor_master, root or SUDO_USER, so the existing matrix (and macOS, where conda-forge's htcondor is client-only) is unaffected. A new e2e CI job on ubuntu-latest installs HTCondor via get.htcondor.org, runs it, and fails unless the test actually passed.

On Ubuntu 24.04 with HTCondor 25.14.1 it takes 8–13s including the ~3s pool startup (11.7s in CI).

Things found along the way

The first, second and fourth are now documented in the README's HTCondor configuration section.

• The OAuth credential directory needs root:condor mode 2770; with 0700 root credd fails with Error 13, attempting to create OAuth cred subdir.
• condor_store_cred -u requires user@domain.
• HTCondor tools run as root authenticate as condor, so root can't stand in for a CRED_SUPER_USERS identity.
• condor_master needs the condor user to reach the credmon executable: with the pixi env under a 0750 home directory it fails with Cannot execute (errno=13, Permission denied) and retries forever. A world-readable install location like /opt/af-credmon (as the README suggests) is fine; the CI job does chmod o+x $HOME.

🤖 Generated with Claude Code

[kratsg]

@bbockelm this is the end-to-end test you suggested: a personal HTCondor pool with credd and af-credmon as CREDMON_OAUTH, a job with use_oauth_services, and the credentials showing up in $_CONDOR_CREDS. It runs in CI in about 12s. Would you be able to take a look, especially at whether the HTCondor configuration in tests/test_e2e_htcondor.py matches how you'd expect an access point to be set up?