feat(s3): tag S3 requests with a luigi user agent, document endpoint_url

[goanpeca]

Description

Two small, independent changes to the S3 contrib module and its documentation.

1. Identify luigi in the S3 user agent (luigi/contrib/s3.py, test/contrib/s3_test.py)

S3Client.s3 now adds luigi/<version> to botocore's user_agent_extra before building the boto3 resource. If the caller already supplies a botocore.config.Config (via [s3] options or kwargs), its user_agent_extra is preserved and the luigi token is appended through Config.merge(), so no other client setting is overwritten. If there is no config, a fresh Config(user_agent_extra="luigi/<version>") is created. botocore.config is imported locally in the property, matching the existing lazy import boto3 a few lines above.

2. Document the [s3] config section (doc/configuration.rst)

Adds an [s3] section (alphabetically, before [scalding]) that notes keys in the section are passed to the underlying boto3 S3 client, and documents endpoint_url and region_name with an example. region_name is worth documenting next to endpoint_url: with a custom endpoint and no region configured, botocore falls back to us-east-1 and then signs requests for the wrong region against a bucket that lives elsewhere. No behavior change.

Motivation and Context

user_agent_extra is boto3's supported extension point for this, so signing and request behavior are untouched: the only difference is one extra token in the User-Agent header. Having it there makes luigi traffic identifiable in object store access logs, which helps when tracing throttling, retries, or unexpected request patterns back to the tool that generated them. Several other data tools tag their S3 clients the same way.

The [s3] section is already read by S3Client._get_s3_config() and every key in it is forwarded to boto3, but doc/configuration.rst never mentioned the section at all. endpoint_url is the option that lets S3Target work against any S3-compatible object store (Amazon S3, Backblaze B2, Cloudflare R2, MinIO), and today it is discoverable only by reading the source.

Have you tested this? If so, how?

Correcting an earlier version of this description: existing coverage was not unaffected. Adding the config kwarg to the boto3.resource() call changed the exact call signature that two existing tests pin, so test_init_without_init_or_config and test_init_with_config in test/contrib/s3_test.py failed on this branch with AssertionError: Expected call: resource('s3', aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None) against an actual call that also carries config=<botocore.config.Config object>. Both tests are updated in this PR.

The two updated tests now go through a small assert_resource_called_with() helper that still asserts the exact credential kwargs and additionally asserts that the config carries the luigi/<version> user agent suffix, so the assertions are tightened rather than loosened to ANY. One test is new, test_init_appends_user_agent_to_given_config, covering the merge branch: a caller supplied Config(user_agent_extra="caller/1.0", read_timeout=42) becomes caller/1.0 luigi/<version> with read_timeout preserved.

Run locally against botocore 1.43.x and moto 4.x, on Python 3.12:

pytest test/contrib/s3_test.py -k test_init: 5 passed.
pytest test/contrib/s3_test.py: 68 passed, 1 skipped, plus 4 failures in the multipart size tests. Those 4 also fail on the parent commit 715f65c in the same environment (a local moto version artifact in the part size accounting), so they are unrelated to this change.
ruff check . and ruff format --check .: clean.
sphinx-build -W -b html doc: build succeeded, and the rendered [s3] section shows both options as a definition list alongside the neighboring sections.

I have not run the full tox matrix locally. Happy to split the two commits into separate PRs if you would rather they land separately.