Add S3 storage and multi-hop transfer tutorial section#766
Conversation
Addresses feedback from PR rucio#744: - Remove Jupyter notebook - Inline script content directly as code blocks - Replace inline script comments with prose descriptions - Remove docker exec wrapping; assume reader is in a Rucio admin environment - Narrow scope to S3 RSE setup only; remove environment initialization steps
for more information, see https://pre-commit.ci
|
Hi @alessio94 , have you gotten a chance to look at these comments? |
|
Hi @voetberg, thanks for the follow-up and apologies for the slow response. I'll address the remaining comments, but I want to be transparent: this PR is now at its third or fourth iteration and I'm finding it difficult to prioritize given that my ATLAS qualification task effectively concluded weeks ago. I'd appreciate if you could consolidate any remaining feedback in one pass once I push the next update, so we can move toward a final review without further back-and-forth. A couple of questions before I start: for the suggested changes you left inline, should I simply accept them as-is, or are there parts you'd like me to rework independently? Also, given how many commits have landed on main since this branch was created, would it make more sense to open a fresh PR on a rebased branch rather than continuing here? I'll aim to push the changes within the next few days. |
Added detailed instructions for configuring S3 storage and multi-hop transfers in Rucio, including setting up MinIO instances, registering RSEs, and verifying the setup. This PR supersedes rucio#744, rucio#739, and rucio#766. Addresses feedback from PR rucio#766. Thanks for the review, @voetberg. I have implemented the requested documentation changes: - Updated the introduction to reflect that `rucio` and `rucio-admin` have been merged since Rucio 38, and that admin access is now handled through permission policies. - Reduced the duplicated MinIO RSE registration commands to a loop over `MINIO1` and `MINIO2`. - Added explanatory comments for the RSE attributes used in the S3 protocol configuration. - Reworked the `rse-accounts.cfg` section to separate the commands used to generate the file from the rendered configuration output, using a `console` block for the inspected file content. - Simplified the explanation of RSE distances and added a Mermaid diagram to make the multi-hop topology clearer. - Removed the Jupyter notebook material from the documentation, keeping the demo setup focused on the operator workflow.
I will do my best on this, but if you make changes that need additional comments after this, I cannot make promises.
It makes more sense to just modify this existing PR with the suggested changes, you can simply use
All the content on this page is new, so it shouldn't have to be rebased. The PR doesn't show any conflicts so making a new PR or rebasing won't do anything. |
Co-authored-by: voetberg <magpie127@gmail.com>
for more information, see https://pre-commit.ci
I have addressed the inline review comments. In the meanwhile Mario asked me to look into trying the same setup with another S3-compatible backend other than MinIO, RustFS so I added a short RustFS compatibility section at the end.
for more information, see https://pre-commit.ci
2 participants
This PR supersedes #744 and #739.
Addresses feedback from PR #744:
@voetberg