UCP/WORKER: retire a local device at runtime — restricted address packing and lane-selection exclusion - #11
Open
wilsonliu-b10 wants to merge 3 commits into
Open
UCP/WORKER: retire a local device at runtime — restricted address packing and lane-selection exclusion#11wilsonliu-b10 wants to merge 3 commits into
wilsonliu-b10 wants to merge 3 commits into
Conversation
ucp_worker_exclude_device removes a device from selection but the packed worker address still advertised it, so the two disagreed: selection refused the device while the address kept inviting peers to it. That is the exact combination that strands traffic on a dead NIC.
Collaborator
Author
|
Consumers: ucxx bindings in basetenlabs/ucxx#6, and the b10 NIC-failover work in basetenlabs/trt-llm#422. Land order: this -> ucxx -> trt-llm. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two new UCP worker entry points, needed to retire a local NIC at runtime.
Consumed by ucxx
wilson/local-device-pinand, above that, the b10 NIC-failoverwork in
basetenlabs/trt-llm.master, 3 commits, 7 files, +267 / −12unless an application calls it
Why
A worker cannot currently stop using one of its own NICs. That matters when a
port dies under an established job:
published. Nothing it does locally can steer away from an entry we gave it.
So an address listing a dead port keeps inviting traffic to that port, and the
only fix available to us is to publish a shorter address.
afterwards still looks usable to lane selection, so our own outbound wireup
keeps choosing it and the UD connect fails against it indefinitely.
Measured on a two-node b300 bed: a local NIC loss produces no endpoint error
for at least 30 s — the transfer dies of its own deadline instead. With two
rails advertised an incoming stream splits 50/50 across them; with one
advertised, 100 % lands there at identical throughput. The advertised set is the
only lever over the remote half of a pair.
API
get_address_with_devicespacks an address containing only the resources of thenamed devices, instead of every device on the worker. Unknown names are skipped
with a diagnostic; if no named device has usable resources the call fails
rather than returning an empty address that would break obscurely on the peer.
exclude_deviceremoves a device's resources from the set every subsequentendpoint draws from, on this worker's context. Endpoints already created keep
their lanes. Not reversible — restoring a device needs a new context, the same
constraint
UCX_NET_DEVICESalready has.Both are needed together, which is the third commit: the first version let
them disagree — selection refused the device while the packed address still
advertised it. That combination is precisely what strands traffic on a dead NIC,
so the exclusion now also removes the device from what gets packed.
Scope and risks
ucp_context.c/hgains an excluded-resource mask;select.cconsults it; theaddress packer filters by device. Nothing on the fast path.
the b300 bed (real NIC kill via GID delete, 400 transfers byte-verified per
run), not by a gtest — a self-contained gtest would need a NIC it is allowed
to take down.