Skip to content

UCP/WORKER: retire a local device at runtime — restricted address packing and lane-selection exclusion - #11

Open
wilsonliu-b10 wants to merge 3 commits into
masterfrom
wilson/local-device-pin
Open

UCP/WORKER: retire a local device at runtime — restricted address packing and lane-selection exclusion#11
wilsonliu-b10 wants to merge 3 commits into
masterfrom
wilson/local-device-pin

Conversation

@wilsonliu-b10

Copy link
Copy Markdown
Collaborator

Two new UCP worker entry points, needed to retire a local NIC at runtime.
Consumed by ucxx wilson/local-device-pin and, above that, the b10 NIC-failover
work in basetenlabs/trt-llm.

  • base master, 3 commits, 7 files, +267 / −12
  • additive only: no existing entry point changes behaviour, and nothing new runs
    unless an application calls it

Why

A worker cannot currently stop using one of its own NICs. That matters when a
port dies under an established job:

  • A peer picks which of our devices to write to purely from the address we
    published.
    Nothing it does locally can steer away from an entry we gave it.
    So an address listing a dead port keeps inviting traffic to that port, and the
    only fix available to us is to publish a shorter address.
  • Port state is read at device init and never revisited. A port that dies
    afterwards still looks usable to lane selection, so our own outbound wireup
    keeps choosing it and the UD connect fails against it indefinitely.

Measured on a two-node b300 bed: a local NIC loss produces no endpoint error
for at least 30 s
— the transfer dies of its own deadline instead. With two
rails advertised an incoming stream splits 50/50 across them; with one
advertised, 100 % lands there at identical throughput. The advertised set is the
only lever over the remote half of a pair.

API

ucs_status_t ucp_worker_get_address_with_devices(
        ucp_worker_h worker, const char *const *dev_names,
        unsigned num_dev_names, ucp_address_t **address_p,
        size_t *address_length_p);

ucs_status_t ucp_worker_exclude_device(ucp_worker_h worker,
                                       const char *dev_name);

get_address_with_devices packs an address containing only the resources of the
named devices, instead of every device on the worker. Unknown names are skipped
with a diagnostic; if no named device has usable resources the call fails
rather than returning an empty address that would break obscurely on the peer.

exclude_device removes a device's resources from the set every subsequent
endpoint draws from, on this worker's context. Endpoints already created keep
their lanes. Not reversible — restoring a device needs a new context, the same
constraint UCX_NET_DEVICES already has.

Both are needed together, which is the third commit: the first version let
them disagree — selection refused the device while the packed address still
advertised it. That combination is precisely what strands traffic on a dead NIC,
so the exclusion now also removes the device from what gets packed.

Scope and risks

  • ucp_context.c/h gains an excluded-resource mask; select.c consults it; the
    address packer filters by device. Nothing on the fast path.
  • Not reversible by design. A device is retired for the life of the context.
  • No test coverage in this PR. It is exercised end-to-end by the b10 harness on
    the b300 bed (real NIC kill via GID delete, 400 transfers byte-verified per
    run), not by a gtest — a self-contained gtest would need a NIC it is allowed
    to take down.

ucp_worker_exclude_device removes a device from selection but the packed
worker address still advertised it, so the two disagreed: selection refused
the device while the address kept inviting peers to it. That is the exact
combination that strands traffic on a dead NIC.
@wilsonliu-b10

Copy link
Copy Markdown
Collaborator Author

Consumers: ucxx bindings in basetenlabs/ucxx#6, and the b10 NIC-failover work in basetenlabs/trt-llm#422. Land order: this -> ucxx -> trt-llm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant