Route multi-host RoCE rails over direct cables (#232)

* Route multi-host RoCE over direct cables and add --roce-switch-only

Multi-host RoCE serving leaves direct cables between ranks idle when
the ranks also share a switch. Helios paired rails by device index:
rail h used device h on every rank whenever each such pair shared a
subnet, and otherwise preferred the lowest-numbered subnet. On DGX
Sparks whose first ConnectX-7 port joins a switch and whose second
port is cabled to one partner, both rules send the partner's traffic
over the switch. Rails also landed on the two PCIe functions behind
one x4 link, which share that link's bandwidth. A subnet that only two
ranks hold identifies their cable, and the PCIe address identifies the
link behind each function, so the planner can choose both from the
endpoints every rank already exchanges at rendezvous.

Helios's route planner (helios/src/execution/roce_fabric.rs) computes
each subnet's breadth, the number of ranks holding an address on it:
two for a direct cable, the whole world for a switch. Index pairing
applies only when every subnet joining two or more ranks joins all of
them and no rank's rail devices share a PCIe link; otherwise per-peer
pairing places one rail behind each PCIe link and then prefers the
narrowest subnets. IPv4 link-local subnets count as world-wide,
because sharing 169.254.0.0/16 does not imply a cable. Each endpoint
record gains its PCIe link, the function's PCI address without the
function number, read from sysfs.

Fabric TCP transfers (serving core distribution, Orbit prestaging and
Ignition checkpoint transfers) plan each edge with
roce_fabric::plan_link over every rank's endpoints, in place of a plan
from the edge's two ranks, which cannot tell a cable from a switch.
Every pair of ranks then connects over the devices its RDMA rails use.

--roce-switch-only, a serving option of flare run, flare orbit run and
flare orbit push that is forwarded to every rank, restricts RDMA rails
and fabric transfers to subnets every rank holds
(roce_fabric::Links::Shared): a switch-only baseline, or a way to keep
traffic off a faulty cable. With two ranks every subnet joins both, so
the option cannot separate a cable from a switch. Ranks exchange
"mesh switch-only" instead of "mesh" in the transport shape record and
stop at transport connection when they disagree; peers adopt rank
zero's choice from the serving distribution plan. The Connect RoCE
transport startup phase lists each peer's local devices in rail order
and marks a peer whose rails all ride direct cables, for example
"rank 2 direct roceP2p1s0f1,rocep1s0f1".

The expected result is that each rank reaches a cabled partner over
the cable's two functions, one behind each PCIe link, and every other
peer over the switch. Cores are unchanged, because routes are planned
at serve time. Without --roce-switch-only, the transport shape record
and the serving distribution plan encode as before. The rendezvous
endpoint line gains an optional third field carrying the PCIe link;
builds without this change reject such lines as malformed, and on
fabrics with direct cables or shared PCIe links the two planners choose
different rails, so every rank must run a build containing this
change.

Evidence: on bbad44f with this change, on Gracie (ARM64),
`cargo test --lib --features roce,runtime-package,server roce` in
helios passed 32 tests. The route tests cover partner cables in either
device order, a cable absent at one end, link-local addresses on
uncabled ports, switch-only routing, single-edge planning equal to full
planning, a cabled pair's PCIe links, endpoint encoding with links and
per-peer route summaries. `cargo test --features roce --no-fail-fast`
in flare-cli passed 177 unit and 24 integration tests; they include
transfer edges over partner cables and with --roce-switch-only,
forwarding of the option to agent-started ranks, and shape-record
agreement. Two Orbit agent tests,
groups_run_together_on_disjoint_gpus_and_status_lists_them and
running_groups_are_stopped_and_outlive_their_controller_session, fail
with the same core-size assertion on bbad44f without this change. A
build that combined this planner with other transport and model changes
served GLM-5.3-Flash at TP4 on four DGX Sparks, each with one
ConnectX-7 port on a switch and the other cabled to one partner. Every
rank's Connect RoCE transport phase reached its partner over
`direct roceP2p1s0f1,rocep1s0f1` and the other two ranks over the
switch functions, and the out-of-sequence, packet-sequence-error and
local-ACK-timeout counters of all 16 RoCE devices were unchanged across
answer checks and prefill and decode benchmarks at 1 to 16 concurrent
requests. A build of this commit alone has not served traffic.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Route a two-rank RoCE group over its cable by LLDP adjacency

A two-rank RoCE group on a cabled pair of hosts that also share a
switch, such as GLM-5.3 TP2 on two cabled DGX Sparks, routes both rails
over the switch functions and leaves the cable idle. Helios's route
planner (helios/src/execution/roce_fabric.rs) identifies a direct cable
as a subnet held by fewer ranks than the world, but the two ranks of
such a group hold every subnet in common. Breadth then ties, and the
lowest network, here the switch's, carries both rails. LLDP gives the
missing evidence: LLDP frames do not cross a bridge, so devices of two
ranks that list each other as neighbors share a cable.

Discovery records each RoCE device's network device MAC address, read
from sysfs under the device's PCI function. It also records the MAC
addresses its LLDP neighbors advertise as port IDs, from
`lldpctl -f keyvalue` with a two-second limit; the list is empty when
lldpctl is missing or fails. Index pairing is invalid when any devices
of two ranks are LLDP neighbors. Per-peer matching prefers links
between LLDP neighbors after PCIe link separation and before subnet
breadth. Links::Shared (`--roce-switch-only`) excludes such links. The
`direct` marker in the `Connect RoCE transport` phase also covers
LLDP-adjacent rails. The endpoint record line gains the MAC address
and the comma-separated neighbor list after the PCIe link, `-` when
unknown, with trailing unknown fields omitted.

The expected result is that a two-rank group on a cabled pair running
lldpd uses both cable functions, one behind each PCIe link, and uses
the switch with `--roce-switch-only`. On fabrics with three or more
ranks, the LLDP neighbors of DGX Sparks coincide with subnets two ranks
hold, so their routes and markers are unchanged. Hosts without lldpd
record no neighbors and keep breadth routing. Core distribution and
Orbit fabric transfers plan links with the same planner, so they follow
the RDMA rails. Cores are unaffected; the fabric is discovered at
serving time. Ranks, Orbit controllers and agents must run builds that
read the five-field record, since a reader of the three-field record
rejects these lines as malformed, and every rank of a group must run
this planner, because ranks with and without it choose different rails
for a two-rank cabled pair.

Evidence: on bbad44f with the direct-cable planner and this change, on
Gracie (ARM64), `cargo test --lib --features roce,runtime-package,server
roce` in helios passed 35 tests. The tests added here cover lldpctl
parsing, MAC and neighbor discovery and the five-field record, and a
cabled pair routed over its cable with the `direct` marker,
`--roce-switch-only` on the switch and fabric-transfer links on the
cable. Routes and direct markers of a four-rank fabric with partner
cables, and of four ranks in two cabled pairs, are the same with and
without LLDP data. `cargo test --features roce --no-fail-fast` in
flare-cli passed 177 unit and 24 integration tests; the two Orbit agent
tests that fail on bbad44f without this change fail the same way. A
build that combined this planner with other transport and model changes
served GLM-5.3-Flash at TP2 on two DGX Sparks that are cabled to each
other and share a switch, both running lldpd. Its Connect RoCE
transport phase reported `rank 0 direct roceP2p1s0f1,rocep1s0f1`, and
the out-of-sequence, packet-sequence-error and local-ACK-timeout
counters of all 8 RoCE devices were unchanged across answer checks and
prefill and decode benchmarks at 1 to 16 concurrent requests. A build
of this commit alone has not served traffic.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Plan ring and chain RoCE rails over every rank's endpoints

In ring and chain mode, each rank planned its RoCE routes over the
endpoints of itself and its linked neighbors only. Subnet breadth and
index-pairing validity then came from a different subset at each end of
an edge, so the two ranks of one edge could choose different devices.
The native proxy connects the device indexes each rank supplies without
exchanging them, so a disagreement surfaces as an RDMA failure or a
collective timeout rather than a route error. With one rail and four
ranks, where rank 0 holds a 10.0.0.1/24 and b 10.0.1.1/24, rank 1
a 10.0.0.2/24 and b 10.0.1.2/24, rank 2 a 10.0.0.3/24 and c 10.0.2.3/24,
and rank 3 b 10.0.1.4/24 and c 10.0.2.4/24, rank 0 chose devices (0, 0)
toward rank 1 while rank 1 chose (1, 1) toward rank 0. Mesh routing and
the fabric TCP transfer planner already plan over every rank's
endpoints.

plan_shape_routes builds one route planner over every rank's endpoints
and asks it only for the peers the shape links, in place of planning
over the neighborhood subset. Peers the shape does not link still get
no route, so their queue pairs stay unconnected.

The expected result is that both ends of every ring or chain link
compute the same breadths and index-pairing validity and choose
mirrored devices. Mesh routes are unchanged.

Evidence: ring_and_chain_ends_agree_when_subnets_differ_between_neighborhoods
plans the four-rank example above in ring and chain mode and checks
that every linked pair's routes mirror each other over a shared subnet
and that unlinked pairs have none. On the parent commit it fails with
rank 0 choosing (0, 0) and rank 1 choosing (1, 1). On Gracie (ARM64),
`cargo test --lib --features roce,runtime-package,server roce` in
helios passed 36 tests, including the ring, line and variable-size
ring tests. No ring or chain fabric has served traffic with this
change.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Require evidence before link-local or unaddressed RoCE links carry rails

Per-peer RoCE route matching accepted two kinds of link without evidence
that the devices are connected. Any two IPv4 link-local addresses
(169.254.0.0/16) counted as a link, although hosts using link-local
addressing share that subnet whether or not their ports are connected.
Link-local subnets have world-wide breadth, and 169.254 sorts below
192.168, so a link-local pair on unrelated ports could displace a switch
rail. With four ranks on 192.168.0.0/24 and 192.168.1.0/24, a private
cable subnet between ranks 0 and 2 (which rules out index pairing), and
link-local addresses on unrelated ports of ranks 1 and 3, rank 1 routed
rank 3 over devices [(2, 2), (0, 0)], where device 2 is the link-local
port. Separately, --roce-switch-only (Links::Shared) promises rails only
over subnets every rank holds, yet a device whose interface prefix
discovery could not resolve still received same-index pairing on the
index-pairing and per-peer paths, even when LLDP showed that the two
devices share a cable.

Per-peer matching admits a link-local pair only when its two devices are
LLDP neighbors, and then ranks it as a direct cable. Under Links::Shared,
index pairing requires every rail device to have a known address outside
link-local space, and per-peer matching admits no same-index pair whose
subnet is unknown. With Links::All, same-index pairs without an address
are still trusted in device order, and index pairing still accepts
same-index link-local pairs.

The expected result is that a peer with verified switch or cable rails
uses them instead of a link-local port, and --roce-switch-only fails
with "0 of N RoCE rails have a link" when no verified shared subnet
exists instead of falling back to device order. A fabric whose RoCE
devices carry only link-local addresses and no LLDP data has no
per-peer candidates; it is still routed when index pairing applies,
which requires no subnet narrower than the world, no PCIe link shared
by one rank's rail devices, and same-index devices on one subnet.

Evidence: link_local_pairs_carry_rails_only_between_lldp_neighbors
plans the four-rank example above. Rank 1 reaches rank 3 over both
switch rails, ranks 0 and 2 use their cable and one switch rail, an
LLDP-adjacent link-local pair carries a rail, and switch-only routing
uses neither the cable nor a link-local pair. On the parent commit it
fails with rank 1 routing rank 3 over [(2, 2), (0, 0)].
switch_only_routing_rejects_links_whose_subnet_is_unknown covers
unaddressed devices with and without LLDP adjacency, and a known switch
subnet beside an unaddressed device; on the parent commit switch-only
routing returns [(0, 0)] for the LLDP-adjacent pair. On Gracie (ARM64),
`cargo test --lib --features roce,runtime-package,server roce` in
helios passed 38 tests. No fabric with link-local RoCE addresses has
served traffic with this change.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Bound lldpctl by one deadline for its output and exit

RoCE endpoint discovery runs `lldpctl -f keyvalue` under a documented
two-second limit, but the limit covered only reading its standard
output. After the output closed, discovery waited for the process
without bound, so an lldpctl that closed its output and stayed alive
would block transport startup indefinitely.

bounded_output runs a command under one deadline for both its output
and its exit: a thread collects standard output, the caller polls for
exit until the deadline, and a command still running at the deadline
is killed and reaped. lldp_neighbors uses it with a two-second limit. A
command that cannot start, fails, or misses the deadline yields no
neighbors, as a failing lldpctl did before.

The expected result is that endpoint discovery spends at most about two
seconds on lldpctl, whichever way lldpctl stalls, and keeps breadth
routing when lldpctl does not answer in time.

Evidence: a_command_is_bounded_by_one_deadline_for_its_output_and_exit
runs sh commands under a 300 ms limit. A successful command returns its
output, a failing one and a missing executable return none, and a
command that closes its output and sleeps for ten seconds, or sleeps
with its output open, returns none in under five seconds. On Gracie
(ARM64), `cargo test --lib --features roce,runtime-package,server roce`
in helios passed 39 tests. lldpctl itself was not run with this change.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Reject RoCE endpoint records and Orbit agents from builds without the route planner

Ranks, Orbit controllers and agents with and without Helios's RoCE
route planner choose different rails and transfer links on cabled
fabrics, so every participant must run a build with this planner. A
build without it reads endpoint lines of exactly two fields (device name
and address) and rejects longer lines as malformed, but this planner
accepted its two-field lines, and Orbit controllers and agents with and
without the planner both spoke agent protocol 6. A controller with the
planner therefore accepted agents without it, and a rank with the
planner decoded endpoints from a rank without it. Only the older side
reported an error, and "malformed RoCE endpoint" does not name the
build mismatch.

roce_fabric::encode writes the PCIe link field on every line, `-` when
unknown, so every line has at least three fields. roce_fabric::decode
rejects a two-field line with "RoCE endpoint record has no PCIe link
field: its rank runs a build that plans rails from addresses alone, and
every rank needs a build that records PCIe links and LLDP neighbors".
The Orbit agent protocol becomes 7, so a session handshake or HELLO
exchange between protocol 6 and protocol 7 peers fails with "peer
speaks agent protocol 6; this agent speaks 7". docs/orbit.md states
what protocol 7 carries in FABRIC replies, and docs/multi-host.md lists
the endpoint record error. Test fixtures that encoded endpoint records
by hand carry the link field.

The expected result is that a group mixing builds with and without the
route planner stops at rank rendezvous or at the Orbit handshake on
every side, with an error naming the incompatibility, instead of
planning different rails. Orbit controllers and agents must be updated
together, as for earlier agent protocol changes. Cores are unaffected.

Evidence: endpoints_survive_the_rendezvous_encoding in helios checks
that an endpoint with only a name and an address encodes with the link
field and decodes back, and that a two-field line fails with the record
error. tampered_records_and_wrong_versions_are_refused in flare-cli
checks that an agent refuses handshakes from protocol 6 and protocol 8
peers. On Gracie (ARM64), `cargo test --lib --features
roce,runtime-package,server roce` in helios passed 39 tests, and
`cargo test --features roce --no-fail-fast` in flare-cli passed 177
unit and 24 integration tests. The same two Orbit agent tests that fail
on bbad44f, groups_run_together_on_disjoint_gpus_and_status_lists_them
and running_groups_are_stopped_and_outlive_their_controller_session,
fail with the same core-size assertion. A mixed-build group has not
been run on hardware.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Document the evidence and limits of RoCE route inference

The multi-host guide stated that a two-rank group on a cabled pair uses
the switch whenever lldpd does not run on both hosts. The planner
treats two devices as LLDP neighbors when either lists the other, so
lldpd on one host identifies the cable. Without LLDP data such a group
has no evidence of its cable, and its rails take one device behind each
PCIe link and then the lowest-numbered network, which is the switch's
only when the switch's subnets sort first. The guide and CLI design
also described the `direct` startup marker as a peer reached over a
cable and `--roce-switch-only` as leaving every cable unused, while the
marker reports the topology inferred from endpoint records and the
option can exclude only cables that subnet breadth or LLDP identifies.

docs/multi-host.md states that lldpd on one host suffices, the
fallback without LLDP data, that two link-local addresses form a link
only between LLDP neighbors, and that the `direct` marker reports
inferred topology rather than verified reachability. docs/cli-design.md
states which cables `--roce-switch-only` excludes and that devices
without a known subnet carry no rails under it. The roce_fabric module
documentation and Topology::direct carry the same limits.

These changes correct documentation only; behavior is unchanged.

Evidence: the statements follow the planner in
helios/src/execution/roce_fabric.rs. lldp_adjacent accepts a listing in
either direction; a_cabled_pair_keeps_its_two_rails_on_separate_pcie_links
shows a two-rank cabled pair without LLDP data placing one rail behind
each PCIe link on the lowest network;
link_local_pairs_carry_rails_only_between_lldp_neighbors and
switch_only_routing_rejects_links_whose_subnet_is_unknown cover the
link-local and switch-only statements. Documentation changes need no
GPU tests.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

* Keep link-local RoCE addresses out of index pairing

Per-peer route matching admits a link-local pair (169.254.0.0/16) only
between LLDP neighbors, but index pairing still accepted same-index
devices whose link-local addresses match, although hosts using
link-local addressing share that subnet whether or not their ports are
connected. With four ranks and no LLDP data, where device 0 holds
169.254.1.r/16 on an unconnected private port and devices 1 and 2 hold
192.168.0.r/24 and 192.168.1.r/24 on a switch, every subnet joins every
rank, so index pairing applied, and every rank routed every peer over
devices [(0, 0), (1, 1)]. The rail on device 0 has no path. Index
pairing cannot use LLDP evidence, because any LLDP adjacency already
rules it out, so a link-local rail device must rule it out as well.

index_pairing_valid rejects index pairing when any rank's rail device
has a link-local address, under every link selection; under
--roce-switch-only it also rejects a rail device without a known
address, as before. Per-peer matching then applies the LLDP rule. When
per-peer matching finds too few rails and skipped a link-local pair
without LLDP adjacency, its error adds "link-local addresses link only
devices that are LLDP neighbors". docs/multi-host.md states that a
device whose only RoCE address is link-local carries rails only over a
cable LLDP identifies, so switch ports need configured subnets, and its
troubleshooting table repeats this for rail errors.

The expected result is that an unconnected link-local port never
carries a rail: in the example above every rank reaches every peer over
devices [(1, 1), (2, 2)], and a fabric with link-local devices after its
rail devices keeps index pairing. A fabric whose RoCE devices carry only
link-local addresses now plans rails only between LLDP neighbors. Such
a fabric on a switch, which `main` routes by index pairing, fails
planning with the error above until its switch ports have configured
subnets.

Evidence: index_pairing_admits_no_link_local_rail_device plans the
example above for every pair of ranks, checks that index pairing still
applies with the link-local device after the rail devices, and checks
that two ranks addressed only by link-local addresses fail with the
link-local error without LLDP data and route over the pair with it. On
the parent commit it fails with [(0, 0), (1, 1)]. On Gracie (ARM64),
`cargo test --lib --features roce,runtime-package,server roce` in
helios passed 40 tests, and `cargo test --lib --features
runtime-core,server roce_fabric` passed 24. No fabric with link-local
RoCE addresses has served traffic with this change.

Design-Signed-Off-By: flare-02; design-review=cfcf
Signed-off-by: Jason Cook <jasonc@maxlyn.com>

---------

Signed-off-by: Jason Cook <jasonc@maxlyn.com>
Co-authored-by: Luke Alonso <lalonso@gmail.com>

Author: original-el8

docs/multi-host.md

@@ -94,8 +94,47 @@ also listen on their fabric addresses. Flare does not configure interfaces,
 addresses, routes, switches, or firewalls.
 
 The serving ranks select up to two rails per peer from active devices. The
-core does not pin the physical fabric used during preparation. For direct
-cabling and neighbor forwarding, see [experimental rings](#experimental-rings).
+core does not pin the physical fabric used during preparation. On a switch,
+rail `h` uses device `h` on every rank. Hosts can also add direct cables beside
+the switch, for example DGX Sparks whose second port is cabled to one partner:
+each rank then reaches a cabled peer over the cable's devices and every other
+peer over the switch. The planner identifies a cable as an IPv4 subnet that only
+the two cabled ranks hold, so every cable needs subnets of its own, distinct from
+the switch's and from every other cable's. The two ranks of a group on one
+cabled pair hold every subnet in common, so the planner also identifies a cable
+by LLDP: each rank records its devices' MAC addresses and, when lldpd runs on the
+host, the port MAC addresses that `lldpctl` reports for their neighbors. LLDP
+frames do not cross a switch, so devices of two ranks share a cable when either
+lists the other; lldpd on one host of the pair is enough. Without LLDP data, a
+two-rank group on a cabled pair has no evidence of its cable: its rails take one
+device behind each PCIe link and then the lowest-numbered network, which is the
+switch only when the switch's subnets sort below the cable's. IPv4 link-local
+addresses (169.254.0.0/16) never identify a cable, and two link-local addresses
+form a link only between LLDP neighbors. A device whose only RoCE address is
+link-local therefore carries rails only over a cable that LLDP identifies; LLDP
+does not cross a switch, so switch ports need configured subnets. Among equally
+verified links, a peer's two rails use devices behind different PCIe links: a
+DGX Spark's two ConnectX-7 functions behind one PCIe x4 link share its
+bandwidth, so a two-rank group on a cabled pair uses one function behind each
+link. Core distribution
+and Orbit prestaging connect each pair of ranks over the same devices as its RDMA
+rails.
+
+`--roce-switch-only` routes every peer over subnets every rank holds between
+devices that are not LLDP neighbors, as on the switch alone, for example to
+measure a switch-only baseline or to keep traffic off a faulty cable. With two
+ranks every subnet joins both, so the option leaves a cable unused only where
+LLDP identifies it. Peers started without it follow rank zero's
+choice for core distribution, and ranks that still disagree stop when the
+transport connects, naming the rank. Every rank's `Connect RoCE transport`
+startup phase lists each peer's local devices in rail order and marks a peer
+reached over a direct cable, for example
+`2 rails per peer: rank 0 roceP2p1s0f0,rocep1s0f0; rank 2 direct roceP2p1s0f1,rocep1s0f1; rank 3 roceP2p1s0f0,rocep1s0f0`,
+so a cable that is down or unaddressed appears as its partner routed without
+`direct`. The marker reports the topology inferred from subnets and LLDP, not
+verified reachability; only a connected transport shows that the devices reach
+each other. For direct cabling and neighbor forwarding, see
+[experimental rings](#experimental-rings).
 
 ## Manual serving
 
@@ -505,7 +544,8 @@ check `orbit status` and restart saved groups explicitly.
 | Checkpoint source changed | Keep source checkpoint files unchanged through loading. |
 | Rank identity or graph I/O schema differs | Matching export, prepared world size, serving options, and compatible executables. |
 | Core RoCE proxy ABI mismatch | Align preparation and serving dependencies; re-export when the executable cannot load that ABI. |
-| Missing GID or rail/queue-pair error | Active ports, IPv4 RoCEv2 GIDs, reachable endpoints, and the intended rank order. |
+| Missing GID or rail/queue-pair error | Active ports, IPv4 RoCEv2 GIDs, reachable endpoints, and the intended rank order. Link-local addresses link only LLDP neighbors; configure subnets on switch ports. |
+| `RoCE endpoint record has no PCIe link field` | A rank runs a build that plans rails from addresses alone; run builds with the same route planner on every rank. |
 
 Inspect rank output and `journalctl --user -u flare-agent` before restarting a
 failed group. A collective failure invalidates the serving run; the agent does

docs/orbit.md

@@ -806,9 +806,12 @@ refuses an agent whose HELLO reports another version. Serving executables must
 also support the core's GPU architecture, prepared graph contract, and RoCE
 proxy ABI. A controller's CPU architecture does not select the serving build.
 
-Fabric TCP distribution requires agent protocol 6 on the controller and every
-agent, including relays. Updating only a serving executable does not update the
-resident agents that distribute cores.
+Fabric TCP distribution requires agent protocol 7 on the controller and every
+agent, including relays. In protocol 7, each RoCE endpoint in a FABRIC reply
+carries its PCIe link, MAC address and LLDP neighbors, and the controller plans
+transfer links with the same route planner as the ranks' RDMA rails. Updating
+only a serving executable does not update the resident agents that distribute
+cores.
 
 HELLO's optional `executable` path advertises support for using the installed
 agent binary in RUN. An agent that omits the field requires
@@ -818,7 +821,7 @@ also updates the default serving executable; restarting the agent stops its
 running ranks.
 
 HELLO's optional `download_progress` flag negotiates live DOWNLOAD progress
-within protocol 6. A controller sets it only when it can consume progress
+within one protocol version. A controller sets it only when it can consume progress
 frames; an agent sends DOWNLOAD_PROGRESS only to a controller that set it.
 A peer that omits the field exchanges DOWNLOAD as one request and one reply,
 with no progress frames.

flare-cli/src/cli.rs

@@ -21,6 +21,9 @@ use flare_helios::{
     runtime_core::{KvCacheLimit, LoadOptions, MemoryLimit},
 };
 
+/// The serving option that routes every peer over subnets every rank holds.
+pub const ROCE_SWITCH_ONLY: &str = "--roce-switch-only";
+
 #[derive(Debug, Parser)]
 #[command(
     name = "flare",
@@ -854,6 +857,11 @@ pub struct ServingOptions {
     )]
     pub memory_limit: Option<MemoryLimit>,
 
+    /// Route RoCE rails and fabric transfers only over subnets every rank holds, as on a switch
+    /// alone, leaving direct cables between ranks unused.
+    #[arg(long, help_heading = "Multi-host options")]
+    pub roce_switch_only: bool,
+
     /// Maximum simultaneous requests [default: model/default configuration, then core setting].
     #[arg(long, value_name = "N", help_heading = "Serving options")]
     pub max_concurrency: Option<NonZeroUsize>,
@@ -1063,6 +1071,9 @@ impl ServingOptions {
             Shape::Ring => options.push("--roce-ring".into()),
             Shape::Chain => options.push("--roce-chain".into()),
         }
+        if self.roce_switch_only {
+            options.push(ROCE_SWITCH_ONLY.into());
+        }
         if self.verify_weights {
             options.push("--verify-weights".into());
         }
@@ -1758,6 +1769,21 @@ mod tests {
         assert_eq!(args.rank_options(), vec!["--roce-ring"]);
     }
 
+    #[test]
+    fn switch_only_flag_is_forwarded_to_every_agent_started_rank() {
+        let args = run(&["flare", "run", "model.safetensors", "--roce-switch-only"]);
+        assert!(args.serving.roce_switch_only);
+        assert_eq!(args.rank_options(), vec![ROCE_SWITCH_ONLY]);
+        let args = run(&[
+            "flare",
+            "run",
+            "model.safetensors",
+            "--roce-chain",
+            "--roce-switch-only",
+        ]);
+        assert_eq!(args.rank_options(), vec!["--roce-chain", ROCE_SWITCH_ONLY]);
+    }
+
     #[test]
     fn verification_token_budget_reaches_loader_and_peer_ranks() {
         for budget in ["24", "32"] {

flare-cli/src/cluster.rs

@@ -6,9 +6,16 @@
 
 //! Ranks of one model on separate hosts: rendezvous at rank zero and connect their RoCE transport.
 
-pub use flare_helios::execution::roce_fabric::Shape;
+pub use flare_helios::execution::roce_fabric::{Links, Shape};
 use std::sync::Arc;
 
+/// How a rank's transport links to its peers: the shape, and which links may carry rails.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct Routing {
+    pub shape: Shape,
+    pub links: Links,
+}
+
 use flare_helios::{
     execution::{CollectiveHealth, RankStream},
     runtime_core::{ClusterRank, NativeModel},
@@ -66,7 +73,7 @@ fn open(
     owner: &NativeModel,
     fabric: &flare_helios::execution::roce_fabric::Fabric,
     endpoints: &[Vec<u8>],
-    shape: Shape,
+    routing: Routing,
 ) -> Result<RoceTransport, Failure> {
     use flare_helios::execution::roce_fabric;
 
@@ -77,37 +84,49 @@ fn open(
         .iter()
         .map(|bytes| roce_fabric::decode(bytes))
         .collect::<flare_helios::Result<Vec<_>>>()?;
-    let topology = fabric.topology(&endpoints, spec.rank, shape)?;
+    let topology = fabric.topology(&endpoints, spec.rank, routing.shape, routing.links)?;
     // SAFETY: The core restored the region at its recorded address; owner outlives the
     // transport, and no graph runs before every rank connected.
     Ok(unsafe { RoceTransport::open(spec, &topology)? })
 }
 
 #[cfg(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce")))]
 fn connected(transport: &RoceTransport) -> String {
+    let topology = transport.topology();
     format!(
-        "{} rails per {} over {}",
-        transport.topology().rails,
-        match transport.topology().shape {
+        "{} rails per {}{}: {}",
+        topology.rails,
+        match topology.shape {
             Shape::Mesh => "peer",
             Shape::Ring => "neighbor (experimental CPU relay ring)",
             Shape::Chain => "neighbor (experimental CPU relay chain)",
         },
-        transport.topology().hcas.join(",")
+        match topology.links {
+            Links::All => "",
+            Links::Shared => ", switch only",
+        },
+        topology.peer_routes().join("; ")
     )
 }
 
-/// The transport shape a rank was started with, as the ranks exchange it before planning routes.
+/// The transport shape a rank was started with and the links it may route over, as the ranks
+/// exchange them before planning routes: `mesh`, `ring` or `chain`, followed by ` switch-only`
+/// when direct cables are excluded.
 #[cfg(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce")))]
-fn shape_record(shape: Shape) -> Vec<u8> {
-    shape.name().as_bytes().to_vec()
+fn shape_record(routing: Routing) -> Vec<u8> {
+    let links = match routing.links {
+        Links::All => "",
+        Links::Shared => " switch-only",
+    };
+    format!("{}{links}", routing.shape.name()).into_bytes()
 }
 
-/// Every rank must select the same shape: a chain rank plans no route to the far end that a ring
-/// rank would wait for.
+/// Every rank must select the same shape and links: a chain rank plans no route to the far end
+/// that a ring rank would wait for, and a switch-only rank routes a cabled partner over the switch
+/// while the partner would wait on the cable.
 #[cfg(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce")))]
-fn agree_shape(records: &[Vec<u8>], shape: Shape) -> Result<(), Failure> {
-    let local = shape_record(shape);
+fn agree_shape(records: &[Vec<u8>], routing: Routing) -> Result<(), Failure> {
+    let local = shape_record(routing);
     if let Some(rank) = records.iter().position(|record| *record != local) {
         return Err(format!(
             "rank {rank} selected RoCE transport shape {} where this rank selected {}",
@@ -165,7 +184,7 @@ pub fn rank_zero(
     owner: &NativeModel,
     streams: Vec<RankStream>,
     progress: &StartupProgress,
-    shape: Shape,
+    routing: Routing,
 ) -> Result<Joined<Vec<RankStream>>, Failure> {
     #[cfg(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce")))]
     {
@@ -175,11 +194,11 @@ pub fn rank_zero(
         let phase = progress.phase("Connect RoCE transport", None);
         // Every rank learns every rank's devices and the shape, then plans the same routes from them.
         let fabric = roce_fabric::discover()?;
-        let shapes = rendezvous::share_records(&mut streams, shape_record(shape))?;
-        agree_shape(&shapes, shape)?;
+        let shapes = rendezvous::share_records(&mut streams, shape_record(routing))?;
+        agree_shape(&shapes, routing)?;
         let endpoints =
             rendezvous::share_records(&mut streams, roce_fabric::encode(&fabric.endpoints))?;
-        let mut transport = open(owner, &fabric, &endpoints, shape)?;
+        let mut transport = open(owner, &fabric, &endpoints, routing)?;
         let records = rendezvous::share_records(&mut streams, transport.local_record()?)?;
         transport.connect(&records)?;
         rendezvous::await_connected(&mut streams)?;
@@ -197,7 +216,7 @@ pub fn rank_zero(
     }
     #[cfg(not(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce"))))]
     {
-        let _ = (owner, streams, progress, shape);
+        let _ = (owner, streams, progress, routing);
         Err(NO_TRANSPORT.into())
     }
 }
@@ -208,7 +227,7 @@ pub fn peer(
     cluster: &ClusterRank,
     stream: RankStream,
     progress: &StartupProgress,
-    shape: Shape,
+    routing: Routing,
 ) -> Result<Joined<RankStream>, Failure> {
     #[cfg(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce")))]
     {
@@ -217,14 +236,15 @@ pub fn peer(
 
         let phase = progress.phase("Connect RoCE transport", None);
         let fabric = roce_fabric::discover()?;
-        let shapes = rendezvous::exchange_record(&mut stream, &shape_record(shape), cluster.world)?;
-        agree_shape(&shapes, shape)?;
+        let shapes =
+            rendezvous::exchange_record(&mut stream, &shape_record(routing), cluster.world)?;
+        agree_shape(&shapes, routing)?;
         let endpoints = rendezvous::exchange_record(
             &mut stream,
             &roce_fabric::encode(&fabric.endpoints),
             cluster.world,
         )?;
-        let mut transport = open(owner, &fabric, &endpoints, shape)?;
+        let mut transport = open(owner, &fabric, &endpoints, routing)?;
         let records =
             rendezvous::exchange_record(&mut stream, &transport.local_record()?, cluster.world)?;
         transport.connect(&records)?;
@@ -249,7 +269,7 @@ pub fn peer(
     }
     #[cfg(not(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce"))))]
     {
-        let _ = (owner, cluster, stream, progress, shape);
+        let _ = (owner, cluster, stream, progress, routing);
         Err(NO_TRANSPORT.into())
     }
 }
@@ -289,6 +309,37 @@ mod pool_tests {
     }
 }
 
+#[cfg(all(
+    test,
+    target_os = "linux",
+    any(target_arch = "aarch64", feature = "roce")
+))]
+mod routing_tests {
+    use super::*;
+
+    #[test]
+    fn ranks_agree_on_switch_only_links_and_mesh_keeps_its_record() {
+        let mesh = Routing {
+            shape: Shape::Mesh,
+            links: Links::All,
+        };
+        let switched = Routing {
+            shape: Shape::Mesh,
+            links: Links::Shared,
+        };
+        assert_eq!(shape_record(mesh), b"mesh");
+        assert_eq!(shape_record(switched), b"mesh switch-only");
+        assert!(agree_shape(&vec![shape_record(switched); 2], switched).is_ok());
+        let error = agree_shape(&[shape_record(mesh), shape_record(switched)], mesh)
+            .unwrap_err()
+            .to_string();
+        assert!(
+            error.contains("rank 1 selected RoCE transport shape mesh switch-only where this rank selected mesh"),
+            "{error}"
+        );
+    }
+}
+
 #[cfg(all(
     test,
     not(all(target_os = "linux", any(target_arch = "aarch64", feature = "roce")))

flare-cli/src/distribution.rs

@@ -226,6 +226,7 @@ pub fn root(
         &endpoints.iter().map(|e| e.port).collect::<Vec<_>>(),
         args.roce_ring,
         args.roce_chain,
+        args.serving.roce_switch_only,
     )?;
     let relay = Relay::start(0, streams, topology.clone(), network, source.port)?;
     for rank in 1..world {
@@ -383,11 +384,15 @@ pub fn peer(args: &mut Run, progress: &mut StartupProgress) -> Result<Received,
     if args.roce_chain && !plan.topology.chain {
         return Err("rank zero did not select --roce-chain".into());
     }
+    if args.serving.roce_switch_only && !plan.topology.switch_only {
+        return Err("rank zero did not select --roce-switch-only".into());
+    }
     if args.stage_only && !plan.stage_only {
         return Err("rank zero did not select --stage-only".into());
     }
     args.roce_ring = plan.topology.ring;
     args.roce_chain = plan.topology.chain;
+    args.serving.roce_switch_only = plan.topology.switch_only;
     args.stage_only = plan.stage_only;
     select_plan_tensor_parallel(args, plan.tensor_parallel)?;
     let relay = Relay::start(rank, plan.streams, plan.topology.clone(), network, 0)?;
@@ -1427,7 +1432,7 @@ mod tests {
                 .map(|s| s.local_addr().unwrap().port())
                 .collect::<Vec<_>>();
             let topology =
-                Topology::plan(&vec![b"a 127.0.0.1/8\n".to_vec(); world], &ports, ring).unwrap();
+                Topology::plan(&vec![b"a 127.0.0.1/8 -\n".to_vec(); world], &ports, ring).unwrap();
             let relays = network
                 .into_iter()
                 .enumerate()