Skip to content

Restore end-to-end client exception messages to the hub for Client-API tasks #987

Description

@atriaybagur

Context

With the legacy Executor syntax retired (#983), nothing sets the "exception" header on task results any more: the retired RUN_TRAINER/RUN_VALIDATOR/RUN_EVALUATOR executors were the only components that attached the client-side traceback, and the Client-API path (InProcessClientAPIExecutor) returns bare replies — TASK_ABORTED when the training script dies pre-flare.init (e.g. the spleen trainer's deliberate "no image/label pairs — was data enrichment run?" ValueError), EXECUTION_EXCEPTION without a header otherwise.

#983 added a stopgap in FLIP's ScatterAndGather._accept_train_result: every non-OK return code is now relayed to the hub, falling back to a pointed generic ("Client task failed with return code X — the full traceback is in the fl-client container logs at the trust") when no header is present. That makes failures visible but not self-explanatory — a missing enrichment step, an OOM, and a typo still read the same from the hub.

Ask

Carry the actual client-side error text end-to-end for Client-API tasks:

  • NVFLARE's TaskScriptRunner puts the full traceback in the TOPIC_ABORT event data when the script thread dies; capture it client-side (a small task_result_filter or a thin InProcessClientAPIExecutor subclass) and attach it as the "exception" header the server-side relay already consumes.
  • Same for EXECUTION_EXCEPTION replies, which today carry no header on the Client-API path.
  • ClientExceptionReporter (still wired on validate/post_validation) is dead code for the same reason — either feed it the same header or retire it.

Acceptance criteria

  • A Client-API trainer that raises before/after flare.init produces a hub-visible model error containing the raised message (verified via the spleen tutorial's no-labels guard on a dev-stack run).
  • No behaviour change for successful tasks.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions