[Docs]: Correct Model Files / Training Configuration in the user guide - #869
Conversation
There was a problem hiding this comment.
Pull request overview
This PR corrects and restructures the ReadTheDocs documentation around Model Files and Training Configuration, removing misleading backend-specific guidance from the common user guide and moving canonical backend/job-type specifics to the FL nodes component reference page.
Changes:
- Rewrites the user guide’s Model Files section to describe required files as a per-job-type contract surfaced by the UI, rather than a fixed list.
- Replaces the user guide’s detailed NVFLARE-only training config key reference with a short backend-neutral explanation, keeping the existing
training-configurationanchor. - Expands the FL nodes component page with new canonical anchors for required files and training configuration, and repoints the Flower app guide to the new reference anchor.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| docs/source/working-with-flip-apps/create-flip-app-from-flower.rst | Updates the Flower app submission guidance to link to the new canonical required-files anchor. |
| docs/source/user-guides/user-common.rst | Rewrites common guidance on model files and training configuration to be job-type/back-end aware without embedding backend-specific key listings. |
| docs/source/components/component-fl-nodes.rst | Adds canonical per-job-type required-files tables and backend-specific training-configuration reference sections with stable anchors. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The user guide's "Model Files" section stated a fixed minimum file set (validator.py, trainer.py) for NVFLARE apps and described config.json as optional. Both are wrong: required files are declared per job type in fl-apps/<backend>/<job_type>/required_files.json, config.json is required for every NVFLARE job type, and standard_client_api does not require validator.py while evaluation requires evaluator.py instead of a trainer. The "Training Configuration" subsection also carried a full NVFLARE key reference in a page that is meant to be backend-neutral, with several inaccuracies: the round range is 1-1000 (not "greater than 0, less than 100") and out-of-range values are discarded in favour of the default rather than rejected; AGGREGATION_WEIGHTS keys are FL client site names (the kit slot each Trust occupies, e.g. Trust_1), not Trust codes such as KCH, so the documented example would have had no effect. Rework the user guide to describe the job-type-driven contract and point at where the UI surfaces the applicable list, and move the corrected per-backend reference to the FL nodes component page, which is already the documented home for framework-specific file requirements and job types. The new reference covers the required-files manifests for both backends, the validated NVFLARE config.json keys (including AGGREGATE_ONLY_REGEX and the BEST_MODEL_METRIC pair) and Flower's pyproject.toml/config.toml run configuration. Also correct the FL nodes page's claim that pyproject.toml is a researcher-supplied file for Flower apps: it comes from the platform's base template, and per-run overrides go in config.toml. Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
Both edited pages described the job type as declared by the job_type key
in config.json without noting that config.json is itself a required file
only for the NVFLARE job types. A Flower app may omit it entirely — the
bundler falls back to job_type=standard when no config.json is present —
so the previous wording ("if that key is absent") covered a missing key
but read as though the file were mandatory for every app.
Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
Review of the new reference sections surfaced six inaccuracies, five of them in prose this branch adds. The invalid-rounds note claimed a discarded GLOBAL_ROUNDS or LOCAL_ROUNDS is replaced by the default of 1. That holds only for GLOBAL_ROUNDS, which the platform reads and writes into the server's job configuration. LOCAL_ROUNDS is read by the app's own trainer straight out of config.json, and configure_config fills in the default only when the key is absent — a key that is present but invalid is left as written and reaches the trainer verbatim, so the documented safety net does not exist for it. Also note the BEST_MODEL_METRIC exception, where a discarded GLOBAL_ROUNDS fails the job rather than defaulting, and that neither key is rewritten in the deployed config.json. 'Any additional file is bundled into the app' omitted two carve-outs: a model file colliding with a base-template name is skipped with only a server-side log line (which for Flower silently drops server_app.py, strategy.py and __init__.py — exactly what the Flower tutorials ship), and a checkpoint named by SERVER_CHECKPOINT or an evaluation job's models entries is staged server-side and never reaches a client. Drop 'pre-trained checkpoints' as an example of what gets shipped, since it is the case with the carve-out. 'Keys the platform does not recognise are passed through untouched' skipped three keys FLIP's own NVFLARE components do read: SERVER_CHECKPOINT, GLOBAL_ROUNDS_AE/GLOBAL_ROUNDS_DM and models. In the user guide, the same pass-through sentence sat under a both-backends bullet list but is backwards for Flower, where flwr rejects a run-config key the template does not declare instead of ignoring it. Scope it to NVFLARE. Finally, the page's own Disclaimer section still told Flower users to upload server_app.py (the template supplies it, and an uploaded copy is dropped) and listed evaluation among the job types taking trainer/validator/models/config. Both contradicted the new tables on the same page, so point that bullet at fl-required-files and remove the obsolete Flower one. Signed-off-by: at24_bioeng625-pc <alexandre.triay_bagur@kcl.ac.uk>
|
Ready for review @garciadias. What this is. The Model Files / Training Configuration section of the user guide was factually wrong in several places. Two of the errors were the kind that waste someone's afternoon: the required-file list omitted Every claim is traced to a source file. The required-files tables against the per-template and aggregate A self-review round then found six more inaccuracies ( The question for you is scope, not correctness. The split applied here assumes the user guide answers "what do I have to do?" and the FL nodes component page answers "what exactly does the platform accept?". If you would rather the detailed reference lived next to the code ( Two follow-ups I deliberately did not fold in:
Docs build with zero warnings, full CI green, no conflicts with |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
docs/source/components/component-fl-nodes.rst:221
- In this note, “discarded” is a bit misleading because an out-of-range/non-numeric value is ignored by the FL API validator (it isn’t rejected), and the original value can still remain in the deployed
config.json. Rephrasing to “ignored by the platform validator” (and using the same term consistently) will better match the actual behavior and avoid implying the key is removed from the file.
.. note::
A value that is not a number, or that falls outside the accepted range, is discarded rather
than reported. What happens next differs between the two keys, because the platform consumes
them differently.
Description
Follow-up to #811. The Model Files section of the user guide (and its Training Configuration subsection) is factually wrong in several places and carries backend-specific detail that does not belong on a page documenting functions common to all users.
What was wrong
Required files. The page stated a fixed minimum set —
validator.pyandtrainer.py— for NVFLARE apps, and describedconfig.jsonas an optional extra. Required files are actually declared per job type infl-apps/<backend>/<job_type>/required_files.json:config.jsonis required by every NVFLARE job type (it carriesjob_type), so it is not optional.models.pyis required by every NVFLARE job type exceptevaluation, and was not mentioned at all.standard_client_apidoes not requirevalidator.py.evaluation/evaluation_client_apirequireevaluator.pyand no trainer at all.pyproject.tomlas a required user-supplied file. It is not — it comes from the platform's base template, and the required set isclient_app.py+models.py.Training configuration. The subsection carried a full NVFLARE key reference, correctly noted as FLARE-only but still sitting in a backend-neutral page. It also had its own inaccuracies:
AGGREGATION_WEIGHTSwas documented as keyed on Trust abbreviations (KCH,UCLH). The FL server knows clients by their FL kit slot name (Trust_1,Trust_2, …), so weights keyed on a Trust code match no participating client and have no effect. The documented example would have silently done nothing.AGGREGATE_ONLY_REGEXand theBEST_MODEL_METRIC/BEST_MODEL_METRIC_MINIMIZEpair added in [Feature]: Save both best and final models — NVFLARE backend #811 were absent.What this PR does
docs/source/user-guides/user-common.rst— rewrites Model Files around the job-type contract and points at where the UI surfaces the applicable list (the Model Files panel shows the detected job type; the Training panel lists required and missing files). Trims Training Configuration to a short backend-neutral explanation of where configuration lives per backend, keeping thetraining-configurationlabel so inbound references still resolve.docs/source/components/component-fl-nodes.rst— becomes the canonical reference it was already being pointed at as. Adds_fl-required-files(per-job-type tables for both backends, with a note that the UI/manifests are the source of truth) and_fl-training-configuration(the validated NVFLAREconfig.jsonkeys with accepted values and defaults, plus Flower'spyproject.toml/config.tomlrun configuration). Also corrects the page's own claim thatpyproject.tomlis researcher-supplied.docs/source/working-with-flip-apps/create-flip-app-from-flower.rst— repoints its "canonical list" reference at the new anchor instead of back at the user guide.Corrections from a self-review round (
bfca0385, after the sections above were written). Sixfurther inaccuracies surfaced when every claim was re-checked against the source — five of them in
prose this PR itself adds:
LOCAL_ROUNDS. It claimed a discardedGLOBAL_ROUNDSor
LOCAL_ROUNDSis replaced by the default of 1. That holds only forGLOBAL_ROUNDS, whichthe platform reads and writes into the server's job configuration.
LOCAL_ROUNDSis read by theapp's own trainer straight out of
config.json, andconfigure_configfills in the default onlywhen the key is absent — a key that is present but invalid is left as written and reaches the
trainer verbatim, so
"LOCAL_ROUNDS": 5000really does run 5000 local iterations. The note nowsplits the two keys, and adds the
BEST_MODEL_METRICexception (a discardedGLOBAL_ROUNDSfailsthe job outright rather than defaulting) and the fact that neither key is rewritten in the
deployed
config.json.base-template name is skipped with only a server-side log line — for Flower that silently drops
server_app.py,strategy.pyand__init__.py, which is exactly what the Flower tutorials ship.And a checkpoint named by
SERVER_CHECKPOINTor an evaluation job'smodelsentries is stagedserver-side and never reaches a client.
pre-trained checkpointshas been dropped as an exampleof what does get shipped, since it is the case with the exception.
by FLIP's own NVFLARE components:
SERVER_CHECKPOINT,GLOBAL_ROUNDS_AE/GLOBAL_ROUNDS_DMandmodels.both-backends bullet list:
flwrrejects a run-config key the template does not declare ratherthan ignoring it, failing the run at submission. Now scoped to NVFLARE, with Flower's behaviour
stated.
upload
server_app.py(the template supplies it, and an uploaded copy is dropped) and listedevaluationamong the job types takingtrainer.py/validator.py/models.py/config.json.Both contradicted the new tables on the same page, so that bullet now points at
fl-required-filesand the obsolete Flower one is gone.Linked Issues
Follow-up to #811. No issue filed yet — happy to open one if you'd prefer this tracked.
Checklist
Type of Change
make -C docs/ docs.Testing
Documentation-only change — no source files touched.
make -C docs/ docsaftermake clean: build succeeds with zero warnings. (When this PR was opened there were 8, all pre-existingautoapi-generated ones; docs: add Security and Governance and compliance pages #850 has since fixed them ondevelop.)fl-apps/<backend>/required_files.json; the config keys, their ranges and defaults againstvalidate_config/upload.pyinfl-services/nvflare/fl-api-base; the client site naming against the kit-slot lookup infl_scheduler_service(confirmed by [Feature]: Save both best and final models — NVFLARE backend #811's own live e2e log, which shows the server reportingTrust_1/Trust_2); the Flower run-config path againstfl-api-flower's--run-confighandling and the base-template bundling infl_service.py; and the UI behaviour againstModelUpload.vue/Training.vue.fl-required-filesreferences resolve, and the rewritten note renders as a proper admonition. Full CI green onbfca0385.Additional Notes
The open question for review is scope, not correctness. This sat as a draft while the claims were
being verified; that work is done and every assertion is now traced to a source file. What remains is
a judgement call that is yours:
The split I've applied assumes the user guide should answer "what do I have to do?" and the FL nodes component page should answer "what exactly does the platform accept?". If you'd rather the detail moved out of ReadTheDocs entirely and lived next to the code (
fl-apps/READMEs, with ReadTheDocs linking to it), that is a smaller edit from here — the two new sections are self-contained and can be lifted wholesale.One thing I deliberately did not do: reconcile this with the "What FLIP actually validates in
config.json" section ofworking-with-flip-apps/package-model-as-map.rst. That section says FLIP reads onlyjob_typeand that everything else is unvalidated, which is true of the Central Hub API but not of the FL API, which does validate the keys documented here. It is accurate in its own frame but reads as contradictory next to the new reference. Worth a follow-up, and worth a second opinion on the right framing before I touch it.