Add Hydra composition for GiGL configs - #726
Conversation
…mposition # Conflicts: # pyproject.toml
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop a redundant genericity test and an implementation-pinning assertion, and assert compiled-pipeline wiring structurally instead of matching the KFP compiler's text layout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Forcing enable_caching=False re-ran composition on every relaunch, so a retry could overwrite the job's resolved snapshots with different dynamically resolved values while downstream cached steps kept the old ones. Recompose by launching with a new job name or with run-level caching disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every YAML read now composes; plain configs and materialized snapshots compose to themselves. compose_yaml_config snapshots and restores any foreign GlobalHydra context instead of refusing to run, so reads work inside a user application under @hydra.main. Local files not named *.yaml are staged like remote URIs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![semgrep-code-snapchat[bot]](https://avatars.githubusercontent.com/u/40505220?s=60&v=4){kind=small}
| file_uri_dst=local_uri, | ||
| ) | ||
| obj_dict = compose_yaml_config(uri=local_uri) | ||
| proto = ParseDict(js_dict=obj_dict, message=proto_cls()) |
There was a problem hiding this comment.
High severity and reachable issue identified in your code:
Line 65 has a vulnerable usage of protobuf, introducing a high severity vulnerability.
ℹ️ Why this is reachable
A reachable issue is a real security risk because your project actually executes the vulnerable code. This issue is reachable because your code uses a certain version of protobuf.
Affected versions of protobuf are vulnerable to Uncontrolled Recursion. A denial-of-service vulnerability in the Python protobuf library's JSON parser allows deeply nested google.protobuf.Any messages to bypass the configured max_recursion_depth in json_format.ParseDict. Because the internal Any-handling logic does not update the recursion counter, an attacker supplying a JSON payload with repeatedly nested Any messages can exhaust Python's recursion stack (raising RecursionError) instead of a controlled ParseError, potentially crashing or disrupting services that parse untrusted JSON.
To resolve this comment:
Upgrade this dependency to at least version 5.29.6 at uv.lock.
💬 Ignore this finding
To ignore this, reply with:
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
You can view more details on this finding in the Semgrep AppSec Platform here.
proto_to_yaml re-serializes the parsed protobuf, so source comments cannot survive composition. Prepend a provenance comment naming the source config instead, prefixed with the docker image when the source is a container-local path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
/all_test |
GiGL Automation@ 17:42:32UTC : 🔄 @ 17:44:34UTC : ✅ Workflow completed successfully. |
GiGL Automation@ 17:42:34UTC : 🔄 @ 19:20:31UTC : ✅ Workflow completed successfully. |
GiGL Automation@ 17:42:35UTC : 🔄 |
GiGL Automation@ 17:42:36UTC : 🔄 @ 19:36:12UTC : ❌ Workflow failed. |
GiGL Automation@ 17:42:36UTC : 🔄 @ 17:52:37UTC : ✅ Workflow completed successfully. |
GiGL Automation@ 17:42:36UTC : 🔄 @ 17:48:58UTC : ❌ Workflow failed. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Why
GiGL configs today are monolithic YAML files, so anything shared between pipelines has to be copy-pasted. Our own e2e resource configs carried three near-identical copies of the same infrastructure and preprocessing sections, and they drifted apart the way copies always do. Users maintaining families
of related pipelines (per-region, per-experiment, per-environment) face the same problem at a larger scale.
This PR lets task and resource configs use Hydra Defaults Lists so shared fragments live in one place and each pipeline's config declares only what makes it different. Dynamic values (dates, env vars, git hashes) resolve through OmegaConf resolvers instead of being hand-edited before every launch.
Design decisions
result is what keeps a pipeline run reproducible and every component in agreement about its config. Retries reuse the frozen snapshots rather than re-resolving (the validator follows run-level KFP caching).
pipeline. Composition also snapshots and restores any active Hydra context, so GiGL reads keep working inside user applications that use Hydra themselves.
traced back.
The e2e resource configs are migrated to shared fragments as the in-repo example. A new user guide (docs/user_guide/config_guides/hydra_composition.md) covers authoring, remote configs, and the pipeline behavior.
Testing