Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ pkg_creator(
|----------|:-:|:-:|:-:|---------|
| `DefaultInfo` | **must** return | — | yes | Defines the executable and runfiles tree. Used as fallback when `RunfilesGroupInfo` is missing or the consumer doesn't support it. |
| `RunfilesGroupInfo` | may return | — | no | Splits `DefaultInfo.default_runfiles.files` into named groups. |
| `RunfilesGroupMetadataInfo` | may return | may add | no | Per-group metadata (rank, do_not_merge, weight) controlling ordering and merge behavior. |
| `RunfilesGroupMetadataInfo` | may return | may add | no | Per-group metadata (rank, do_not_merge, weight, executable_group) controlling ordering, merge behavior, and executable placement. |
| `RunfilesGroupTransformInfo` | — | may add | no | Transforms groups and metadata (e.g., exclude a group, remap names). |

> **Full worked example:** The [`example/`](example/) directory contains a complete end-to-end demo. Look at [`example/producer/`](example/producer/) for `*_binary` rule implementation, [`example/consumer/`](example/consumer/) for packaging rule implementation, and [`example/src/`](example/src/) for user-facing `BUILD` files.
Expand Down Expand Up @@ -95,6 +95,7 @@ Each group can have:
| `rank` | int | 0 | Partial ordering key. Lower rank = earlier in the output. Groups at different ranks are never merged together. |
| `do_not_merge` | bool | False | If True, packaging rules must not merge this group with others. |
| `weight` | int >= 0 or None | None | Hint for merge priority. Lighter groups are merged first when reducing group count. If None, the packager may apply its own default. |
| `executable_group` | bool | False | If True, signals that the packager should place the executable file, runfiles symlinks, repo mapping manifest, and other supporting files for the main entrypoint into this group. Only meaningful at the top level — `collect_groups` strips this bit by default (see below). |

Groups not listed in the metadata dict get default values for all fields (the same applies if `RunfilesGroupMetadataInfo` is missing).

Expand All @@ -109,7 +110,7 @@ providers.append(RunfilesGroupInfo(**groups))
providers.append(RunfilesGroupMetadataInfo(groups = {
"interpreter": lib.group_metadata(rank = -2, do_not_merge = True),
"std": lib.group_metadata(rank = -1),
"app_code": lib.group_metadata(rank = 0, weight = 100),
"app_code": lib.group_metadata(rank = 0, weight = 100, executable_group = True),
}))
```

Expand Down Expand Up @@ -146,6 +147,8 @@ Most rules have the attributes `deps` and `data`. You should implement support f

**`data`** can be arbitrary targets. Some may provide `RunfilesGroupInfo` (e.g., a `*_binary` from a ruleset that supports it), while others won't. Add ungrouped files (when `RunfilesGroupInfo` is missing) to a runfiles group (the default for the current target) so they are not lost.

By default, `collect_groups` strips the `executable_group` bit from all collected metadata entries. This is the correct behavior for `data` deps: when a binary appears as a data dependency of another binary, its `executable_group` annotation is meaningless because the outer binary has its own entrypoint. The top-level `*_binary` target should set `executable_group` on its own group instead.

```starlark
dep_groups = lib.collect_groups(ctx.attr.deps)
data_groups = lib.collect_groups(ctx.attr.data)
Expand All @@ -156,6 +159,11 @@ groups.update(data_groups.groups)
groups["app_code"] = depset(my_own_files, transitive = data_groups.ungrouped)

metadata = lib.merge_metadata(dep_groups.metadata, data_groups.metadata)
# executable_group has been stripped from dep metadata by collect_groups.
# Set it on our own group instead:
metadata = lib.merge_metadata(metadata, RunfilesGroupMetadataInfo(groups = {
"app_code": lib.group_metadata(executable_group = True),
}))
```

### Group count limits
Expand Down Expand Up @@ -210,7 +218,7 @@ When resolving runfiles groups from a binary target, follow this well-defined or

4. **Optionally merge:** If you need to enforce a maximum group count, call `lib.merge_to_limit(runfiles_group_info, metadata_info, max_groups = N)` before ordering. This merges the lightest same-rank groups until the count fits within the limit. Note: packagers may wish to implement their own group merging strategies instead of `lib.merge_to_limit`.

5. **Apply ordering:** Call `lib.ordered_groups(runfiles_group_info, metadata_info)` to get the final ordered list of `(group_name, depset[File])` tuples, sorted by rank.
5. **Apply ordering:** Call `lib.ordered_groups(runfiles_group_info, metadata_info)` to get the final ordered list of `struct(name, files, metadata)` entries, sorted by rank. Each entry has `name` (string), `files` (depset[File]), and `metadata` (the group's metadata struct, or None if no explicit metadata was set for that group). When a group has `metadata.executable_group == True`, the packager should add the executable file, runfiles symlinks, repo mapping manifest, and other supporting files for the main entrypoint to that group's files.

### Using the library

Expand All @@ -237,7 +245,14 @@ for hint in ctx.rule.attr.aspect_hints:

# Order by rank
ordered = lib.ordered_groups(rgi, metadata)
for group_name, files_depset in ordered:
for entry in ordered:
# entry.name: group name (string)
# entry.files: depset[File]
# entry.metadata: group_metadata struct or None
if entry.metadata and entry.metadata.executable_group:
# Add executable, runfiles symlinks, repo mapping manifest
# to this group's layer.
...
# Create a layer / archive entry / etc.
...

Expand All @@ -254,7 +269,11 @@ Note that ordering may not matter for some kinds of packages. In that case, it's

### Packaging the executable file itself along with other supporting files

`RunfilesGroupInfo` only covers the files inside `DefaultInfo.default_runfiles.files`. A well-behaved packager should also handle the remaining pieces of the executable: the binary file itself, the runfiles symlinks, the repo mapping manifest, etc. These are not part of any runfiles group. It is up to the packager to decide where they go — they could be added to an existing group, placed in a dedicated group, or handled out of band entirely.
`RunfilesGroupInfo` only covers the files inside `DefaultInfo.default_runfiles.files`. A well-behaved packager should also handle the remaining pieces of the executable: the binary file itself, the runfiles symlinks, the repo mapping manifest, etc. These are not part of any runfiles group.

When a group's metadata has `executable_group = True`, the packager should add these supporting files to that group. This is the `*_binary` rule's way of saying "this is where my entrypoint lives." If no group is marked `executable_group`, the packager decides where to place them — they could be added to an existing group, placed in a dedicated group, or handled out of band entirely.

The `executable_group` bit is only meaningful at the top level. When a binary appears as a `data` dependency of another binary, the outer binary has its own entrypoint. For this reason, `lib.collect_groups()` strips `executable_group` from collected metadata by default. The `*_binary` rule should set `executable_group` on its own group rather than inheriting it from deps.

---

Expand Down
10 changes: 5 additions & 5 deletions example/consumer/rules/fake_package.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ load(
_FakePackageGroupsInfo = provider(
doc = "Resolved and ordered runfiles groups from the aspect pipeline.",
fields = {
"ordered_groups": "list of (group_name, depset[File]) tuples.",
"ordered_groups": "list of struct(name, files, metadata) entries.",
},
)

Expand Down Expand Up @@ -51,15 +51,15 @@ def _fake_package_impl(ctx):

# Build JSON debug output (list to preserve order).
groups_list = []
for name, files_depset in ordered:
groups_list.append({"group": name, "files": [f.path for f in files_depset.to_list()]})
for entry in ordered:
groups_list.append({"group": entry.name, "files": [f.path for f in entry.files.to_list()]})
json_file = ctx.actions.declare_file(ctx.label.name + ".json")
ctx.actions.write(json_file, json.encode(groups_list))

# Build OutputGroupInfo.
output_groups = {}
for name, files_depset in ordered:
output_groups[name] = files_depset
for entry in ordered:
output_groups[entry.name] = entry.files

return [
DefaultInfo(files = depset([json_file])),
Expand Down
4 changes: 2 additions & 2 deletions example/producer/rules/starlark_binary.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ def _starlark_binary_impl(ctx):
if ctx.attr.runfiles_grouping == "by_target":
groups.update(data_groups.groups)
groups["entrypoint"] = depset(transitive = [entrypoint_files] + data_groups.ungrouped)
metadata["entrypoint"] = lib.group_metadata(rank = 2)
metadata["entrypoint"] = lib.group_metadata(rank = 2, executable_group = True)
for name in data_groups.groups:
dep_weight = _get_dep_weight(dep_metadata, name)
if _extract_repo(name) == own_repo:
Expand Down Expand Up @@ -177,7 +177,7 @@ def _starlark_binary_impl(ctx):
for repo, ds in repo_depsets.items():
groups[repo or "_main"] = depset(transitive = ds)
if repo == own_repo:
metadata[repo or "_main"] = lib.group_metadata(rank = 1, weight = repo_weights.get(repo, None))
metadata[repo or "_main"] = lib.group_metadata(rank = 1, weight = repo_weights.get(repo, None), executable_group = True)
elif repo in repo_weights:
metadata[repo or "_main"] = lib.group_metadata(weight = repo_weights[repo])

Expand Down
54 changes: 47 additions & 7 deletions runfiles_group/private/lib.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,17 @@ lib.group_names(runfiles_group_info)
Returns the list of group names in a RunfilesGroupInfo instance.

lib.ordered_groups(runfiles_group_info, metadata_info = None)
Returns a list of (group_name, depset[File]) tuples, ordered by rank
(ascending). Within the same rank, order is deterministc,
Returns a list of struct(name, files, metadata) entries, ordered by rank
(ascending). name is the group name (string), files is depset[File],
and metadata is the group_metadata struct (or None if no explicit
metadata exists for that group).

Within the same rank, order is deterministc,
but consumers should not rely on intra-rank order.

If metadata_info is None, all groups are included in deterministic order.
Groups not present in metadata get default rank (0).
If metadata_info is None, all groups are included in deterministic order
with metadata set to None.
Groups not present in metadata get None as metadata.

lib.transform_groups(runfiles_group_info, metadata_info = None, transform_info = None)
Applies a transform to (RunfilesGroupInfo, RunfilesGroupMetadataInfo).
Expand All @@ -30,11 +35,16 @@ lib.merge_metadata(*metadatas)
Dict-merges any number of RunfilesGroupMetadataInfo instances (or None).
Returns RunfilesGroupMetadataInfo or None. Per-key last-wins.

lib.collect_groups(deps)
lib.collect_groups(deps, *, strip_executable_group = True)
Extracts RunfilesGroupInfo and RunfilesGroupMetadataInfo from a list of
dependency targets. For deps providing RunfilesGroupInfo, extracts all
groups and metadata. For deps without it, collects
DefaultInfo.default_runfiles.files as ungrouped.
If strip_executable_group is True (default), the executable_group bit
is cleared on all collected metadata entries. This is the correct
default when collecting from data deps: the executable_group annotation
is only meaningful for the top-level *_binary target, not for binaries
that appear as data dependencies of another binary.
Returns struct(groups, metadata, ungrouped) where:
groups: dict[str, depset[File]]
metadata: RunfilesGroupMetadataInfo or None
Expand Down Expand Up @@ -76,7 +86,18 @@ def _ordered_groups(runfiles_group_info, runfiles_group_metadata_info = None):
),
)

return [(name, getattr(runfiles_group_info, name)) for name in ordered]
return [
struct(
name = name,
files = getattr(runfiles_group_info, name),
metadata = (
runfiles_group_metadata_info.groups[name]
if runfiles_group_metadata_info != None and name in runfiles_group_metadata_info.groups
else None
),
)
for name in ordered
]

def _transform_groups(runfiles_group_info, runfiles_group_metadata_info = None, runfiles_transform_info = None):
if runfiles_transform_info == None:
Expand Down Expand Up @@ -133,6 +154,7 @@ def _merge_pair(groups, meta, lighter, heavier, default_weight, merged_group_nam
rank = meta[heavier].rank,
do_not_merge = False,
weight = merged_weight,
executable_group = meta[lighter].executable_group or meta[heavier].executable_group,
)

if merged_group_name_fn != None:
Expand Down Expand Up @@ -194,7 +216,7 @@ def _merge_metadata(*metadatas):
result = RunfilesGroupMetadataInfo(groups = merged)
return result

def _collect_groups(deps):
def _collect_groups(deps, *, strip_executable_group = True):
groups = {}
metadata = None
ungrouped = []
Expand All @@ -206,6 +228,24 @@ def _collect_groups(deps):
metadata = _merge_metadata(metadata, dep[RunfilesGroupMetadataInfo])
else:
ungrouped.append(depset(transitive = [dep[DefaultInfo].files, dep[DefaultInfo].default_runfiles.files]))
if strip_executable_group and metadata != None:
needs_strip = False
for entry in metadata.groups.values():
if entry.executable_group:
needs_strip = True
break
if needs_strip:
stripped = {}
for name, entry in metadata.groups.items():
if entry.executable_group:
stripped[name] = group_metadata(
rank = entry.rank,
do_not_merge = entry.do_not_merge,
weight = entry.weight,
)
else:
stripped[name] = entry
metadata = RunfilesGroupMetadataInfo(groups = stripped)
return struct(groups = groups, metadata = metadata, ungrouped = ungrouped)

lib = struct(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,31 @@ Each entry maps a group name to a struct with:
- do_not_merge (bool): If True, packager must not merge this group. Default False.
- weight (int or None): Hint for merge priority. Lighter groups merge first.
If None, the packager may apply an undefined default. Default None.
- executable_group (bool): If True, signals that the packager should place
the executable file, runfiles symlinks, repo mapping manifest, and other
supporting files for the main entrypoint into this group. Default False.

Groups not present in the dict are treated as having default metadata
(rank=0, do_not_merge=False, weight=None).
(rank=0, do_not_merge=False, weight=None, executable_group=False).
"""

_DEFAULT_RANK = 0
_DEFAULT_DO_NOT_MERGE = False
_DEFAULT_WEIGHT = None
_DEFAULT_EXECUTABLE_GROUP = False

def group_metadata(*, rank = _DEFAULT_RANK, do_not_merge = _DEFAULT_DO_NOT_MERGE, weight = _DEFAULT_WEIGHT):
def group_metadata(*, rank = _DEFAULT_RANK, do_not_merge = _DEFAULT_DO_NOT_MERGE, weight = _DEFAULT_WEIGHT, executable_group = _DEFAULT_EXECUTABLE_GROUP):
"""Creates a validated group metadata struct.

Args:
rank: Partial ordering key. Lower rank = earlier. Default 0.
do_not_merge: If True, packager must not merge this group. Default False.
weight: Merge priority hint (int >= 0 or None). Default None.
executable_group: If True, the packager should place the executable
and supporting files into this group. Default False.

Returns:
A struct with rank, do_not_merge, and weight fields.
A struct with rank, do_not_merge, weight, and executable_group fields.
"""
if type(rank) != "int":
fail("group_metadata: rank must be an int, got ", type(rank))
Expand All @@ -41,7 +47,9 @@ def group_metadata(*, rank = _DEFAULT_RANK, do_not_merge = _DEFAULT_DO_NOT_MERGE
fail("group_metadata: weight must be an int or None, got ", type(weight))
if weight < 0:
fail("group_metadata: weight must be >= 0, got ", weight)
return struct(rank = rank, do_not_merge = do_not_merge, weight = weight)
if type(executable_group) != "bool":
fail("group_metadata: executable_group must be a bool, got ", type(executable_group))
return struct(rank = rank, do_not_merge = do_not_merge, weight = weight, executable_group = executable_group)

_DEFAULT_METADATA = group_metadata()

Expand All @@ -50,12 +58,14 @@ def _normalize_entry(name, entry):
rank = getattr(entry, "rank", _DEFAULT_RANK)
do_not_merge = getattr(entry, "do_not_merge", _DEFAULT_DO_NOT_MERGE)
weight = getattr(entry, "weight", _DEFAULT_WEIGHT)
return group_metadata(rank = rank, do_not_merge = do_not_merge, weight = weight)
executable_group = getattr(entry, "executable_group", _DEFAULT_EXECUTABLE_GROUP)
return group_metadata(rank = rank, do_not_merge = do_not_merge, weight = weight, executable_group = executable_group)
if type(entry) == "dict":
return group_metadata(
rank = entry.get("rank", _DEFAULT_RANK),
do_not_merge = entry.get("do_not_merge", _DEFAULT_DO_NOT_MERGE),
weight = entry.get("weight", _DEFAULT_WEIGHT),
executable_group = entry.get("executable_group", _DEFAULT_EXECUTABLE_GROUP),
)
fail("RunfilesGroupMetadataInfo: entry for group '{}' must be a struct or dict, got {}".format(name, type(entry)))

Expand All @@ -72,8 +82,8 @@ RunfilesGroupMetadataInfo, _ = provider(
init = _make_runfilesgroupmetadatainfo_init,
fields = {
"groups": """\
A dict mapping group name (string) to a struct with rank, do_not_merge, and weight fields.
Groups not present get default metadata (rank=0, do_not_merge=False, weight=None).
A dict mapping group name (string) to a struct with rank, do_not_merge, weight, and executable_group fields.
Groups not present get default metadata (rank=0, do_not_merge=False, weight=None, executable_group=False).
""",
},
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ def _test_one(ctx, binary_attr):
)

ordered = lib.ordered_groups(rgi, metadata)
actual_names = [name for name, _ in ordered]
actual_names = [entry.name for entry in ordered]

if ctx.attr.expected_group_names:
if actual_names != ctx.attr.expected_group_names:
Expand All @@ -132,6 +132,14 @@ def _test_one(ctx, binary_attr):
_INDENT + str(actual_names),
)

executable_groups = [entry.name for entry in ordered if entry.metadata and entry.metadata.executable_group]
if len(executable_groups) > 1:
success = False
issues.append(
"at most one group may set executable_group = True, but found {}:\n".format(len(executable_groups)) +
"\n".join([_INDENT + name for name in executable_groups]),
)

return (success, issues)

def _runfiles_group_analysis_test_impl(ctx):
Expand Down
Loading