stdenv: PURL fetcher introduction

[h0nIg]

#421125 was merged and reverted later, because of regressions.

the background is described here: #421125 (comment)

@wolfgangwalther outlined the conditions and would like to enhance CI - over time. This is a continuous approach, which is in line with packages which have been found to be defunct and which need a fix. There may be more packages which have problems and we would like to prevent further fallout by a feature flag (prevents accessing + inheritance of drv.src / drv.srcs).

packages list: #453322 (comment)
list of real broken packages: #453322 (comment)
broken packages fix (deferrable PR: #457769)

With the old PR + the broken&platform check fix from #453291 + the feature flag, we enable maintainers to gather experience with PURL and set appropriate information (e.g. jq example, where fetchurl is used instead of fetchFromGithub)

nix-repl> xx = (import /my/nixpkgs {config={derivationPURLInheritance = true;};})
nix-repl> xx.python3Packages.boto3.meta.identifiers
{
  cpeParts = { ... };
  possibleCPEs = [ ... ];
  purl = "pkg:github/boto/boto3@1.40.18";
  purlParts = { ... };
  purls = [ ... ];
  v1 = { ... };
}

nix-repl> xx = (import /my/nixpkgs {})
nix-repl> xx.python3Packages.boto3.meta.identifiers
{
  cpeParts = { ... };
  possibleCPEs = [ ... ];
  purlParts = { ... };
  purls = [ ... ];
  v1 = { ... };
}

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[h0nIg]

we need to squash all commits later, just for transparency let's keep the commits separated in order to understand what has been changed.

[h0nIg]

(I have not digested the design decisions of that part of this PR, but it seems to me it might help to discuss those changes in isolation in their own PR, which we might be able to merge more quickly?)

as agreed in our call together with @raboof, i removed the inheritance and its feature flag

[raboof]

Since the transition to Github Actions we no longer have a CI which performs those checks so this needs to be done manually.

You can see the performance report in the PR summary page: https://github.com/NixOS/nixpkgs/actions/runs/18756048808?pr=454333#summary-53508514968

@wolfgangwalther that link is now dead. This PR was significantly simplified since - would it be possible to do another performance check? How would one do that?

[wolfgangwalther]

would it be possible to do another performance check? How would one do that?

The performance data is available in every CI run. Just click your way to the details page of the CI jobs listed at the bottom of the PR and scroll your way through.

[raboof]

would it be possible to do another performance check? How would one do that?

The performance data is available in every CI run. Just click your way to the details page of the CI jobs listed at the bottom of the PR and scroll your way through.

Gotcha, for the 'Eval Summary' job, so https://github.com/NixOS/nixpkgs/actions/runs/22830597798?pr=454333 - so no noticeable change in speed, and a size increase of 0.05%-0.3%.

[raboof]

Starting to look attractive! Some comments here and there.

[raboof]

"Package URL's shall default to the mkDerivation.src" is no longer true 'in nixpkgs', but consuming tools are expected to make this association (at least for now). That seems like a good start to me.

[h0nIg]

src or srcs is implementation specific, the tool outside should not try to guess which one is the right approach. The interface is drv.meta.identifiers.purl and the implementation (the drv) should define the "where to search". I would like to keep this

[RossComputerGuy]

I am in agreement that we shouldn't push tools to trying to guess. Especially with tools that try to be efficient and can parallelize things, it makes the concurrency more difficult. Imo, it would be great if drv.meta.identifiers.purl defaults to drv.src.meta.identifiers.purl. I know people would be concerned with eval performance.

[h0nIg]

@RossComputerGuy it would be helpful that the SBOM team meets and agrees on an approach. CRA is due in september this year and this should get included in nixos 26.05 - at least the reduced fetcher part

[RossComputerGuy]

I get that, though I don't think this is something we can or should rush. I want PURL's just as much as anyone else does. However, we should do this properly. 26.05 is close but I am not super confident that it's possible at this point to get PURL's in.

I think my concern about drv.src.meta vs drv.meta is something that's big enough (considering that it impacts how the identifier should be discovered & consumed). I don't think it's a bad idea for the SBOM team to sync up. However, it is difficult to get people to sync up for something like this with these sorts of projects. I do think that there should be more of a discussion and thorough review before we attempt to merge this PR.

[raboof]

I agree 'deeper' integration would likely be too ambitious for 26.05, but 'just' being able to populate the drv.meta seems feasible and already a helpful step in the right direction, doesn't it?

[RossComputerGuy]

Yes, as long as whatever is scanning nixpkgs doesn't have to figure out where the identifiers coming from and can just assume drv.meta like with CPE's, I think that is good. With these sorts of things, usually the first NixOS release doesn't have much use of it. Usually it gets flushed out over the following 2 or 3 releases.

The only blocker I see is not needing to figure out if it's drv.src or drv.srcs for the identifiers. Once that's solved, I don't have any objections to merging this PR.

[h0nIg]

populate the drv.meta seems feasible and already a helpful step in the right direction, doesn't it?

@raboof the current PR is feasible. Please keep in mind that despite 95% of purls coming out of the box, it is required to manually maintain the remaining ones. This is something which may land 26.11, but no one is investing into something which is an open PR. My ask would be to merge this as a first step into the right direction, i can then work on contributing missing purl informations - since this takes time.

The only blocker I see is not needing to figure out if it's drv.src or drv.srcs for the identifiers. Once that's solved, I don't have any objections to merging this PR.

@RossComputerGuy blocking the full PR does not make sense IMHO, we may go and land this as a first milestone? Maybe even mark it as experimental, which gives time and others confidence that it may not be "stable"?

[Eveeifyeve]

This looks to good to me, however I would like to see the purl parts be automatically defined by fetchers as well in a followup.

[raboof]

I don't think this PR is perfect (it still doesn't quite make sense to me to split the purlParts rather than using a simple string, and I'm not sure the field name spec makes sense to me).

Nonetheless, having a place to record upstream purls is valuable, so I'm reluctantly approving this PR.

I think we should feel free to possibly refactor the structure later.

[h0nIg]

I think we should feel free to possibly refactor the structure later.

i think we should outline a list of possible enhancements, which will come later once this PR will get merged:

• validation of purl scheme (which requires easy to parse spec to determine the validation schema)
• manual enhancements / maintenance of data in case of fetchurl is used
• further fetcher support getting added (and missing data getting maintained by local ecosystem experts) - but this does not require stdenv changes
• ...