enable capywfa to detect sources available locally

[samastek]

CaPyWfa has 6 passes. The first three are:

1. Map BOM to SW360: Matches packages against existing SW360 entries and produces a new SBOM with match results.

2. Verify SW360 sources: Checks whether packages found in SW360 have source files attached and verified.

3. Identify packages needing download: Identifies packages that are either found as unapproved in SW360 or not found at all.

Currently, pass 3 exits with error code 80 if there are sources that must be downloaded.

The Problem

Our workflow starts with all source packages already present in a local directory (whose path is passed via --sources), but pass 3 is exiting without detecting the local sources.

The Suggestion

After examining the code, I think we can update the pass3_download_sources method to search for each source package. When a match is found locally under the specified sources dir, the method:

• Adds an external reference to the SBOM pointing to the local file path
• Sets SourceFileDownload to "skip" to prevent the exit code 80

[gernot-h]

Thanks for your idea! The problem is that I didn't think about this kind of workflow yet, the CaPyWfa design assumed that you need to download the sources during clearing CI jobs and this step would add the necessary information to the SBOM.

Now, my main question is, where does your SBOM and your downloads come from? Because what CaPyWfa expects is

• one source file per SW360 release. In Siemens, we e.g. use one SW360 release per Debian "source package", that means we have to combine the Debian orig and patches files into one before SW360 uploads.
• external references to your sources,
• (optionally?) SHA1 hashes so we can decide whether an existing SW360 attachment is identical. This assumes that a possible process to combine sources is reproducible
• a capycli:sourceFileComment used as attachment description when uploading your sources. The idea is the for cases where you had to post-process your sources, e.g. to combine them into one, the comment should describe how you did that.

And I don't think CaPyWfa should populate these fields, but your SBOM should already provide them, as CaPyWfa could only guess which files belong to which component and has no idea how the source files were create.

[gernot-h]

So in a nutshell: The current workflow assumes you first run capywfa to identify missing sources (or sources missing verification), then download them and then re-run capywfa with an extended SBOM. However, if you provide the extended SBOM already in the beginning, it should run through all steps in one pass (untested), if not, we need to fix that.

Also, we definitely lack

• documentation what users need to add to the SBOM so CaPyWfa can pickup the sources
• a reference implementation for source download. My main use case is debian, so the reference implementation will likely be based on https://github.com/siemens/debsbom/.

[gernot-h]

We should probably also add a new cmdline option to provide the comment text so we don't need a proprietary property in the SBOM for it.

[samastek]

We have a package list and we generate the SBOM using the provided script lst_to_sbom.py. our source packages are being downloaded from a generic registry. we selfhost an instance of nexus sonatype. and it does not provide a debian src registry to push and download the sources from using apt-get. So we fetch the sources using an external tool.

[gernot-h]

I think lst_to_sbom.py is only useful for the 1st step in my workflow where you start with capywfa in an empty environment as it can't generate the necessary source information which is a heavy domain (e.g. Debian) specific task.

I guess you already have some package lists floating around in your tools, so you probably have to extend those to generate an SBOM which has all necessary information for CaPyWfa as outlined above. If you want to avoid adding the ugly proprietary capycli:sourceFileComment property, we can think about a new cmdline parameter. Or probably there's even a standard solution available using latest CycloneDX features, need to think about that first.

[gernot-h]

Side note: Our Siemens process expects the patches to be applied when creating a combined source package as done by apt-get source. So if you can't run apt-get in your environment, you should look at how https://github.com/siemens/debsbom/ does it (iirc, they also don't use apt-get) or probably even check if debsbom can't do that for you.

[samastek]

The way I understand your comment is we should implement an extra tool that would take a package-list as input and does what lst_to_sbom.py already does to generate an sbom and then extend it by setting external references pointing to the pkg directory where the sources are?

[gernot-h]

If you don't have internal tooling already which does detailed handling of source packages and has the needed data anyways, then I'd really recommend to have a look at https://github.com/siemens/debsbom/.

If I remember correctly, it also accepts a simple package list as input, and should extend it with all the required information. As of now, I'd assume that capycli:sourceFileComment should be the only missing bit when using its output as input for CaPyWfa, but I didn't have time to test it yet, unfortunately. But I'm definitely willing to help with either adding the missing bits parts in CaPyWfa or requesting missing features in debsbom if any.