-
Notifications
You must be signed in to change notification settings - Fork 24
Support semver version ranges for dependencies #311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
heatonmatthew
wants to merge
24
commits into
helsing-ai:main
Choose a base branch
from
heatonmatthew:versioning
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
de6d421
feat(cli): [WIP] Support semantic version ranges.
heatonmatthew ce2b42f
feat(cli): [WIP] Additional unit tests for semantic version ranges.
heatonmatthew 99ed6e7
feat(cli): Update documentation
heatonmatthew 95326c5
fix(cli): Incomplete handling of parts in "name version" Deserialize
heatonmatthew a9ae96c
fix(cli): Incorrect handling of CWD when installing a workspace.
heatonmatthew 65ce199
chore: fix lint fmt
heatonmatthew 46604d8
chore(cli): re-add debug traces for Artifactory calls
heatonmatthew d33ebc7
doc: Re-add incorrectly removed content
heatonmatthew 19596ee
Remove unused function.
heatonmatthew 09e53cf
Tidy-up imports
heatonmatthew 5f14c66
chore: ignore .worktrees/ directory
heatonmatthew 8f3db35
manifest: sort dependencies_as_vec output deterministically
heatonmatthew 7964fae
style: remove redundant type annotation in dependencies_as_vec
heatonmatthew d3289fc
resolver: replace VersionConflict with NoCompatibleVersion and Lockfi…
heatonmatthew 7f7238e
fix: scope unused_assignments allow to file level for thiserror v2 co…
heatonmatthew 8134655
resolver: drop LockfileStale variant
heatonmatthew 9c0d86e
resolver: replace recursion with worklist + per-package requirement m…
heatonmatthew a416a81
test: cascade re-resolution when a package's chosen version changes
heatonmatthew a6deab1
test: lockfile pins that fail merged requirements are re-resolved
heatonmatthew 7c4fcd1
docs: describe merge-and-resolve version selection
heatonmatthew 308a0cf
test: cover retraction cascade and survivor rescheduling directly
heatonmatthew c1c8e41
Merge remote-tracking branch 'upstream/main' into versioning
heatonmatthew 4d4a09f
test: lockfile pins are not re-resolved for range criteria
heatonmatthew ff08773
doc: keep examples in original single line format
heatonmatthew File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,3 +7,4 @@ target | |
| .DS_Store | ||
| .vscode | ||
| .envrc | ||
| .worktrees/ | ||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,81 @@ | ||
| # Dependency Resolution | ||
|
|
||
| When you run `buffrs install`, the resolver builds a complete dependency graph | ||
| for your project — including all transitive dependencies — and determines the | ||
| concrete version to install for each package. | ||
|
|
||
| ## Resolution algorithm | ||
|
|
||
| For each dependency (direct or transitive), the resolver follows this priority | ||
| order: | ||
|
|
||
| 1. **Lockfile hit** — if `Proto.lock` records a version of the package that | ||
| satisfies *every* requirement gathered for it so far, that version is used | ||
| immediately without contacting the registry. This makes repeated installs | ||
| fast and reproducible. A pin that does not satisfy them all is simply not a | ||
| usable answer, and resolution falls through to the registry. | ||
|
|
||
| 2. **Registry resolution** — if no locked version qualifies, the resolver | ||
| queries the registry for all available versions of the package, then selects | ||
| the **highest** version that satisfies **all** of the requirements at once. | ||
|
|
||
| 3. **Download and cache** — the resolved version is downloaded, stored in the | ||
| local cache, and its digest is recorded in the lockfile for future installs. | ||
|
|
||
| Transitive dependencies are discovered by reading the `Proto.toml` bundled | ||
| inside each downloaded package archive, then resolved using the same steps | ||
| above. Because a package's full set of requirements is only known once every | ||
| path to it has been walked, choosing a lower version can retract dependencies | ||
| that a previously-chosen higher version had pulled in; those become unreachable | ||
| and are dropped from the graph. | ||
|
|
||
| ## Version conflict detection | ||
|
|
||
| If the same package is required by more than one path in the dependency tree, | ||
| the resolver merges every requirement and picks the highest version satisfying | ||
| all of them together. Requirements are never evaluated pairwise or in | ||
| encounter order, so a later, tighter requirement can lower an earlier pick | ||
| rather than conflicting with it. | ||
|
|
||
| The install fails only when the intersection is empty — no published version | ||
| satisfies every requirement: | ||
|
|
||
| ``` | ||
| no version of leaf-lib satisfies all requirements: [^1.0.0, ^2.0.0]; | ||
| available versions: [2.0.0, 1.0.0] | ||
| ``` | ||
|
|
||
| To fix a conflict, update the requiring packages so their version requirements | ||
| overlap, or introduce a package that bridges the incompatible requirements. | ||
|
|
||
| Note that this conflict detection operates **within a single package's | ||
| dependency graph**. In a workspace, different members may independently resolve | ||
| different versions of the same package — the workspace lockfile records them | ||
| separately using a `(name, version)` composite key. | ||
|
|
||
| ## Workspace resolution | ||
|
|
||
| In a workspace, each member package's dependency graph is resolved | ||
| independently. The workspace lockfile (`Proto.lock` at the workspace root) | ||
| accumulates all resolved packages across all members. Because the workspace | ||
| lockfile allows multiple versions of the same package, two members that require | ||
| incompatible versions of a shared library can co-exist. | ||
|
|
||
| If a subsequent install finds a workspace lockfile, it reuses those locked | ||
| versions (subject to satisfying each member's requirements) to avoid redundant | ||
| registry queries. | ||
|
|
||
| ## Topological ordering | ||
|
|
||
| After the full graph is built, packages are sorted topologically so that each | ||
| dependency is installed before its dependants. This guarantees that vendored | ||
| proto sources are available in the correct order during compilation. | ||
|
|
||
| ## Determinism and the lockfile | ||
|
|
||
| The resolver always picks the **highest** version satisfying every requirement | ||
| when multiple candidates exist. This is deterministic given the same set of available | ||
| registry versions. Once a version is recorded in `Proto.lock`, it is used | ||
| as-is on all subsequent installs, regardless of newer versions that may have | ||
| been published since. Run `buffrs install` after deleting or modifying | ||
| `Proto.lock` to re-resolve against the current registry state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,88 @@ | ||
| # SemVer Compatibility | ||
|
|
||
| buffrs uses [Semantic Versioning](https://semver.org/) for all packages. | ||
| Version requirements in `Proto.toml` follow the same syntax as | ||
| [Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html). | ||
|
|
||
| ## Version requirement syntax | ||
|
|
||
| A version requirement is placed in the `version` field of a dependency: | ||
|
|
||
| ```toml | ||
| [dependencies.my-lib] | ||
| version = "^1.2.0" | ||
| registry = "https://my-registry.example.com" | ||
| repository = "my-repo" | ||
| ``` | ||
|
|
||
| The following operators are supported: | ||
|
|
||
| ### Caret (`^`) — default for ranges | ||
|
|
||
| Allows minor and patch updates within the same major version. | ||
| This is the recommended operator for most dependencies. | ||
|
|
||
| | Requirement | Resolves versions | | ||
| |-------------|-------------------| | ||
| | `^1.2.3` | `>=1.2.3, <2.0.0` | | ||
| | `^1.2` | `>=1.2.0, <2.0.0` | | ||
| | `^1` | `>=1.0.0, <2.0.0` | | ||
| | `^0.2.3` | `>=0.2.3, <0.3.0` | | ||
| | `^0.0.3` | `>=0.0.3, <0.0.4` | | ||
|
|
||
| Note that `0.x` versions are treated as unstable: `^0.2` only allows `0.2.x`, | ||
| not `0.3.x`, since breaking changes are expected in pre-1.0 packages. | ||
|
|
||
| ### Tilde (`~`) — patch-level updates only | ||
|
|
||
| Allows patch updates within the same minor version. | ||
|
|
||
| | Requirement | Resolves versions | | ||
| |-------------|-------------------| | ||
| | `~1.2.3` | `>=1.2.3, <1.3.0` | | ||
| | `~1.2` | `>=1.2.0, <1.3.0` | | ||
| | `~1` | `>=1.0.0, <2.0.0` | | ||
|
|
||
| ### Exact (`=`) — pin to a specific version | ||
|
|
||
| Resolves to exactly the stated version, with no flexibility. | ||
|
|
||
| ```toml | ||
| version = "=1.2.3" | ||
| ``` | ||
|
|
||
| Use exact pins when you need bit-for-bit reproducibility in the manifest | ||
| itself, or when you are distributing a library whose consumers should be | ||
| in full control of the version. | ||
|
|
||
| ### Comparison operators | ||
|
|
||
| For more control, the standard comparison operators are available: | ||
|
|
||
| | Requirement | Meaning | | ||
| |-----------------|----------------------------------| | ||
| | `>=1.2.0` | Any version at or above 1.2.0 | | ||
| | `>1.2.0` | Any version strictly above 1.2.0 | | ||
| | `<2.0.0` | Any version strictly below 2.0.0 | | ||
| | `<=2.0.0` | Any version at or below 2.0.0 | | ||
| | `>=1.0.0, <2.0.0` | Intersection (multiple constraints) | | ||
|
|
||
| ## How the resolver picks a version | ||
|
|
||
| When a requirement matches more than one available version, buffrs always | ||
| selects the **highest** satisfying version. The resolved concrete version is | ||
| written to `Proto.lock` to ensure reproducible installs — re-running | ||
| `buffrs install` will use the locked version rather than querying the registry | ||
| again. | ||
|
|
||
| See [Dependency Resolution](./resolver.md) for a full description of the | ||
| resolution algorithm. | ||
|
|
||
| ## Choosing between pinning and ranges | ||
|
|
||
| | Situation | Recommended style | | ||
| |-----------|-------------------| | ||
| | Public library — let consumers decide | `^1.0.0` | | ||
| | Internal service — stable dependency set | `^1.0.0` or `~1.2.0` | | ||
| | Security patch must be applied exactly | `=1.2.5` | | ||
| | Compatibility ceiling known | `>=1.0.0, <3.0.0` | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we keep the removed lines:
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. Fixed.
Please check you're happy.