Milestone: Eliminate reliance on `npm`

[savetheclocktower]

Have you checked for existing feature requests?

• Completed

Summary

ppm relies on npm to do a lot of grunt work. But since ppm bundles its own Node, it also must bundle its own npm.

For reasons that I’m sure are complex and better left to @DeeDeeG to explain (or decide not to explain to avoid exhaustion), we maintain our own fork of the npm-cli repository. Of the commits that exist in that repo, I can identify ten that are definitely unique to our fork, and they suggest that the reasons to maintain an ongoing fork of npm-cli include

• bumping node-gyp occasionally, and
• bumping libnpm, libcipm, and npm-lifecycle (three dependencies of npm-cli that are themselves deprecated in favor of packages I'll mention below).

This is not the world’s most onerous chore, but there’s only one of @DeeDeeG, and I’d love to use their talents as politely as possible.

Can we outright eliminate dependence on npm? Let’s think it through.

How do we use NPM?

These are the places in the codebase I can identify where we explicitly shell out to npm:

• ci.js shells out to npm ci
• clean.js shells out to npm prune
• config.js shells out to npm config
• dedupe.js shells out to npm dedupe
• install.js shells out to npm install
• publish.js shells out to npm version
• rebuild.js shells out to npm rebuild

There are other places where we use npm more like a library — require('npm'), then call methods on it — but these mainly involve reading configuration. Using npm in this manner is also deprecated and is worth replacing if we can.

So we can group these into tasks and then figure out other ways to accomplish the same tasks:

Dependency installation/cleanup

ci.js and install.js use npm for managing node_modules and reconciling it with package.json. We can use arborist for this; see #159.

arborist can also probably do the stuff done by npm prune, and I imagine by npm dedupe as well.

Reading NPM configuration

config.js delegates to npm for reading from and writing to configuration, since ppm has a need for many of the configuration keys that npm does, and wants to manage them the same way.

We could instead manage this with @npmcli/config.

Building/rebuilding modules

ppm famously needs to compile packages with native module dependencies, so that’s why it shells out to npm rebuild. But npm itself mainly delegates this task to node-gyp, so we could do the same.

That said: build scripts don’t actually have to use node-gyp. So what we really want here is the ability to run arbitrary package scripts, including build and rebuild — hence we also want the @npmcli/run-script package.

Creating tags and commits

Publishing a Pulsar package is quite different from publishing an NPM package, so publish.js doesn’t actually call npm publish. But it does leverage npm version to handle the chore of bumping a version number and generating a Git commit and tag.

This doesn’t have a direct analog in library form. But the version incrementing is simple enough as to be achievable via the semver library, and the Git-related tasks are straightforward as well. We could just shell out to git for those, since we require it for package publishing anyway.

What benefits does this feature provide?

Atom never seemed to have their own fork of npm — so why did we have to fork it? The original PR says it was to allow us to bump node-gyp to 9.4.

I don't have enough context to know why forking was necessary to achieve that goal. But I do know that ppm also depends directly on node-gyp, so it would make things simpler if we didn't have to bump it in two different places.

Any alternatives?

Not realistically. The options are the status quo (ad-hoc usage of npm for all its management tasks) and the more modern approach of relying on single-purpose libraries that offer more hygienic access to the same functionality.

Other examples:

No response

[DeeDeeG]

Explaining past nonsense and shenanigans:

The forking of npm was indeed just a hassle, a hoop I needed to jump through to get the node-gyp bump to actually stick... in the sense of our copy of npm actually getting the desired (bumped) version of node-gyp under it in the right place(s) under ppm/node_modules/ and/or ppm/node_modules/npm/node_modules/. (Given the vagaries of package deduplication and/or hoisting up/down the dependency tree.)

Top-level dependency on node-gyp is for specs purposes only.

---

Longer, optional explanation of "why do we depend on node-gyp directly in package.json and then proceed to not use it via a direct require('node-gyp') in production?"

Details

tl;dr: it's a specs-only dependency, basically. And trying to wrangle the hoisting algorithm of npm/yarn to put the dep in a predictable location for said spec's sake -- I forget if we still need to do that or not, TBH.

IIRC, ppm's top-level dependency on node-gyp (direct dependency on node-gyp as listed in package.json) is also a bit of a kludge. Or put differently, it is a spec-only thing. Kinda should be a devDependency I guess, IDK. It is an attempt to help wrangle the install location of node-gyp in the hoisting logic under node_modules to be as top-level consistent as possible. Since its location is expected to be at a hard-coded location relative to the repo root in the setup phase of a certain spec. The node-gyp library files have to be copied from that fixed location to another, to set up for running the actual test. Or something.

Technically speaking: the way the spec is currently written, require.resolve('node-gyp') needs to work for this spec to pass:

ppm/spec/install-spec.js

Lines 586 to 598 in d288b09

	describe("with a space in node-gyp's path", () => {
	const nodeModules = fs.realpathSync(path.join(__dirname, '..', 'node_modules'));
	beforeEach(() => {
	// Normally npm_config_node_gyp would be ignored, but it works here because we're calling apm
	// directly and not through the scripts in bin/
	const nodeGypPath = path.dirname(path.dirname(require.resolve('node-gyp'))); // find an installed node-gyp
	fs.copySync(nodeGypPath, path.join(nodeModules, 'with a space'));
	process.env.npm_config_node_gyp = path.join(nodeModules, 'with a space', 'bin', 'node-gyp.js');
	// Read + execute permission
	return fs.chmodSync(process.env.npm_config_node_gyp, fs.constants.S_IRUSR | fs.constants.S_IXUSR);
	});

*Nota Bene: I don't believe we actually require('node-gyp') anywhere in the repo!!! If it weren't for this one weird spec threatening to redden/fail CI runs, we would not have that node-gyp direct dependency in package.json.

There's probably a better way. Like downloading a spec-specific copy of node-gyp during the spec's setup phase and putting it where we want it or some such. IDK.

---

Longer, optional explanation of "Why did @DeeDeeG fork npm?" below:

Details

IIRC, we had to fork because OG npm repo, circa npm 6, was set up in a rather "unique" way where many dependencies were checked in on-disk. This seemed to allow them to do certain things in CI and simplify their bootstrapping of npm repo to be ready to install the rest of its own dependencies or some such. So the repo is self-hosting enough to only need a NodeJS binary and the checked-in packages in the repo to finish building the rest of npm -- a standalone npm was not needed to hydrate the repo, if you will.

Because of several dependencies being checked-in on-disk, overrides such as yarn's resolutions: field of package.json were insufficient to get the new copy of node-gyp to live under ppm/node_modules/npm/node_modules -- a stale copy would persist, no matter what we might do anywhere in ppm repo files without a modified npm that stopped checking in so many dependencies.

The bumps of sub-dependencies of npm was partly a failed experiment. I forked all of those packages to depend on a newer node-gyp. And bumped to our forked packages to get the newer node-gyp in our npm fork. In the end that somehow still wasn't enough. I/we ended up having to un-check-in all of those files in our forked npm and just rely on yarn resolutions in ppm repo anyhow. See:

ppm/package.json

Line 61 in d288b09

"npm/**/node-gyp": "^10.2.0",

tl;dr: strictly to make the node-gyp bump work. Nothing more or less.

---

tl;dr lotta kludges, weirdnesses, bribing the technical debt collector to look the other way, at least for the moment. Mistakes were arguably made. These hacky solutions could be said to have been expedient, perhaps. But tinkering around the edges where a better fix might have been warranted? Absolutely.

Floating replacing much of our npm use with arborist and such is an idea that has been tossed around for a long while, and somewhat often. This inventory of the needs/steps to do so looks the most complete, though! It sounds doable. Pleased to chip in as/when I am able.

[DeeDeeG]

We no-longer have to touch the npm fork, since it has been made compatible with yarn resolutions: field in this repo as a mechanism to bump node-gyp. That chapter of fiddling with that is over for now.

But getting rid of full-fat npm would be a blessing to this repo. Many older npm@6 era dependencies are not amenable to running on newer node or perhaps electron if we ever make all of ppm's native C/C++ deps compatible with [building for, and] running under an electron runtime rather than a node one.

Besides the directness and control of asking libs to do what we want them to do, rather than having to shell out to a large and complex standalone CLI program (npm).

[savetheclocktower]

OK, this will go on my list. A sanity check I did last night with arborist and pacote worked well, and managed to pass at least some of the install specs while also eliminating the requirement to shell out to npm. The next thing I ought to verify is that native module building works — hence getting the specs around ppm install --check to pass.

If we gain the ability to run scripts directly without going through npm (via run-script) then that might create a new need for our direct dependence on node-gyp, making it less of a kludge.