Skip to content

feat(cli): add progress bar to generation pipeline #680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shivxmsharma wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat(cli): add progress bar to generation pipeline #680

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
57a244e
feat(cli): add progress bar to generation pipeline
shivxmsharma Mar 15, 2026
806c0bf
Update src/utils/configuration/types.d.ts
shivxmsharma Mar 19, 2026
87b8843
Update src/generators.mjs
shivxmsharma Mar 19, 2026
b74b39e
refactor: address review feedback - use PassThrough stream, remove op…
shivxmsharma Mar 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions bin/commands/generate.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ const { runGenerators } = createGenerator();
* @property {string} index
* @property {boolean} minify
* @property {string} typeMap
* @property {boolean} progress
*/

export default new Command('generate')
Expand Down Expand Up @@ -58,6 +59,7 @@ export default new Command('generate')
.addOption(new Option('--index <url>', 'index.md URL or path'))
.addOption(new Option('--minify', 'Minify?'))
.addOption(new Option('--type-map <url>', 'Type map URL or path'))
.addOption(new Option('--no-progress', 'Disable the progress bar'))

.action(
errorWrap(async opts => {
Expand Down
142 changes: 63 additions & 79 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
"@rollup/plugin-virtual": "^3.0.2",
"@swc/html-wasm": "^1.15.18",
"acorn": "^8.16.0",
"cli-progress": "^3.12.0",
Copy link
Copy Markdown
Member

@ovflowd ovflowd Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last release was 3y ago, although I'm fine with not having releases that often, any particular reason why this package instead of others?

Copy link
Copy Markdown
Contributor Author

@shivxmsharma shivxmsharma Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems stable and mature — no breaking changes or open security issues. The last release is just because there's nothing to fix. Open to switching if you have a preference though!

"commander": "^14.0.3",
"dedent": "^1.7.1",
"estree-util-to-js": "^2.0.0",
Expand Down
18 changes: 12 additions & 6 deletions src/generators.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import logger from './logger/index.mjs';
import { isAsyncGenerator, createStreamingCache } from './streaming.mjs';
import createWorkerPool from './threading/index.mjs';
import createParallelWorker from './threading/parallel.mjs';
import createProgressBar from './utils/progressBar.mjs';

const generatorsLogger = logger.child('generators');

Expand Down Expand Up @@ -92,7 +93,7 @@ const createGenerator = () => {
/**
* Runs all requested generators with their dependencies.
*
* @param {import('./utils/configuration/types').Configuration} options - Runtime options
* @param {import('./utils/configuration/types').Configuration} configuration - Runtime options
* @returns {Promise<unknown[]>} Results of all requested generators
*/
const runGenerators = async configuration => {
Expand All @@ -111,6 +112,9 @@ const createGenerator = () => {
await scheduleGenerator(name, configuration);
}

const progress = createProgressBar({ enabled: configuration.progress });
progress?.start(generators.length, 0, { phase: 'Starting...' });
Copy link
Copy Markdown
Member

@ovflowd ovflowd Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why progress?.?

Copy link
Copy Markdown
Member

@avivkeller avivkeller Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the user disabled the progress bar, it'll be undefined

Copy link
Copy Markdown
Member

@ovflowd ovflowd Mar 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ehh, I'd argue that

if (!enabled || !process.stderr.isTTY) {
return null;
}

Shouldn't be inside createProgressBar, what's the point of calling createProgressBar just to return null? Id recommend instead of not creating the instance, simply making a mock stream.

so:

import { PassThrough } from 'node:stream';

// ...
stream: shouldEnable ? process.stdout : new PassThrough();

Copy link
Copy Markdown
Contributor Author

@shivxmsharma shivxmsharma Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion, done! Now using a PassThrough stream when disabled instead of returning null. Much cleaner.


// Start all collections in parallel (don't await sequentially)
const resultPromises = generators.map(async name => {
let result = await cachedGenerators[name];
Expand All @@ -119,14 +123,16 @@ const createGenerator = () => {
result = await streamingCache.getOrCollect(name, result);
}

progress?.increment({ phase: name });
Copy link
Copy Markdown
Member

@ovflowd ovflowd Mar 20, 2026 •
edited by AugustinMauroy
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why progress?.?

return result;
});

const results = await Promise.all(resultPromises);

await pool.destroy();

return results;
try {
Copy link
Copy Markdown
Member

@ovflowd ovflowd Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this change 🙇

Copy link
Copy Markdown
Contributor Author

@shivxmsharma shivxmsharma Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, reverted! 👍

return Promise.all(resultPromises);
} finally {
await pool.destroy();
progress?.stop();
}
};

return { runGenerators };
Expand Down
2 changes: 2 additions & 0 deletions src/utils/configuration/__tests__/index.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ describe('config.mjs', () => {
target: 'json',
threads: 4,
chunkSize: 5,
progress: true,
};

const config = createConfigFromCLIOptions(options);
Expand All @@ -112,6 +113,7 @@ describe('config.mjs', () => {
target: 'json',
threads: 4,
chunkSize: 5,
progress: true,
});
});

Expand Down
2 changes: 2 additions & 0 deletions src/utils/configuration/index.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ export const getDefaultConfig = lazy(() =>

threads: cpus().length,
chunkSize: 10,
progress: true,
})
)
);
Expand Down Expand Up @@ -96,6 +97,7 @@ export const createConfigFromCLIOptions = options => ({
target: options.target,
threads: options.threads,
chunkSize: options.chunkSize,
progress: options.progress,
});

/**
Expand Down
3 changes: 3 additions & 0 deletions src/utils/configuration/types.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ export type Configuration = {

// Number of items to process per worker thread
chunkSize: number;

// Whether or not the progress bar is enabled
progress: boolean;
} & {
[K in keyof AllGenerators]: GlobalConfiguration &
AllGenerators[K]['defaultConfiguration'];
Expand Down
30 changes: 30 additions & 0 deletions src/utils/progressBar.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
'use strict';

import { SingleBar, Presets } from 'cli-progress';

/**
* Creates a progress bar for the generation pipeline.
* Writes to stderr to avoid conflicts with the logger (stdout).
* Returns a no-op object if disabled or if stderr is not a TTY (e.g. CI).
*
* @param {object} [options]
* @param {boolean} [options.enabled=true] Whether to render the progress bar
* @returns {SingleBar | null}
*/
const createProgressBar = ({ enabled = true } = {}) => {
if (!enabled || !process.stderr.isTTY) {
return null;
}

return new SingleBar(
Copy link
Copy Markdown
Member

@ovflowd ovflowd Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OOC, why this package versus other packages?

Copy link
Copy Markdown
Contributor Author

@shivxmsharma shivxmsharma Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went with cli-progress mainly because of its large user base (~4M weekly downloads) and the simple API. It does exactly what we need without any extra bloat.

{
stream: process.stderr,
format: ' {phase} [{bar}] {percentage}% | {value}/{total}',
hideCursor: true,
clearOnComplete: false,
},
Presets.shades_grey
);
};

export default createProgressBar;
Loading