Add Scrape Autopilot components#21238
Open
scrappilot wants to merge 4 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
dannyroosevelt
moved this to Ready for PR Review
in Component (Source and Action) Backlog
Collaborator
|
Thank you so much for submitting this! We've added it to our backlog to review, and our team has been notified. |
Collaborator
|
Thanks for submitting this PR! When we review PRs, we follow the Pipedream component guidelines. If you're not familiar, here's a quick checklist:
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds a new ChangesScrape Autopilot Integration
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/scrape_autopilot/actions/get-balance/get-balance.mjs`:
- Line 5: The component description for getBalance (and the other two new action
definitions) is missing the required documentation link suffix. Update the
description text in each action’s metadata so it ends with a “See the
documentation” markdown link pointing to the appropriate ScrapePilot API
reference page, keeping the existing summary text and appending the link at the
end.
In `@components/scrape_autopilot/actions/scrape-url/scrape-url.mjs`:
- Line 20: The scrape-url action’s description is missing the required
documentation link. Update the description in scrape-url.mjs so it still
describes the action, but ends with the exact “See the documentation” link
format requested; use the description field in the scrape-url action object and
append the docs URL to it.
In `@components/scrape_autopilot/actions/scrape-urls/scrape-urls.mjs`:
- Line 21: The action description in scrapeUrlsAction is missing the required
documentation link at the end. Update the description field so it still
describes the component’s purpose and ends with the exact "[See the
documentation](https://...)" link format required by the review; make this
change in the scrape-urls action definition where the description string is
defined.
- Around line 56-62: The pre-call validation in the scrape-urls action currently
throws generic Error instances for the empty urls list and the MAX_URLS limit
check. Update the validation logic in the scrape-urls flow to use
ConfigurationError from `@pipedream/platform` for both cases so these user-input
issues are surfaced as configuration mistakes; keep the existing messages, and
make sure the checks around urls.length and MAX_URLS still run before any API
call.
In `@components/scrape_autopilot/scrape_autopilot.app.mjs`:
- Around line 6-13: Move the shared `format` and `js` props out of the
individual scrape components and into `scrapeAutopilot`’s `propDefinitions`,
since they are duplicated in both `scrape-url.mjs` and `scrape-urls.mjs`. Add
them as shared definitions on the app object, then update the components to
reference them via `propDefinition` while keeping any component-specific
overrides in place. Also consolidate the duplicated `FORMATS` constant so it is
defined once alongside the shared prop definitions and reused by both
components.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: cadc2987-92bb-4bc9-b6f8-65a6e8f1e79d
📒 Files selected for processing (6)
components/scrape_autopilot/README.mdcomponents/scrape_autopilot/actions/get-balance/get-balance.mjscomponents/scrape_autopilot/actions/scrape-url/scrape-url.mjscomponents/scrape_autopilot/actions/scrape-urls/scrape-urls.mjscomponents/scrape_autopilot/package.jsoncomponents/scrape_autopilot/scrape_autopilot.app.mjs
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
components/scrape_autopilot/actions/scrape-urls/scrape-urls.mjs (1)
8-8: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winComponent description must end with a documentation link.
Append
[See the documentation](https://...)pointing to the ScrapePilot scrape endpoint reference.As per path instructions: "Component
descriptionmust end with a documentation link[See the documentation](https://...)".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/scrape_autopilot/actions/scrape-urls/scrape-urls.mjs` at line 8, The component description in the scrape-urls action must end with the required documentation link. Update the description field in scrape-urls.mjs so it appends “[See the documentation](https://...)” pointing to the ScrapePilot scrape endpoint reference, and ensure the final text ends with that link exactly.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@components/scrape_autopilot/actions/scrape-urls/scrape-urls.mjs`:
- Line 8: The component description in the scrape-urls action must end with the
required documentation link. Update the description field in scrape-urls.mjs so
it appends “[See the documentation](https://...)” pointing to the ScrapePilot
scrape endpoint reference, and ensure the final text ends with that link
exactly.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 94b597b2-efeb-42ab-9120-30918143f69b
📒 Files selected for processing (4)
components/scrape_autopilot/actions/get-balance/get-balance.mjscomponents/scrape_autopilot/actions/scrape-url/scrape-url.mjscomponents/scrape_autopilot/actions/scrape-urls/scrape-urls.mjscomponents/scrape_autopilot/scrape_autopilot.app.mjs
ashwins01
reviewed
Jun 30, 2026
ashwins01
left a comment
ashwins01
left a comment
Contributor
There was a problem hiding this comment.
Hi @scrappilot, thank you for your contribution! I see you've already raised a request for a base integration to be setup in pipedream for scrappilot, we can review and test these changes once a base integration is in place.
ashwins01
moved this from Ready for PR Review
to Blocked
in Component (Source and Action) Backlog
Author
|
Thanks! Sounds good. I'll wait until the base integration (#21239) is in place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@pipedream/platformaxios; no official Node.js SDK is currently available.Tests
node scripts/findBadKeys.js components/scrape_autopilot/actions/scrape-url/scrape-url.mjs,components/scrape_autopilot/actions/scrape-urls/scrape-urls.mjs,components/scrape_autopilot/actions/get-balance/get-balance.mjsnode scripts/findDuplicateKeys.jsnode scripts/generate-package-report.js --package=scrape_autopilot --verbosepnpm exec eslint components/scrape_autopilotChecklist
Please check the following items before your PR can be reviewed:
Versioning
0.0.1for new ones)package.json's version updatedNew app
App integration request submitted: #21239 (#21239)
CodeRabbit review
Summary by CodeRabbit
New Features
Documentation