Sync upstream Apache DevLake (v1.0.3-beta9 → beta15) - #22
Merged
Conversation
…pache#8699) * chore(deps): update graphql dependency from merico-dev to merico-ai - Update github.com/merico-dev/graphql to github.com/merico-ai/graphql - Update import paths across all files using the graphql package - Update go.mod and go.sum with new dependency version Co-Authored-By: Claude <noreply@anthropic.com> * fix(github): fix ineffassign lint error in remote_api.go Use named return values consistently with bare returns to avoid variable shadowing of err in listGithubOwnerRepos. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…eport paths (apache#8725) User now provides basePath and accountId instead of manually typing the full S3 prefix. The collector automatically constructs and scans both by_user_analytic and user_report paths under {basePath}/AWSLogs/{accountId}/KiroLogs/…/{region}/{year}[/{month}]. Includes migration, updated blueprint, multi-prefix collector iteration, user_report model/extractor/dashboard, and frontend Account ID input field. Old scopes without accountId continue to work unchanged.
* feat(github_graphql): add finished_date to GithubDeployment model Introduce FinishedDate field in GithubDeployment to represent success-priority deployment completion timestamp. This field allows distinguishing SUCCESS timestamp from latest status update time while preserving backward compatibility. * feat(github_graphql): compute success-priority deployment finish time Add logic to derive FinishedDate from deployment statuses. Preference order: 1. Latest SUCCESS timestamp 2. Latest terminal timestamp (FAILURE/ERROR/INACTIVE/ACTIVE) This improves deployment lifecycle accuracy without modifying existing LatestStatus semantics. * feat(github_graphql): use finished_date in deployment domain conversion Update ConvertDeployment to use tool-layer FinishedDate when populating domain TaskDatesInfo.FinishedDate. Duration is now calculated using resolved finished time instead of UpdatedDate. * chore(github_graphql): add migration for finished_date column Add migration script to auto-migrate GithubDeployment and create nullable finished_date column. * test(github_graphql): update e2e snapshots for deployment finished date and duration Introduce adjustments to existing e2e tests to validate backward compatibility after adding deployment finished date and duration calculation logic. Add `deployment_finished_date` field to `_tool_github_deployments.csv` and `duration_seconds` field to `cicd_deployments.csv`. Ensure historical data remains compatible with the updated behavior. * test(github_graphql): add tests for deployment finished date and duration calculation logic Add test coverage for deployment finished date and duration calculation based on the SUCCESS state instead of relying on the inactive state. Verify fallback behavior when a SUCCESS state is not present. * fix(github_graphql): decouple add finished_date migration from active model Define a dedicated versioned struct for the finished_date migration instead of referencing the evolving GithubDeployment model to ensure deterministic and reproducible schema changes. * chore(github_graphql): fix duplicate import and lint issues in deployment extractor Remove redundant restTasks alias and use githubTasks constants consistently to resolve lint warnings and improve import clarity.
…apache#8714) * perf(dora): optimize change lead time calculation using batch queries Eliminates N+1 query problem by implementing batch fetching for: - First commits per PR (batchFetchFirstCommits) - First reviews per PR (batchFetchFirstReviews) - Deployments per project (batchFetchDeployments) Performance improvement: - Before: 3 queries per PR (30,001 queries for 10K PRs) - After: 3 batch queries total (99.99% reduction) Also fixes NULL author_id handling in review queries to properly handle PRs with empty author_id field. Tested with E2E tests confirming correctness and performance gains. Signed-off-by: Ankesh <athakur@g2.com> * fix(dora): remove obsolete single-query functions replaced by batch queries Signed-off-by: Ankesh <athakur@g2.com> * ci: add workflow to build and push amd64 image to GHCR Signed-off-by: Ankesh <athakur@g2.com> * revert: remove GHCR build workflow Signed-off-by: Ankesh <athakur@g2.com> --------- Signed-off-by: Ankesh <athakur@g2.com>
* feat(gh-copilot): add backend plugin * feat(config-ui): add gh-copilot connection UI * feat(grafana): add GitHub Copilot dashboards * fix: gh-copilot CI fixes
…en using devcontainers (apache#8645) * fix(devcontainer): Make config-ui dev server accessible outside the container when started by listening on 0.0.0.0 with the `--host` flag. Also ensure it uses port 4000 or fails to start with `--strictPort` to prevent silent port change that will mismatch with the exported dev container port. * fix(devcontainer): make Config UI and Grafana accessible from host when using devcontainers
…#8716) * feat(github): add username filtering helper for bot exclusion Implements shouldSkipByUsername() function to filter bot accounts by username using GITHUB_PR_EXCLUDELIST environment variable. - Case-insensitive matching - Comma-separated list support - Whitespace trimming - Returns false for empty usernames or empty exclusion list Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: AvivGuiser <avivguiser@gmail.com> * feat(github): filter bot PRs in extractor Adds username filtering to PR extractor to skip bot-authored PRs when GITHUB_PR_EXCLUDELIST is set. - Checks author username before extraction - Logs debug message when PR is skipped - Includes unit test for bot filtering - Includes e2e test data for bot filtering Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot reviews in extractor Adds username filtering to review extractor to skip bot reviews when GITHUB_PR_EXCLUDELIST is set. - Checks reviewer username before extraction - Logs debug message when review is skipped - Includes e2e test for bot filtering Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot PR review comments in extractor Adds username filtering to PR review comment extractor to skip bot comments when GITHUB_PR_EXCLUDELIST is set. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot issue comments in extractor Adds username filtering to issue comment extractor to skip bot comments when GITHUB_PR_EXCLUDELIST is set. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs(github): add bot filtering documentation Documents GITHUB_PR_EXCLUDELIST configuration and usage. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix(github): fix CI errors in bot filtering PR - Add Apache license header to README_FILTERING.md - Export ResetExcludedUsernamesForTest to fix unused lint error - Call ResetExcludedUsernamesForTest in e2e tests to reset sync.Once cache before setting GITHUB_PR_EXCLUDELIST env var Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: AvivGuiser <avivguiser@gmail.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Add automatic URL enrichment to PutScopes function to fetch missing htmlUrl and cloneUrl fields from GitHub API when only fullName is provided. This ensures gitextractor has the required URLs for cloning repositories. - Fetch repo details from GitHub API when URLs are missing - Populate htmlUrl and cloneUrl from API response - Add graceful error handling for inaccessible repos
…les (apache#8737) * fix(q_dev): prevent data duplication in user_report and user_data tables Replace auto-increment ID with composite primary keys so that CreateOrUpdate can properly deduplicate rows on re-extraction. - user_report PK: (connection_id, scope_id, user_id, date, client_type) - user_data PK: (connection_id, scope_id, user_id, date) - Switch db.Create() to db.CreateOrUpdate() in s3_data_extractor - Migration drops old tables, rebuilds with new PKs, resets s3_file_meta processed flag to trigger re-extraction * fix(q_dev): gofmt archived user_data_v2 model
…pache#8719) * feat(github): extend PR size exclusion for specified file extension to github plugin * fix: register migration script * fix: move PR size to 'Additional settings' and change so the comma doesn't get removed while typing * fix: linting
The Slack invite links in README.md were expired and returning "This link is no longer active." Updated both occurrences (badge and community section) to match the current link on the official DevLake website. Closes apache#8738 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
…che#8733) Adds gh-devlake as a third installation method alongside Docker Compose and Helm. gh-devlake is a GitHub CLI extension that automates DevLake deployment, configuration, and monitoring from the terminal. Closes apache#8732
GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
…a sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
* fix: modify cicd_deployments name from varchar to text * fix: update the year
…thods for MySQL 8.x compatibility (apache#8745)
…in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
…pesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
…apache#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
…#8758) Add a new Asana plugin that integrates with Asana's REST API to collect projects, sections, tasks, subtasks, stories (comments), tags, and users, mapping them to DevLake's ticket/board domain model. Backend: - Plugin implementation with all required interfaces (PluginMeta, PluginTask, PluginModel, PluginMigration, PluginSource, PluginApi, DataSourcePluginBlueprintV200) - Collectors, extractors, and converters for projects, sections, tasks, subtasks, stories, tags, and users - Remote API scope picker (Workspaces -> Teams/Portfolios -> Projects) - Scope config with issue-type regex transformation rules - Migration scripts for schema evolution - E2E tests with CSV fixtures for project and task data flows Config UI: - Plugin registration with connection form (PAT auth, endpoint, proxy) - Scope config transformation form for issue-type mapping - Dashboard URL integration for onboarding flow Grafana: - Asana dashboard with task metrics and visualizations Made-with: Cursor
* feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (apache#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (apache#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com>
…pache#8767) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards Add support for ingesting S3 logging data (GenerateAssistantResponse and GenerateCompletions events) into new database tables, and enrich all three Kiro Grafana dashboards with additional metrics. Changes: - New models: QDevChatLog and QDevCompletionLog for logging event data - New extractor: s3_logging_extractor.go parses JSON.gz logging files - Updated S3 collector to also handle .json.gz files - Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions) - New dashboard: "Kiro AI Activity Insights" with 10 panels including model usage distribution, active hours, conversation depth, feature adoption (Steering/Spec), file type usage, and prompt/response trends - Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration, and Dev (Agentic) metric panels - Fixed "Kiro Usage Dashboard" per-user table to sort by user_id - Migration script for new tables * fix(q-dev): use separate base path for logging S3 prefixes Logging data lives under a different S3 prefix ("logging/") than user report data ("user-report/"). Add LoggingBasePath option (defaults to "logging") so logging prefixes are constructed correctly. * fix(q-dev): auto-scan logging path without extra config Kiro exports to two well-known S3 prefixes in the same bucket: - user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports) - logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs) When AccountId is set, automatically scan both paths. The "logging" prefix is hardcoded since it's a standard Kiro export convention. No additional configuration needed. * fix(q-dev): update scope tooltip to mention logging data scanning * fix(q-dev): fix scope ID routing and CSV/JSON file separation Three fixes: 1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope IDs containing "/" (e.g. "034362076319/2026") work in URL paths 2. CSV extractor now filters for .csv files only, preventing it from trying to parse .json.gz logging files as CSV 3. Frontend scope API calls now encodeURIComponent(scopeId) for safe URL encoding * fix(q-dev): resolve *scopeId route conflict with dispatcher pattern The catch-all *scopeId route conflicts with *scopeId/latest-sync-state. Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a GetScopeDispatcher that checks for /latest-sync-state suffix and dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId. * fix(q-dev): use URL-safe scope ID format (underscore separator) Scope IDs like "034362076319/2026" break URL routing because "/" is a path separator. Change ID format to "034362076319_2026" (underscore) when AccountId is set. The Prefix field still uses "/" for S3 path matching. Revert to standard :scopeId routes since IDs are now safe. Note: existing scopes need to be recreated after this change. * fix(q-dev): use NoPKModel instead of Model in archived logging models archived.Model only has ID+timestamps, missing RawDataOrigin fields (_raw_data_params etc.) that common.NoPKModel includes. This caused "Unknown column '_raw_data_params'" errors at runtime. * fix(q-dev): fix GROUP BY in per-user table to merge display_name variants Remove display_name from GROUP BY so same user_id with different display_name values gets merged. Use MAX(display_name) in SELECT. * fix(q-dev): normalize logging user IDs to match CSV short UUID format Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to one user_id across both data sources. * fix(q-dev): normalize user IDs in CSV extractors and sort table DESC Apply normalizeUserId to both createUserReportData and createUserDataWithDisplayName so user_report CSV data also strips the "d-{directoryId}." prefix. Change per-user table sort to ORDER BY user_id DESC. * style(q-dev): fix gofmt formatting in chat_log models * perf(q-dev): parallelize logging S3 downloads and batch DB writes Optimize logging extractor performance: - 10 goroutine workers for parallel S3 file downloads - Batch 50 files per DB transaction instead of 1-per-file - sync.Map cache for display name resolution (avoid repeated IAM calls) - Parse records in memory during download, write all at once This should improve throughput from ~1.5 files/sec to ~15+ files/sec for typical logging file sizes. * fix(q-dev): check tx.Rollback error return to satisfy errcheck lint * feat(q-dev): add per-user model usage table and models column Add "Per-User Model Usage" table (panel 11) showing each user's request count and avg prompt/response length per model_id. Also add "Models Used" column to the Per-User Activity table. * fix(q-dev): remove per-user model usage table, keep models column only * feat(q-dev): add Kiro Executive Dashboard with cross-source analytics New dashboard "Kiro Executive Dashboard" with 12 panels covering: - KPIs: WAU, credits efficiency, acceptance rate, steering adoption - Trends: weekly active users, new vs returning users - Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec - Cost: credits pace vs projected monthly, idle power users - Quality: acceptance rate trends, code review findings, test generation - Efficiency: per-user productivity table with credits/line ratio Correlates data across user_report (credits), user_data (code metrics), and chat_log (interaction patterns) for holistic Kiro usage insights. * fix(q-dev): fix pie charts to show per-row slices instead of single total Set reduceOptions.values=true so Grafana treats each SQL result row as a separate pie slice. Fixes Model Usage Distribution, File Type Usage, Kiro Feature Adoption, and Active File Types pie charts. * fix(q-dev): cast Hour to string for Active Hours bar chart x-axis * fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues 1. qdev_user_report Panel 4 (Subscription Tier Distribution): set reduceOptions.values=true to show per-tier slices 2. qdev_user_data Panel 6 (User Interactions): remove display_name from GROUP BY, use MAX(display_name) to merge same user * fix(q-dev): prevent data inflation in user_report JOIN user_data user_report has multiple rows per (user_id, date) due to client_type (KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date). A direct JOIN causes user_data metrics to be counted multiple times. Fix: pre-aggregate user_report by (user_id, date) in a subquery before joining, so the JOIN is always 1:1. Affects: Credits Efficiency stat and User Productivity table.
* feat(qa): add is_invalid field to qa_test_case_executions Add is_invalid boolean field to the domain layer qa_test_case_executions table to allow QA teams to flag test executions as invalid due to environmental issues, flaky tests, false positives, or false negatives. Changes: - Add IsInvalid field to QaTestCaseExecution domain model - Create migration script (20260313_add_is_invalid_to_qa_test_case_executions) - Register migration in migrationscripts/register.go - Update customize service to set default value for is_invalid - Update E2E test data to include new column Resolves apache#8763 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(qa): handle missing is_invalid column in CSV import Fix PostgreSQL compatibility issue when CSV files don't contain the is_invalid column. The field now defaults to false instead of an empty string. Changes: - Update qaTestCaseExecutionHandler to check for empty string values - Add E2E test for backward compatibility with CSV files lacking is_invalid - Add explicit IsInvalid initialization in Testmo plugin converter Resolves apache#8763 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(linker): branch names containing issue keys * chore: add testing data
… time range (apache#8956) * Skip 500 (corrupt CircleCI Server records) alongside 404 in AfterResponse hook so a single bad pipeline does not abort the entire subtask * Apply SyncPolicy.TimeAfter to workflow and job DB iterators on full sync to avoid calling the API for every historical tool-layer row Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
Contrary to what the PR for the feature mentioned, the setting was enabled by default, instead of disabled. Follow up to apache#8854
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…xclusion (apache#8931) * feat(jira): collect issues via search API to avoid kanban sub-filter exclusion * Switch from board Agile API to search API with saved filter JQL so resolved issues (e.g. released fixVersions) are no longer silently excluded by kanban sub-filters * Add sub_query field to boards model and migration to track kanban sub-filter changes across syncs * Support JIRA Cloud v3 search/jql endpoint alongside v2 for Server * fix(jira,config-ui): correct SubQuery parsing and connection form crashes * Fix Jira board SubQuery struct to match nested API response shape * Prevent click event from being stored as plugin state in connection list * Default initialValues to empty object when plugin config omits it * feat(jira): Add BoardConfiguration unmarshal tests * Verify SubQuery parsing for kanban boards with, without, and empty sub-filter objects * Include a test using a full Jira Cloud response payload to validate all fields
* feat(jira): Allow extra JQL when querying JIRA board * fix: revert code so we just do extrajql, no dynamic dev lake project name * fix: remove reference to devlakeprojectname as a template option
Co-authored-by: Louis.z <louis.s4372121@gmail.com>
…70 (apache#8916) * fix(bitbucket): migrate off cross-workspace APIs removed by CHANGE-2770 - listBitbucketWorkspaces: replace GET /user/permissions/workspaces with the supported GET /workspaces (lists the current user's workspaces). Workspace slug/name now parsed from the top level of each value instead of a nested "workspace" object. - searchBitbucketRepos: replace cross-workspace GET /repositories?role=member with per-workspace GET /repositories/{workspace}, enumerating the user's workspaces first and aggregating up to PageSize matches. - models: flatten GroupResponse to match the /workspaces response shape. * fix(bitbucket): list workspaces via supported /user/workspaces (CHANGE-2770) GET /2.0/workspaces is also deprecated, not just /user/permissions/workspaces. The supported replacement for enumerating the current user's workspaces is GET /2.0/user/workspaces, which nests slug/name under a workspace object. - remote_api.go: listBitbucketWorkspaces / listAllBitbucketWorkspaces now call /user/workspaces with workspace.* fields and read via GroupId()/GroupName(). - models/repo.go: re-nest GroupResponse under WorkspaceItem to match the /user/workspaces response shape. Repo listing stays on the supported GET /2.0/repositories/{workspace}. * fix(bitbucket): drop sort/fields on /user/workspaces (400 Invalid field name) GET /2.0/user/workspaces rejects sort=workspace.slug with HTTP 400 'Invalid field name'. The bare paginated call returns the nested workspace objects we parse, so drop the sort/fields query params in both listBitbucketWorkspaces and listAllBitbucketWorkspaces. * fix(bitbucket): ignore 410 Gone on sunset issues API Bitbucket has sunset the issue tracker/wiki API; the issues endpoint now returns 410 Gone instead of 404. ignoreHTTPStatus404 only mapped 404 to ErrIgnoreAndContinue, so a 410 fell through to nil, DoAsync retried 3x and failed the whole pipeline ("Collect Issues ended unexpectedly"). Treat 410 like 404 (ignore and continue). Shared helper also covers issue comments and pr commits. Adds a table test for the status-code mapping.
Co-authored-by: Bujjibabukatta <bujjubabukatta6@gmail.com> Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com> Co-authored-by: Eldrick Wega <ewega@github.com>
apache#8942) * fix(dora): use two-phase deployment lookup to prevent first-deployment over-mapping in lead time calculator * test(dora): add e2e coverage for Phase 1 direct deployment match (issue apache#8790) * test(dora): add direct-match deployment case to change lead time e2e test * fix(dora): use straight double quotes in doc comment to satisfy gofmt --------- Co-authored-by: Bujjibabukatta <bujjubabukatta6@gmail.com> Co-authored-by: Klesh Wong <klesh@qq.com>
apache#8983) * feat(sonarqube): add project-level metrics history collection Add collector, extractor, and convertor subtasks that pull historical metric snapshots from the SonarQube measures/search_history and project_analyses/search APIs into a new cq_project_metrics_history domain table, enabling trend analysis for coverage, bugs, code smells, complexity, vulnerabilities, and other quality indicators over time. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * fix(sonarqube): Require user token at test * Reject Global and Project analysis tokens on SonarQube Server before authentication validate runs. * Skip prefix validation for SonarCloud endpoints where token prefixes differ. * feat(grafana): add historical trends to SonarQube dashboards * Add collapsed "Historical Trends" row with coverage, bugs, vulnerabilities, code smells, and duplication time-series panels to both Server and Cloud dashboards * Panels query cq_project_metrics_history and honor the Grafana time picker via $__timeFilter Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * style(sonarqube): Fix gofmt formatting in code quality files * Align struct fields and ApiCollectorArgs literals to satisfy golangci-lint gofmt checks. * Updates cq_project_metrics_history and SonarQube collector tasks flagged in CI. --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
…r context (apache#8989) Make the backend container build inputs reproducible and shrink the Docker build context: - Align the swag CLI pin to v1.16.6 across backend/Makefile, backend/Dockerfile and backend/Dockerfile.local so local, CI and image builds generate Swagger docs with the same tool version. - Pin the oauth2-proxy dev auth proxy image to v7.15.3-amd64 in both docker-compose-dev-mysql.yml and docker-compose-dev-postgresql.yml. - Add backend/.dockerignore to keep regenerated build artifacts (bin/, mocks/, logs/, Python venv/build output, coverage files and OS junk) out of the build context. Source under python/ is retained. No functional/runtime code changes. Signed-off-by: DoDiODev <DoDiDev@proton.me>
…pache#8995) Co-authored-by: Klesh Wong <klesh@qq.com>
Signed-off-by: DoDiODev <DoDiDev@proton.me>
Signed-off-by: DoDiODev <DoDiDev@proton.me> Co-authored-by: Klesh Wong <klesh@qq.com>
…he#8988) The `fix_versions` field on the jira issue model is declared as `gorm:"type:text"`, but the original migration (20250619_add_fix_versions_to_issue) created the column as `varchar(255)`. Installations that ran the earlier migration therefore have a `varchar(255)` column that no longer matches the model and truncates long fix version lists. This migration alters `_tool_jira_issues.fix_versions` to `text` so the database schema matches the model definition and can store values longer than 255 characters. Signed-off-by: DoDiODev <DoDiDev@proton.me> Co-authored-by: Klesh Wong <klesh@qq.com>
…ache#8992) Update the Grafana SQL converter dependency to sqlglot 30.12.0. Upgrade Grafana from 11.6.2 through 12.4.5 to 13.0.2 and harden dashboard provisioning, home-dashboard resolution, legacy-volume handling, and datasource cleanup. Align dashboard datasource references and plugin metadata with Grafana 13, and remove duplicate AI Model ROI dashboards. Keep postgresVersion at 1400; the PostgreSQL 18 server-version bump remains scoped to its dedicated change. Signed-off-by: DoDiODev <DoDiDev@proton.me>
Brings in 127 upstream commits (37 features, 71 fixes) since the 2026-01-15 divergence. 27 files conflicted; resolutions below. Mechanical (Phase 1): - core + github migration registers: unioned (kept both fork and upstream migration registrations). github: fork's fixNullableTokenExpiry and upstream's modifyTokenExpiresAtToNullable both kept (both make token expiry columns nullable; re-applying is a MySQL no-op). - q_dev/testmo dated migrations: took upstream's AutoMigrateTables reimplementation (same version number, only one ever runs). - table_info_test.go, config-ui register/index.ts: unioned fork plugins (cursor/findevops) with upstream plugins (claude_code/gh-copilot/tempo). - build/infra (Dockerfile[.local], Makefile, lake-builder, env.example): took upstream toolchain (Go 1.26, newer mockery/swag/golangci-lint v2); dropped fork's golang-tools stage split. - asf-header-check.yml, commit-msg.yml: kept deleted (fork CI intent). GitHub cluster (Phase 2): - token_provider.go: took upstream's GitHub App (refreshFn) + RefreshToken guards in needsRefresh; token tests taken from upstream (match new logic). - comment/pr/pr_review extractors: took upstream (both sides added identical bot-filtering; upstream's uses the correct `body` variable name). - cicd_run_collector.go(+test): took upstream — it is a superset of the fork's PR apache#8844 plus upstream's incremental collection (apache#8858). - blueprint_v200.go: kept fork's skipCommitFiles/noShallowClone (aidetector). - remote_api.go: took upstream (authUserID hoisted before switch). - deployment_convertor.go: kept fork's deployment dedup (seenDedupKey). - github transformation.tsx: took upstream (formatting-only). Duplicates / fixtures (Phase 3): - pushapi.go: took upstream (apache#8879) — extracts validation to pushapiaccess.ValidateTable (already imported), adds disabled-by-default and _devlake_* protections. Same InsertRow signature; consumer unaffected. - dora change_lead_time/project_pr_metrics.csv: merged calculator outputs milliseconds (fork's cltime-precision fix won the auto-merge), so kept fork's ms rows and converted upstream's new pr7 row to ms (x60000). NOTE: pr7 expected values are hand-derived; MUST be verified by running the dora e2e test (regenerate snapshot). Not yet done: enable opt-in upstream features (OIDC, AES-GCM encryption, Linear/Asana/Tempo/Rootly/Taiga plugins); optional Claude Code token/cost port into upstream's claude_code plugin. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three issues surfaced by CI on the merge commit: 1. dora change_lead_time_calculator.go — the auto-merge silently included BOTH the fork's and upstream's copies of deploymentCommitWithMergeSha, batchFetchFirstCommits, batchFetchFirstReviews, and batchFetchDeployments, causing "redeclared in this block" build failures. Removed the fork's duplicate block; kept upstream's two-phase batchFetchDeployments (apache#8942) — the other three are byte-identical. The ms-precision computeTimeSpan (the fork's cltime fix) is a single, untouched declaration. 2/3. golangci-lint.yml and test.yml — the fork added a step that downloads mockery v2.53.5 (a go1.25-built binary) and overwrites the container's mockery. That binary cannot parse the now go1.26 modules ("package requires newer Go version go1.26"). Removed the step (aligned both workflows to upstream, which runs `make mock` against the container mockery and passes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merged Makefile kept the fork's `mock:` target, which required an
exact mockery version (v2.53.5) that the fork's now-removed CI step used
to install. The lake-builder CI container ships mockery v2.20.0, so the
version check failed ("v2.20.0 detected, expected v2.53.5").
Upstream's mock target has no version check and runs the container's
mockery directly. Validated locally in mericodev/lake-builder:latest
(go1.20.4 + mockery v2.20.0): `make mock` now exits 0. v2.20.0 tolerates
the go1.26 module directive; v2.53.5 (go1.25-built) did not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fact) Another silently-broken auto-merge: TokenExpiresAt/RefreshTokenExpiresAt are *time.Time and UpdateToken's params are already *time.Time, but the merged code assigned &expiry (**time.Time), failing to compile: cannot use &expiry (**time.Time) as *time.Time value in assignment Assign the pointers directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…artifact) Auto-merge kept both upstream's (uses body) and the fork's (uses the now-undefined apiPullRequestReview) bot-filter blocks. Removed the fork's dead duplicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Merges 127 upstream commits (37 features, 71 fixes) accumulated since the fork's 2026-01-15 divergence. 27 files conflicted; all resolved (see the merge commit for the full per-file resolution log).
Highlights pulled in
claude_code, gh-copilotInsertRowsignatureConflict resolution summary
skipCommitFiles/noShallowClone(aidetector), deployment dedup (seenDedupKey), DORA millisecond precision, disabled CI workflows kept deleted.cicd_run_collector(fork's fix(github): bisect adaptive time windows for workflow runs 40k pagination cap apache/devlake#8844 + upstream feat(github): Add incremental data collection apache/devlake#8858), token-refresh logic, bot-filtering extractors, pushapi.go.modclean at go 1.26.dorae2e fixture needs verification:change_lead_time/project_pr_metrics.csv— the merged calculator outputs milliseconds, so the new upstreampr7row was hand-converted (×60000). Run the dora change_lead_time e2e test and regenerate the snapshot to confirm before merging.Not included (deliberate follow-ups)
claude_code.Note
The fork's in-progress config-ui WIP (28 files) was stashed before this merge, not included here.
🤖 Generated with Claude Code