Skip to content

feat: add YouTube knowledge source ingestion and retrieval - #59

Closed
chaudharyvinay4 wants to merge 56 commits into
mosip:masterfrom
chaudharyvinay4:issue-48-youtube-ingestion-v2
Closed

feat: add YouTube knowledge source ingestion and retrieval#59
chaudharyvinay4 wants to merge 56 commits into
mosip:masterfrom
chaudharyvinay4:issue-48-youtube-ingestion-v2

Conversation

@chaudharyvinay4

Copy link
Copy Markdown
Collaborator

Summary

This PR adds YouTube as a new knowledge source to MOSIP Nexus, allowing relevant YouTube video transcripts to be indexed, retrieved, and used during response generation.

Changes

YouTube Ingestion

  • Added YouTube crawler for collecting video metadata and transcripts.
  • Added ingestion pipeline for processing YouTube content.
  • Chunked and embedded transcripts into a dedicated pgvector collection.

Retrieval

  • Integrated YouTube into the unified retrieval pipeline.
  • Included YouTube results in the retrieval context for RAG.
  • Added support for YouTube source attribution in responses.

UI

  • Added YouTube support in the source display.
  • Updated the application to surface YouTube references alongside other knowledge sources.

Configuration

  • Added YouTube collection configuration.
  • Updated application settings to support YouTube retrieval.

Testing

Verified the following:

  • YouTube transcripts are successfully ingested.
  • Embeddings are stored in the vector database.
  • Relevant YouTube content is retrieved for matching queries.
  • Responses use YouTube transcript context when applicable.
  • Existing Docs, Community, GitHub, and Code retrieval continue to work as expected.

Example Query

Query

What is OpenBQ?

Result

  • Retrieves the relevant YouTube transcript.
  • Generates an answer using the retrieved transcript.
  • Displays the YouTube video as a source.

Notes

This change extends the existing multi-source RAG pipeline without affecting existing retrieval functionality.

Madhuravas and others added 30 commits March 16, 2026 12:54
Signed-off-by: Madhuravas <madhuravasreddy1998@gmail.com>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
issue-15 created apis to sync github apis
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
Signed-off-by: Madhuravas reddy <madhu@mosip.io>
implemented apis for UI data and developed UI pages
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
Signed-off-by: Mohanraj209 <mohan1715@gmail.com>
…t.lock

Signed-off-by: Mohan E <mohanraj1715@gmail.com>
Signed-off-by: Mohanachandran S <mohanachandran.s@technoforte.co.in>
Add MOSIP Nexus AI — production RAG chatbot for MOSIP knowledge base
[MOSIP-44779] Created helm charts for github tracker activity.
Signed-off-by: Mohanachandran S <mohanachandran.s@technoforte.co.in>
Migrated to pgvector and introduced docker compose
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
kharodejayesh and others added 26 commits July 1, 2026 13:01
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
MOSIP-45244- added origination dropdown
…ation (Fine-Tuning & MCP Integration Changes)

Signed-off-by: Mohanachandran S <mohanachandran.s@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
MOSIP-45245-Github Trakker add Apis to fetch user details in backend along with roles
Restore pre-PR helm templates and deploy values so chart packaging matches the known-good layout, while keeping RDS/GitHub env wiring and GITHUB_ORG/USER_ROLES in values.yaml so the backend still starts correctly.

Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Co-authored-by: Cursor <cursoragent@cursor.com>
Revert Helm chart template and deploy values changes from PR mosip#52.
Use mosip-helm instead of helm so chart-lint-publish clones the existing mosip/mosip-helm repository.

Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Co-authored-by: Cursor <cursoragent@cursor.com>
Fix chart publish target repo name in CI workflow.
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
Signed-off-by: Jayesh Kharode <jayesh.kharode@technoforte.co.in>
MOSIP-45293-Github trakker maintain new apis arc param
MOSIP-45284 - RAG Answer Quality Improvements + Claude Desktop Integration (Fine-Tuning & MCP Integration Changes)
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 190 files, which is 40 over the limit of 150.

To get a review, narrow the scope:
• coderabbit review --type committed # exclude uncommitted changes
• coderabbit review --dir # limit to a subdirectory
• coderabbit review --base # compare against a closer base

Upgrade to a paid plan to raise the limit.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: cf55eae9-bd9f-49d7-8953-3b0447305bfa

📥 Commits

Reviewing files that changed from the base of the PR and between bb03212 and a54cc79.

⛔ Files ignored due to path filters (20)
  • MosipNexus/app/assets/logo.png is excluded by !**/*.png
  • MosipNexus/docs/MOSIP_Nexus_Developer_Guide.docx is excluded by !**/*.docx
  • MosipNexus/docs/MOSIP_Nexus_Presentation.pptx is excluded by !**/*.pptx
  • MosipNexus/uv.lock is excluded by !**/*.lock
  • github-activity-tracker/backend/package-lock.json is excluded by !**/package-lock.json, !**/package-lock.json
  • github-activity-tracker/frontend/package-lock.json is excluded by !**/package-lock.json, !**/package-lock.json
  • github-activity-tracker/frontend/public/vite.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/BronzeIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/CodeReviewIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/CommitIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/DashboardIconBlack.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/DashboardIconWhite.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/DownloadIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/GoldIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/LeaderboardIconBlack.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/LeaderboardIconWhite.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/PRIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/SilverIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/TotalActivityIcon.svg is excluded by !**/*.svg
  • github-activity-tracker/frontend/src/assets/react.svg is excluded by !**/*.svg
📒 Files selected for processing (190)
  • .github/workflows/chart-lint-publish.yml
  • .github/workflows/mosip-nexus-docker-publish.yml
  • .github/workflows/push-trigger.yml
  • MosipNexus/.dockerignore
  • MosipNexus/.env.example
  • MosipNexus/.gitignore
  • MosipNexus/.python-version
  • MosipNexus/.streamlit/config.toml
  • MosipNexus/Dockerfile
  • MosipNexus/README.md
  • MosipNexus/api/__init__.py
  • MosipNexus/api/main.py
  • MosipNexus/app/__init__.py
  • MosipNexus/app/app.py
  • MosipNexus/app/pages/settings.py
  • MosipNexus/chain/__init__.py
  • MosipNexus/chain/query_engine.py
  • MosipNexus/chain/summarizer.py
  • MosipNexus/config/__init__.py
  • MosipNexus/config/settings.py
  • MosipNexus/crawler/__init__.py
  • MosipNexus/crawler/code_crawler.py
  • MosipNexus/crawler/community_crawler.py
  • MosipNexus/crawler/confluence_crawler.py
  • MosipNexus/crawler/docs_crawler.py
  • MosipNexus/crawler/github_crawler.py
  • MosipNexus/crawler/jira_crawler.py
  • MosipNexus/crawler/state.py
  • MosipNexus/crawler/utils.py
  • MosipNexus/crawler/youtube_crawler.py
  • MosipNexus/data/mosip_code.json
  • MosipNexus/data/mosip_community.json
  • MosipNexus/data/mosip_docs.json
  • MosipNexus/data/mosip_github.json
  • MosipNexus/data/mosip_youtube.json
  • MosipNexus/docker-compose.yml
  • MosipNexus/docs/MOSIP_Nexus_Rancher_Deployment_Guide.md
  • MosipNexus/docs/generate_docs.py
  • MosipNexus/ingestion/__init__.py
  • MosipNexus/ingestion/ingest_youtube.py
  • MosipNexus/ingestion/store.py
  • MosipNexus/k8s/00-namespace.yaml
  • MosipNexus/k8s/01-postgres.yaml
  • MosipNexus/k8s/02-secret.yaml
  • MosipNexus/k8s/03-deployment-api.yaml
  • MosipNexus/k8s/04-deployment-ui.yaml
  • MosipNexus/k8s/05-services.yaml
  • MosipNexus/k8s/06-ingress.yaml
  • MosipNexus/k8s/07-cronjob.yaml
  • MosipNexus/k8s/08-initial-ingest-job.yaml
  • MosipNexus/k8s/postgres-init.sql
  • MosipNexus/mcp_server/__init__.py
  • MosipNexus/mcp_server/server.py
  • MosipNexus/memory/__init__.py
  • MosipNexus/memory/session.py
  • MosipNexus/notifications/__init__.py
  • MosipNexus/notifications/email_notifier.py
  • MosipNexus/pyproject.toml
  • MosipNexus/retrieval/__init__.py
  • MosipNexus/retrieval/dedup.py
  • MosipNexus/retrieval/retriever.py
  • MosipNexus/run_update.py
  • MosipNexus/tests/data/mosip_community.json
  • MosipNexus/tests/data/mosip_docs.json
  • github-activity-tracker/.env
  • github-activity-tracker/.env.example
  • github-activity-tracker/backend/.dockerignore
  • github-activity-tracker/backend/.env.example
  • github-activity-tracker/backend/.gitignore
  • github-activity-tracker/backend/Dockerfile
  • github-activity-tracker/backend/app.js
  • github-activity-tracker/backend/config.properties
  • github-activity-tracker/backend/config/errorCodes.js
  • github-activity-tracker/backend/config/excludedGitHubLogins.js
  • github-activity-tracker/backend/config/syncConfig.js
  • github-activity-tracker/backend/db/dbPool.js
  • github-activity-tracker/backend/db/initLookupTables.js
  • github-activity-tracker/backend/migrations/001_create_repos_table.sql
  • github-activity-tracker/backend/migrations/002_create_github_users_table.sql
  • github-activity-tracker/backend/migrations/003_create_repo_users_table.sql
  • github-activity-tracker/backend/migrations/004_create_activity_events_table.sql
  • github-activity-tracker/backend/migrations/005_add_name_to_github_users.sql
  • github-activity-tracker/backend/migrations/006_create_lookup_tables.sql
  • github-activity-tracker/backend/migrations/007_create_user_details_table.sql
  • github-activity-tracker/backend/migrations/008_backfill_user_details_active_from.sql
  • github-activity-tracker/backend/migrations/runMigrations.js
  • github-activity-tracker/backend/package.json
  • github-activity-tracker/backend/routes/commitSyncRoute.js
  • github-activity-tracker/backend/routes/leaderBoardRoute.js
  • github-activity-tracker/backend/routes/orgActivityRoute.js
  • github-activity-tracker/backend/routes/orgSummaryRoute.js
  • github-activity-tracker/backend/routes/orgUsersRoute.js
  • github-activity-tracker/backend/routes/organizationsRoute.js
  • github-activity-tracker/backend/routes/prSyncRoute.js
  • github-activity-tracker/backend/routes/repoSyncRoute.js
  • github-activity-tracker/backend/routes/reviewSyncRoute.js
  • github-activity-tracker/backend/routes/userDetailsRoute.js
  • github-activity-tracker/backend/routes/userRoleRoute.js
  • github-activity-tracker/backend/routes/userRolesRoute.js
  • github-activity-tracker/backend/services/commitSyncService.js
  • github-activity-tracker/backend/services/githubUserService.js
  • github-activity-tracker/backend/services/leaderBoardService.js
  • github-activity-tracker/backend/services/orgActivityService.js
  • github-activity-tracker/backend/services/orgSummaryService.js
  • github-activity-tracker/backend/services/orgUsersService.js
  • github-activity-tracker/backend/services/organizationsService.js
  • github-activity-tracker/backend/services/prSyncService.js
  • github-activity-tracker/backend/services/reviewSyncService.js
  • github-activity-tracker/backend/services/syncRepos.js
  • github-activity-tracker/backend/services/userDetailsService.js
  • github-activity-tracker/backend/services/userRoleService.js
  • github-activity-tracker/backend/services/userRolesService.js
  • github-activity-tracker/backend/utils/githubClient.js
  • github-activity-tracker/backend/utils/userRoleSql.js
  • github-activity-tracker/deploy/README.md
  • github-activity-tracker/deploy/delete.sh
  • github-activity-tracker/deploy/gh-tracker-ui-values.yaml
  • github-activity-tracker/deploy/gh-tracker-values.yaml
  • github-activity-tracker/deploy/install.sh
  • github-activity-tracker/deploy/restart.sh
  • github-activity-tracker/docker-compose.yml
  • github-activity-tracker/frontend/.dockerignore
  • github-activity-tracker/frontend/.env
  • github-activity-tracker/frontend/.env.example
  • github-activity-tracker/frontend/.gitignore
  • github-activity-tracker/frontend/Dockerfile
  • github-activity-tracker/frontend/README.md
  • github-activity-tracker/frontend/eslint.config.cjs
  • github-activity-tracker/frontend/index.html
  • github-activity-tracker/frontend/nginx.conf
  • github-activity-tracker/frontend/package.json
  • github-activity-tracker/frontend/postcss.config.js
  • github-activity-tracker/frontend/src/App.css
  • github-activity-tracker/frontend/src/App.tsx
  • github-activity-tracker/frontend/src/Sidebar.tsx
  • github-activity-tracker/frontend/src/components/ActivityCard.tsx
  • github-activity-tracker/frontend/src/components/ActivityChart.tsx
  • github-activity-tracker/frontend/src/components/ActivityItem.tsx
  • github-activity-tracker/frontend/src/components/ActivityTable.tsx
  • github-activity-tracker/frontend/src/components/ActivityTrend.tsx
  • github-activity-tracker/frontend/src/components/DetailView.tsx
  • github-activity-tracker/frontend/src/components/LeaderboardCard.tsx
  • github-activity-tracker/frontend/src/components/Modal.tsx
  • github-activity-tracker/frontend/src/components/StatsCard.tsx
  • github-activity-tracker/frontend/src/components/TeamMembers.tsx
  • github-activity-tracker/frontend/src/components/TopNav.tsx
  • github-activity-tracker/frontend/src/components/UserActivityStats.tsx
  • github-activity-tracker/frontend/src/components/UserActivityTable.tsx
  • github-activity-tracker/frontend/src/components/UserProfile.tsx
  • github-activity-tracker/frontend/src/index.css
  • github-activity-tracker/frontend/src/lib/api.ts
  • github-activity-tracker/frontend/src/lib/database.types.ts
  • github-activity-tracker/frontend/src/lib/hooks.ts
  • github-activity-tracker/frontend/src/lib/organizations.ts
  • github-activity-tracker/frontend/src/lib/periods.ts
  • github-activity-tracker/frontend/src/main.tsx
  • github-activity-tracker/frontend/src/vite-env.d.ts
  • github-activity-tracker/frontend/tailwind.config.js
  • github-activity-tracker/frontend/tsconfig.app.json
  • github-activity-tracker/frontend/tsconfig.json
  • github-activity-tracker/frontend/tsconfig.node.json
  • github-activity-tracker/frontend/vite.config.ts
  • github-activity-tracker/frontend/workflow.md
  • github-activity-tracker/helm/gh-tracker-service/Chart.yaml
  • github-activity-tracker/helm/gh-tracker-service/README.md
  • github-activity-tracker/helm/gh-tracker-service/charts/common-1.17.1.tgz
  • github-activity-tracker/helm/gh-tracker-service/templates/NOTES.txt
  • github-activity-tracker/helm/gh-tracker-service/templates/_helpers.tpl
  • github-activity-tracker/helm/gh-tracker-service/templates/configmap.yaml
  • github-activity-tracker/helm/gh-tracker-service/templates/cronjob.yaml
  • github-activity-tracker/helm/gh-tracker-service/templates/deployment.yaml
  • github-activity-tracker/helm/gh-tracker-service/templates/secrets.yaml
  • github-activity-tracker/helm/gh-tracker-service/templates/service.yaml
  • github-activity-tracker/helm/gh-tracker-service/templates/serviceaccount.yaml
  • github-activity-tracker/helm/gh-tracker-service/templates/servicemonitor.yaml
  • github-activity-tracker/helm/gh-tracker-service/values.yaml
  • github-activity-tracker/helm/gh-tracker-ui/Chart.yaml
  • github-activity-tracker/helm/gh-tracker-ui/README.md
  • github-activity-tracker/helm/gh-tracker-ui/charts/common-1.17.1.tgz
  • github-activity-tracker/helm/gh-tracker-ui/templates/NOTES.txt
  • github-activity-tracker/helm/gh-tracker-ui/templates/_helpers.tpl
  • github-activity-tracker/helm/gh-tracker-ui/templates/configmap.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/deployment.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/extra-list.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/gateway.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/service-account.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/service.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/servicemonitor.yaml
  • github-activity-tracker/helm/gh-tracker-ui/templates/virtualservice.yaml
  • github-activity-tracker/helm/gh-tracker-ui/values.yaml

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chaudharyvinay4

Copy link
Copy Markdown
Collaborator Author

Closing this PR because it was created from an incorrect base and included unrelated repository history.

The clean replacement PR containing only the YouTube knowledge source integration is:

#60

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants