feat: add YouTube knowledge source ingestion and retrieval - #60
feat: add YouTube knowledge source ingestion and retrieval#60chaudharyvinay4 wants to merge 4 commits into
Conversation
WalkthroughAdds YouTube crawling, transcript chunking, pgvector retrieval, response attribution, and UI support. Groq configuration is loaded from environment settings, and the maximum LLM context document count is reduced. ChangesYouTube knowledge-source integration
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant YouTubeDataAPI
participant youtube_crawler
participant ingest_youtube
participant retriever
participant query_engine
YouTubeDataAPI->>youtube_crawler: video metadata and transcripts
youtube_crawler->>ingest_youtube: JSON video documents
ingest_youtube->>retriever: transcript chunks in YouTube collection
retriever->>query_engine: YouTube and other retrieved chunks
query_engine-->>retriever: source classification and attributions
Possibly related issues
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@MosipNexus/.env.example`:
- Line 40: Replace the real credential assigned to YOUTUBE_API_KEY in the
environment template with a clearly fake placeholder value, preserving the
variable name and configuration format.
In `@MosipNexus/app/pages/settings.py`:
- Around line 253-260: Remove the plaintext GROQ_API_KEY from the st.text_input
value in the settings UI; display only a non-secret masked placeholder or empty
value while retaining the disabled password field and explanatory help text.
Keep api_key server-side for backend use without passing the secret to the
client DOM.
In `@MosipNexus/ingestion/ingest_youtube.py`:
- Around line 66-83: Update the ingestion flow after loading videos to iterate
over every video, chunk each transcript, generate embeddings, and persist the
resulting chunks and metadata into the dedicated pgvector collection. Replace
the single-video console-only handling around first, chunk_transcript, and the
print statements with database ingestion while preserving the existing
transcript chunking behavior and collection configuration.
In `@MosipNexus/pyproject.toml`:
- Around line 27-28: Update the dependencies list in pyproject.toml to restore
the runtime packages mcp, langchain-anthropic, and langchain-openai. Keep the
existing dependency declarations intact and add these packages so the imports
used by query_engine.py and server.py resolve at runtime.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: eea6ea44-ec83-4b6e-a40b-5d03854bb651
⛔ Files ignored due to path filters (1)
MosipNexus/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
MosipNexus/.env.exampleMosipNexus/app/app.pyMosipNexus/app/pages/settings.pyMosipNexus/chain/query_engine.pyMosipNexus/config/settings.pyMosipNexus/crawler/youtube_crawler.pyMosipNexus/data/mosip_youtube.jsonMosipNexus/ingestion/ingest_youtube.pyMosipNexus/pyproject.tomlMosipNexus/retrieval/retriever.py
| with open(YOUTUBE_FILE, encoding="utf-8") as f: | ||
| videos = json.load(f) | ||
|
|
||
| print(f"Loaded {len(videos)} videos") | ||
|
|
||
| first = videos[0] | ||
|
|
||
| chunks = chunk_transcript(first["transcript"]) | ||
|
|
||
| print() | ||
|
|
||
| print(first["title"]) | ||
|
|
||
| print(f"Chunks: {len(chunks)}") | ||
|
|
||
| print(chunks[0]["start"]) | ||
|
|
||
| print(chunks[0]["text"][:250]) No newline at end of file |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift
Implement full ingestion and database embedding.
The script currently only processes the first video (videos[0]) and prints the output to the console. It does not iterate over all videos or embed them into the pgvector collection. Failing to embed the data violates MOSIP data integrity standards by leaving the knowledge base unpopulated. As per PR objectives, transcript chunking and embedding into a dedicated pgvector collection is required.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@MosipNexus/ingestion/ingest_youtube.py` around lines 66 - 83, Update the
ingestion flow after loading videos to iterate over every video, chunk each
transcript, generate embeddings, and persist the resulting chunks and metadata
into the dedicated pgvector collection. Replace the single-video console-only
handling around first, chunk_transcript, and the print statements with database
ingestion while preserving the existing transcript chunking behavior and
collection configuration.
| "google-api-python-client>=2.100.0", | ||
| "youtube-transcript-api>=0.6.2", |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Check if langchain or mcp dependencies are still present in pyproject.toml
rg -e "langchain" -e "mcp" pyproject.tomlRepository: mosip/mosip-labs
Length of output: 209
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate pyproject.toml files and inspect the relevant one(s)
git ls-files | rg '(^|/)pyproject\.toml$'
# Show the relevant section for each candidate without reading too much
for f in $(git ls-files | rg '(^|/)pyproject\.toml$'); do
echo "===== $f ====="
wc -l "$f"
sed -n '1,120p' "$f"
doneRepository: mosip/mosip-labs
Length of output: 1118
🏁 Script executed:
#!/bin/bash
set -euo pipefail
git ls-files | rg '(^|/)pyproject\.toml$'
for f in $(git ls-files | rg '(^|/)pyproject\.toml$'); do
echo "===== $f ====="
wc -l "$f"
sed -n '1,120p' "$f"
doneRepository: mosip/mosip-labs
Length of output: 1118
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find the repository path that contains the targeted pyproject.toml
fd -a --hidden --glob 'pyproject.toml' .
# If MosipNexus/pyproject.toml exists, print the relevant dependency block
if [ -f MosipNexus/pyproject.toml ]; then
echo "===== MosipNexus/pyproject.toml ====="
wc -l MosipNexus/pyproject.toml
sed -n '1,120p' MosipNexus/pyproject.toml
fiRepository: mosip/mosip-labs
Length of output: 1137
🏁 Script executed:
#!/bin/bash
set -euo pipefail
cd MosipNexus
echo "== package refs =="
rg -n --hidden -S '\b(mcp|langchain-anthropic|langchain-openai)\b' .
echo "== langchain provider imports =="
rg -n --hidden -S 'from langchain_(anthropic|openai)|import langchain_(anthropic|openai)' .Repository: mosip/mosip-labs
Length of output: 3715
🏁 Script executed:
#!/bin/bash
set -euo pipefail
cd MosipNexus
# Inspect the query engine around the provider imports and fallback logic
wc -l chain/query_engine.py
sed -n '1,160p' chain/query_engine.py
# Also inspect the MCP server entrypoint imports to see if the MCP package is required directly
wc -l mcp_server/server.py
sed -n '1,120p' mcp_server/server.pyRepository: mosip/mosip-labs
Length of output: 11058
Restore the missing runtime dependencies MosipNexus/pyproject.toml no longer declares mcp, langchain-anthropic, or langchain-openai, but chain/query_engine.py and mcp_server/server.py import them directly. That breaks the MCP server and Anthropic/OpenAI provider paths at runtime.
dependencies = [
...
"mcp",
"langchain-anthropic",
"langchain-openai",
]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@MosipNexus/pyproject.toml` around lines 27 - 28, Update the dependencies list
in pyproject.toml to restore the runtime packages mcp, langchain-anthropic, and
langchain-openai. Keep the existing dependency declarations intact and add these
packages so the imports used by query_engine.py and server.py resolve at
runtime.
Signed-off-by: Vinay Kumar <chaudharyvinay44477@gmail.com>
Signed-off-by: Vinay Kumar <chaudharyvinay44477@gmail.com>
72be2d3 to
7ec9ec3
Compare
Signed-off-by: Vinay Kumar <chaudharyvinay44477@gmail.com>
Signed-off-by: Vinay Kumar <chaudharyvinay44477@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (2)
MosipNexus/ingestion/ingest_youtube.py (1)
66-83: 🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy liftIngestion pipeline still doesn't embed anything — unresolved from prior review.
Same gap flagged previously: this script only loads
videos[0], chunks its transcript, and prints debug output — no loop over all videos, no embeddings, no writes to the pgvectorYOUTUBE_COLLECTION. As per PR objectives, transcript chunking and embedding into a dedicated pgvector collection is required, but nothing is persisted here.Also, this runs unconditionally at module import time (no
if __name__ == "__main__":guard) — worth wrapping in amain()/guard as part of the rewrite so the module can be safely imported by an orchestrator without side effects.if __name__ == "__main__": main()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@MosipNexus/ingestion/ingest_youtube.py` around lines 66 - 83, Rewrite the module-level ingestion flow around YOUTUBE_FILE so a main() function processes every video, chunks each transcript, generates embeddings, and persists the resulting chunks to the pgvector YOUTUBE_COLLECTION instead of only printing videos[0] debug data. Preserve the existing chunk_transcript behavior and collection configuration, and invoke main() only under an if __name__ == "__main__": guard so importing the module has no side effects.MosipNexus/pyproject.toml (1)
27-28: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick winRuntime deps still missing:
mcp,langchain-anthropic,langchain-openai.This is the same gap flagged in a previous review: those packages were dropped and this diff again modifies the same block without restoring them, while
chain/query_engine.pyandmcp_server/server.pyimport them directly.🔧 Proposed fix
dependencies = [ + "mcp", + "langchain-anthropic", + "langchain-openai", "google-api-python-client>=2.100.0", "youtube-transcript-api>=0.6.2", ]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@MosipNexus/pyproject.toml` around lines 27 - 28, Restore the missing runtime dependencies in the project dependency list: add mcp, langchain-anthropic, and langchain-openai alongside the existing entries. Ensure the versions are compatible with the direct imports used by chain/query_engine.py and mcp_server/server.py.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@MosipNexus/app/pages/settings.py`:
- Around line 253-260: Fix the syntax error in the API key widget arguments by
adding the missing comma after the value argument in the settings page, before
the type argument. Keep the existing disabled password-field configuration and
GROQ_API_KEY assignment unchanged.
In `@MosipNexus/config/settings.py`:
- Line 59: Increase MAX_CONTEXT_DOCS from 10 or add per-source minimum
reservations before the global deduplication slice, ensuring YouTube results
cannot crowd out existing prod-code, docs, community, GitHub, and demo-code
context; preserve the final context limit and existing source-priority behavior
where applicable.
In `@MosipNexus/ingestion/ingest_youtube.py`:
- Around line 7-11: Update the chunk_transcript call in the ingestion flow to
pass the imported CHUNK_SIZE and CHUNK_OVERLAP configuration values explicitly,
replacing reliance on its hardcoded defaults. Keep the existing transcript and
chunking flow unchanged.
---
Duplicate comments:
In `@MosipNexus/ingestion/ingest_youtube.py`:
- Around line 66-83: Rewrite the module-level ingestion flow around YOUTUBE_FILE
so a main() function processes every video, chunks each transcript, generates
embeddings, and persists the resulting chunks to the pgvector YOUTUBE_COLLECTION
instead of only printing videos[0] debug data. Preserve the existing
chunk_transcript behavior and collection configuration, and invoke main() only
under an if __name__ == "__main__": guard so importing the module has no side
effects.
In `@MosipNexus/pyproject.toml`:
- Around line 27-28: Restore the missing runtime dependencies in the project
dependency list: add mcp, langchain-anthropic, and langchain-openai alongside
the existing entries. Ensure the versions are compatible with the direct imports
used by chain/query_engine.py and mcp_server/server.py.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7af049d1-32b4-44db-84aa-baf36fe13124
⛔ Files ignored due to path filters (1)
MosipNexus/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
MosipNexus/.env.exampleMosipNexus/app/app.pyMosipNexus/app/pages/settings.pyMosipNexus/chain/query_engine.pyMosipNexus/config/settings.pyMosipNexus/crawler/youtube_crawler.pyMosipNexus/data/mosip_youtube.jsonMosipNexus/ingestion/ingest_youtube.pyMosipNexus/pyproject.tomlMosipNexus/retrieval/retriever.py
| value="Loaded from .env" | ||
| type="password", | ||
| placeholder=key_placeholder, | ||
| key="_api_key_widget", | ||
| on_change=_sync_api_key, | ||
| disabled=True, | ||
| label_visibility="collapsed", | ||
| help="API key is loaded from the .env file." | ||
| ) | ||
| api_key = st.session_state.get("llm_api_key", "") | ||
|
|
||
| api_key = GROQ_API_KEY |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Fix the syntax error.
The missing comma after the value argument causes a SyntaxError that crashes the application. Such service outages violate MOSIP operational availability and compliance baselines.
🐛 Proposed fix
- value="Loaded from .env"
+ value="Loaded from .env",
type="password",
disabled=True,
label_visibility="collapsed",
help="API key is loaded from the .env file."
)
api_key = GROQ_API_KEY📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| value="Loaded from .env" | |
| type="password", | |
| placeholder=key_placeholder, | |
| key="_api_key_widget", | |
| on_change=_sync_api_key, | |
| disabled=True, | |
| label_visibility="collapsed", | |
| help="API key is loaded from the .env file." | |
| ) | |
| api_key = st.session_state.get("llm_api_key", "") | |
| api_key = GROQ_API_KEY | |
| value="Loaded from .env", | |
| type="password", | |
| disabled=True, | |
| label_visibility="collapsed", | |
| help="API key is loaded from the .env file." | |
| ) | |
| api_key = GROQ_API_KEY |
🧰 Tools
🪛 Ruff (0.15.21)
[warning] 254-254: Expected ,, found type
(invalid-syntax)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@MosipNexus/app/pages/settings.py` around lines 253 - 260, Fix the syntax
error in the API key widget arguments by adding the missing comma after the
value argument in the settings page, before the type argument. Keep the existing
disabled password-field configuration and GROQ_API_KEY assignment unchanged.
Source: Linters/SAST tools
| RETRIEVAL_K = 8 # final chunks returned per collection | ||
| RETRIEVAL_FETCH_K = 30 # MMR candidate pool | ||
| MAX_CONTEXT_DOCS = 40 # hard cap on total docs passed to LLM (prevents context overflow) | ||
| MAX_CONTEXT_DOCS = 10 |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
MAX_CONTEXT_DOCS cut 75% (40→10) while adding a 5th competing source.
Per retriever.py, the final context is _deduped[:MAX_CONTEXT_DOCS] over the concatenation of prod-code, docs, community, github, youtube, and demo-code results — there's no per-source floor. Dropping the cap to 10 at the same time YouTube is added as a new competitor for that slice risks starving existing sources (docs/community/github/code) of context, potentially degrading answer quality/citations for previously working queries.
Consider either a higher cap or per-source minimums (e.g., reserve N slots per collection before the global truncation) so YouTube doesn't crowd out existing sources.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@MosipNexus/config/settings.py` at line 59, Increase MAX_CONTEXT_DOCS from 10
or add per-source minimum reservations before the global deduplication slice,
ensuring YouTube results cannot crowd out existing prod-code, docs, community,
GitHub, and demo-code context; preserve the final context limit and existing
source-priority behavior where applicable.
| from config.settings import ( | ||
| YOUTUBE_FILE, | ||
| CHUNK_SIZE, | ||
| CHUNK_OVERLAP, | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
CHUNK_SIZE/CHUNK_OVERLAP imported but unused — chunking silently ignores configured values.
chunk_transcript keeps its hardcoded defaults (chunk_size=500, overlap=50); the imported CHUNK_SIZE/CHUNK_OVERLAP from config.settings are never passed in at the call site (Line 73), so this ingestion path diverges from the shared chunking config used elsewhere.
🔧 Proposed fix
-chunks = chunk_transcript(first["transcript"])
+chunks = chunk_transcript(first["transcript"], chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP)Also applies to: 13-13
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@MosipNexus/ingestion/ingest_youtube.py` around lines 7 - 11, Update the
chunk_transcript call in the ingestion flow to pass the imported CHUNK_SIZE and
CHUNK_OVERLAP configuration values explicitly, replacing reliance on its
hardcoded defaults. Keep the existing transcript and chunking flow unchanged.
1 participant
Summary
This PR adds YouTube as a new knowledge source to MOSIP Nexus, enabling relevant MOSIP YouTube video transcripts to be indexed, retrieved, and used during RAG response generation.
Changes
YouTube Ingestion
Retrieval
Configuration
Testing
Summary by CodeRabbit
New Features
Improvements