feat: add sitemap.xml and robots.txt for SEO - #55
Conversation
|
@coderabbitai review please |
|
✅ Actions performedReview triggered.
|
📝 WalkthroughWalkthroughAdds sitemap and robots.txt generation and routes; integrates sitemap generation with the database. Also introduces unknowns collection/fetching, bech32 relay-hinting, extended rendering for embedded quotes/highlights, and several RelayPool and render API changes. Changes
sequenceDiagram
participant Client as "Client (browser/crawler)"
participant Server as "HTTP server (main.rs)"
participant Sitemap as "sitemap module"
participant Ndb as "Ndb (nostrdb)"
Client->>Server: GET /sitemap.xml
Server->>Sitemap: generate_sitemap(&app.ndb)
Sitemap->>Ndb: query notes, articles, profiles
Ndb-->>Sitemap: results
Sitemap-->>Server: sitemap XML
Server-->>Client: 200 application/xml (with cache headers)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 5 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Tip Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@src/sitemap.rs`:
- Around line 269-276: Update the metrics calls to use the metrics 0.21.0 API:
replace the incorrect usage of counter!("sitemap_generations_total", 1) with
either the increment_counter! convenience macro or call
counter!("sitemap_generations_total").increment(1), and replace
gauge!("sitemap_generation_duration_seconds", duration.as_secs_f64()) and the
other gauge calls for "sitemap_urls_total", "sitemap_notes_count",
"sitemap_articles_count", and "sitemap_profiles_count" with
gauge("...").set(value) (or use a suitable gauge-set helper) so the macros
return handles before invoking .increment(...) or .set(...); use the existing
local variables start, duration (computed from start.elapsed()), entries.len(),
notes_count, articles_count, and profiles_count as the values passed to the
handle methods.
🧹 Nitpick comments (2)
src/sitemap.rs (2)
71-79: Redundant datetime conversion.Lines 75-79 create a
datetimefromtimestamp, then immediately extract the same value back assecs_since_epoch. This is unnecessary; you can usetimestampdirectly.♻️ Suggested simplification
fn format_lastmod(timestamp: u64) -> String { - use std::time::{Duration, UNIX_EPOCH}; - - let datetime = UNIX_EPOCH + Duration::from_secs(timestamp); - let secs_since_epoch = datetime - .duration_since(UNIX_EPOCH) - .unwrap_or_default() - .as_secs(); - // Simple date formatting without external dependencies - let days_since_epoch = secs_since_epoch / 86400; + let days_since_epoch = timestamp / 86400; let mut year = 1970i32; let mut remaining_days = days_since_epoch as i32;
160-160: Consider logging query failures for observability.Using
unwrap_or_default()provides resilience, but query failures are silently ignored. Consider logging a warning when queries fail to help diagnose issues in production. The same applies to lines 187 and 235.♻️ Optional: Log query errors
- let results = ndb.query(&txn, &[notes_filter], MAX_SITEMAP_URLS as i32).unwrap_or_default(); + let results = ndb + .query(&txn, &[notes_filter], MAX_SITEMAP_URLS as i32) + .unwrap_or_else(|e| { + tracing::warn!("Failed to query notes for sitemap: {e}"); + vec![] + });
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/main.rssrc/sitemap.rs
🧰 Additional context used
🧬 Code graph analysis (1)
src/main.rs (1)
src/sitemap.rs (2)
generate_robots_txt(282-294)generate_sitemap(129-279)
🔇 Additional comments (10)
src/sitemap.rs (7)
1-22: LGTM!Constants are well-documented and reasonable. Using
OnceLockfor lazy initialization of the base URL is appropriate for avoiding repeated environment variable lookups.
24-44: LGTM!The lazy initialization pattern with a one-time warning for missing configuration is a good approach. The normalization correctly handles trailing slashes.
46-53: LGTM!Good defensive coding with
unwrap_or_default()andsaturating_sub()to handle edge cases gracefully.
55-69: LGTM!Correct XML escaping implementation covering all required special characters with appropriate capacity pre-allocation.
80-118: LGTM!The date calculation logic is correct, including proper leap year handling with the standard formula.
281-294: LGTM!The robots.txt output is well-structured with appropriate directives. The wildcard pattern
/*.jsonis supported by major search engines (Google, Bing).
296-356: LGTM!Good test coverage for the utility functions with meaningful test cases including edge cases like leap years.
src/main.rs (3)
36-36: LGTM!Module declaration is correctly placed.
143-150: LGTM!The route follows the established pattern for static content routes. The 24-hour cache duration is appropriate for robots.txt.
151-167: LGTM!Good error handling pattern with logging before returning 500. The 1-hour cache is appropriate for dynamic content, and the XML content type is correct.
Note:
generate_sitemapperforms synchronous database queries, which is consistent with how other routes in this file usendb. If sitemap generation becomes slow with larger databases, consider offloading to a blocking task withtokio::task::spawn_blocking.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
| // Record metrics (aggregate stats, not user-tracking) | ||
| let duration = start.elapsed(); | ||
| metrics::counter!("sitemap_generations_total", 1); | ||
| metrics::gauge!("sitemap_generation_duration_seconds", duration.as_secs_f64()); | ||
| metrics::gauge!("sitemap_urls_total", entries.len() as f64); | ||
| metrics::gauge!("sitemap_notes_count", notes_count as f64); | ||
| metrics::gauge!("sitemap_articles_count", articles_count as f64); | ||
| metrics::gauge!("sitemap_profiles_count", profiles_count as f64); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check for metrics crate version in Cargo.toml
find . -name "Cargo.toml" -type f | head -5 | xargs grep -l "metrics" 2>/dev/nullRepository: damus-io/notecrumbs
Length of output: 75
🏁 Script executed:
# Also verify the actual code at the specified lines
if [ -f "src/sitemap.rs" ]; then
sed -n '269,276p' src/sitemap.rs
fiRepository: damus-io/notecrumbs
Length of output: 565
🌐 Web query:
metrics crate Rust 0.21.0 counter macro syntax increment
💡 Result:
Use the counter! macro to get a Counter handle, then call increment(n); or use the convenience macro increment_counter! to add 1 (or pass labels). Examples:
-
Get handle and increment by n:
let c = counter!("my_metric");
c.increment(5);
[1] -
Increment by 1 directly:
increment_counter!("my_metric");
// or with labels:
increment_counter!("my_metric", "service" => "http");
[2]
Docs (macro reference and examples) show the counter! and increment_counter! usage. [1] [2]
Sources:
[1] docs.rs: metrics::counter! macro.
[2] docs.rs / metrics macro increment_counter! (examples).
🏁 Script executed:
# Read the Cargo.toml to find the metrics version
cat Cargo.toml | grep -A 2 "^\[dependencies\]" | head -20Repository: damus-io/notecrumbs
Length of output: 147
🏁 Script executed:
# Or search more directly for metrics
grep "metrics" Cargo.tomlRepository: damus-io/notecrumbs
Length of output: 116
🌐 Web query:
metrics crate Rust gauge macro syntax 0.21
💡 Result:
The gauge! macro (as of metrics 0.21) accepts these forms:
- gauge!(
$name:expr $ (,$label_key:expr $ (=>$label_value:expr)? )* $ (,)? ) - gauge!( level: $level:expr,
$name:expr $ (,$label_key:expr $ (=>$label_value:expr)? )* $ (,)? ) - gauge!( target: $target:expr,
$name:expr $ (,$label_key:expr $ (=>$label_value:expr)? )* $ (,)? ) - gauge!( target: $target:expr, level: $level:expr,
$name:expr $ (,$label_key:expr $ (=>$label_value:expr)? )* $ (,)? )
Examples:
- let g = gauge!("my_gauge");
- let g = gauge!("my_gauge", "service" => "http");
- let g = gauge!(level: Level::DEBUG, "my_gauge");
- let g = gauge!(format!("{}{}", prefix, "g"), &labels);
(You get a Gauge handle which supports .increment(), .decrement(), .set().) (docs.rs)
If you want the docs page for that macro/version, see the metrics macro docs and the 0.21.0 changelog. (docs.rs)
Citations:
- 1: https://docs.rs/metrics/latest/metrics/macro.gauge.html?utm_source=openai
- 2: https://docs.rs/metrics/latest/metrics/macro.gauge.html?utm_source=openai
Fix metrics macro syntax for version 0.21.0.
The counter!() and gauge!() macros in metrics 0.21.0 return handles that require method calls. The current syntax is incorrect:
counter!("name", value)should becounter!("name").increment(value)(or useincrement_counter!("name")convenience macro)gauge!("name", value)should begauge!("name").set(value)
All metrics calls in lines 269-276 need to be updated to use the correct API.
🤖 Prompt for AI Agents
In `@src/sitemap.rs` around lines 269 - 276, Update the metrics calls to use the
metrics 0.21.0 API: replace the incorrect usage of
counter!("sitemap_generations_total", 1) with either the increment_counter!
convenience macro or call counter!("sitemap_generations_total").increment(1),
and replace gauge!("sitemap_generation_duration_seconds",
duration.as_secs_f64()) and the other gauge calls for "sitemap_urls_total",
"sitemap_notes_count", "sitemap_articles_count", and "sitemap_profiles_count"
with gauge("...").set(value) (or use a suitable gauge-set helper) so the macros
return handles before invoking .increment(...) or .set(...); use the existing
local variables start, duration (computed from start.elapsed()), entries.len(),
notes_count, articles_count, and profiles_count as the values passed to the
handle methods.
Fixes 404 errors when Ubuntu package versions change on mirrors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update to nostrdb 0.9.0 and fix breaking API changes: - FilterBuilder.tags() now takes &str instead of String - Handle new FilterField variants (Search, Relays, Custom) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
NIP-84 Highlights (kind:9802): - Extract and render highlight metadata (context, comment) - Source attribution for web URLs, notes, and articles - Blockquote styling with left border accent NIP-18 Embedded Quotes: - Parse q tags and inline nevent/note/naddr mentions - Rich quote cards with avatar, name, @handle, relative time - Reply detection using nostrdb's NoteReply - Type indicators for articles/highlights/drafts Other improvements: - Draft badge for unpublished articles (kind:30024) - @username handles displayed under profile names - Human-readable @mentions (resolve npub to display names) Closes: damus-io#51 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Co-Authored-By: alltheseas <alltheseas@users.noreply.github.com>
Embedded article quotes now display as cards matching iOS Damus: - Hero image (if available) - Bold article title - Summary text (if available) - Word count - DRAFT badge via CSS for kind 30024 Co-Authored-By: alltheseas <alltheseas@users.noreply.github.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add an UnknownIds pattern (adapted from notedeck) to fetch quoted events referenced in q tags and inline mentions. Events are fetched using relay hints from nevent/naddr bech32 and q tag relay fields. - Add src/unknowns.rs with UnknownId enum and UnknownIds collection - Update QuoteRef to include relay hints (Vec<RelayUrl>) - Extract relay hints from nevent/naddr bech32 and q tag third element - Add collect_quote_unknowns() and fetch_unknowns() to render.rs - Fetch quote unknowns in main.rs after primary note is loaded Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Expand the unknowns pattern beyond just quoted events to collect: - Author profiles - Reply chain (root/reply) using nostrdb's NoteReply (NIP-10) - Mentioned profiles (npub/nprofile with relay hints) - Mentioned events (nevent/note1 with relay hints) - Quoted events (q tags, inline mentions) Move unknowns collection to main.rs for consistent handling regardless of whether primary note was cached. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: William Casarin <jb55@jb55.com>
Previously, background profile refreshes only fetched kind 1 (notes), never updating kind 0 (profile metadata). This caused profiles to remain stale indefinitely after initial cache. Now fetch_profile_feed also fetches the latest profile metadata from relays, allowing nostrdb to update cached profiles with newer versions. Fixes: damus-io#52 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…d_quotes_html Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Integrate relay provenance tracking from rust-nostr PR #1172 to enable proper NIP-19 bech32 links with relay hints for better content discoverability. Changes: - Update nostr-sdk to alltheseas/rust-nostr relay-provenance-tracking branch - RelayPool::stream_events returns BoxedStream<RelayEvent> with source relay URL - NoteAndProfileRenderData stores source_relays captured during fetch - Generate bech32 links with relay hints for all event types (notes, articles, highlights) - Filter profile (kind 0) relays from note hints - Prioritize default relays in source_relays for reliability - Preserve author/kind fields when rebuilding nevent bech32 - Graceful fallback to original nip19 on encoding failure with metric - Add bech32_with_relays() helper with 8 unit tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use process_event_with instead of process_event to pass the source relay URL to nostrdb when ingesting events from relay streams. Fixes: fffaa9e ("feat: integrate rust-nostr PR #1172 relay provenance tracking") Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add dynamic sitemap generation from nostrdb cache to improve search engine discoverability of Nostr content. New routes: - GET /robots.txt - crawler directives with sitemap reference - GET /sitemap.xml - dynamic sitemap from cached notes/profiles/articles The sitemap queries local nostrdb for: - Notes (kind:1) → note1xxx URLs - Long-form articles (kind:30023) → naddr1xxx URLs - Profiles (kind:0) → npub1xxx URLs Ref: damus-io#26 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Track aggregate stats (privacy-preserving, no user tracking): - sitemap_generations_total: counter for generation requests - sitemap_generation_duration_seconds: time to generate - sitemap_urls_total: total URLs in sitemap - sitemap_notes_count: notes included - sitemap_articles_count: articles included - sitemap_profiles_count: profiles included Metrics available at /metrics endpoint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Skip kind:30023 entries with missing/empty d-tag to avoid ambiguous URLs and potential collisions across authors - Add since filter (90 days) to notes and articles queries to prioritize recent content for SEO freshness - Log warning when NOTECRUMBS_BASE_URL is not set, to surface potential misconfiguration in production 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Cache base URL with OnceLock to avoid logging warning on every request - Use separate lookback periods: 90 days for notes, 365 days for evergreen article content (kind:30023) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use early returns and let-else patterns to reduce nesting depth in generate_sitemap loops. Improves readability by making the happy path linear instead of deeply indented. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
34db03a to
b632279
Compare
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/main.rs (1)
502-502:⚠️ Potential issue | 🔴 CriticalSame metrics syntax issue.
Line 502 uses the incorrect two-argument
gauge!()form.- metrics::gauge!("relay_pool_known_relays", tracked as f64); + metrics::gauge!("relay_pool_known_relays").set(tracked as f64);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/main.rs` at line 502, The gauge! macro is being used with the wrong two-argument form; replace the current call metrics::gauge!("relay_pool_known_relays", tracked as f64); with the correct form that passes a single numeric value expression (cast to f64 if needed), e.g. call metrics::gauge!("relay_pool_known_relays", tracked_value_as_f64) where tracked_value_as_f64 is the tracked variable cast to f64 before/inside the macro invocation; update the usage around the gauge! macro and the tracked identifier accordingly.src/relay_pool.rs (1)
63-63:⚠️ Potential issue | 🔴 CriticalMetrics macro syntax is invalid for
metrics0.21. Allcounter!()andgauge!()calls use an unsupported two-argument form.The macros
counter!("name", value)andgauge!("name", value)are incorrect. Themetrics0.21 API registers metrics by returning a handle; numeric values must be passed via methods on that handle. This affects lines 63, 80, 100, 103, and 139.Proposed fix for all occurrences
- metrics::counter!("relay_pool_ensure_calls_total", 1); + metrics::counter!("relay_pool_ensure_calls_total").increment(1);- metrics::counter!("relay_pool_relays_added_total", relays_added); + metrics::counter!("relay_pool_relays_added_total").increment(relays_added);- metrics::counter!("relay_pool_connect_success_total", connect_success); + metrics::counter!("relay_pool_connect_success_total").increment(connect_success);- metrics::counter!("relay_pool_connect_failure_total", connect_failure); + metrics::counter!("relay_pool_connect_failure_total").increment(connect_failure);- metrics::gauge!("relay_pool_known_relays", tracked as f64); + metrics::gauge!("relay_pool_known_relays").set(tracked as f64);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/relay_pool.rs` at line 63, The metrics macro calls use the old two-argument form; replace each two-argument call with a handle + method call: call metrics::counter!("relay_pool_ensure_calls_total") to get the counter handle and then call .increment(1) (or .add(1)) on it; do the same for the other counters (relay_pool_check_ok_total, relay_pool_check_errors_total) and for gauges call metrics::gauge!("relay_pool_up_gauge") and metrics::gauge!("relay_pool_backoff_ms_gauge") to get gauge handles and then set their values via .set(f64) (or appropriate update method). Update all occurrences referenced (the counter/gauge invocations around the symbols relay_pool_ensure_calls_total, relay_pool_check_ok_total, relay_pool_check_errors_total, relay_pool_up_gauge, relay_pool_backoff_ms_gauge) so no call uses the unsupported two-argument macro form.
🤖 Fix all issues with AI agents
Verify each finding against the current code and only fix it if needed.
In `@assets/damus.css`:
- Around line 503-660: The CSS repeats the draft-orange hex (`#ff6b35`) in
.damus-article-draft, .damus-embedded-quote-type-draft, and
.damus-embedded-article-title.damus-embedded-article-draft::after; extract this
value into a custom property (e.g. --damus-draft) in :root and replace the
hardcoded `#ff6b35` occurrences with var(--damus-draft) so the draft color is
centralized and consistent across .damus-article-draft,
.damus-embedded-quote-type-draft, and the ::after pseudo-element.
In `@Cargo.toml`:
- Around line 22-24: Replace the two git dependencies that point to the personal
fork with references to the upstream rust-nostr repository: update the nostr-sdk
and nostr entries in Cargo.toml to use the official rust-nostr repo (use the
appropriate upstream git URL or, preferably, a crate version constraint) and, if
you must keep a git dependency, pin it with rev = "..." for reproducible builds;
ensure you edit the dependency keys "nostr-sdk" and "nostr" to reference the
upstream source and/or a fixed version instead of
"https://github.com/alltheseas/rust-nostr.git".
In `@src/html.rs`:
- Line 438: Replace the direct unwrap on block.as_mention() with a safe guard
like the other blocks: use if let Some(mention) = block.as_mention() { ... } (or
match) and bail/continue early in the else branch so the code only uses the
mention variable when it is present; update the block handling that currently
does let mention = block.as_mention().unwrap() to follow the same guard pattern
used elsewhere for BlockType::MentionBech32.
- Line 2068: The metrics macro call uses the wrong syntax—replace the incorrect
metrics::counter!("bech32_encode_fallback_total", 1) with the proper increment
macro metrics::increment_counter!("bech32_encode_fallback_total") (or use
metrics::counter!("bech32_encode_fallback_total", 1.0) only if your metrics
crate expects a float value); update the invocation in src/html.rs and ensure
the metrics macros are imported/available so the new metrics::increment_counter!
call compiles.
- Around line 707-728: In lookup_article_by_addr, the use of the `?` on
`tag.get_str(0)?` will return None and abort the whole lookup if any tag lacks a
name; replace that with a non-failing check (e.g., use `if let Some(tag_name) =
tag.get_str(0) { ... } else { continue }`) so malformed tags are skipped rather
than propagating None, keeping the existing match arms for "d" and "title" and
preserving the logic that sets `found_d_match` and `title`.
- Around line 852-855: The fallback avatar URL used in the embedded quote avatar
unwrap_or_else returns an image src of "/img/no-profile.svg" which is not
served; update that fallback to a served asset (e.g. change the returned string
to use "/assets/default_pfp.jpg" or another existing asset) so embedded quote
avatars load correctly, locating the expression that builds the img tag in
src/html.rs (the unwrap_or_else returning the img tag) and replace the path
accordingly; alternatively, if you prefer server-side, add "/img/no-profile.svg"
to the static asset routes in main.rs so the existing path is served.
In `@src/main.rs`:
- Around line 217-225: Currently render::fetch_unknowns is awaited inline during
handling of RenderData::Note (after render::collect_note_unknowns), which adds
request latency; instead, call fetch_unknowns asynchronously and do not await it
on the request path. Spawn a background task (e.g., tokio::spawn) that takes
clones/handles of app.relay_pool and app.ndb and calls render::fetch_unknowns(…,
unknowns).await inside the task, and log any errors there—do not block the
current request; keep the existing collect_note_unknowns and unknowns handling
but replace the inline await with a detached task that handles errors locally.
- Line 502: The gauge! macro is being used with the wrong two-argument form;
replace the current call metrics::gauge!("relay_pool_known_relays", tracked as
f64); with the correct form that passes a single numeric value expression (cast
to f64 if needed), e.g. call metrics::gauge!("relay_pool_known_relays",
tracked_value_as_f64) where tracked_value_as_f64 is the tracked variable cast to
f64 before/inside the macro invocation; update the usage around the gauge! macro
and the tracked identifier accordingly.
In `@src/relay_pool.rs`:
- Line 63: The metrics macro calls use the old two-argument form; replace each
two-argument call with a handle + method call: call
metrics::counter!("relay_pool_ensure_calls_total") to get the counter handle and
then call .increment(1) (or .add(1)) on it; do the same for the other counters
(relay_pool_check_ok_total, relay_pool_check_errors_total) and for gauges call
metrics::gauge!("relay_pool_up_gauge") and
metrics::gauge!("relay_pool_backoff_ms_gauge") to get gauge handles and then set
their values via .set(f64) (or appropriate update method). Update all
occurrences referenced (the counter/gauge invocations around the symbols
relay_pool_ensure_calls_total, relay_pool_check_ok_total,
relay_pool_check_errors_total, relay_pool_up_gauge, relay_pool_backoff_ms_gauge)
so no call uses the unsupported two-argument macro form.
In `@src/sitemap.rs`:
- Around line 254-267: The sitemap builder writes base_url directly into the
XML, so unescaped characters in NOTECRUMBS_BASE_URL can break the sitemap;
update the code that constructs the loc value (either where SitemapEntry is
built or in this loop that writes entries) to XML-escape the entire loc string
(not just the bech32 portion) before writing it into xml—e.g., apply the
existing xml_escape function (or equivalent) to entry.loc or escape base_url at
initialization so that the final loc written by the loop is always XML-safe.
- Around line 152-177: The sitemap is regenerated synchronously on each cache
miss in generate_sitemap and should be server-cached to avoid blocking HTTP
handlers; add a shared in-memory cache (e.g., Mutex<Option<(std::time::Instant,
String)>> or RwLock) accessible to the sitemap handler and modify
generate_sitemap to first check the cache age and return the cached XML if
fresh, otherwise regenerate the XML, then update the cache with the new String
and timestamp; use double-checked locking so you don't hold the lock while
calling heavy functions like ndb.query, and reference generate_sitemap,
ndb.query, MAX_SITEMAP_URLS, and NOTES_LOOKBACK_DAYS when locating where to add
the cache logic.
In `@src/unknowns.rs`:
- Around line 233-251: When ndb.get_note_by_id(txn, ev.id()) returns Err we
currently call add_note_if_missing which performs the same lookup again;
instead, avoid the redundant lookup by directly recording the missing event
(e.g. insert the event id into the unknowns map) and extend relays there, then
still handle the optional author lookup: keep the check using ev.pubkey() and
insert UnknownId::Profile via self.ids.entry(...).or_default().extend(relays).
Update the Mention::Event branch to perform the direct insert for the missing
event rather than calling add_note_if_missing (or refactor add_note_if_missing
to accept a fast-path that skips the second ndb.get_note_by_id).
🧹 Nitpick comments (4)
🤖 Fix all nitpicks with AI agents
Verify each finding against the current code and only fix it if needed. In `@assets/damus.css`: - Around line 503-660: The CSS repeats the draft-orange hex (`#ff6b35`) in .damus-article-draft, .damus-embedded-quote-type-draft, and .damus-embedded-article-title.damus-embedded-article-draft::after; extract this value into a custom property (e.g. --damus-draft) in :root and replace the hardcoded `#ff6b35` occurrences with var(--damus-draft) so the draft color is centralized and consistent across .damus-article-draft, .damus-embedded-quote-type-draft, and the ::after pseudo-element. In `@src/main.rs`: - Around line 217-225: Currently render::fetch_unknowns is awaited inline during handling of RenderData::Note (after render::collect_note_unknowns), which adds request latency; instead, call fetch_unknowns asynchronously and do not await it on the request path. Spawn a background task (e.g., tokio::spawn) that takes clones/handles of app.relay_pool and app.ndb and calls render::fetch_unknowns(…, unknowns).await inside the task, and log any errors there—do not block the current request; keep the existing collect_note_unknowns and unknowns handling but replace the inline await with a detached task that handles errors locally. In `@src/sitemap.rs`: - Around line 152-177: The sitemap is regenerated synchronously on each cache miss in generate_sitemap and should be server-cached to avoid blocking HTTP handlers; add a shared in-memory cache (e.g., Mutex<Option<(std::time::Instant, String)>> or RwLock) accessible to the sitemap handler and modify generate_sitemap to first check the cache age and return the cached XML if fresh, otherwise regenerate the XML, then update the cache with the new String and timestamp; use double-checked locking so you don't hold the lock while calling heavy functions like ndb.query, and reference generate_sitemap, ndb.query, MAX_SITEMAP_URLS, and NOTES_LOOKBACK_DAYS when locating where to add the cache logic. In `@src/unknowns.rs`: - Around line 233-251: When ndb.get_note_by_id(txn, ev.id()) returns Err we currently call add_note_if_missing which performs the same lookup again; instead, avoid the redundant lookup by directly recording the missing event (e.g. insert the event id into the unknowns map) and extend relays there, then still handle the optional author lookup: keep the check using ev.pubkey() and insert UnknownId::Profile via self.ids.entry(...).or_default().extend(relays). Update the Mention::Event branch to perform the direct insert for the missing event rather than calling add_note_if_missing (or refactor add_note_if_missing to accept a fast-path that skips the second ndb.get_note_by_id).assets/damus.css (1)
503-660: Consider extracting the repeated draft-orange color into a CSS variable.The draft color
#ff6b35appears in four places across this file (lines 360, 571–572, 617) — in.damus-article-draft,.damus-embedded-quote-type-draft, and the::afterpseudo-element. Extracting it to a--damus-draftcustom property in:rootwould make future palette changes easier and keep the pattern consistent with the rest of the design system.Otherwise the embedded-quote and article-card blocks look clean and consistent.
♻️ Suggested extraction
Add to
:root::root { /* ... existing vars ... */ + --damus-draft: `#ff6b35`; + --damus-draft-end: `#f7931a`; }Then replace hardcoded values, e.g.:
.damus-article-draft { - background: linear-gradient(135deg, `#ff6b35`, `#f7931a`); + background: linear-gradient(135deg, var(--damus-draft), var(--damus-draft-end));.damus-embedded-quote-type-draft { - background: rgba(255, 107, 53, 0.2); - color: `#ff6b35`; + background: rgba(255, 107, 53, 0.2); + color: var(--damus-draft); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@assets/damus.css` around lines 503 - 660, The CSS repeats the draft-orange hex (`#ff6b35`) in .damus-article-draft, .damus-embedded-quote-type-draft, and .damus-embedded-article-title.damus-embedded-article-draft::after; extract this value into a custom property (e.g. --damus-draft) in :root and replace the hardcoded `#ff6b35` occurrences with var(--damus-draft) so the draft color is centralized and consistent across .damus-article-draft, .damus-embedded-quote-type-draft, and the ::after pseudo-element.src/main.rs (1)
217-225: Unknowns fetch adds latency to every note request.
fetch_unknownsis awaited inline on the request path (bounded by a 1.5s timeout per filter). For notes with many quoted events/profiles, this could add noticeable latency. This is acceptable for now since it improves rendering quality, but consider making this non-blocking in a future iteration (render with what's available, then fetch unknowns asynchronously for subsequent requests).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/main.rs` around lines 217 - 225, Currently render::fetch_unknowns is awaited inline during handling of RenderData::Note (after render::collect_note_unknowns), which adds request latency; instead, call fetch_unknowns asynchronously and do not await it on the request path. Spawn a background task (e.g., tokio::spawn) that takes clones/handles of app.relay_pool and app.ndb and calls render::fetch_unknowns(…, unknowns).await inside the task, and log any errors there—do not block the current request; keep the existing collect_note_unknowns and unknowns handling but replace the inline await with a detached task that handles errors locally.src/sitemap.rs (1)
152-177: Sitemap generation is synchronous on the request thread.
generate_sitemapperforms three potentially large NDB queries (up to 10k results each) synchronously. On a hot path (every sitemap request that misses the 1-hour cache), this blocks the HTTP handler.This is mitigated by the
max-age=3600cache header inmain.rs, but the cache is client-side only — there's no server-side caching, so every unique client triggers a full regeneration. Consider caching the generated XML in memory (e.g., behind aMutex<Option<(Instant, String)>>) to avoid redundant work under load.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/sitemap.rs` around lines 152 - 177, The sitemap is regenerated synchronously on each cache miss in generate_sitemap and should be server-cached to avoid blocking HTTP handlers; add a shared in-memory cache (e.g., Mutex<Option<(std::time::Instant, String)>> or RwLock) accessible to the sitemap handler and modify generate_sitemap to first check the cache age and return the cached XML if fresh, otherwise regenerate the XML, then update the cache with the new String and timestamp; use double-checked locking so you don't hold the lock while calling heavy functions like ndb.query, and reference generate_sitemap, ndb.query, MAX_SITEMAP_URLS, and NOTES_LOOKBACK_DAYS when locating where to add the cache logic.src/unknowns.rs (1)
233-251: Minor: redundant ndb lookup inMention::Eventhandling.At Line 233,
ndb.get_note_by_id(txn, ev.id())is called. When it returnsErr, Line 236 callsadd_note_if_missingwhich callsndb.get_note_by_idagain internally. You could use a direct insert here to skip the second lookup.Optional optimization
Mention::Event(ev) => { let relays: HashSet<RelayUrl> = ev .relays_iter() .filter_map(|s| RelayUrl::parse(s).ok()) .collect(); match ndb.get_note_by_id(txn, ev.id()) { Err(_) => { - // Event not found - add it and its author if specified - self.add_note_if_missing(ndb, txn, ev.id(), relays.clone()); + // Event not found - add directly (already checked) + let unknown_id = UnknownId::NoteId(*ev.id()); + self.ids.entry(unknown_id).or_default().extend(relays.clone()); if let Some(pk) = ev.pubkey() {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/unknowns.rs` around lines 233 - 251, When ndb.get_note_by_id(txn, ev.id()) returns Err we currently call add_note_if_missing which performs the same lookup again; instead, avoid the redundant lookup by directly recording the missing event (e.g. insert the event id into the unknowns map) and extend relays there, then still handle the optional author lookup: keep the check using ev.pubkey() and insert UnknownId::Profile via self.ids.entry(...).or_default().extend(relays). Update the Mention::Event branch to perform the direct insert for the missing event rather than calling add_note_if_missing (or refactor add_note_if_missing to accept a fast-path that skips the second ndb.get_note_by_id).
| # PR #1172: relay provenance tracking - https://github.com/rust-nostr/nostr/pull/1172 | ||
| nostr-sdk = { git = "https://github.com/alltheseas/rust-nostr.git", branch = "relay-provenance-tracking" } | ||
| nostr = { git = "https://github.com/alltheseas/rust-nostr.git", branch = "relay-provenance-tracking" } |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
rust-nostr PR 1172 relay provenance tracking
💡 Result:
PR #1172 in rust-nostr/nostr (“Add relay provenance tracking to streaming APIs”, opened/updated Dec 18–19, 2025, now closed) adds relay provenance to the SDK’s event streaming interfaces by introducing a new RelayEvent wrapper and parallel “with source” streaming methods. (github.com)
What it changes
- Adds a new type
RelayEvent= anEventbundled with the relay URL(s) where it was seen (tracksVec<RelayUrl>and provides accessors likerelay_url()/relay_urls(), plus conversion helpers likeinto_event()/into_parts()). (github.com) - Adds new non-breaking, opt-in streaming APIs alongside the existing ones:
Client::stream_events_with_source*RelayPool::stream_events_with_source*(including variants like*_from_*and*_targeted_*)Relay::stream_events_with_source- and exports
RelayEventfromnostr-sdk. (github.com)
Why it exists (the “provenance” use case)
The main goal is to let downstream apps know which relay delivered each streamed event, enabling:
- Proper NIP-19 identifiers (
nevent,nprofile) that include relay hints (so shared links are more discoverable). (github.com) - Better debugging/analytics and support for routing models like NIP-65 (outbox/inbox decisions based on where content is actually seen). (github.com)
Citations:
- 1: Add relay provenance tracking to streaming APIs nostrdevkit/nostr#1172
- 2: Add relay provenance tracking to streaming APIs nostrdevkit/nostr#1172
- 3: Add relay provenance tracking to streaming APIs nostrdevkit/nostr#1172
- 4: Add relay provenance tracking to streaming APIs nostrdevkit/nostr#1172
- 5: Add relay provenance tracking to streaming APIs nostrdevkit/nostr#1172
Migrate back to the upstream rust-nostr repository; PR #1172 is merged.
PR #1172 ("Add relay provenance tracking to streaming APIs") is now closed and merged into the official rust-nostr repo. There is no reason to depend on the personal fork alltheseas/rust-nostr any longer—it should create ongoing supply chain risk and maintenance burden.
Update both dependencies to use the official repositories instead. If you need a specific version, use a version constraint (= "0.x.y") or, if using a git dependency temporarily, always pin to a specific revision with rev = "..." for reproducible builds.
Example migration
-nostr-sdk = { git = "https://github.com/alltheseas/rust-nostr.git", branch = "relay-provenance-tracking" }
-nostr = { git = "https://github.com/alltheseas/rust-nostr.git", branch = "relay-provenance-tracking" }
+nostr-sdk = { version = "0.x.y" }
+nostr = { version = "0.x.y" }(or with git + rev if a git dependency is required):
+nostr-sdk = { git = "https://github.com/rust-nostr/nostr", rev = "<commit-sha>" }
+nostr = { git = "https://github.com/rust-nostr/nostr", rev = "<commit-sha>" }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@Cargo.toml` around lines 22 - 24, Replace the two git dependencies that point
to the personal fork with references to the upstream rust-nostr repository:
update the nostr-sdk and nostr entries in Cargo.toml to use the official
rust-nostr repo (use the appropriate upstream git URL or, preferably, a crate
version constraint) and, if you must keep a git dependency, pin it with rev =
"..." for reproducible builds; ensure you edit the dependency keys "nostr-sdk"
and "nostr" to reference the upstream source and/or a fixed version instead of
"https://github.com/alltheseas/rust-nostr.git".
| block.as_str(), | ||
| &abbrev_str(block.as_str()) | ||
| ); | ||
| let mention = block.as_mention().unwrap(); |
There was a problem hiding this comment.
unwrap() on block.as_mention() can panic.
While as_mention() should always succeed for BlockType::MentionBech32, an unwrap() is unnecessarily risky. Use the same guard pattern used elsewhere in this file (e.g., Line 506).
Proposed fix
BlockType::MentionBech32 => {
- let mention = block.as_mention().unwrap();
+ let Some(mention) = block.as_mention() else {
+ continue;
+ };
let pubkey = match mention {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| let mention = block.as_mention().unwrap(); | |
| let Some(mention) = block.as_mention() else { | |
| continue; | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/html.rs` at line 438, Replace the direct unwrap on block.as_mention()
with a safe guard like the other blocks: use if let Some(mention) =
block.as_mention() { ... } (or match) and bail/continue early in the else branch
so the code only uses the mention variable when it is present; update the block
handling that currently does let mention = block.as_mention().unwrap() to follow
the same guard pattern used elsewhere for BlockType::MentionBech32.
| for result in results { | ||
| let mut found_d_match = false; | ||
| let mut title = None; | ||
|
|
||
| for tag in result.note.tags() { | ||
| let tag_name = tag.get_str(0)?; | ||
| match tag_name { | ||
| "d" => { | ||
| if tag.get_str(1) == Some(d_identifier) { | ||
| found_d_match = true; | ||
| } | ||
| } | ||
| "title" => { | ||
| if let Some(t) = tag.get_str(1) { | ||
| if !t.trim().is_empty() { | ||
| title = Some(t.to_owned()); | ||
| } | ||
| } | ||
| } | ||
| _ => {} | ||
| } | ||
| } |
There was a problem hiding this comment.
Bug: ? operator in lookup_article_by_addr aborts the entire lookup if any tag lacks a name.
Line 712 uses let tag_name = tag.get_str(0)?; inside a for tag in result.note.tags() loop. Since the function returns Option<...>, the ? will cause the function to return None if any single tag in the note has no string at index 0 — skipping all remaining tags and results.
This should use continue to skip malformed tags instead:
Proposed fix
for tag in result.note.tags() {
- let tag_name = tag.get_str(0)?;
+ let Some(tag_name) = tag.get_str(0) else {
+ continue;
+ };
match tag_name {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for result in results { | |
| let mut found_d_match = false; | |
| let mut title = None; | |
| for tag in result.note.tags() { | |
| let tag_name = tag.get_str(0)?; | |
| match tag_name { | |
| "d" => { | |
| if tag.get_str(1) == Some(d_identifier) { | |
| found_d_match = true; | |
| } | |
| } | |
| "title" => { | |
| if let Some(t) = tag.get_str(1) { | |
| if !t.trim().is_empty() { | |
| title = Some(t.to_owned()); | |
| } | |
| } | |
| } | |
| _ => {} | |
| } | |
| } | |
| for result in results { | |
| let mut found_d_match = false; | |
| let mut title = None; | |
| for tag in result.note.tags() { | |
| let Some(tag_name) = tag.get_str(0) else { | |
| continue; | |
| }; | |
| match tag_name { | |
| "d" => { | |
| if tag.get_str(1) == Some(d_identifier) { | |
| found_d_match = true; | |
| } | |
| } | |
| "title" => { | |
| if let Some(t) = tag.get_str(1) { | |
| if !t.trim().is_empty() { | |
| title = Some(t.to_owned()); | |
| } | |
| } | |
| } | |
| _ => {} | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/html.rs` around lines 707 - 728, In lookup_article_by_addr, the use of
the `?` on `tag.get_str(0)?` will return None and abort the whole lookup if any
tag lacks a name; replace that with a non-failing check (e.g., use `if let
Some(tag_name) = tag.get_str(0) { ... } else { continue }`) so malformed tags
are skipped rather than propagating None, keeping the existing match arms for
"d" and "title" and preserving the logic that sets `found_d_match` and `title`.
| .unwrap_or_else(|| { | ||
| r#"<img src="/img/no-profile.svg" class="damus-embedded-quote-avatar" alt="" />"# | ||
| .to_string() | ||
| }); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if /img/no-profile.svg is served anywhere or exists as a static file
rg -n "no-profile" --type rust
fd "no-profile"Repository: damus-io/notecrumbs
Length of output: 339
🏁 Script executed:
#!/bin/bash
# Check main.rs for static file serving configuration
rg -n "/img|/assets" src/main.rs
rg -n "no-profile" src/main.rs
# Look for static file configuration or route setup
rg -n "static|assets" src/main.rs -iRepository: damus-io/notecrumbs
Length of output: 1097
🏁 Script executed:
#!/bin/bash
# Check context around the no-profile references
echo "=== Context around line 853 ==="
sed -n '840,865p' src/html.rs
echo -e "\n=== Context around line 1541 ==="
sed -n '1530,1550p' src/html.rs
echo -e "\n=== Context around line 1605 ==="
sed -n '1595,1615p' src/html.rs
# Check for any default route or catch-all
echo -e "\n=== Checking for catch-all or default routes ==="
rg -n "default|catch.*all|\*" src/main.rs | head -20Repository: damus-io/notecrumbs
Length of output: 3965
🏁 Script executed:
#!/bin/bash
# Find get_base_url function and /img handling
rg -n "fn get_base_url|get_base_url()" src/
rg -n '"/img' src/main.rs
# Check if there's any wildcard or catch-all route
sed -n '100,150p' src/main.rsRepository: damus-io/notecrumbs
Length of output: 2662
🏁 Script executed:
#!/bin/bash
# Check get_base_url() implementations
echo "=== get_base_url in sitemap.rs ==="
sed -n '26,35p' src/sitemap.rs
echo -e "\n=== get_base_url in html.rs ==="
sed -n '2018,2030p' src/html.rs
# Check full serve function for /img route
echo -e "\n=== Searching for /img in serve function ==="
rg -n '"/img' src/main.rs
# Check if there's a catch-all route at the end of serve
echo -e "\n=== End of serve function path matching ==="
rg -n 'path\(\)' src/main.rs -A 5Repository: damus-io/notecrumbs
Length of output: 2242
🏁 Script executed:
#!/bin/bash
# Verify if the file actually exists in assets directory
ls -la assets/ 2>/dev/null | grep -i "no-profile\|img"
# Double-check: search for any /img route definition
rg -n '"/img' src/
# Check entire serve function to ensure we didn't miss anything
wc -l src/main.rsRepository: damus-io/notecrumbs
Length of output: 192
Fix fallback avatar path in embedded quote avatars.
The fallback avatar at line 853 references /img/no-profile.svg, but this path is not served by the application. The server only serves static assets from /assets/ and specific routes like /damus.css and /fonts/. Embedded quote avatars without a profile picture URL will display broken images.
Change the path to point to a served asset, such as /assets/default_pfp.jpg, or add /img/no-profile.svg to the static asset routes in main.rs.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/html.rs` around lines 852 - 855, The fallback avatar URL used in the
embedded quote avatar unwrap_or_else returns an image src of
"/img/no-profile.svg" which is not served; update that fallback to a served
asset (e.g. change the returned string to use "/assets/default_pfp.jpg" or
another existing asset) so embedded quote avatars load correctly, locating the
expression that builds the img tag in src/html.rs (the unwrap_or_else returning
the img tag) and replace the path accordingly; alternatively, if you prefer
server-side, add "/img/no-profile.svg" to the static asset routes in main.rs so
the existing path is served.
| "failed to encode bech32 with relays for nip19: {:?}, falling back to original", | ||
| nip19 | ||
| ); | ||
| metrics::counter!("bech32_encode_fallback_total", 1); |
There was a problem hiding this comment.
Same incorrect metrics macro syntax.
- metrics::counter!("bech32_encode_fallback_total", 1);
+ metrics::counter!("bech32_encode_fallback_total").increment(1);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| metrics::counter!("bech32_encode_fallback_total", 1); | |
| metrics::counter!("bech32_encode_fallback_total").increment(1); |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/html.rs` at line 2068, The metrics macro call uses the wrong
syntax—replace the incorrect metrics::counter!("bech32_encode_fallback_total",
1) with the proper increment macro
metrics::increment_counter!("bech32_encode_fallback_total") (or use
metrics::counter!("bech32_encode_fallback_total", 1.0) only if your metrics
crate expects a float value); update the invocation in src/html.rs and ensure
the metrics macros are imported/available so the new metrics::increment_counter!
call compiles.
| // Build XML | ||
| let mut xml = String::with_capacity(entries.len() * 200); | ||
| xml.push_str("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"); | ||
| xml.push_str("<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n"); | ||
|
|
||
| for entry in &entries { | ||
| let _ = write!( | ||
| xml, | ||
| " <url>\n <loc>{}</loc>\n <lastmod>{}</lastmod>\n <changefreq>{}</changefreq>\n <priority>{}</priority>\n </url>\n", | ||
| entry.loc, entry.lastmod, entry.changefreq, entry.priority | ||
| ); | ||
| } | ||
|
|
||
| xml.push_str("</urlset>\n"); |
There was a problem hiding this comment.
base_url is not XML-escaped in sitemap output.
On Line 256–257, base_url is written directly into the XML <urlset> output. While the loc bech32 portion is escaped via xml_escape, base_url itself (from env var) is never escaped. If someone sets NOTECRUMBS_BASE_URL to a value containing & or other XML-special characters, the sitemap XML will be malformed.
Proposed fix — escape loc as a whole, or escape base_url at init
fn normalize_base_url(url: &str) -> String {
- url.trim_end_matches('/').to_string()
+ let trimmed = url.trim_end_matches('/');
+ xml_escape(trimmed)
}Alternatively, escape the entire loc value when building SitemapEntry:
entries.push(SitemapEntry {
- loc: format!("{}/{}", base_url, xml_escape(&bech32)),
+ loc: xml_escape(&format!("{}/{}", base_url, bech32)),🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/sitemap.rs` around lines 254 - 267, The sitemap builder writes base_url
directly into the XML, so unescaped characters in NOTECRUMBS_BASE_URL can break
the sitemap; update the code that constructs the loc value (either where
SitemapEntry is built or in this loop that writes entries) to XML-escape the
entire loc string (not just the bech32 portion) before writing it into xml—e.g.,
apply the existing xml_escape function (or equivalent) to entry.loc or escape
base_url at initialization so that the final loc written by the loop is always
XML-safe.
2 participants
Summary
Adds dynamic sitemap generation and robots.txt to improve search engine discoverability of Nostr content.
Closes #26
New Routes
/robots.txt/sitemap.xmlSitemap Content
Queries local nostrdb cache for:
note1xxxURLs, 90-day lookbacknaddr1xxxURLs, 365-day lookback (evergreen content)npub1xxxURLsLimited to 10,000 URLs per content type.
robots.txt
Prometheus Metrics
sitemap_generations_totalsitemap_generation_duration_secondssitemap_urls_totalsitemap_notes_countsitemap_articles_countsitemap_profiles_countConfiguration
NOTECRUMBS_BASE_URL=https://damus.io # Required for correct sitemap URLsTest plan
/robots.txtreturns correct content/sitemap.xmlgenerates valid XML/metrics🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Improvements
Style