Modernize eXide: CM6 editor, REx parser, REST API, LSP integration

[joewiz]

Summary

A comprehensive modernization of eXide, replacing legacy dependencies with modern alternatives and adding LSP-powered language intelligence. This consolidates work from multiple prior PRs (#774, #777) into a single reviewable PR with grouped commits.

Commits (grouped for review)

1. Replace xqlint parser with REx-generated XQuery 3.1 parser

REx-generated parser from W3C EBNF grammar (XQuery 3.1 + Update 3.0 + Full Text 1.0). Adapter layer preserves the existing visitor infrastructure. Includes unit tests for parser, visitors, and static analysis.

2. Replace Ace editor with CodeMirror 6

Full migration from Ace to CM6 with language support, syntax highlighting, keybinding infrastructure, and editor-utils compatibility layer.

3. Add Prettier-based code formatting

Multi-language formatting via Prettier + plugins (XQuery, XML, HTML, CSS, Less, Markdown), bundled with esbuild. Replaces xqlint's CodeFormatter.

4. Remove jQuery, jQuery UI, and AG Grid dependencies

All AJAX calls use fetch, dialogs use native <dialog> via dialog-utils.js, DOM manipulation uses vanilla JS. AG Grid replaced with vanilla HTML tables.

5. Add OpenAPI/Roaster REST API and remove legacy XQuery modules

Replace 15+ legacy XQuery modules with a Roaster-based OpenAPI REST API. All backend operations (storage, query, packages, auth, sync, search, admin) handled through typed JSON endpoints.

6. Redesign UI

Toolbar, status bar with mode selector, native dialog system, tab overflow controls, filterable outline panel, dark mode with proper syntax highlighting (One Dark palette via CSS classHighlighter), system theme detection, keyboard shortcuts dialog, monitoring panel, and responsive layout.

7. Add CM6 autocomplete, semantic highlighting, and multi-language helpers

CM6 autocompletion with LSP-aware function completions (prefixed and unprefixed default namespace), semantic highlighting via decorations, hover tooltips with AST fallback, go-to-definition (same-file and cross-module), multi-cursor rename, and language helpers for JSON, Less, Markdown, JavaScript, CSS, and XML. Native HTML tag/attribute, CSS property/value, and JavaScript local variable completions via CM6 language packs.

8. Add Cypress integration tests and remaining frontend updates

E2E tests for autocomplete, diagnostics, outline, query execution, preferences, formatting, and more (172+ tests). Remaining frontend updates: layout, find/replace, menus, directory browser, drag-and-drop, build scripts, login page.

9. Add tag matching, exist-next compat, and CI fixes

• Matching tag highlight for XML/HTML (cursor on open tag highlights close tag)
• XML/HTML scope breadcrumb in status bar (XPath at cursor position)
• XML Rename Element with multi-cursor (both open + close tags)
• XPath signatures with positional predicates in outline and QuickPicker
• Navigation flash highlight on all jump-to actions
• Selection match highlighting, Cmd+D multi-select, diagnostics panel
• Autocomplete-on-type preference, dynamic namespace registry from server
• Local hover fallback, unprefixed default namespace completions
• Deferred splash screen, dialog focus on primary button
• exist-next compatibility: xs:dayTimeDuration casts, dynamic file module detection, package lookup fix
• CI: skip LSP-dependent tests when lsp:* module unavailable

10. Add Find All References

• Shift+F3 / Navigate > Find All References: QuickPicker listing all references to function/variable at cursor
• Server-side lsp:references() with AST visitor fallback
• Fix hover error handling for non-200 responses

11. WebSocket transport and monitoring

• WebSocket transport layer (ws-transport.js) for real-time LSP and monitoring push
• Monitor panel with WebSocket-based metrics, running/recent queries, kill query button
• LSP diagnostics push via WebSocket after compilation
• Auto-connect with exponential backoff reconnect

12. Server-side cursor query execution

Replaces the old XQueryServlet + session.xq flow with a cursor-based architecture using lsp:eval/lsp:fetch/lsp:close from the exist-lsp package.

Before:

User clicks Eval → HTTP POST to XQueryServlet → session.xq stores results
  → 10 GET requests for results/1..10 → pagination via more GETs

After:

User clicks Eval → POST /api/query → lsp:eval() stores result in Caffeine cache
  → GET /api/query/{id}/results?start=1&count=10&method=xml&indent=yes
  → lsp:fetch() serializes just that page with requested serialization params
  → changing serialization mode re-fetches current page (no re-execution)

Key properties:

• Lazy serialization — only the viewed page is serialized, with configurable W3C serialization parameters
• No client-side buffering — results stay on the server as live node references
• Document links — documentURI and nodeId available per result item
• Dynamic serialization — change output method, indent, or highlight-matches without re-executing
• Pagination — identical UX with next/previous/first/last

This follows the same pattern used by SQL clients (server-side cursors), BaseX GUI (in-process result references), and eXist's own Java admin client (XML-RPC result handles).

Cursor store cache policy

Results are held in a Caffeine cache with configurable eviction:

Setting	Default	Description
cursor.maximumSize	100	Max concurrent cursors (LRU eviction when exceeded)
cursor.expireAfterAccess	300000 (5 min)	Inactivity timeout in ms
cursor.maximumWeight	0 (unlimited)	Max estimated total memory in bytes. When > 0, replaces count-based eviction. Weight per cursor = itemCount × 1KB.

Configuration via exist.xml module parameters:

<parameter name="cursor.maximumSize" value="50"/>
<parameter name="cursor.expireAfterAccess" value="600000"/>
<parameter name="cursor.maximumWeight" value="536870912"/>

13. Query cancellation

What it does:

1. Client-side: AbortController.abort() drops the HTTP connection
2. Server-side: Kills running queries via system:kill-running-xquery() (admin only)

Risks — cancelling state-modifying queries can leave the database inconsistent:

Expression type	Risk	Notes
XQUF (insert/delete/replace/rename)	Medium	PUL may be partially applied
Legacy update (update value, update insert)	High	Applied immediately, no batching
xmldb:store()/xmldb:remove()	High	Side effects immediate
sm:chmod()/sm:chown()	Medium	Permission changes immediate
file:write()/file:delete()	High	Cannot be rolled back

Mitigations:

1. Warning dialog for queries containing update/store/delete/chmod/file expressions
2. Admin-only kill — guest users can only abort the HTTP connection (server query runs to completion)
3. Preclaiming locks (eXist-db PR #6112) prevents deadlocks during partial update application
4. Detection regex covers XQUF, legacy update, xmldb:*, sm:*, file:* mutations
5. Monitor sidebar shows all running queries server-wide with per-query kill buttons

14. Results toolbar and menubar polish

• Copy all results: items are now separated by newlines when pasted
• Number-of-results dropdown: changing the page size now immediately re-fetches from page 1 (previously only updated the internal variable)
• Active toggle buttons: Indent and Highlight Matches buttons now show a solid blue background when active, clearly distinct from the hover state
• Menubar toggle: clicking an already-open menu header now closes it, consistent with standard menubar behaviour

Temporary workarounds (remove after upstream merges)

• LSP test skips: Autocomplete and outline Cypress tests are skipped when lsp:* module is not available. Remove once [feature] Add LSP XQuery function module exist#6130 is merged.
• XQuery 4.0 Prettier guard: Formatting rejects xquery version "4.0" code with a user-friendly message. Remove once prettier-plugin-xquery adds 4.0 support.
• Dynamic file module import: admin.xqm and sync.xqm use util:eval to dynamically load the file module. Can be simplified once eXist-db standardizes on the EXPath File module namespace.
• Error line fallback: compileError parses line/column from error message text when lsp:diagnostics() returns line 0. Remove once lsp:diagnostics() line fix is deployed.
• function-lookup fallback: query.xqm uses function-lookup for 4-arity lsp:fetch when the new overload isn't available. Remove once exist-lsp with serialization parameter support is deployed.

Known limitations

• lsp:hover() does not return info for all registered module functions (e.g., sm:is-dba). Filed as server-side issue.
• XQuery 4.0 parser is lazy-loaded as a separate 664KB bundle to avoid doubling the main bundle size.
• Query evaluation is outside the LSP spec's scope — lsp:eval/lsp:fetch/lsp:close are eXist-specific IDE support functions in the exist-lsp package. Reviewers are invited to weigh in on whether this scope is appropriate or if a separate package would be preferred.

Requirements

• eXist-db 7.0+ with exist-lsp package installed
• Roaster 1.8.0+

Test plan

• Build and install XAR (npm run build && xst package install local build/eXide-3.5.5.xar --force)
• Unit tests pass (npm test — 178 tests)
• Cypress E2E tests pass (213 tests: 201 pass, 11 pending, 1 flaky monitor kill)
• Full smoke test — 213 Cypress tests cover the vast majority; remaining visual/manual items tracked for future sessions
• All LSP endpoints working — autocomplete_spec (11), diagnostics_panel_spec (5), error_status_spec (8), native_autocomplete_spec (7)
• Cmd+Click go-to-definition with centered scroll and flash — goto_references_spec (F3 jump + clickable class)
• Find All References (Shift+F3) with QuickPicker — goto_references_spec (QuickPicker opens + Escape closes)
• Auto-triggered completions with prefix/unprefixed namespace support — autocomplete_spec covers prefixed, unprefixed, fn:, repo:, variable completions
• Dark mode with One Dark syntax highlighting — layout_spec (dark mode layout)
• Prettier formatting for XQuery 3.1, XML, HTML, CSS, JS, Markdown — prettier_format_spec (4 tests)
• Outline panel with nested tree, XPath signatures, and filter — outline_navigation_spec (12 tests)
• Tag matching, scope breadcrumb, selection match highlighting — tag_scope_selection_spec (CSS defined, XML/XQuery scope, selection match)
• exist-next Docker compatibility verified — all development and testing done against exist-next Docker (eXist 7.0.0-SNAPSHOT)
• Query execution with cursor pagination — query_execution_spec "paginates results with next/previous buttons"
• Large result set (1 to 10000) — query_execution_spec "handles large result sets" (10 DOM nodes for 10K items)
• Change serialization mode — query_execution_spec "re-fetches page when serialization mode changes without re-executing"
• Toggle indent/highlight-matches — query_execution_spec "re-fetches page when indent toggle changes"
• Cancel query with warning dialog — query_execution_spec cancel button tests (existence, visibility)
• Monitor sidebar kill button — monitor_spec "shows running query and allows killing it" (flaky timing)

🤖 Generated with Claude Code

[duncdrum]

I would recommend using gh cli, and download both latest release and the min version as per expath-pkg.xml in a matrix

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Done in commits ea7d771–1ddca80: replaced curl with gh release download (using the built-in GITHUB_TOKEN) for the Roaster dependency. The existing release/latest Docker matrix covers the most important cases — testing against the specific semver-min from expath-pkg.xml (6.1.0) would require an old Docker image that predates the Roaster/lsp:* dependencies this PR needs. Happy to add that as a separate matrix entry once we settle on the minimum version the full feature set requires.

[duncdrum]

random timeouts are usually an anti pattern, is there something we can put on a named route, so we can wait on a request and its response?

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Line 22 is already cy.wait('@compile', { timeout: 10000 }) — a named intercept alias set up in beforeEach via cy.intercept('POST', '**/api/query/compile').as('compile'). The remaining cy.wait(50) at line 30 is waiting for a browser MutationObserver microtask to fire after writing to #error-status, which can't be replaced with a network route. It's a minimal delay for a synchronous DOM write, not a timing guess.

[duncdrum]

s.a.

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Same as above — this is the MutationObserver microtask wait (50ms), not a network timing guess.

[duncdrum]

is this covered in our tests?

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Not yet — the deployment dialog involves a multi-step UI (selecting server, entering credentials, triggering package install) that's harder to automate without a test server with the public repo accessible. Tracked in the future test gaps list.

[duncdrum]

Why do we store and commit binaries and java here?

[joewiz]

[This response was co-authored with Claude Code. -Joe]

REx.java is the source for the REx parser generator (by Gunther Rademacher, bottlecaps.de), included for transparency and reproducible builds — the site hosts the tool but isn't always reachable. The compiled REx.class has been removed from git (commit ea7d771); npm run generate-parser now compiles it on first use via javac. The .java source stays so contributors can inspect and rebuild without depending on an external download.

[duncdrum]

I m starting to be a bit concerned about a mix of Java, Rust, Js,… it adds quite a bit of complexity.

That being said love tauri.

Should be build on ci though.

[line-o]

@joewiz is it possible to split this PR up into

1. add features and modernize application XAR
2. eXide desktop

I do like the direction you are heading in and I would like to get this into the next release as a bundled application.
But for this we need to be able to grasp the changes.
A desktop application will increase the complexity and maintenance effort at this point that will put this goal at risk.
The desktop application can even be a separate project that has this repo as a dependency.

[joewiz]

@line-o Thanks for considering this PR for the next release! Sure, I can update the PR to set aside the eXide Desktop feature.

[joewiz]

Superseded by #794, which contains only the web-app modernization (no eXide Desktop). The desktop scope will follow in a separate PR. Closing this PR but leaving the pr/modernize-eXide branch in place since it still holds the desktop commits needed for the follow-on PR.