Processing Cross Database Lineage not supported for Mssql

[igoresz]

Hello
Is "Processing Cross Database Lineage for Mssql" in roadmap of OpenMetaData team?
Many thanks in advance.

[sqlparser]

This one sat unanswered for a while — sorry to bump it late, but the cross-database MSSQL case has a couple of sharp edges worth writing up for anyone else landing here from search.

Two things have to work for cross-database lineage to show up:

1) The parser has to recognize [DatabaseA].[dbo].[Table] as a 3-part name.

OpenMetadata's parser path for MSSQL has historically had trouble with square-bracket identifiers that span databases — this is tracked as #16424, and the root cause was a greedy regex r"\[(.*)\]" that matched across bracket pairs, so [DatabaseA].[dbo].[Customers] was being collapsed into a single mangled identifier rather than a proper 3-part FQN. Part of that has been addressed in newer sqlfluff versions, but if the same query also sits inside a stored procedure (CREATE PROCEDURE ... BEGIN ... END), the outer T-SQL wrapper still defeats the parser chain (see #16737, #17586, #25299) and lineage silently drops to zero.

2) OpenMetadata has to have both databases registered as entities.

Even when the parser returns a clean 3-part name, the lineage push (PUT /api/v1/lineage) needs an entity UUID on both ends. For cross-database lineage that means both DatabaseA and DatabaseB must have been ingested through the same MSSQL service connector (or a connector that covers both), so their tables exist in OpenMetadata with resolvable FQNs like mssql_prod.DatabaseA.dbo.Customers. If DatabaseB is only ever referenced in a query but never ingested as metadata, there's nothing to link the edge to.

---

A pragmatic path that works today:

We've been maintaining a small open companion tool — gsp-openmetadata-sidecar — that handles both of the parser-layer issues above (square brackets across databases + T-SQL procedural wrappers) and pushes lineage through the public PUT /api/v1/lineage API. It doesn't patch OpenMetadata core; it just fills in the edges the core parser drops.

Concrete example on the exact shape of SQL this discussion is asking about:

CREATE VIEW [ReportDB].[dbo].[vw_CustomerOrders] AS
SELECT
    [SalesDB].[dbo].[Customers].[CustomerID],
    [SalesDB].[dbo].[Customers].[CustomerName],
    [SalesDB].[dbo].[Orders].[OrderID],
    [SalesDB].[dbo].[Orders].[OrderDate]
FROM [SalesDB].[dbo].[Customers]
INNER JOIN [SalesDB].[dbo].[Orders]
    ON [SalesDB].[dbo].[Customers].[CustomerID] = [SalesDB].[dbo].[Orders].[CustomerID]

Dry-run output:

SalesDB.dbo.Customers --> ReportDB.dbo.vw_CustomerOrders (CustomerID, CustomerName)
SalesDB.dbo.Orders   --> ReportDB.dbo.vw_CustomerOrders (OrderID, OrderDate)

Usage:

pip install gsp-openmetadata-sidecar

gsp-openmetadata-sidecar \
  --sql-file your_crossdb_query.sql \
  --db-vendor dbvmssql \
  --dry-run

Caveat it shares with any other tool: both SalesDB and ReportDB still need to be registered in OpenMetadata as database entities for the lineage edge to have UUIDs to bind to. That's an ingestion-config answer, not a parser answer — but once both are ingested, the sidecar will resolve the cross-database FQNs correctly.

More details if useful — the cross-database bracket case, temp-table chains, and MERGE are all walked through with runnable examples:

• Why MSSQL stored-procedure lineage is silently empty in OpenMetadata (and the fix) — narrative deep-dive
• OpenMetadata MSSQL sidecar — install + CLI quickstart
• gudusoftware/gsp-openmetadata-sidecar — source and example SQL files

Happy to look at a scrubbed sample of your actual cross-database query and report back what lineage it pulls out — often the specific shape of the SQL (stored-proc-wrapped? view? synonym? linked-server?) changes which layer is actually breaking.

(Disclosure: I work at Gudu Software, which builds the SQLFlow engine behind the sidecar. Sidecar is free to use; backend has an anonymous tier and a self-hosted Docker option for teams that need data to stay inside their network.)

[harshach]

@ulixius9 @mohittilala can you answer this