Skip to content

fix: resolve lineage extraction bug for MySQL 5.7.3 #28036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cchenan wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: resolve lineage extraction bug for MySQL 5.7.3 #28036

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ceb6be7
fix: resolve lineage extraction bug for MySQL 5.7.3
cchenan May 11, 2026
f1f1c51
fix: also convert slow_log.sql_text
cchenan May 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions ingestion/src/metadata/ingestion/source/database/mysql/queries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,13 @@

import textwrap

# general_log.argument and slow_log.sql_text are MEDIUMTEXT on older MySQL and MEDIUMBLOB on 5.7+;
# CONVERT(... USING utf8mb4) unifies behavior for SELECT/WHERE.
MYSQL_SQL_STATEMENT = textwrap.dedent(
"""
SELECT
NULL `database_name`,
argument `query_text`,
CONVERT(argument USING utf8mb4) `query_text`,
Copy link
Copy Markdown
Collaborator

@harshach harshach May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this work with only msyql 5.7 or any version after that too?

Copy link
Copy Markdown
Author

@cchenan cchenan May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this work with only msyql 5.7 or any version after that too?

It works for both MySQL 5.7.3+ and 8.0. I've tested with 5.7.23 and 8.0, no regression. The CONVERT function is supported in all MySQL 5.7+ and 8.0 versions.

Copy link
Copy Markdown
Author

@cchenan cchenan May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The static check errors reported (e.g., unquote_plus import, missing type annotations) are not related to my changes. They exist in the current main branch. Could you please advise how to proceed? Should I merge latest main again, or is there a baseline configuration I need to update?

Copy link
Copy Markdown
Author

@cchenan cchenan May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for testing! I see the deployment hit some failures. Is there anything on my side that I can help fix? Or is

event_time `start_time`,
NULL `end_time`,
NULL `duration`,
Expand All @@ -29,8 +31,8 @@
FROM mysql.general_log
WHERE command_type = 'Query'
AND event_time between '{start_time}' and '{end_time}'
AND argument NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
AND argument NOT LIKE '/* {{"app": "dbt", %%}} */%%'
AND CONVERT(argument USING utf8mb4) NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
AND CONVERT(argument USING utf8mb4) NOT LIKE '/* {{"app": "dbt", %%}} */%%'
{filters}
ORDER BY event_time desc
LIMIT {result_limit};
Expand All @@ -42,7 +44,7 @@
"""
SELECT
NULL `database_name`,
sql_text `query_text`,
CONVERT(sql_text USING utf8mb4) `query_text`,
start_time `start_time`,
NULL `end_time`,
NULL `duration`,
Expand All @@ -52,8 +54,8 @@
NULL `aborted`
FROM mysql.slow_log
WHERE start_time between '{start_time}' and '{end_time}'
AND sql_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
AND sql_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
AND CONVERT(sql_text USING utf8mb4) NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
AND CONVERT(sql_text USING utf8mb4) NOT LIKE '/* {{"app": "dbt", %%}} */%%'
{filters}
ORDER BY start_time desc
LIMIT {result_limit};
Expand All @@ -62,13 +64,13 @@

MYSQL_TEST_GET_QUERIES = textwrap.dedent(
"""
SELECT `argument` from mysql.general_log limit 1;
SELECT CONVERT(argument USING utf8mb4) AS query_text FROM mysql.general_log LIMIT 1;
"""
)

MYSQL_TEST_GET_QUERIES_SLOW_LOGS = textwrap.dedent(
"""
SELECT `sql_text` from mysql.slow_log limit 1;
SELECT CONVERT(sql_text USING utf8mb4) AS query_text FROM mysql.slow_log LIMIT 1;
"""
)

Expand Down
Loading