[DE-4902] enforcing string casting when column type is INTEGER with (38,0) precision and scale

[gregosaurus]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• New Query Runner (Data Source)
• New Alert Destination
• Other

Description

Fixing JS Int overflow by casting all INTEGER types to String

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

[gregosaurus]

[self-review] this isn't going to work since line 93 the type is changed.

[inytar]

Hey Greg,
Let's take an other approach here.
The issue is clearly in the json -> javascript object. As we can see in this screenshot.

Let's try and tackle that. A bit of investigations shows that Redash uses axios as helper to do http calls. And axios is automatically transforming the json response to a javascript object.

The good part is that we can overwrite this transformResponse function in the axios config.
Now we just use a json library that works with big numbers, and we should be good.

[inytar]

What are you doing here? Is this a if scale is 0? Could you add a comment to make that clear.

[gregosaurus]

I'll enhance, i refactored to make this part "shorter" but i do agree it's blurry now.

Basically cursor.description has different metadata for each index (see: https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api#description), index 1 being the data_type code.

The snowflake doc is blurry in terms of what is provided as metadata in description in case of INTEGER:

Synonymous with NUMBER, except that precision and scale can’t be specified (that is, it always defaults to NUMBER(38, 0))

Does that mean that precision and scale are set to None in the description object when it's a pure integer ? but when it is a float it is ? the existing code from redash team in snowflake query_runner might lead me to think so:

def determine_type(cls, data_type, scale):
        t = TYPES_MAP.get(data_type, None)
        if t == TYPE_INTEGER and scale > 0:
            return TYPE_FLOAT
        return t

TL;DR, to fix the issue i had to check on the type as it was done by the snippet above, minus the scale.

(further read https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api#label-python-connector-type-codes and https://docs.snowflake.com/en/sql-reference/data-types-numeric)

[inytar]

Doesn't this change all values types that aren't number to strings?

Suggested change

	if column[1] == 0 and column[5] > 0:
	row_dict[column_name] = value
	else:
	# Cast INT to STRING to avoid potential overflow in JS
	row_dict[column_name] = str(value)
	if column[1] == 0 and column[5] == 0:
	# Cast INT to STRING to avoid potential overflow in JS
	row_dict[column_name] = str(value)
	else:
	row_dict[column_name] = value

[gregosaurus]

I just pushed the wrong commit...