Fix precision timestamp#28919
Conversation
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
3 similar comments
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
- Update type assertions from timestamp(3) to timestamp(6) with time zone - Replace TIMESTAMP_TZ_MILLIS with TIMESTAMP_TZ_MICROS in column handle construction for Delta-specific tests - Add end-to-end test verifying microsecond values are preserved on INSERT/SELECT roundtrip - Add backward compatibility test verifying MILLIS-encoded parquet files written by older Trino versions still read correctly with zero sub-millisecond digits
Remove comment
…e precision - update values for timestamp(6)
- update values for timestamp(6)
patrickmoch
force-pushed
the
fix-precision-timestamp
branch
from
356e8c6 to
c9f7b44
Compare
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
Bumps [io.github.gitflow-incremental-builder:gitflow-incremental-builder](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder) from 4.5.6 to 4.6.0. - [Release notes](https://github.com/gitflow-incremental-builder/gitflow-incremental-builder/releases) - [Commits](gitflow-incremental-builder/gitflow-incremental-builder@v4.5.6...v4.6.0) --- updated-dependencies: - dependency-name: io.github.gitflow-incremental-builder:gitflow-incremental-builder dependency-version: 4.6.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
- Update type assertions from timestamp(3) to timestamp(6) with time zone - Replace TIMESTAMP_TZ_MILLIS with TIMESTAMP_TZ_MICROS in column handle construction for Delta-specific tests - Add end-to-end test verifying microsecond values are preserved on INSERT/SELECT roundtrip - Add backward compatibility test verifying MILLIS-encoded parquet files written by older Trino versions still read correctly with zero sub-millisecond digits
Remove comment
…e precision - update values for timestamp(6)
- update values for timestamp(6)
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
- Update type assertions from timestamp(3) to timestamp(6) with time zone - Replace TIMESTAMP_TZ_MILLIS with TIMESTAMP_TZ_MICROS in column handle construction for Delta-specific tests - Add end-to-end test verifying microsecond values are preserved on INSERT/SELECT roundtrip - Add backward compatibility test verifying MILLIS-encoded parquet files written by older Trino versions still read correctly with zero sub-millisecond digits
Remove comment
…e precision - update values for timestamp(6)
- update values for timestamp(6)
…och/trino into fix-precision-timestamp
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
…x-precision-timestamp
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack. |
|
Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time. |
3 participants
Description
Fixes precision loss when reading timestamp with time zone columns from Delta Lake tables backed by Parquet files written with microsecond precision (e.g. by Spark).
Additional context and related issues
The Delta Lake protocol spec defines timestamp as microsecond precision stored as int64 with isAdjustedToUTC=true in Parquet. Trino was mapping this to TIMESTAMP_TZ_MILLIS (precision 3), causing the sub-millisecond digits to be silently dropped on read. For example, a value written by Spark as 2024-01-12 09:30:47.405423 UTC would be read by Trino as 2024-01-12 09:30:47.405 UTC.
The fix maps the Delta "timestamp" type to TIMESTAMP_TZ_MICROS (precision 6) and updates the validation guard in validatePrimitiveType accordingly.
Backward compatibility with Parquet files written by older Trino versions (which used TIMESTAMP(MILLIS) physical encoding) is preserved by handling the MILLIS case in ColumnReaderFactory for LongTimestampWithTimeZone rather than throwing. These files will read correctly with zeros in the sub-millisecond digits, which accurately reflects the precision they were written with.
Note: timestamp_ntz (without timezone) was already correctly mapped to TIMESTAMP_MICROS (precision 6) — this change brings timestamp (with timezone) into alignment with it.
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(X ) Release notes are required, with the following suggested text: