-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Add Teradata connector #26574
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Add Teradata connector #26574
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
8f3ba59
Initial setup
sc250072 3f1bb34
Streamlined Teradata plugin, added DEW tests, and integrated with Tri…
sc250072 31d76a6
Update config.properties
sc250072 c305582
Add Teradata connector (#27)
sc250072 c53a1a4
Updated files with license headers
sc250072 49da91d
Checkstyle errors resolved
sc250072 4106e3b
Skipping testDataMappingSmokeTest due to issue with ClearScape instance
sc250072 d2cd258
Removed not required tests
sc250072 e128e28
Fixed PR workflow reported issues
sc250072 a92fa94
Fixed github error prone check issues
sc250072 e4b2713
Added token to JWT authentication mechanism
sc250072 a448b10
Merge branch 'trinodb:master' into master
sc250072 a0b8ea7
addressed PR review comments of trino team
sc250072 db0ee55
Refactor Teradata documentation for clarity and formatting improvements
sc250072 f512297
Fix package declaration in Region.java for consistency
sc250072 92d4c91
Remove unused imports and variables in TeradataClient and TeradataCli…
sc250072 f4bf536
Remove unused imports in TeradataClient.java
sc250072 033d2e7
Remove commented-out code in TeradataClient.java
sc250072 06bc33b
Add @Override annotations to test methods in TeradataConnectorTest.java
sc250072 8dda285
Fix formatting in TeradataClient.java by removing trailing whitespace
sc250072 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| # Teradata connector | ||
|
|
||
| ```{raw} html | ||
| <img src="../_static/img/teradata.png" class="connector-logo"> | ||
| ``` | ||
|
|
||
| The Teradata connector allows querying and creating tables in an external | ||
| [Teradata](https://www.teradata.com/) database. This can be used to join | ||
| data between different systems like Teradata and Hive, or between different Teradata instances. | ||
|
|
||
| ## Requirements | ||
|
|
||
| To connect to Teradata, you need: | ||
|
|
||
| - Teradata Database | ||
| - Network access from the Trino coordinator and workers to Teradata. Port | ||
| 1025 is the default port | ||
|
|
||
| ## Configuration | ||
|
|
||
| To configure the Teradata connector, create a catalog properties file in | ||
| `etc/catalog` named, for example, `teradata.properties`, to mount the Teradata | ||
| connector as the `teradata` catalog. Create the file with the following | ||
| contents, replacing the connection properties as appropriate for your setup: | ||
|
|
||
| ```properties | ||
| connector.name=teradata | ||
| connection-url=jdbc:teradata://example.teradata.com/CHARSET=UTF8,TMODE=ANSI,LOGMECH=TD2 | ||
| connection-user=*** | ||
| connection-password=*** | ||
| ``` | ||
|
|
||
| The `connection-url` defines the connection information and parameters to pass | ||
| to the Teradata JDBC driver. The supported parameters for the URL are | ||
| available in the | ||
| [Teradata JDBC documentation](https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#BABJIHBJ). | ||
| For example, the following `connection-url` configures character encoding, | ||
| transaction mode, and authentication. | ||
|
|
||
| ```properties | ||
| connection-url=jdbc:teradata://example.teradata.com/CHARSET=UTF8,TMODE=ANSI,LOGMECH=TD2 | ||
| ``` | ||
|
|
||
| The `connection-user` and `connection-password` are typically required and | ||
| determine the user credentials for the connection, often a service user. | ||
|
|
||
| ### Connection security | ||
|
|
||
| If you have TLS configured with a globally-trusted certificate installed on | ||
| your data source, you can enable TLS between your cluster and the data | ||
| source by appending parameters to the JDBC connection string set in the | ||
| connection-url catalog configuration property. | ||
|
|
||
| For example, to specify SSLMODE: | ||
|
|
||
| ```properties | ||
| connection-url=jdbc:teradata://example.teradata.com/SSLMODE=REQUIRED | ||
| ``` | ||
|
|
||
| For more information on TLS configuration options, see the | ||
| Teradata [JDBC documentation](https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#URL_SSLMODE). | ||
|
|
||
| ```{include} jdbc-authentication.fragment | ||
| ``` | ||
|
|
||
| ### Multiple Teradata databases | ||
|
|
||
| You can have as many catalogs as you need, so if you have additional Teradata | ||
| databases, simply add another properties file to etc/catalog with a different | ||
| name, making sure it ends in .properties. | ||
| For example, if you name the property file sales.properties, Trino creates a | ||
| catalog named sales using the configured connector. | ||
|
|
||
| ## Type mapping | ||
|
|
||
| Because Trino and Teradata each support types that the other does not, this | ||
| connector {ref}`modifies some types <type-mapping-overview>` when reading data. | ||
| Refer to the following sections for type mapping in when reading data from | ||
| Teradata to Trino. | ||
|
|
||
| ### Teradata type to Trino type mapping | ||
|
|
||
| The connector maps Teradata types to the corresponding Trino types following | ||
| this table: | ||
|
|
||
| :::{list-table} Teradata type to Trino type mapping | ||
| :widths: 40, 40, 20 | ||
| :header-rows: 1 | ||
|
|
||
| * - Teradata type | ||
| - Trino type | ||
| - Notes | ||
| * - `TINYINT` | ||
| - `TINYINT` | ||
| - | ||
| * - `SMALLINT` | ||
| - `SMALLINT` | ||
| - | ||
| * - `INTEGER` | ||
| - `INTEGER` | ||
| - | ||
| * - `BIGINT` | ||
| - `BIGINT` | ||
| - | ||
| * - `REAL` | ||
| - `DOUBLE` | ||
| - | ||
| * - `DOUBLE` | ||
| - `DOUBLE` | ||
| - | ||
| * - `FLOAT` | ||
| - `DOUBLE` | ||
| - | ||
| * - `NUMBER(p, s)` | ||
| - `DECIMAL(p, s)` | ||
| - | ||
| * - `NUMERIC(p, s)` | ||
| - `DECIMAL(p, s)` | ||
| - | ||
| * - `DECIMAL(p, s)` | ||
| - `DECIMAL(p, s)` | ||
| - | ||
| * - `CHAR(n)` | ||
| - `CHAR(n)` | ||
| - | ||
| * - `CHARACTER(n)` | ||
| - `CHAR(n)` | ||
| - | ||
| * - `VARCHAR(n)` | ||
| - `VARCHAR(n)` | ||
| - | ||
| * - `DATE` | ||
| - `DATE` | ||
| - | ||
| ::: | ||
|
|
||
| No other types are supported. | ||
|
|
||
| ```{include} jdbc-type-mapping.fragment | ||
| ``` | ||
|
|
||
| ## Querying Teradata | ||
|
|
||
| The Teradata connector provides a schema for every Teradata database. You can | ||
| see the available Teradata databases by running SHOW SCHEMAS: | ||
|
|
||
| ``` | ||
| SHOW SCHEMAS FROM teradata; | ||
| ``` | ||
|
|
||
| If you have a Teradata database named sales, you can view the tables in this | ||
| database by running SHOW TABLES: | ||
|
|
||
| ``` | ||
| SHOW TABLES FROM teradata.sales; | ||
| ``` | ||
|
|
||
| You can see a list of the columns in the orders table in the sales database | ||
| using either of the following: | ||
|
|
||
| ``` | ||
| DESCRIBE teradata.sales.orders; | ||
| SHOW COLUMNS FROM teradata.sales.orders; | ||
| ``` | ||
|
|
||
| Finally, you can access the orders table in the sales database: | ||
|
|
||
| ``` | ||
| SELECT * FROM teradata.sales.orders; | ||
| ``` | ||
|
|
||
| ## SQL support | ||
|
|
||
| The connector provides read access access to data and metadata in | ||
| a Teradata database. The connector supports the {ref}`globally available | ||
| <sql-globally-available>` and {ref}`read operation <sql-read-operations>` | ||
| statements. | ||
|
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Teradata Connector Developer Notes | ||
|
|
||
| The Teradata connector module has both unit tests and integration tests. | ||
| The integration tests require access to a [Teradata ClearScape Analytics™ Experience](https://clearscape.teradata.com/sign-in). | ||
| You can follow the steps below to run the integration tests locally. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| #### 1. Create a new ClearScape Analytics™ Experience account | ||
|
|
||
| If you don't already have one, sign up at: | ||
|
|
||
| [Teradata ClearScape Analytics™ Experience](https://www.teradata.com/getting-started/demos/clearscape-analytics) | ||
|
|
||
| #### 2. Login | ||
|
|
||
| Sign in with your new account at: | ||
|
|
||
| [ClearScape Analytics™ Experience Login](https://clearscape.teradata.com/sign-in) | ||
|
|
||
| #### 3. Collect the API Token | ||
|
|
||
| Use the **Copy API Token** button in the UI to retrieve your token. | ||
|
|
||
| #### 4. Define the following environment variables | ||
|
|
||
| ⚠️ **Note:** The Teradata database password must be **at least 8 characters long**. | ||
|
|
||
| ``` | ||
| export CLEARSCAPE_TOKEN=<API Token> | ||
| export CLEARSCAPE_PASSWORD=<Password for Teradata database (min 8 chars)> | ||
| ``` | ||
|
|
||
| ## Running Integration Tests | ||
|
|
||
| Once the environment variables are set, run the integration tests with: | ||
|
|
||
| ⚠️ **Note:** Run the following command from the Trino parent directory. | ||
|
|
||
| ``` | ||
| ./mvnw clean install -pl :trino-teradata | ||
| ``` |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update ci.yml so we can run tests on CI. You can ask maintainers to add secrets to this repository in Slack.