Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion amazon-redshift-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Amazon Redshift plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion aurora-mysql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Aurora DB MySQL plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion aurora-postgresql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Aurora DB PostgreSQL plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion cloudsql-mysql-plugin/docs/CloudSQLMySQL-batchsink.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Can be found in the instance overview page.

**Password:** Password to use to connect to the specified database.

**Transaction Isolation Level:** Transaction isolation level for queries run by this sink.
**Transaction Isolation Level:** Transaction isolation level for queries run by this sink.

**Connection Timeout:** The timeout value (in seconds) used for socket connect operations. If connecting to the server
takes longer than this value, the connection is broken. A value of 0 means that it is disabled.
Expand Down
8 changes: 8 additions & 0 deletions cloudsql-mysql-plugin/docs/CloudSQLMySQL-batchsource.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,14 @@ For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is s

**Password:** Password to use to connect to the specified database.

**Transaction Isolation Level:** The transaction isolation level of the database connection.
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.

For more details on the Transaction Isolation Levels supported in CloudSQL MySQL, refer to the [CloudSQL MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html)

**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

Expand Down
8 changes: 8 additions & 0 deletions cloudsql-mysql-plugin/docs/CloudSQLMySQL-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@ authentication. Optional for databases that do not require authentication.

**Password:** Password to use to connect to the specified database.

**Transaction Isolation Level:** The transaction isolation level of the database connection.
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
- TRANSACTION_READ_UNCOMMITTED: Allows dirty reads (reading uncommitted changes from other transactions). Non-repeatable reads and phantom reads are possible.

For more details on the Transaction Isolation Levels supported in CloudSQL MySQL, refer to the [CloudSQL MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html)

**Connection Arguments:** A list of arbitrary string tag/value pairs as connection arguments. These arguments
will be passed to the JDBC driver, as connection arguments, for JDBC drivers that may need additional configurations.
This is a semicolon-separated list of key-value pairs, where each pair is separated by a equals '=' and specifies
Expand Down
2 changes: 1 addition & 1 deletion cloudsql-mysql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>CloudSQL MySQL plugin</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,13 @@ public static class CloudSQLMySQLSourceConfig extends AbstractDBSpecificSourceCo
@Description("The existing connection to use.")
private CloudSQLMySQLConnectorConfig connection;

@Name(TRANSACTION_ISOLATION_LEVEL)
@Description("Transaction isolation level for queries run by this source.")
@Nullable
private String transactionIsolationLevel;
Comment on lines +155 to +158

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The transactionIsolationLevel field is missing the @Macro annotation. Adding @Macro allows users to dynamically configure the transaction isolation level using pipeline arguments or macros at runtime.

Suggested change
@Name(TRANSACTION_ISOLATION_LEVEL)
@Description("Transaction isolation level for queries run by this source.")
@Nullable
private String transactionIsolationLevel;
@Name(TRANSACTION_ISOLATION_LEVEL)
@Description("Transaction isolation level for queries run by this source.")
@Nullable
@Macro
private String transactionIsolationLevel;


private static final String DEFAULT_TRANSACTION_ISOLATION_LEVEL = "TRANSACTION_REPEATABLE_READ";

@Override
protected Map<String, String> getDBSpecificArguments() {
if (getFetchSize() == null || getFetchSize() <= 0) {
Expand All @@ -164,6 +171,11 @@ protected Map<String, String> getDBSpecificArguments() {
return arguments;
}

@Override
public String getTransactionIsolationLevel() {
return transactionIsolationLevel != null ? transactionIsolationLevel : DEFAULT_TRANSACTION_ISOLATION_LEVEL;
}
Comment on lines +174 to +177

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When useConnection is enabled, the source should inherit the transaction isolation level configured in the connection. Currently, getTransactionIsolationLevel() completely ignores the connection's configuration and only checks the stage-level property, which means any setting configured in the Connection is ignored.

Please update the method to delegate to the connection config when useConnection is true.

    @Override
    public String getTransactionIsolationLevel() {
      if (useConnection != null && useConnection && connection != null) {
        return connection.getTransactionIsolationLevel() != null ? connection.getTransactionIsolationLevel() : DEFAULT_TRANSACTION_ISOLATION_LEVEL;
      }
      return transactionIsolationLevel != null ? transactionIsolationLevel : DEFAULT_TRANSACTION_ISOLATION_LEVEL;
    }


@Override
public void validate(FailureCollector collector) {
ConfigUtil.validateConnection(this, useConnection, connection, collector);
Expand Down
14 changes: 14 additions & 0 deletions cloudsql-mysql-plugin/widgets/CloudSQLMySQL-batchsource.json
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,20 @@
"label": "Password",
"name": "password"
},
{
"widget-type": "select",
"label": "Transaction Isolation Level",
"name": "transactionIsolationLevel",
"widget-attributes": {
"default": "TRANSACTION_REPEATABLE_READ",
"values": [
"TRANSACTION_REPEATABLE_READ",
"TRANSACTION_READ_UNCOMMITTED",
"TRANSACTION_READ_COMMITTED",
"TRANSACTION_SERIALIZABLE"
]
}
},
Comment on lines +87 to +100

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The transactionIsolationLevel property is not included in the showConnectionProperties filter's show list. This means that when useConnection is set to true, the transactionIsolationLevel dropdown will still be visible in the Source stage UI, which is confusing since the setting should be inherited from the Connection.

Please update the showConnectionProperties filter (further down in this file) to include "transactionIsolationLevel" in its show list so that it is hidden when a connection is used.

{
"widget-type": "keyvalue",
"label": "Connection Arguments",
Expand Down
14 changes: 14 additions & 0 deletions cloudsql-mysql-plugin/widgets/CloudSQLMySQL-connector.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,20 @@
"widget-attributes": {
"default": "3306"
}
},
{
"widget-type": "select",
"label": "Transaction Isolation Level",
"name": "transactionIsolationLevel",
"widget-attributes": {
"default": "TRANSACTION_REPEATABLE_READ",
"values": [
"TRANSACTION_REPEATABLE_READ",
"TRANSACTION_READ_UNCOMMITTED",
"TRANSACTION_READ_COMMITTED",
"TRANSACTION_SERIALIZABLE"
]
}
}
]
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Can be found in the instance overview page.

**Password:** Password to use to connect to the specified database.

**Transaction Isolation Level:** Transaction isolation level for queries run by this sink.
**Transaction Isolation Level:** Transaction isolation level for queries run by this sink.

**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,14 @@ For example, 'SELECT MIN(id),MAX(id) FROM table'. Not required if numSplits is s

**Password:** Password to use to connect to the specified database.

**Transaction Isolation Level:** The transaction isolation level of the database connection.
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
- Note: PostgreSQL does not implement `TRANSACTION_READ_UNCOMMITTED` as a distinct isolation level. Instead, this mode behaves identically to`TRANSACTION_READ_COMMITTED`, which is why it is not exposed as a separate option.

For more details on the Transaction Isolation Levels supported in CloudSQL PostgreSQL, refer to the [CloudSQL PostgreSQL documentation](https://www.postgresql.org/docs/current/transaction-iso.html)

**Connection Arguments:** A list of arbitrary string key/value pairs as connection arguments. These arguments
will be passed to the JDBC driver as connection arguments for JDBC drivers that may need additional configurations.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@ authentication. Optional for databases that do not require authentication.

**Password:** Password to use to connect to the specified database.

**Transaction Isolation Level:** The transaction isolation level of the database connection.
- TRANSACTION_READ_COMMITTED: No dirty reads. Non-repeatable reads and phantom reads are possible.
- TRANSACTION_SERIALIZABLE: No dirty reads. Non-repeatable and phantom reads are prevented.
- TRANSACTION_REPEATABLE_READ: No dirty reads. Prevents non-repeatable reads, but phantom reads are still possible.
- Note: PostgreSQL does not implement `TRANSACTION_READ_UNCOMMITTED` as a distinct isolation level. Instead, this mode behaves identically to`TRANSACTION_READ_COMMITTED`, which is why it is not exposed as a separate option.

For more details on the Transaction Isolation Levels supported in CloudSQL PostgreSQL, refer to the [CloudSQL PostgreSQL documentation](https://www.postgresql.org/docs/current/transaction-iso.html)

**Connection Arguments:** A list of arbitrary string tag/value pairs as connection arguments. These arguments
will be passed to the JDBC driver, as connection arguments, for JDBC drivers that may need additional configurations.
This is a semicolon-separated list of key-value pairs, where each pair is separated by a equals '=' and specifies
Expand Down
2 changes: 1 addition & 1 deletion cloudsql-postgresql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>CloudSQL PostgreSQL plugin</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,13 @@ public static class CloudSQLPostgreSQLSourceConfig extends AbstractDBSpecificSou
@Description("The existing connection to use.")
private CloudSQLPostgreSQLConnectorConfig connection;

@Name(TRANSACTION_ISOLATION_LEVEL)
@Description("Transaction isolation level for queries run by this source.")
@Nullable
private String transactionIsolationLevel;
Comment on lines +156 to +159

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The transactionIsolationLevel field is missing the @Macro annotation. Adding @Macro allows users to dynamically configure the transaction isolation level using pipeline arguments or macros at runtime.

Suggested change
@Name(TRANSACTION_ISOLATION_LEVEL)
@Description("Transaction isolation level for queries run by this source.")
@Nullable
private String transactionIsolationLevel;
@Name(TRANSACTION_ISOLATION_LEVEL)
@Description("Transaction isolation level for queries run by this source.")
@Nullable
@Macro
private String transactionIsolationLevel;


private static final String DEFAULT_TRANSACTION_ISOLATION_LEVEL = "TRANSACTION_READ_COMMITTED";

@Override
protected Map<String, String> getDBSpecificArguments() {
return Collections.emptyMap();
Expand All @@ -169,6 +176,11 @@ protected CloudSQLPostgreSQLConnectorConfig getConnection() {
return connection;
}

@Override
public String getTransactionIsolationLevel() {
return transactionIsolationLevel != null ? transactionIsolationLevel : DEFAULT_TRANSACTION_ISOLATION_LEVEL;
}
Comment on lines +179 to +182

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When useConnection is enabled, the source should inherit the transaction isolation level configured in the connection. Currently, getTransactionIsolationLevel() completely ignores the connection's configuration and only checks the stage-level property, which means any setting configured in the Connection is ignored.

Please update the method to delegate to the connection config when useConnection is true.

    @Override
    public String getTransactionIsolationLevel() {
      if (useConnection != null && useConnection && connection != null) {
        return connection.getTransactionIsolationLevel() != null ? connection.getTransactionIsolationLevel() : DEFAULT_TRANSACTION_ISOLATION_LEVEL;
      }
      return transactionIsolationLevel != null ? transactionIsolationLevel : DEFAULT_TRANSACTION_ISOLATION_LEVEL;
    }


@Override
public void validate(FailureCollector collector) {
ConfigUtil.validateConnection(this, useConnection, connection, collector);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,19 @@
"label": "Password",
"name": "password"
},
{
"widget-type": "select",
"label": "Transaction Isolation Level",
"name": "transactionIsolationLevel",
"widget-attributes": {
"default": "TRANSACTION_READ_COMMITTED",
"values": [
"TRANSACTION_READ_COMMITTED",
"TRANSACTION_REPEATABLE_READ",
"TRANSACTION_SERIALIZABLE"
]
}
},
Comment on lines +87 to +99

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The transactionIsolationLevel property is not included in the showConnectionProperties filter's show list. This means that when useConnection is set to true, the transactionIsolationLevel dropdown will still be visible in the Source stage UI, which is confusing since the setting should be inherited from the Connection.

Please update the showConnectionProperties filter (further down in this file) to include "transactionIsolationLevel" in its show list so that it is hidden when a connection is used.

{
"widget-type": "keyvalue",
"label": "Connection Arguments",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,19 @@
"widget-attributes": {
"default": "5432"
}
},
{
"widget-type": "select",
"label": "Transaction Isolation Level",
"name": "transactionIsolationLevel",
"widget-attributes": {
"default": "TRANSACTION_READ_COMMITTED",
"values": [
"TRANSACTION_READ_COMMITTED",
"TRANSACTION_REPEATABLE_READ",
"TRANSACTION_SERIALIZABLE"
]
}
}
]
},
Expand Down
2 changes: 1 addition & 1 deletion database-commons/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Database Commons</name>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ public abstract class AbstractDBSpecificSourceConfig extends PluginConfig implem
public static final String DATABASE = "database";
public static final String FETCH_SIZE = "fetchSize";
public static final String DEFAULT_FETCH_SIZE = "1000";
public static final String TRANSACTION_ISOLATION_LEVEL = "transactionIsolationLevel";

@Name(Constants.Reference.REFERENCE_NAME)
@Description(Constants.Reference.REFERENCE_NAME_DESCRIPTION)
Expand Down
2 changes: 1 addition & 1 deletion db2-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>IBM DB2 plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion generic-database-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Generic database plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion generic-db-argument-setter/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Generic database argument setter plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion mariadb-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Maria DB plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion memsql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Memsql plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion mssql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Microsoft SQL Server plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion mysql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Mysql plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion netezza-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Netezza plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion oracle-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>Oracle plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

<groupId>io.cdap.plugin</groupId>
<artifactId>database-plugins-parent</artifactId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Database Plugins</name>
<description>Collection of database plugins</description>
Expand Down
2 changes: 1 addition & 1 deletion postgresql-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>PostgreSQL plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion saphana-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<name>SAP HANA plugin</name>
Expand Down
2 changes: 1 addition & 1 deletion teradata-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<artifactId>database-plugins-parent</artifactId>
<groupId>io.cdap.plugin</groupId>
<version>1.12.4</version>
<version>1.12.5-SNAPSHOT</version>
</parent>

<artifactId>teradata-plugin</artifactId>
Expand Down
Loading