Skip to content

Enable writing PKs per extraction batch #2329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JFriel merged 4 commits into from
Apr 16, 2026
Merged

Enable writing PKs per extraction batch #2329

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64e326e
enable pk ordering
JFriel Apr 14, 2026
a770fe8
add to mssql
JFriel Apr 14, 2026
d1d8430
add space
JFriel Apr 15, 2026
490aa41
fix test
JFriel Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [9.2.2] - Unreleased
- Add ability to force primary keys at each batch of data when extracting to a database

## [9.2.1] - 2026-04-07
- Update Sql Merge component to force destination column types
- Update Sql merge to correctly match updates
Expand Down
4 changes: 4 additions & 0 deletions ...e/DataExport/DataExtraction/Pipeline/Destinations/ExecuteFullExtractionToDatabaseMSSql.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ You must have either $a or $d
[DemandsInitialization(DataTableUploadDestination.AlterTimeout_Description, DefaultValue = 300)]
public int AlterTimeout { get; set; }

[DemandsInitialization("By applying the primary keys after writing the data, it ensures all data is extracted. Disabling this configuration may improve performance but will quickly raise issues with poorly keyed data.",DefaultValue =true)]
public bool WriteDataBeforeApplyingPrimaryKeys { get; set; }

[DemandsInitialization(
"True to copy the column collations from the source database when creating the destination database. Only works if both the source and destination have the same DatabaseType. Excludes columns which feature a transform as part of extraction.",
DefaultValue = false)]
Expand Down Expand Up @@ -281,6 +284,7 @@ private DataTableUploadDestination PrepareDestination(IDataLoadEventListener lis

_destination.AllowResizingColumnsAtUploadTime = true;
_destination.AlterTimeout = AlterTimeout;
_destination.WriteDataBeforeApplyingPrimaryKeys = WriteDataBeforeApplyingPrimaryKeys;
_destination.AppendDataIfTableExists = AppendDataIfTableExists;
_destination.IncludeTimeStamp = IncludeTimeStamp;
_destination.UseTrigger = AppendDataIfTableExists;
Expand Down
8 changes: 7 additions & 1 deletion Rdmp.Core/DataLoad/Engine/Pipeline/Destinations/DataTableUploadDestination.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,9 @@ public CultureInfo Culture
set => _culture = value;
}

[DemandsInitialization("By applying the primary keys after writing the data, it ensures all data is extracted. Disabling this configuration may improve performance but will quickly raise issues with poorly keyed data.")]
public bool WriteDataBeforeApplyingPrimaryKeys { get; set; }

public string TargetTableName { get; private set; }

/// <summary>
Expand Down Expand Up @@ -181,7 +184,10 @@ public DataTable ProcessPipelineData(DataTable toProcess, IDataLoadEventListener
}
}

ClearPrimaryKeyFromDataTableAndExplicitWriteTypes(toProcess);
if (WriteDataBeforeApplyingPrimaryKeys)
{
ClearPrimaryKeyFromDataTableAndExplicitWriteTypes(toProcess);
}

StartAuditIfExists(TargetTableName);

Expand Down
8 changes: 4 additions & 4 deletions SharedAssemblyInfo.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

[assembly: AssemblyCompany("Health Informatics Centre, University of Dundee")]
[assembly: AssemblyProduct("Research Data Management Platform (RDMP)")]
[assembly: AssemblyCopyright("Copyright (c) 2018 - 2025")]
[assembly: AssemblyCopyright("Copyright (c) 2018 - 2026")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]

[assembly: AssemblyVersion("9.2.1")]
[assembly: AssemblyFileVersion("9.2.1")]
[assembly: AssemblyInformationalVersion("9.2.1")]
[assembly: AssemblyVersion("9.2.2")]
[assembly: AssemblyFileVersion("9.2.2")]
[assembly: AssemblyInformationalVersion("9.2.2")]
4 changes: 2 additions & 2 deletions rdmp-client.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<item>
<version>9.2.0.0</version>
<url>https://github.com/HicServices/RDMP/releases/download/v9.2.1/rdmp-9.2.1-client.zip</url>
<version>9.2.2.0</version>
<url>https://github.com/HicServices/RDMP/releases/download/v9.2.2/rdmp-9.2.2-client.zip</url>
<changelog>https://github.com/HicServices/RDMP/blob/main/CHANGELOG.md#7</changelog>
<mandatory>true</mandatory>
</item>
Loading