Skip to content

Commit fb2476d

Browse files
SteveL-MSFTGijsreynCopilotmichaeltlombardi
authored
3.3 rc.1 backport fixes (#1662)
* feat: implement export filter functionality for resource exports (#1621) * feat: implement export filter functionality for resource exports * Remove comment * feat: implement export filter functionality for resource exports * Remove comment * Fix Copilot remarks * Wrong commit * Remove the wildcard support for resources * Fix Copilot remarks * Attempt to increase code coverage and fix test * Add additional test for coverage * Add test and fix dism_dsc * fix: correct code coverage calculation for uninstrumented files Files without LCOV data (e.g., platform-specific code behind #[cfg(windows)] when only Linux coverage is collected) were incorrectly counting ALL added lines as uncovered, including non-executable lines like comments and blanks. This caused coverage to drop significantly when Windows-only files were modified in a PR. Fix: skip files without LCOV data since coverage cannot be determined for uninstrumented code. Also handle the '\ No newline at end of file' diff marker which could cause off-by-one line number errors after deleted lines. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refactor work on services * Remove unused key * Revert change * Restore native resource filtering and add engine filtering fallback * Update resource definitions * Fix Copilot remark * remove directive * Change comment wording and update tests * Add crate * Revert change and fix test * Update resource manifest --------- Co-authored-by: Steve Lee <slee@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * (GH-1563) Enhance the PSScript resources (#1657) * (GH-1563) Canonicalize `_inDesiredState` for PSScript schema Prior to this change, the PSScript resource manifests embedded a schema that minimally defined the `_inDesiredState` canonical property instead of referencing the canonical schema for that property. This change updates the embedded schemas to reference the canonical schema and adds the embedded schema by ID to the `$defs` keyword. * (GH-1563) Canonicalize script properties for PSScript schemas Prior to this change, the embedded schemas in the manifests for the transitional PowerShell script resources defined the script properties with a minimal schema allowing the value to be any string or null. They didn't include any other keywords. This change: 1. Defines a shared subschema for the script properties, which they now reference instead of redefining the constraints for each property. 1. Removed `null` from the allowed types, since defining a script property as `null` isn't valid but _not_ specifying the property at all _is_ valid. 1. Added the `writeOnly` keyword to the script properties, since they're only used for input and not output. 1. Added the `contentMediaType` keyword to the script properties to clearly indicate---but not validate---that the value is expected to be a PowerShell script. 1. Added the `minLength` keyword to the script properties to ensure that an empty string isn't used as a script value. 1. Adds the `title` and `description` keywords to the script properties to provide more context for each property in the schema. * (GH-1563) Canonicalize `input` for PSScript resource schemas Prior to this change, the `input` property for the PSScript resource schemas defined the valid types with two problems: 1. It allowed `null` as a valid type, which caused failures when the resource invoked the scripts. 1. It didn't allow non-integer numbers, erroneously preventing users from defining input values like `3.14`, even though `[3.14]` was accepted. This change: 1. Removes `null` from the valid types for `input`. 1. Adds `number` to the valid types for `input`, allowing non-integer numbers to be used as input values. 1. Adds the `title` and `description` keywords to the `input` property, providing better documentation for users. * (GH-1563) Add constraints to PSScript resource schema Prior to this change, the embedded schemas for the PSScript resources didn't require any properties to be defined. For input, a user should always define at least one script property. For output, the resource should always return one of three value shapes: 1. For `test` operations, the resource should always define the `inDesiredState` property. 1. For implemented `get` and `set` operations, the resource should define the `output` property when the script emits any output to the success stream. 1. When `get` and `set` operations aren't implemented or don't emit any output, the resource should return an empty object. This change adds constraints to the embedded schemas with the `oneOf` keyword and nested `oneOf`/`anyOf` keywords to enforce these rules. * (GH-1563) Add docs keywords for PSScript resources Prior to this change, the embedded schema for the PSScript resources didn't define the `title` or `description` keywords. This change: 1. Adds `title` and `description` keywords to the embedded schema for the PSScript resources to provide some documentation for them. 1. Defines the `$schema` keyword explicitly. * (GH-1563) Canonicalize output for PSScript resources Prior to this change, the PSScript resources: 1. Always returned an array of output objects, even if there was only one object. 1. Didn't ensure that enums in script output were serialized as strings, causing enums to emit as integers, which is likely not what the user intended and makes review more difficult. This change: 1. Ensures that if a script returns a single object, it's serialized as that object instead of nested in an array. 1. Adds the `-EnumAsString` parameter to the `ConvertTo-Json` call for emitting script output when invoked through PowerShell to ensure that enums are serialized as strings. This enhancement doesn't affect Windows PowerShell, which doesn't support the `-EnumAsString` parameter. 1. Updates the tests to reflect the new behavior of returning a single object instead of an array. 1. Fixes the tests to use correct casing for the script properties, now that the schema validation checks for those properties. * (GH-1563) Fix tests using invalid casing for PSScript resource Prior to this change, some tests were using invalid casing for the PSScript resource type, which didn't previously cause errors during the JSON Schema validation because the schema didn't forbid additional properties. The tests passed because PowerShell isn't case-sensitive, so was able to retrieve the properties even with incorrect casing. With the changes to the JSON schema _requiring_ one or more script properties, these tests began to fail because the invalidly-cased property names didn't satisfy the JSON Schema. This change corrects the casing in the test definitions to ensure both that the tests are correct and pass the schema validation. * Apply suggestions from review Co-authored-by: Steve Lee <slee@microsoft.com> --------- Co-authored-by: Steve Lee <slee@microsoft.com> * Add JSON schema caching (#1637) * Add JSON schema caching * address copilot feedback * fix copilot feedback to not deserializing the json * rename function to be singular * make helpers scoped to crate * make cache crate only * fix: add support for preserving quoted group names in sshd_config (#1639) * fix: add support for preserving quoted group names in sshd_config * Update with remarks Tess --------- Co-authored-by: Gijs Reijn <26114636+Gijsreyn@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Mikey Lombardi (He/Him) <michael.t.lombardi@gmail.com>
1 parent def8803 commit fb2476d

32 files changed

Lines changed: 1286 additions & 398 deletions

‎dsc/tests/dsc_config_get.tests.ps1‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,4 +182,54 @@ Describe 'dsc config get tests' {
182182
$result.results[0].result.actualState.family | Should -BeIn @('Windows', 'Linux', 'macOS')
183183
$LASTEXITCODE | Should -Be 0
184184
}
185+
186+
It 'embedded schema should only be retrieved once for multiple instances of the same resource' {
187+
$config_yaml = @"
188+
`$schema: https://aka.ms/dsc/schemas/v3/bundled/config/document.json
189+
resources:
190+
- name: Echo 1
191+
type: Microsoft.DSC.Debug/Echo
192+
properties:
193+
output: hello
194+
- name: Echo 2
195+
type: Microsoft.DSC.Debug/Echo
196+
properties:
197+
output: world
198+
"@
199+
$result = dsc -l debug config get -i $config_yaml 2>$TestDrive/error.log | ConvertFrom-Json
200+
$errorLog = Get-Content $TestDrive/error.log -Raw
201+
$LASTEXITCODE | Should -Be 0 -Because $errorLog
202+
$result.hadErrors | Should -BeFalse
203+
$result.results.Count | Should -Be 2
204+
$errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"
205+
}
206+
207+
It 'retrieves command resource schema from cache for multiple instances of the same resource' {
208+
$config_yaml = @"
209+
`$schema: https://aka.ms/dsc/schemas/v3/bundled/config/document.json
210+
resources:
211+
- name: one
212+
type: Test/Version
213+
requireVersion: =1.1.0
214+
properties:
215+
version: 1.1.0
216+
- name: two
217+
type: Test/Version
218+
requireVersion: =1.1.0
219+
properties:
220+
version: 1.1.0
221+
- name: three
222+
type: Test/Version
223+
requireVersion: =2.0.0
224+
properties:
225+
version: 2.0.0
226+
"@
227+
$result = dsc -l debug config get -i $config_yaml 2>$TestDrive/error.log | ConvertFrom-Json
228+
$errorLog = Get-Content $TestDrive/error.log -Raw
229+
$LASTEXITCODE | Should -Be 0 -Because $errorLog
230+
$result.hadErrors | Should -BeFalse
231+
$result.results.Count | Should -Be 3
232+
$errorLog | Should -BeLike "*Retrieved schema for resource 'Test/Version' with version '1.1.0' from cache*"
233+
$errorLog | Should -BeLike "*Retrieved schema for resource 'Test/Version' with version '2.0.0' from cache*"
234+
}
185235
}

‎dsc/tests/dsc_export.tests.ps1‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,3 +213,53 @@ resources:
213213
$out.resources[0].properties.id | Should -Be 1
214214
}
215215
}
216+
217+
Describe 'engine export filtering tests' {
218+
It 'engine filters properties for a resource that does not support filtering natively' {
219+
$yaml = @'
220+
$schema: https://aka.ms/dsc/schemas/v3/bundled/config/document.json
221+
resources:
222+
- name: No Native Filtering
223+
type: Test/ExportSchemaNoFiltering
224+
properties:
225+
name: '*e*'
226+
'@
227+
$out = dsc --trace-level info config export -i $yaml 2>$TESTDRIVE/error.log | ConvertFrom-Json
228+
$errorlog = Get-Content "$TESTDRIVE/error.log" -Raw
229+
$LASTEXITCODE | Should -Be 0 -Because $errorlog
230+
@($out.resources).Count | Should -Be 2
231+
$out.resources.properties.name | Should -Be @('Steve', 'Tess')
232+
$errorlog | Should -Match 'does not support export filtering, the engine will filter the exported instances'
233+
}
234+
235+
It 'INFO message indicates engine export filtering is experimental' {
236+
$yaml = @'
237+
$schema: https://aka.ms/dsc/schemas/v3/bundled/config/document.json
238+
resources:
239+
- name: No Native Filtering
240+
type: Test/ExportSchemaNoFiltering
241+
properties:
242+
name: 'Gijs'
243+
'@
244+
$out = dsc --trace-level info config export -i $yaml 2>$TESTDRIVE/error.log | ConvertFrom-Json
245+
$errorlog = Get-Content "$TESTDRIVE/error.log" -Raw
246+
$LASTEXITCODE | Should -Be 0 -Because $errorlog
247+
@($out.resources).Count | Should -Be 1
248+
$out.resources[0].properties.name | Should -BeExactly 'Gijs'
249+
$errorlog | Should -BeLike "*INFO*Resource 'Test/ExportSchemaNoFiltering' does not support export filtering, the engine will filter the exported instances (experimental feature)*"
250+
}
251+
252+
It 'no INFO message is emitted when no filter input is provided' {
253+
$yaml = @'
254+
$schema: https://aka.ms/dsc/schemas/v3/bundled/config/document.json
255+
resources:
256+
- name: No Native Filtering
257+
type: Test/ExportSchemaNoFiltering
258+
'@
259+
$out = dsc --trace-level info config export -i $yaml 2>$TESTDRIVE/error.log | ConvertFrom-Json
260+
$errorlog = Get-Content "$TESTDRIVE/error.log" -Raw
261+
$LASTEXITCODE | Should -Be 0 -Because $errorlog
262+
@($out.resources).Count | Should -Be 3
263+
$errorlog | Should -Not -Match 'the engine will filter the exported instances'
264+
}
265+
}

‎dsc/tests/dsc_metadata.tests.ps1‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ Describe 'metadata tests' {
155155
- name: return variable
156156
type: Microsoft.DSC.Transitional/PowerShellScript
157157
properties:
158-
SetScript: |
158+
setScript: |
159159
if ($IsWindows) {
160160
$env:myTestVariable
161161
}
@@ -189,7 +189,7 @@ Describe 'metadata tests' {
189189
- name: return path
190190
type: Microsoft.DSC.Transitional/PowerShellScript
191191
properties:
192-
SetScript: |
192+
setScript: |
193193
$env:PATH
194194
'@
195195
$oldUserPath = [System.Environment]::GetEnvironmentVariable('PATH', [System.EnvironmentVariableTarget]::User)
@@ -219,7 +219,7 @@ Describe 'metadata tests' {
219219
- name: return variable
220220
type: Microsoft.DSC.Transitional/PowerShellScript
221221
properties:
222-
SetScript: |
222+
setScript: |
223223
if ($IsWindows) {
224224
$env:myTestVariable
225225
}

‎dsc/tests/dsc_resource_export.tests.ps1‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,29 @@ Describe 'Resource export tests' {
1717
$output.resources.properties.name | Should -Be $expected -Because ($output | ConvertTo-Json -Depth 4)
1818
}
1919

20-
It 'Resource not support export filtering will return error' {
20+
It "Engine filters input '<json>' for a resource without native filtering support" -TestCases @(
21+
@{ json = '{ "name": "Gijs" }'; expected = @('Gijs') }
22+
@{ json = '{ "name": "*e*" }'; expected = @('Steve', 'Tess') }
23+
@{ json = '[{ "name": "Gijs" }, { "name": "Steve" }]'; expected = @('Steve', 'Gijs') }
24+
){
25+
param($json, $expected)
26+
27+
$resource = 'Test/ExportSchemaNoFiltering'
28+
$output = dsc --trace-level info resource export -r $resource -i $json 2>$TESTDRIVE/error.log | ConvertFrom-Json
29+
$errorlog = Get-Content "$TESTDRIVE/error.log" -Raw
30+
$LASTEXITCODE | Should -Be 0 -Because $errorlog
31+
$output.resources.count | Should -Be $expected.Count -Because ($output | ConvertTo-Json -Depth 4)
32+
$output.resources.properties.name | Should -Be $expected -Because ($output | ConvertTo-Json -Depth 4)
33+
$errorlog | Should -Match "Resource '$resource' does not support export filtering, the engine will filter the exported instances \(experimental feature\)"
34+
}
35+
36+
It 'Engine filtering rejects input that is not an object or array of objects' {
2137
$resource = 'Test/ExportSchemaNoFiltering'
22-
$json = '{ "name": "Gijs" }'
38+
$json = '5'
2339

24-
$output = dsc resource export -r $resource -i $json 2>$TESTDRIVE/error.log | ConvertFrom-Json
40+
dsc resource export -r $resource -i $json 2>$TESTDRIVE/error.log | Out-Null
2541
$errorlog = Get-Content "$TESTDRIVE/error.log" -Raw
2642
$LASTEXITCODE | Should -Be 2 -Because $errorlog
27-
$errorlog | Should -Match "Resource '$resource' does not support export filtering"
43+
$errorlog | Should -Match 'Export filter input must be a JSON object or an array of JSON objects'
2844
}
2945
}

‎helpers.build.psm1‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2448,6 +2448,8 @@ function Get-CodeCoverageReport {
24482448
$currentLineNum++
24492449
} elseif ($diffLine.StartsWith('-') -and -not $diffLine.StartsWith('---')) {
24502450
# Deleted lines don't advance the new file line counter
2451+
} elseif ($diffLine.StartsWith('\')) {
2452+
# "\ No newline at end of file" marker — not a real line
24512453
} else {
24522454
$currentLineNum++
24532455
}
@@ -2469,10 +2471,7 @@ function Get-CodeCoverageReport {
24692471
}
24702472

24712473
if (-not $fileCoverage) {
2472-
# File not in coverage report means LLVM found no instrumentable
2473-
# executable code in it (e.g., struct/enum definitions with derive
2474-
# macros). Skip it rather than penalizing as uncovered.
2475-
Write-Verbose -Verbose "No LCOV data for '$file' - file has no instrumentable code, skipping"
2474+
Write-Verbose -Verbose "Skipping '$file': not in LCOV data (possibly platform-specific or not instrumented)"
24762475
continue
24772476
}
24782477

‎lib/dsc-lib/locales/en-us.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@ dependencyNotInOrder = "Dependency not found in order"
3535
circularDependency = "Circular dependency detected for resource named '%{resource}'"
3636
invocationOrder = "Resource invocation order"
3737

38+
[configure.export_filter]
39+
filteredInstances = "Export filter reduced %{original} instances to %{retained}"
40+
3841
[configure.mod]
3942
nestedArraysNotSupported = "Nested arrays not supported"
4043
arrayElementCouldNotTransformAsString = "Array element could not be transformed as string"
@@ -48,6 +51,9 @@ groupNotSupportedForDelete = "Group resources not supported for delete"
4851
deleteNotSupported = "Resource '%{resource}' does not support `delete` and does not handle `_exist` as false"
4952
expectedState = "Expected state: %{state}"
5053
exportInput = "Export input: %{input}"
54+
engineExportFiltering = "Resource '%{resource}' does not support export filtering, the engine will filter the exported instances (experimental feature)"
55+
invalidExportFilterInput = "Invalid export filter input: %{error}"
56+
exportFilterNotObject = "Export filter input must be a JSON object or an array of JSON objects"
5157
noParameters = "No parameters defined in configuration and no parameters input"
5258
noParametersDefined = "No parameters defined in configuration"
5359
processingParameter = "Processing parameter '%{name}'"
@@ -202,6 +208,7 @@ securityContextRequired = "Operation '%{operation}' for resource '%{resource}' r
202208
noAdaptedContent = "No adapted content available for resource '%{resource}'"
203209
invalidAdaptedContent = "Invalid adapted content for resource '%{resource}': %{error}"
204210
exportFilteringNotSupported = "Resource '%{resource}' does not support export filtering"
211+
retrievedSchemaFromCache = "Retrieved schema for resource '%{resource}' with version '%{version}' from cache"
205212

206213
[dscresources.dscresource]
207214
invokeGet = "Invoking get for '%{resource}'"
@@ -241,6 +248,7 @@ adapterManifestNotFound = "Adapter manifest for '%{adapter}' not found"
241248
adapterDoesNotSupportDelete = "Adapter '%{adapter}' does not support delete operation"
242249
validatingAgainstSchema = "Validating against resource schema"
243250
deprecationMessage = "Resource '%{resource}' is deprecated: %{message}"
251+
retrievedSchemaFromCache = "Retrieved schema for resource '%{resource}' with version '%{version}' from cache"
244252

245253
[dscresources.resource_manifest]
246254
resourceManifestSchemaTitle = "Resource manifest schema URI"

0 commit comments

Comments
 (0)