Skip to content

Commit e722f6b

Browse files
nohwndCopilot
nohwnd
and
Copilot
committed
Add settings migrator analysis report with telemetry info
Reports which .testsettings properties require legacy mode (most of them) and documents existing telemetry metrics for measuring feature usage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent f5b2ce5 commit e722f6b

1 file changed

Lines changed: 156 additions & 0 deletions

File tree

‎docs/settings-migrator-report.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# SettingsMigrator Report: What It Actually Does
2+
3+
## Summary
4+
5+
The SettingsMigrator reads 8 categories of settings from `.testsettings` files.
6+
**6 out of 8 go into `<LegacySettings>` and force `<MSTest><ForcedLegacyMode>true</ForcedLegacyMode>`.**
7+
Only 2 settings get truly native runsettings placement (+ 1 that goes top-level for web tests).
8+
9+
**Verdict: If your .testsettings uses Deployment, Scripts, Hosts, UnitTestRunConfig,
10+
parallelTestCount, hostProcessPlatform, or testTimeout — the tool forces legacy mode,
11+
making it essentially useless as a "migration" tool. It just wraps the old settings
12+
in a `<LegacySettings>` envelope.**
13+
14+
---
15+
16+
## Property Migration Map
17+
18+
### ❌ Properties that REQUIRE Legacy Mode
19+
20+
These go into `<LegacySettings>` and set `<ForcedLegacyMode>true</ForcedLegacyMode>`:
21+
22+
| .testsettings XPath | Destination in .runsettings | Notes |
23+
|---|---|---|
24+
| `/TestSettings/Deployment` | `<LegacySettings><Deployment .../>` | Copied as-is |
25+
| `/TestSettings/Scripts` | `<LegacySettings><Scripts .../>` | Copied with attributes (setupScript, etc.) |
26+
| `/TestSettings/Execution` attr `parallelTestCount` | `<LegacySettings><Execution parallelTestCount="...">` | Attribute on Execution element |
27+
| `/TestSettings/Execution` attr `hostProcessPlatform` | `<LegacySettings><Execution hostProcessPlatform="...">` | Attribute on Execution element |
28+
| `/TestSettings/Execution/Timeouts` attr `testTimeout` | `<LegacySettings><Execution><Timeouts testTimeout="..."/>` | Child of Execution |
29+
| `/TestSettings/Execution/Hosts` | `<LegacySettings><Execution><Hosts>...</Hosts>` | Copied as-is |
30+
| `/TestSettings/Execution/TestTypeSpecific/UnitTestRunConfig` | `<LegacySettings><Execution><TestTypeSpecific><UnitTestRunConfig>...` | Wrapped in TestTypeSpecific |
31+
32+
If **any** of the above are present → the tool creates `<LegacySettings>` node and forces legacy mode.
33+
34+
### ✅ Properties that get NATIVE runsettings placement
35+
36+
| .testsettings XPath | Destination in .runsettings | Notes |
37+
|---|---|---|
38+
| `/TestSettings/Execution/Timeouts` attr `runTimeout` | `<RunConfiguration><TestSessionTimeout>value</TestSessionTimeout>` | Truly native, no legacy needed |
39+
| `/TestSettings/Execution/AgentRule/DataCollectors/DataCollector` | `<DataCollectionRunSettings><DataCollectors><DataCollector .../>` | Each DataCollector copied as-is |
40+
41+
### ⚠️ Top-level (not legacy, but niche)
42+
43+
| .testsettings XPath | Destination in .runsettings | Notes |
44+
|---|---|---|
45+
| `/TestSettings/Execution/TestTypeSpecific/WebTestRunConfiguration` | Top-level `<WebTestRunConfiguration>` under `<RunSettings>` | Only relevant for web/load tests |
46+
47+
### 🚫 Unsupported (dropped with a warning)
48+
49+
| .testsettings XPath | Result |
50+
|---|---|
51+
| `Timeouts/@agentNotRespondingTimeout` | Dropped, warning printed |
52+
| `Timeouts/@deploymentTimeout` | Dropped, warning printed |
53+
| `Timeouts/@scriptTimeout` | Dropped, warning printed |
54+
55+
---
56+
57+
## Example Output Structure
58+
59+
For a typical .testsettings file that uses Deployment, Scripts, Timeouts, Hosts,
60+
UnitTestRunConfig, and DataCollectors, the migrator produces:
61+
62+
```xml
63+
<?xml version="1.0" encoding="utf-8"?>
64+
<RunSettings>
65+
<!-- ❌ LEGACY: forces old MSTest v1 runner -->
66+
<MSTest>
67+
<ForcedLegacyMode>true</ForcedLegacyMode>
68+
</MSTest>
69+
70+
<!-- ❌ LEGACY: just a wrapper around the old settings -->
71+
<LegacySettings>
72+
<Deployment>
73+
<DeploymentItem filename="..." />
74+
</Deployment>
75+
<Scripts setupScript=".\setup.bat" cleanupScript=".\cleanup.bat" />
76+
<Execution parallelTestCount="2" hostProcessPlatform="MSIL">
77+
<Timeouts testTimeout="120000" />
78+
<Hosts>...</Hosts>
79+
<TestTypeSpecific>
80+
<UnitTestRunConfig>
81+
<AssemblyResolution>
82+
<TestDirectory ... />
83+
</AssemblyResolution>
84+
</UnitTestRunConfig>
85+
</TestTypeSpecific>
86+
</Execution>
87+
</LegacySettings>
88+
89+
<!-- ✅ NATIVE: actually migrated to runsettings -->
90+
<RunConfiguration>
91+
<TestSessionTimeout>60000</TestSessionTimeout>
92+
</RunConfiguration>
93+
94+
<!-- ✅ NATIVE: actually migrated to runsettings -->
95+
<DataCollectionRunSettings>
96+
<DataCollectors>
97+
<DataCollector friendlyName="Event Log" ... />
98+
</DataCollectors>
99+
</DataCollectionRunSettings>
100+
101+
<!-- ⚠️ TOP-LEVEL: web test specific -->
102+
<WebTestRunConfiguration>
103+
<Browser>
104+
<Headers>
105+
<Header ... />
106+
</Headers>
107+
</Browser>
108+
</WebTestRunConfiguration>
109+
</RunSettings>
110+
```
111+
112+
---
113+
114+
## Bottom Line
115+
116+
| Question | Answer |
117+
|---|---|
118+
| Does the tool truly migrate settings to native runsettings? | **Mostly no.** Only `runTimeout` → `TestSessionTimeout` and DataCollectors get native placement. |
119+
| Does it require legacy mode? | **Yes**, for 7 out of 10 property categories. Any .testsettings using Deployment, Scripts, Hosts, UnitTestRunConfig, parallelTestCount, hostProcessPlatform, or testTimeout will force `<ForcedLegacyMode>true</ForcedLegacyMode>`. |
120+
| Is the tool useful if legacy mode makes it useless? | **No.** For any real-world .testsettings file (which almost always has at least Deployment or testTimeout), this tool just wraps old settings in `<LegacySettings>` XML — it doesn't actually migrate them to modern equivalents. |
121+
122+
---
123+
124+
## Existing Telemetry
125+
126+
There is telemetry already in place to measure usage of these features. Query your
127+
telemetry backend for these metrics to decide if removal is safe.
128+
129+
### Telemetry Constants (`TelemetryDataConstants.cs`)
130+
131+
| Metric Name | What It Tracks |
132+
|---|---|
133+
| `VS.TestRun.IsTestSettingsUsed` | Whether a `.testsettings` file is active in the run |
134+
| `VS.TestRun.LegacySettings.Elements` | Which LegacySettings child elements are present (Deployment, Scripts, etc.) |
135+
| `VS.TestRun.LegacySettings.DeploymentAttributes` | Attributes on the `<Deployment>` node |
136+
| `VS.TestRun.LegacySettings.ExecutionAttributes` | Attributes on the `<Execution>` node (parallelTestCount, hostProcessPlatform) |
137+
138+
### Collection Flow
139+
140+
1. **`TestRequestManager.LogTelemetryForLegacySettings()`** — called during `RunTests()` when telemetry is opted in
141+
2. Calls **`InferRunSettingsHelper.TryGetLegacySettingElements()`** which inspects the runsettings XML for:
142+
- `/RunSettings/LegacySettings` child elements
143+
- `/RunSettings/LegacySettings/Execution/TestTypeSpecific/UnitTestRunConfig/AssemblyResolution`
144+
- `/RunSettings/LegacySettings/Execution/Timeouts`
145+
- `/RunSettings/LegacySettings/Execution/Hosts`
146+
- Deployment and Execution attributes
147+
3. All metrics are published as part of the **`vs/testplatform/testrunsession`** event
148+
149+
### What to Query
150+
151+
| Question | Query |
152+
|---|---|
153+
| How many users still use .testsettings? | `VS.TestRun.IsTestSettingsUsed = true` |
154+
| Which legacy features are in use? | `VS.TestRun.LegacySettings.Elements` values |
155+
| Is Deployment used? | `VS.TestRun.LegacySettings.DeploymentAttributes` is non-empty |
156+
| Is parallelTestCount/hostProcessPlatform used? | `VS.TestRun.LegacySettings.ExecutionAttributes` is non-empty |

0 commit comments

Comments
 (0)