Add CI log-size regression checks with verbosity 4 and per-test/node thresholds

[Copilot]

This updates .ci network test scripts to actively catch logfile growth regressions. Tests now run at --verbosity 4 and enforce configurable logfile size caps per test and node type.

• Shared log validation (.ci/utils.sh)

  • Extended check_logs with optional size-limit parameters:
    • max_validator_log_size_bytes
    • max_client_log_size_bytes
  • Added hard-fail checks when a node logfile exceeds its configured bound.
  • Corrected the validator iteration condition in check_logs so validator logs are actually checked.

• Per-test size bounds + verbosity 4

  • Updated scripts to run nodes with verbosity 4 and pass explicit size bounds into check_logs:
    • .ci/test_devnet.sh (validator + client bounds)
    • .ci/test_full_upgrade.sh (validator + client bounds)
    • .ci/test_partial_upgrade.sh (validator bound)
    • .ci/test_db_backup.sh (validator bound)
    • .ci/test_reset_minority.sh (validator bound)
    • .ci/test_restart_majority.sh (validator bound)

• Usage pattern

  check_logs "$log_dir" "$total_validators" "$total_clients" "$max_warnings" \
    "$MAX_VALIDATOR_LOG_SIZE_BYTES" "$MAX_CLIENT_LOG_SIZE_BYTES"

[vicsn]

@ljedrz can you run locally and confirm if the suggested filesize bounds are tight enough? Can you also pick a testfile where we set verbosity to just 1 and check the log file size?

[ljedrz]

These are the log file sizes with these changes in place:

.ci/test_devnet.sh:

 21k ci-runner.log
169k client-0.log
170k client-1.log
169k client-2.log
168k client-3.log
1.1M validator-0.log
1.0M validator-1.log
1.0M validator-2.log
1.0M validator-3.log

.ci/test_full_upgrade.sh:

 12k ci-runner.log
609k client-0.log
584k client-1.log
235k probe-snarkos-validator-0.log
215k probe-snarkos-validator-1.log
215k probe-snarkos-validator-2.log
216k probe-snarkos-validator-3.log
3.9M validator-0.log
3.9M validator-1.log
4.0M validator-2.log
3.9M validator-3.log

.ci/test_partial_upgrade.sh:

6.8k ci-runner.log
234k probe-snarkos-validator-0.log
213k probe-snarkos-validator-1.log
213k probe-snarkos-validator-2.log
213k probe-snarkos-validator-3.log
1.5M validator-0.log
1.5M validator-1.log
1.6M validator-2.log
1.9M validator-3.log

.ci/test_db_backup.sh:

 10k ci-runner.log
672k validator-0.log
576k validator-1.log
567k validator-2.log
544k validator-3.log

.ci/test_reset_minority.sh:

 28k ci-runner.log
3.3M validator-0.log
3.2M validator-1.log
3.0M validator-2.log
2.9M validator-3.log
3.9M validator-4.log
3.3M validator-5.log
3.0M validator-6.log

.ci/test_restart_majority.sh:

 10k ci-runner.log
1.5M validator-0.log
1.3M validator-1.log
1.3M validator-2.log
1.3M validator-3.log
1.4M validator-4.log
1.3M validator-5.log
1.4M validator-6.log

Can you also pick a testfile where we set verbosity to just 1 and check the log file size?

@vicsn you mean an existing script with a verbosity of 1, or one of those listed in the PR with it changed to 1?

[vicsn]

@vicsn you mean an existing script with a verbosity of 1, or one of those listed in the PR with it changed to 1?

I'll vote for applying it to .ci/test_restart_majority.sh. Can you finish off this PR? Thx 🙏

[ljedrz]

.ci/test_restart_majority.sh (verbosity 1):

 10k ci-runner.log
320k validator-0.log
299k validator-1.log
308k validator-2.log
310k validator-3.log
306k validator-4.log
319k validator-5.log
315k validator-6.log

So about a 3-4x decrease in size.

[ljedrz]

The denvet-workflow job failed due to too many WARN logs (293, max permitted is 40); most (224) of these logs are Received signature for an older batch, how should we approach this?

[vicsn]

The denvet-workflow job failed due to too many WARN logs (293, max permitted is 40); most (224) of these logs are Received signature for an older batch, how should we approach this?

The warning looks useful to me, so shall we just increase the limit?

[ljedrz]

.ci/test_devnet.sh failed in a weird fashion - despite outputting 🎉 Test passed! All nodes reached minimum height.