Skip to content

check_last_maintenance : filter out small tables #426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yhuelf wants to merge 5 commits into
base: master
Choose a base branch
from
Open

check_last_maintenance : filter out small tables #426

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d634622
check_last_maintenance : filter out small tables
yhuelf Jun 3, 2026
981678e
check_last_maintenance: fix tests broken by previous commit
yhuelf Jun 3, 2026
6bad3f3
check_last_maintenance: review of Benoit and Christophe
yhuelf Jun 8, 2026
0ad5044
last_maintenance: fix up last commit
yhuelf Jun 8, 2026
8d9141b
last_maintenance: gleu's review
yhuelf Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -772,6 +772,11 @@ COMPATIBILITY

You can use multiple "--exclude REGEX" parameters.

This service supports a "--analyze_table_min_size" parameter to
exclude small relations. The value must be a positive integer
representing the threshold in kilobytes below which the relation
is excluded. This defaults to 1024 (1MB).

@blogh blogh Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should specify that we are talking about the "main fork" and maybe reflect
this in the parameter name.

@blogh blogh Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect units to work here.

I don't know if that's necessary, we will likely filter small table so the value should be easy to
read whthout it. But it would be consistent with other sizes.

@Krysztophe Krysztophe Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main fork only? toast not included?

@yhuelf yhuelf Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main fork only? toast not included?

Yes because the toast table is processed separately by autovacuum. The main fork could be only 8kB, 100 tuples, never vacuumed, while the toast table can be 100MB, 50000 tuples, already vacuumed. If you use pg_table_size instead of pg_relation_size, you will get an alert for the main fork, and we don't want that.

@yhuelf yhuelf Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect units to work here.

fixed in 6bad3f3


Required privileges: unprivileged role able to log in all databases.

last_vacuum (8.2+)
Expand Down Expand Up @@ -810,6 +815,11 @@ COMPATIBILITY

You can use multiple "--exclude REGEX" parameters.

This service supports a "--vacuum_table_min_size" parameter to
exclude small relations. The value must be a positive integer
representing the threshold in kilobytes below which the relation
is excluded. This defaults to 10240 (10MB).

@blogh blogh Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as --analyze_table_min_size


Required privileges: unprivileged role able to log in all databases.

locks (all)
Expand Down
11 changes: 11 additions & 0 deletions check_pgactivity
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -503,6 +503,8 @@ my %args = (
'wal_buffers' => undef,
'checkpoint_segments' => undef,
'effective_cache_size' => undef,
'vacuum_table_min_size' => 10 * 1024,

@Krysztophe Krysztophe Jun 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10 MB is already too much I think.

Most small tables that gives false alarms are only a few blocks wide.

We can discuss at length the right limit, but since we had service that was much too sensitive, perhaps shall we raise the bar not so high.

@yhuelf yhuelf Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about 2 MB ?

@yhuelf yhuelf Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think 1000 tuples would be a better threshold (autovacuum_vacuum_insert_threshold default value).

A 2MB table should contains at least that number of tuples.

'analyze_table_min_size' => 1024,
'no_check_autovacuum' => 0,
'no_check_fsync' => 0,
'no_check_enable' => 0,
Expand Down Expand Up @@ -5270,6 +5272,7 @@ sub check_last_maintenance {
my @rs;
my $c_limit;
my $w_limit;
my $table_min_size;
my @perfdata;
my @msg_crit;
my @msg_warn;
Expand All @@ -5285,6 +5288,8 @@ sub check_last_maintenance {
my @dbexclude = @{ $args{'dbexclude'} };
my $me = 'POSTGRES_LAST_' . uc($type);

$table_min_size = $args{$type . "_table_min_size"} * 1024;

# warning and critical are mandatory.
pod2usage(
-message => "FATAL: you must specify critical and warning thresholds.",
Expand Down Expand Up @@ -5334,6 +5339,7 @@ sub check_last_maintenance {
||'$c_limit $w_limit'))
FROM pg_stat_all_tables a
WHERE schemaname NOT LIKE 'pg_temp_%'
AND pg_relation_size(relid) >= $table_min_size
AND schemaname NOT LIKE 'information_schema'
AND (('${type}' = 'analyze' AND schemaname NOT LIKE 'pg_toast') -- TOAST tables aren't ANALYZEd
OR ('${type}' = 'vacuum'))
Expand Down Expand Up @@ -5364,6 +5370,7 @@ sub check_last_maintenance {
FROM pg_stat_all_tables a
WHERE schemaname NOT LIKE 'pg_temp_%'
AND schemaname NOT LIKE 'pg_toast_temp_%'
AND pg_relation_size(relid) >= $table_min_size
AND schemaname NOT LIKE 'information_schema'
AND (('${type}' = 'analyze' AND schemaname NOT LIKE 'pg_toast') -- TOAST tables aren't ANALYZEd
OR ('${type}' = 'vacuum'))
Expand Down Expand Up @@ -5396,6 +5403,7 @@ sub check_last_maintenance {
FROM pg_stat_all_tables a
WHERE schemaname NOT LIKE 'pg_temp_%'
AND schemaname NOT LIKE 'pg_toast_temp_%'
AND pg_relation_size(relid) >= $table_min_size
AND schemaname NOT LIKE 'information_schema'
AND (('${type}' = 'analyze' AND schemaname NOT LIKE 'pg_toast') -- TOAST tables aren't ANALYZEd
OR ('${type}' = 'vacuum'))
Expand Down Expand Up @@ -5438,6 +5446,7 @@ sub check_last_maintenance {
JOIN pg_class b on a.relid = b.oid
WHERE schemaname NOT LIKE 'pg_temp_%'
AND schemaname NOT LIKE 'pg_toast_temp_%'
AND pg_relation_size(relid) >= $table_min_size
AND schemaname NOT LIKE 'information_schema'
AND (('${type}' = 'analyze' AND schemaname NOT LIKE 'pg_toast') -- TOAST tables aren't ANALYZEd
OR ('${type}' = 'vacuum'))
Expand Down Expand Up @@ -10035,6 +10044,8 @@ GetOptions(
'dump-status-file!',
'dump-bin-file:s',
'effective_cache_size=i',
'vacuum_table_min_size=i',
'analyze_table_min_size=i',
'exclude=s',
'format|F=s',
'global-pattern=s',
Expand Down
4 changes: 4 additions & 0 deletions t/01-last_analyze.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ sleep(1);

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_analyze',
'--analyze_table_min_size' => 0,

@gleu gleu Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should only use dash in a CLI option, so no underline (meaning --analyze-table-min-size).

'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand Down Expand Up @@ -106,6 +107,7 @@ SKIP: {

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_analyze',
'--analyze_table_min_size' => 0,

@gleu gleu Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand Down Expand Up @@ -145,6 +147,7 @@ push @stdout, (

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_analyze',
'--analyze_table_min_size' => 0,

@gleu gleu Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, same everywhere :)

'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand Down Expand Up @@ -182,6 +185,7 @@ push @stdout, (

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_analyze',
'--analyze_table_min_size' => 0,
'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand Down
50 changes: 49 additions & 1 deletion t/01-last_vacuum.t
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ use warnings;
use lib 't/lib';
use pgNode;
use TestLib ();
use Test::More tests => 33;
use Test::More tests => 39;

my $node = pgNode->get_new_node('prod');
my $pga_data = "$TestLib::tmp_check/pga.data";
Expand Down Expand Up @@ -50,6 +50,7 @@ sleep(1);

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_vacuum',
'--vacuum_table_min_size' => 0,
Comment thread
gleu marked this conversation as resolved.
Outdated
'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'template1',
Expand Down Expand Up @@ -107,6 +108,7 @@ SKIP: {

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_vacuum',
'--vacuum_table_min_size' => 0,
'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand Down Expand Up @@ -146,6 +148,7 @@ push @stdout, (

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_vacuum',
'--vacuum_table_min_size' => 0,
'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand Down Expand Up @@ -183,6 +186,7 @@ push @stdout, (

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_vacuum',
'--vacuum_table_min_size' => 0,
'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
Expand All @@ -196,6 +200,50 @@ $node->command_checks_all( [
'test database with two tables, both vacuumed'
);

# test database with three tables, only one never vacuumed, filtering out small tables

# we must track the stat activity on pg_class to make sure there was some stat
# activity to avoid the check_pga shortcut when no activity.
($stdout) = $node->psql('testdb', q{
SELECT n_tup_ins
FROM pg_stat_sys_tables
WHERE relname = 'pg_class'
});

$node->psql('testdb', 'CREATE TABLE boo (bar INT)');

$node->poll_query_until('testdb', qq{
SELECT n_tup_ins > $stdout
FROM pg_stat_sys_tables
WHERE relname = 'pg_class'
});

@stdout = (
qr/^Service *: POSTGRES_LAST_VACUUM$/m,
qr/^Returns *: 0 \(OK\)$/m,
qr/^Message *: 1 database\(s\) checked$/m,
qr/^Perfdata *: testdb=.*s warn=3600 crit=864000$/m
);

# we don't check the [auto][vacuum,analyze]_count here, because we are filtering out
# system tables which were accounted for in the previous test.

$node->command_checks_all( [
'./check_pgactivity', '--service' => 'last_vacuum',
'--vacuum_table_min_size' => 10,
'--username' => $ENV{'USER'} || 'postgres',
'--format' => 'human',
'--dbname' => 'testdb',
'--status-file' => $pga_data,
'--warning' => '1h',
'--critical' => '10d'
],
0,
\@stdout,
[ qr/^$/ ],
'database with two tables, one never vacuumed, filtering out small tables'
);

### End of tests ###

$node->stop( 'immediate' );