ra_log_meta hardcoded 30s timeout causes gen_batch_server:call failures on slow/loaded hosts (not configurable)

[tankerktv]

Summary

ra_log_meta's internal store/delete calls use a hardcoded 30s timeout with no way to override it, so on a sufficiently slow or momentarily-overloaded host, a node can fail to start entirely (rather than just start slowly) — even brand new, single-node, no prior data.

Where

https://github.com/rabbitmq/ra/blob/v3.1.9/src/ra_log_meta.erl#L31-L34

-define(TIMEOUT, 30000).
...
store_sync(Name, UId, Key, Value) ->
    gen_batch_server:call(Name, {store, UId, Key, Value}, ?TIMEOUT).

delete_sync(Name, UId) ->
    gen_batch_server:call(Name, {delete, UId}, ?TIMEOUT).

Confirmed present, unchanged, in ra 3.1.9, which is what RabbitMQ 4.3.4 currently pins (rabbitmq-components.mk: dep_ra = hex 3.1.9) — so this isn't specific to an old release.

What we hit

Via RabbitMQ (Khepri-enabled node, rabbitmq:3.13-management-alpine, but the same ra_log_meta code path), on a host under temporary memory/IO pressure (spinning-HDD-backed volume, load average ~36 on 4 cores):

{application_start_failure,rabbit,
 {{{timeout,{gen_batch_server,call,
    [ra_coordination_log_meta,
     {store,<<"RABBITR6...">>,voted_for,{rabbitmq_metadata,rabbit@...}},
     30000]}},
   {gen_statem,call,
    [{rabbitmq_metadata,rabbit@...},
     {leader_call,{command,normal,
       {'$usr',{register_projection,[rabbit_db_exchange,exchanges,...],
                {khepri_projection,rabbit_db_exchange,copy,[{keypos,2},named_table]}},
                {await_consensus,#{reply_from => local}}}}},
     30000]}},
   {rabbit,start,[normal,[]]}}}}

Once host load came back down, the exact same node started fine on the exact same (still slow) disk — the only thing that changed was whether the call landed inside or outside the fixed 30s window.

Ask

Make the ra_log_meta store/delete timeout configurable (e.g. via ra_system config or an application env var), rather than a compile-time ?TIMEOUT constant. Related discussion, for a different (quorum-queue-recovery) scenario hitting the same constant: rabbitmq/rabbitmq-server#3745.

[michaelklishin]

30s sounds like a reasonable timeout for nearly every environment. The exceptionally low frequency of such reports and conditions like 'an HDD with an OS load average of 36" further reinforce this exact line of thinking.