Outbound GSSAPI bind holds the global krb5_lock with no timeout: one hung supplier stalls replication on all of its peers #7879

Issue Description

When one supplier's ns-slapd hangs, replication stops not only to and from that supplier but also between the remaining healthy suppliers. It stays stopped until the hung process is killed. In our case the hang was the retroCL deadlock in #6652: the kernel still accepts TCP connections, but no LDAP operation is ever answered.

On both healthy suppliers (3-supplier full mesh, FreeIPA, GSSAPI replication agreements) we found the same picture:

• One repl-prot thread, the agreement to the hung supplier, was blocked in poll(fds, 1, -1) (infinite timeout) on its socket to the hung server. On the hung server, that socket had 1839 unread bytes in its Recv-Q, which is the SASL/GSSAPI BindRequest.
• The other three repl-prot threads were all in futex(FUTEX_WAIT_PRIVATE, 2) on one and the same mutex. These include the agreements between the two healthy suppliers, for both dc= and o=ipaca.

/proc/<pid>/task/<tid>/syscall on supplier 1 (supplier 3 looked the same):

207588 repl-prot  7   0x7f90386ce804 0x1 0xffffffff ...   poll(fds, 1, -1); fds[0].fd = 23 = TCP socket to supplier2:389
207589 repl-prot  202 0x7f9038647180 0x80 0x2 ...         futex(FUTEX_WAIT_PRIVATE)
207590 repl-prot  202 0x7f9038647180 0x80 0x2 ...
207591 repl-prot  202 0x7f9038647180 0x80 0x2 ...

Result:

• The two healthy suppliers stopped exchanging changes for about 6 hours. Their last sessions were at 23:35 and 00:00, and the RUV max CSNs diverged by about 6 hours.
• This happened silently. The errors log had no entries, and the agreement status still said "Incremental update succeeded" with an old nsds5replicaLastUpdateEnd. ipa-healthcheck on the healthy suppliers also hung.
• As soon as the hung supplier was SIGKILLed (TCP reset), all agreements on both peers resumed within seconds and the RUVs converged.

Code (389-ds-base-3.2.0)

• For GSSAPI, slapi_ldap_bind() takes the process-wide krb5_lock (ldaputil.c:1171-1195). It holds the lock across slapd_ldap_sasl_interactive_bind() (:1199) → ldap_sasl_interactive_bind_s() (:1507), and only releases it at :1217.
• The only timeout set on the connection is LDAP_OPT_NETWORK_TIMEOUT. slapi_ldap_init_ext() sets it from nsslapd-outbound-ldap-io-timeout (:790, 300000 here), and the replication plugin then overrides it with the agreement's nsds5ReplicaTimeout (repl5_connection.c:1247, 120/300 s here). It covers connect() only. LDAP_OPT_TIMEOUT is never set, so ldap_sasl_interactive_bind_s() waits for every server response without a timeout.
• So a peer that accepts the TCP connection but never answers blocks the bind forever, with krb5_lock held. Every other agreement then blocks on krb5_lock at its next (re)bind, including the agreements to healthy peers.

We did not have gdb on the healthy suppliers at the time, so we inferred which lock the futex belongs to:

• All agreement threads wait on one heap-allocated lock.
• krb5_lock is created lazily with PR_NewLock() via PR_CallOnce(), by the first thread that does a GSSAPI bind.
• The lock and the blocked thread's poll array are in the same malloc arena.

We can capture a backtrace on a peer the next time this happens.

Expected results

An unresponsive peer should only affect the agreements to that peer. An outbound SASL bind should time out, using the agreement timeout or nsslapd-outbound-ldap-io-timeout, and a process-wide lock should not be held across network I/O.

Possible fixes

• Set LDAP_OPT_TIMEOUT for outbound SASL binds, or use the asynchronous SASL bind with a timeout.
• Do not hold krb5_lock across network I/O. There is no separate credential setup step to lock around any more (set_krb5_creds() was removed in Client keytab realm mismanagement #4537): the lock serializes the whole slapd_ldap_sasl_interactive_bind() call, so narrowing it means separating the Kerberos credential handling from the SASL exchange. Alternatively, use per-connection or per-thread credential caches (gss_krb5_ccache_name() is thread-specific), so the bind does not need the global lock at all.

Package Version and Platform

• Rocky Linux 10.2, kernel 6.12.0-211.el10_2
• 389-ds-base-3.2.0-10.el10_2, openldap-2.6.10-1.el10, cyrus-sasl-gssapi-2.1.28-30.el10_2, krb5-libs-1.21.3-10.el10_2, nspr-4.39.0-5.el10_2
• FreeIPA 4.13.1: 3 suppliers, full mesh, GSSAPI replication agreements for dc= and o=ipaca

Steps to Reproduce (suggested; we have not run this synthetically yet)

1. Set up 3 suppliers with GSSAPI replication agreements (e.g. FreeIPA).
2. Make supplier 2 unresponsive without closing its sockets: kill -STOP $(pidof ns-slapd) on supplier 2.
3. Make changes on suppliers 1 and 3, and wait until their agreements to supplier 2 time out and reconnect (new GSSAPI bind).
4. Changes stop flowing between suppliers 1 and 3. /proc/<pid>/task/*/syscall shows the poll(-1) + futex pattern above. kill -CONT on supplier 2 releases it.

Additional context

The trigger in our case was the deadlock in #6652. That comment has the backtraces and the timeline.