On BDB,a subtree search failing to build its candidate list can return the SRCH as successull

[tbordaz]

Issue Description
On BDB, A reader (e.g. SRCH) and a writer (e.g. DEL) can be in db_deadlock ending with reader retry (see #7670 and #7723). dblayer detects this situation and if one of the filter component hit too much retrun then dblayer reports an empty candidate list and error=DBI_RC_RETRY. The problem is that in subtree search the error is ignored and is overwritten when computing the ancestorid.

When filter_candidate returns a DB_RETRY the server should propagate the error up to the operation so that the operation is 'err=1'

From a functional pov, the result of the operation should be valid (even with that bug) if the ancestorid returns the valid subtree and the filter is applied. But there are cases where filter is not tested when returning entries.

Package Version and Platform:

• since BDB range searches intermittently fail with err=1 under write load #7670 and Range search returns an empty result when its start key is removed #7723

Steps to Reproduce
Steps to reproduce the behavior:

1. Using BDB range searches intermittently fail with err=1 under write load #7670 test case

Expected results
If an DB_RETRY is reported on any component of the filter, then the SRCH should fail