Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions src/ray/common/scheduling/label_selector.h
Original file line number Diff line number Diff line change
Expand Up @@ -126,18 +126,18 @@ inline bool operator==(const LabelSelector &lhs, const LabelSelector &rhs) {
}

template <typename H>
H AbslHashValue(H h, const LabelSelector &label_selector) {
h = H::combine(std::move(h), label_selector.GetConstraints().size());
for (const auto &constraint : label_selector.GetConstraints()) {
h = H::combine(std::move(h),
constraint.GetLabelKey(),
static_cast<int>(constraint.GetOperator()));
H AbslHashValue(H h, const LabelConstraint &constraint) {
// Hash the values set as a set, the way operator== above compares it. absl hashes its
// unordered containers order-independently and mixes in the size.
return H::combine(std::move(h),
constraint.GetLabelKey(),
static_cast<int>(constraint.GetOperator()),
constraint.GetLabelValues());
}

for (const auto &value : constraint.GetLabelValues()) {
h = H::combine(std::move(h), value);
}
}
return h;
template <typename H>
H AbslHashValue(H h, const LabelSelector &label_selector) {
return H::combine(std::move(h), label_selector.GetConstraints());
}

inline std::optional<absl::flat_hash_set<std::string>> GetHardNodeAffinityValues(
Expand Down
78 changes: 78 additions & 0 deletions src/ray/common/scheduling/tests/label_selector_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,15 @@
#include "ray/common/scheduling/label_selector.h"

#include <algorithm>
#include <cstddef>
#include <map>
#include <random>
#include <string>
#include <utility>
#include <vector>

#include "absl/container/flat_hash_set.h"
#include "absl/hash/hash.h"
#include "gmock/gmock.h"
#include "gtest/gtest.h"

Expand Down Expand Up @@ -208,4 +212,78 @@ TEST(LabelSelectorTest, Deduplication) {
ASSERT_EQ(selector.GetConstraints().size(), 4);
}

namespace {

LabelSelector OneConstraint(const std::string &key,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could use pass by value here and move to avoid deep copies every iteration in the test:

LabelSelector OneConstraint(std::string key,
                            LabelSelectorOperator op,
                            absl::flat_hash_set<std::string> values) {
  LabelSelector selector;
  selector.AddConstraint(LabelConstraint(std::move(key), op, std::move(values)));
  return selector;
}

LabelSelector RegionSelector(absl::flat_hash_set<std::string> values) {
  return OneConstraint("region", LabelSelectorOperator::LABEL_IN, std::move(values));
}

we'd also change the selector creation to this pattern:

const LabelSelector selector = RegionSelector(std::move(set));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 85f926b, and it turned out to be more than a copy saved. LabelConstraint already takes the set by value and moves it, so the old const & helper meant a copy landed on that parameter, and an absl set copy reinserts into a freshly sized table. The layouts the order sweep was visiting were therefore rehashed ones rather than the ones it built from each shuffle. Moving all the way through keeps the table the test made, which is what the sweep is about.

Re-verified after the change since it touches that mechanism: 10 tests green on 10 consecutive runs, and both failing directions still fail by exit code, with an order-dependent hash failing the sweep on 10 out of 10.

Left the four calls in SelectorsDifferingOnlyInKeyOrOperatorHashDifferently as they are. The copy count there is the same either way, since it used to happen on LabelConstraint's parameter and now happens at the call, and removing it would mean four separate sets or moving only on the last call.

LabelSelectorOperator op,
const absl::flat_hash_set<std::string> &values) {
LabelSelector selector;
selector.AddConstraint(LabelConstraint(key, op, values));
return selector;
}

LabelSelector RegionSelector(const absl::flat_hash_set<std::string> &values) {
return OneConstraint("region", LabelSelectorOperator::LABEL_IN, values);
}

} // namespace

// A constraint holds its values in a flat_hash_set and operator== compares them as a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could shorten some of these comments, I think they err on the verbose side currently

// LabelConstraint values form an unordered set, so their hash must be order-independent
// to match operator==. We shuffle inputs to force diverse internal memory layouts 
// (caused by insertion order and internal salting) and verify hash stability.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took your wording in 85f926b.

One measurement worth passing on, since your text credits salting: I instrumented the 16 rounds and the per-table salt proxy took only 2 distinct values across them, because the allocator kept handing back the same address. So the layout diversity the sweep gets is almost entirely insertion order, with salting contributing close to nothing here. Your parenthetical is still true of absl in general, which is why I left it as you wrote it, but say the word and I will narrow it to insertion order.

I also dropped the older, longer version's mention of the salt coming from the address of the control bytes. That is specific to this absl version, so it would go stale on an upgrade.

// set, so selectors built from the same labels have to hash alike however the set was
// filled. Which slot an element takes depends on the order elements were inserted, when
// two contend for one, and on a salt absl derives from the address of the table's control
// bytes, so sweep shuffled orders rather than trusting one layout.
TEST(LabelSelectorTest, EqualSelectorsHashEquallyWhateverTheValueOrder) {
std::vector<std::string> values;
values.reserve(64);
for (int i = 0; i < 64; i++) {
values.push_back("region-" + std::to_string(i));
}
const LabelSelector reference =
RegionSelector(absl::flat_hash_set<std::string>(values.begin(), values.end()));

absl::flat_hash_set<size_t> hashes;
absl::flat_hash_set<std::vector<std::string>> layouts;
std::mt19937 rng(20260826);
for (int round = 0; round < 16; round++) {
std::shuffle(values.begin(), values.end(), rng);
absl::flat_hash_set<std::string> set(values.begin(), values.end());
const LabelSelector selector = RegionSelector(set);
// operator== compares the constraint vectors, so it also passes when both sides are
// empty; the count needs its own assertion, which also guards the [0] below.
ASSERT_EQ(selector.GetConstraints().size(), 1u);
ASSERT_EQ(selector, reference);
const auto &stored = selector.GetConstraints()[0].GetLabelValues();
layouts.insert(std::vector<std::string>(stored.begin(), stored.end()));
hashes.insert(absl::HashOf(selector));
}

// Without more than one layout the hash assertion below holds for an order-dependent
// hash too, so the test would pass while guarding nothing.
ASSERT_GT(layouts.size(), 1u);
EXPECT_EQ(hashes.size(), 1u);
}

// The order sweep above would also pass if the values stopped reaching the hash at all,
// so pin that they contribute.
TEST(LabelSelectorTest, SelectorsWithDifferentValuesHashDifferently) {
EXPECT_NE(absl::HashOf(RegionSelector({"us-east", "us-west"})),
absl::HashOf(RegionSelector({"eu-central", "ap-south"})));
EXPECT_NE(absl::HashOf(RegionSelector({"us-east", "us-west"})),
absl::HashOf(RegionSelector({"us-east", "us-west", "eu-central"})));
EXPECT_NE(absl::HashOf(RegionSelector({})), absl::HashOf(RegionSelector({"us-east"})));
}

// The key and the operator need their own case: with only the cases above, dropping

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think we could remove this note about the key and operator needing their own test case, if a comment is needed it could just describe the test:

// Verify that both the key and the operator are mixed into the hash.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 85f926b, with your line. I applied the same treatment to the test above it, which argued for its own existence in the same way, so the file does not end up half in each style.

// either one from the hash leaves this file green.
TEST(LabelSelectorTest, SelectorsDifferingOnlyInKeyOrOperatorHashDifferently) {
const absl::flat_hash_set<std::string> values = {"us-east"};
EXPECT_NE(
absl::HashOf(OneConstraint("region", LabelSelectorOperator::LABEL_IN, values)),
absl::HashOf(OneConstraint("zone", LabelSelectorOperator::LABEL_IN, values)));
EXPECT_NE(
absl::HashOf(OneConstraint("region", LabelSelectorOperator::LABEL_IN, values)),
absl::HashOf(OneConstraint("region", LabelSelectorOperator::LABEL_NOT_IN, values)));
}

} // namespace ray
Loading