Skip to content

[Particle] Improve bin weight estimate #2145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vovannikov wants to merge 2 commits into
base: main
Choose a base branch
from
+45 −12
Open

[Particle] Improve bin weight estimate #2145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7d13f4a
Add interactions count to bin weights
vovannikov Jun 2, 2026
6599910
Optimize weights bootstraping stage
vovannikov Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions src/particle/src/algorithm/4C_particle_algorithm.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,16 +134,31 @@ void Particle::ParticleAlgorithm::setup()
// setup initial rigid bodies
if (particlerigidbody_) setup_initial_rigid_bodies();

// distribute load among processors
// distribute load among processors (first pass: by particle count)
distribute_load_among_procs();

// ghost particles on other processors
particleengine_->ghost_particles();

// build global id to local index map
if (particleinteraction_)
{
// build particle-to-particle neighbors only, to populate bin_interaction_costs_ for
// cost-aware rebalancing below; the global id map and wall neighbor relations are skipped
// here since both would be invalidated by the redistribution pass right after anyway
build_potential_neighbor_relation(/*include_wall_neighbors=*/false);

@georghammerl georghammerl Jul 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the assumption of only the particle-particle interaction entering the cost function is fair? The evluation particle-wall is (what I would guess) way more expensive as the nonlinear contact point search needs to be performed (see Core::Geo::nearest_3d_object_on_element calls in interaction_sph_neighbor_pairs and interaction_dem_neighbor_pairs.

@vovannikov vovannikov Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a very good point! I thought of this. But this issue existed even before this modification. Strictly speaking, that's why we have customizable weights per particle type. Here the weights are averaged within each contact, so if a user provided custom weights, then this information is also accounted for. But it would be nice to have this data provided in more automatic manner rather then user provided weights, this will then further resolve the imbalance issue.

@slfuchs slfuchs Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why I neglected particle-wall pairs respectively wall element counts for the bin distribution is, that in most cases the wall is surrounding the particle domain and coupling is performed only on the interface. So my aim was to optimize for ideal distribution of particles among procs rather than wall elements among procs. Note that the structural elements in the structural solver are distributed based on graph partitioning. That said, and also for the sake of ideally cubic processor domains, only the particles are considered for the bin distribution. I think, that considering particle-wall pairs in the bin distribution might lead to more lengthy shaped processor domains, such that each processor gets a share of wall elements and consequently this leads to more communication between the processors due to the worse ratio of processor domain boundary to volume.

@vovannikov vovannikov Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slfuchs that makes sense. It would be actually interesting to measure the effect of particle-wall interactions on the load imbalance. The benchmark I have been using so far has no walls. I guess, the best way is to run the same geometry with the rigid boundary particles and the particle walls and measure the load imbalance. Anyways, this is one of those things that, in my opinion, needs profiling and measuring. And I would postpone it for a different PR.


// second redistribution pass: redistribute load using interaction-pair counts as bin weights
// now that bin_interaction_costs_ is available from the neighbor build above
distribute_load_among_procs();

// re-ghost after second redistribution
particleengine_->ghost_particles();
}

// build global id to local index map (once, for the final distribution)
particleengine_->build_global_id_to_local_index_map();

// build potential neighbor relation
// build potential neighbor relation, including wall neighbors, for the final distribution
if (particleinteraction_) build_potential_neighbor_relation();

// setup initial states
Expand Down Expand Up @@ -970,14 +985,14 @@ void Particle::ParticleAlgorithm::distribute_load_among_procs()
Core::IO::cout(Core::IO::verbose) << "distribute load in step " << step() << Core::IO::endl;
}

void Particle::ParticleAlgorithm::build_potential_neighbor_relation()
void Particle::ParticleAlgorithm::build_potential_neighbor_relation(bool include_wall_neighbors)
{
TEUCHOS_FUNC_TIME_MONITOR("Particle::ParticleAlgorithm::build_potential_neighbor_relation");

// build particle to particle neighbors
particleengine_->build_particle_to_particle_neighbors();

if (particlewall_)
if (particlewall_ && include_wall_neighbors)
{
// relate bins to column wall elements
particlewall_->relate_bins_to_col_wall_eles();
Expand Down
4 changes: 3 additions & 1 deletion src/particle/src/algorithm/4C_particle_algorithm.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,8 +340,10 @@ namespace Particle
/*!
* \brief build potential neighbor relation
*
* \param[in] include_wall_neighbors also build particle to wall neighbors
*
*/
void build_potential_neighbor_relation();
void build_potential_neighbor_relation(bool include_wall_neighbors = true);

/*!
* \brief set initial conditions
Expand Down
25 changes: 19 additions & 6 deletions src/particle/src/engine/4C_particle_engine.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,8 +468,9 @@ void Particle::ParticleEngine::build_particle_to_particle_neighbors()
// relate half neighboring bins to owned bins
if (not binning_->validhalfneighboringbins_) relate_half_neighboring_bins_to_owned_bins();

// clear potential particle neighbors
// clear potential particle neighbors and accumulated interaction costs
potentialparticleneighbors_.clear();
bin_interaction_costs_.clear();

// invalidate flag denoting validity of particle neighbors map
validparticleneighbors_ = false;
Expand Down Expand Up @@ -554,6 +555,11 @@ void Particle::ParticleEngine::build_particle_to_particle_neighbors()
potentialparticleneighbors_.push_back(
std::make_pair(std::make_tuple(type, Owned, ownedindex),
std::make_tuple(neighbortype, neighborstatus, neighborindex)));

// accumulate interaction cost for load balancing weight estimation, weighted by the
// average of the dynamic load balance factors of the two interacting particle types
bin_interaction_costs_[gidofbin] +=

@slfuchs slfuchs Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to move this to a separate function. I like to keep the concept, that one method is doing only one thing. So build_particle_to_particle_neighbors() should only build pontential particle neighbors.

@vovannikov vovannikov Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The potential particle neighbors is exactly the data this criterium relies on. So in that sense bin_interaction_costs_ is not something that is absolutely independent, on the contrary, now the coupling in the code is quite tight, which was the exact intent. However, I don't have a strong opinion on this.

What bothers me more - is actually the overhead when moving the population of bin_interaction_costs_ into a different function. I could have just iterated over potentialparticleneighbors_ as it contains the neighbors types (still, there is overhead here but not significant) but there is no gidofbin info available. Unless there is a quick way to get it from a potentialparticleneighbors_ entry. Is something like this available?

@slfuchs slfuchs Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, since the global id of the bin is needed it probably makes sense to leave it as it is.

0.5 * (typeweights_[type] + typeweights_[neighbortype]);
}
}
}
Expand Down Expand Up @@ -2064,12 +2070,19 @@ void Particle::ParticleEngine::determine_bin_weights()
// get global id of bin
const int gidofbin = binning_->binrowmap_->gid(rowlidofbin);

// iterate over owned particles in current bin
for (const auto& particleIt : particlestobins_[binning_->bincolmap_->lid(gidofbin)])
// use neighbor pair count from the previous interaction evaluation as a proxy for the
// compute cost of this bin; fall back to particle count * type weight when no interaction
// data is available yet (e.g. on the very first load redistribution)
auto costIt = bin_interaction_costs_.find(gidofbin);
if (costIt != bin_interaction_costs_.end())
{
binning_->binweights_->get_vector(0).get_values()[rowlidofbin] += costIt->second;
}
else
{
// add weight of particle of specific type
binning_->binweights_->get_vector(0).get_values()[rowlidofbin] +=
typeweights_[particleIt.first];
for (const auto& particleIt : particlestobins_[binning_->bincolmap_->lid(gidofbin)])
binning_->binweights_->get_vector(0).get_values()[rowlidofbin] +=
typeweights_[particleIt.first];
}
}
}
Expand Down
3 changes: 3 additions & 0 deletions src/particle/src/engine/4C_particle_engine.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -738,6 +738,9 @@ namespace Particle
//! relate potential particle neighbors of all types and statuses
PotentialParticleNeighbors potentialparticleneighbors_;

//! accumulated interaction pair count per bin global id for load balancing weight estimation
std::unordered_map<int, double> bin_interaction_costs_;

//! owned particles being communicated (transfered/distributed) to target processors
std::vector<std::vector<int>> communicatedparticletargets_;

Expand Down
Loading