[Particle] Improve bin weight estimate - #2145
Conversation
82c06ff to
6599910
Compare
| // build particle-to-particle neighbors only, to populate bin_interaction_costs_ for | ||
| // cost-aware rebalancing below; the global id map and wall neighbor relations are skipped | ||
| // here since both would be invalidated by the redistribution pass right after anyway | ||
| build_potential_neighbor_relation(/*include_wall_neighbors=*/false); |
There was a problem hiding this comment.
I wonder if the assumption of only the particle-particle interaction entering the cost function is fair? The evluation particle-wall is (what I would guess) way more expensive as the nonlinear contact point search needs to be performed (see Core::Geo::nearest_3d_object_on_element calls in interaction_sph_neighbor_pairs and interaction_dem_neighbor_pairs.
There was a problem hiding this comment.
That is a very good point! I thought of this. But this issue existed even before this modification. Strictly speaking, that's why we have customizable weights per particle type. Here the weights are averaged within each contact, so if a user provided custom weights, then this information is also accounted for. But it would be nice to have this data provided in more automatic manner rather then user provided weights, this will then further resolve the imbalance issue.
There was a problem hiding this comment.
The reason why I neglected particle-wall pairs respectively wall element counts for the bin distribution is, that in most cases the wall is surrounding the particle domain and coupling is performed only on the interface. So my aim was to optimize for ideal distribution of particles among procs rather than wall elements among procs. Note that the structural elements in the structural solver are distributed based on graph partitioning. That said, and also for the sake of ideally cubic processor domains, only the particles are considered for the bin distribution. I think, that considering particle-wall pairs in the bin distribution might lead to more lengthy shaped processor domains, such that each processor gets a share of wall elements and consequently this leads to more communication between the processors due to the worse ratio of processor domain boundary to volume.
There was a problem hiding this comment.
@slfuchs that makes sense. It would be actually interesting to measure the effect of particle-wall interactions on the load imbalance. The benchmark I have been using so far has no walls. I guess, the best way is to run the same geometry with the rigid boundary particles and the particle walls and measure the load imbalance. Anyways, this is one of those things that, in my opinion, needs profiling and measuring. And I would postpone it for a different PR.
|
|
||
| // accumulate interaction cost for load balancing weight estimation, weighted by the | ||
| // average of the dynamic load balance factors of the two interacting particle types | ||
| bin_interaction_costs_[gidofbin] += |
There was a problem hiding this comment.
I would prefer to move this to a separate function. I like to keep the concept, that one method is doing only one thing. So build_particle_to_particle_neighbors() should only build pontential particle neighbors.
There was a problem hiding this comment.
The potential particle neighbors is exactly the data this criterium relies on. So in that sense bin_interaction_costs_ is not something that is absolutely independent, on the contrary, now the coupling in the code is quite tight, which was the exact intent. However, I don't have a strong opinion on this.
What bothers me more - is actually the overhead when moving the population of bin_interaction_costs_ into a different function. I could have just iterated over potentialparticleneighbors_ as it contains the neighbors types (still, there is overhead here but not significant) but there is no gidofbin info available. Unless there is a quick way to get it from a potentialparticleneighbors_ entry. Is something like this available?
There was a problem hiding this comment.
you are right, since the global id of the bin is needed it probably makes sense to leave it as it is.
|
Two questions here
|
4 participants
Description and Context
This feature evaluates weights for particle bins based on the number of interaction pairs a bin contains. This information is already available, since we construct neighbors anyways. As the outcome, it improves the particle distribution and optimizes the load balance, especially when the work load per rank is rather large, e.g. >300,000 particles/rank. For smaller number of particles per rank, e.g. ~50,000, the effect is less significant but still present.
Related Issues and Pull Requests
#2117, #2074