Parallelize periodic RVE boundary conditions - #2102
Conversation
| if (is_parallel && | ||
| rve_ref_type_ == Constraints::MultiPoint::RveReferenceDeformationDefinition::manual) | ||
| { | ||
| FOUR_C_THROW("Manual RVE reference points are not implemented in parallel."); |
There was a problem hiding this comment.
May be use FOUR_C_ASSERT_ALWAYS here. Same below.
| //! Return the Penalty-Parameter | ||
| double& get_penalty_parameter_ptr() { return penalty_parameter_; } | ||
|
|
||
| //! Set the constraint rows owned by this rank |
There was a problem hiding this comment.
Are these actually row IDs in a matrix or the global IDs of DOFs, that are subject to a constraint?
| //! Suspend floating point exception trapping while in scope. ArborX may raise benign fp | ||
| //! exceptions on ranks whose local search input is empty. |
There was a problem hiding this comment.
| //! Suspend floating point exception trapping while in scope. ArborX may raise benign fp | |
| //! exceptions on ranks whose local search input is empty. | |
| //! Suspend floating point exception trapping while in scope. ArborX may raise benign floating point | |
| //! exceptions on ranks whose local search input is empty. |
| discret_ptr_ = disc_ptr; | ||
| writable_discret_ = std::move(disc_ptr); |
There was a problem hiding this comment.
- Is
discret_ptr_still a valid point with actual data at this point? - Is it still used anywhere?
| // ghosting must not change the dof row map | ||
| const Core::LinAlg::Map& dof_row_map_after = *writable_discret_->dof_row_map(); | ||
| const std::vector<int> dof_gids_after(dof_row_map_after.my_global_elements(), | ||
| dof_row_map_after.my_global_elements() + dof_row_map_after.num_my_elements()); | ||
| if (dof_gids_before != dof_gids_after) | ||
| FOUR_C_THROW("Ghosting the periodic partner nodes changed the dof row map."); |
There was a problem hiding this comment.
That looks like code, that should be run in DEBUG mode only.
bennoschoenstein
left a comment
bennoschoenstein
left a comment
There was a problem hiding this comment.
Thanks a lot! Exactly what I need 🗡️
| std::map<int, std::vector<int>> positive_partners; | ||
| for (const auto& match : matches) | ||
| positive_partners[match.gid_predicate].push_back(match.gid_primitive); | ||
|
|
There was a problem hiding this comment.
What about if nodes don't find any partner?
The old code checked nHits == 0 for every '-' node and threw.
Now positive_partners only contains nodes that actually found a partner. A '-' node without a partner (mesh not perfectly periodic, or POINT_TOLERANCE too small) is now skipped - the simulation continues with a missing constraint and gives wrong results without any warning.
FIX:
if (positive_partners.size() != shifted_negative_nodes.size())
{
std::string unmatched_gids;
for (const auto& [minus_gid, bounding_volume] : shifted_negative_nodes)
if (!positive_partners.contains(minus_gid))
unmatched_gids += std::to_string(minus_gid) + " ";
FOUR_C_THROW(
"Periodic search on the '{}-' boundary found no partner for node(s): {}."
" Check mesh periodicity or increase POINT_TOLERANCE.",
axis, unmatched_gids);
}Maybe also worth a small test with an intentionally non-periodic mesh?
| } | ||
| // assemble the rows owned by this rank | ||
| for (const auto& [coefficient, row_id, dof_id] : equation_data_) | ||
| if (Q_Ld.row_map().my_gid(row_id)) Q_Ld.assemble(coefficient, row_id, dof_id); |
There was a problem hiding this comment.
This if drops rows that are not in the row map. For the periodic constraints this never happens. But for DESIGN POINT COUPLED DOF EQUATION CONDITIONS together with RVE_REFERENCE_POINTS: automatic it does happen.
FIX:
for (const auto& [coefficient, row_id, dof_id] : equation_data_)
{
FOUR_C_ASSERT_ALWAYS(Q_Ld.row_map().my_gid(row_id),
"Constraint equation row {} is not in the constraint row map and would "
"be dropped.",
row_id);
Q_Ld.assemble(coefficient, row_id, dof_id);
}| Core::IO::cout(Core::IO::verbose) | ||
| << "\nNumber of periodic constraint equations on this rank: " | ||
| << constraint_equations_.size() << Core::IO::endl; | ||
| return; |
There was a problem hiding this comment.
This early return skips the "Ensure that no constraint equation is used twice" block further down, which the old automatic path still reached.
Note: two duplicates can be created on different ranks. Maybe the simplest fix is to not create the duplicate in the first place, e.g. skip the known corner nodes in the '-' node loop for all axes but one
| for (const auto& [minus_gid, partners] : positive_partners) | ||
| { | ||
| if (partners.size() != 1) | ||
| FOUR_C_THROW( |
There was a problem hiding this comment.
If only one rank throws, the others can end up waiting instead of aborting
| Core::LinAlg::Map ghosted_node_map(-1, static_cast<int>(ghosted_node_gids.size()), | ||
| ghosted_node_gids.data(), 0, writable_discret_->get_comm()); | ||
| writable_discret_->export_column_nodes(ghosted_node_map); | ||
| writable_discret_->fill_complete(); |
There was a problem hiding this comment.
Note: this fill_complete() rebuilds the column maps and re-assigns the dofs. Anything that grabbed column-map data from this discretization before this constructor runs would silently be stale...
Description and Context
The periodic RVE boundary conditions in the constraint framework were
serial-only and aborted with a FOUR_C_THROW on more than one rank. This
enables them in parallel.
Changes:
(
global_collision_search) instead of a rank-local BVH query.four nodes of a constraint are locally available.
Q_dLand the residual from a distributed mat-vec, so no off-processor
assembly is needed.
Related Issues and Pull Requests