Description
I'd like to have more flexible ways to complete conduit::relay::mpi::Requests from non-blocking communications. Specifically,
- support the way applications store Requests and
- provide a choice to use MPI_Waitsome and MPI_Testsome to complete the MPI side of the requests.
Conduit "wait" functions requires a C array of Requests. Building such an array requires reallocating memory when we run out of space. However, reallocating memory would invalidate the pointers given to MPI. We use a list<Request> in Axom so we don't invalidate any pointers to add more Requests. If the Conduit "wait" functions allowed Request *requests[] instead of just Request requests[] we could use them. We're currently writing our own "wait" code, but since we're using conduit::relay::mpi::Request, it would make sense to let Conduit complete the request.
The second part is for conduit to support waitsome and testsome to check requests. The first always completes at least 1 outstanding request (if there are any) and the second is non-blocking. We currently do it this way:
void check_send_requests(std::list<conduit::relay::mpi::Request>& isendRequests,
bool atLeastOne) const
{
std::vector<MPI_Request> reqs; // Intermediate request container for MPI.
for(auto& isr : isendRequests) reqs.push_back(isr.m_request);
int inCount = static_cast<int>(reqs.size());
int outCount = 0;
std::vector<int> indices(reqs.size(), -1);
if(atLeastOne)
MPI_Waitsome(inCount,
reqs.data(),
&outCount,
indices.data(),
MPI_STATUSES_IGNORE);
else
MPI_Testsome(inCount,
reqs.data(),
&outCount,
indices.data(),
MPI_STATUSES_IGNORE);
indices.resize(outCount);
// Remove the completed requests. (Since they are sends, nothing else is requires.)
auto reqIter = isendRequests.begin();
int prevIdx = 0;
for(const int idx : indices)
{
for(; prevIdx < idx; ++prevIdx) ++reqIter;
reqIter = isendRequests.erase(reqIter);
++prevIdx;
}
}
Obviously, if conduit support lists of Requests, we wouldn't need to generate the intermediate container to pass to the wait functions. But there's probably no way to do this without templates.