TL/MLX5: complete the mcast init

[MamziB]

adding the states regarding the team creation and mcast group join

[samnordmann]

Looks good! about the service bcast, I'm afraid it will use the full team from ctx and not the actual team. Did you try it with a subteam? Im afraid it could hang

[MamziB]

Thanks, @samnordmann for the comments. They are all resolved. @Sergei-Lebedev do you have any comments on this PR?

[MamziB]

@samnordmann @Sergei-Lebedev Thank you, guys, for the comments. I resolved all of them.

[samnordmann]

Looks very good, thanks!

[MamziB]

@Sergei-Lebedev @samnordmann Thanks guys for the commets. Can you please merge this PR? It has some important fixes that we need for the hpcx release. Thanks

[MamziB]

@Sergei-Lebedev @samnordmann I have addressed the rest of the comments. Please let me know if you have more comments.

[samnordmann]

Thanks for addressing the comments. Here are some new ones.

I am also trying to run this branch, using two ray nodes and the following command line:

mpirun -x UCC_TL_MLX5_MCAST_ENABLE=1 -x UCC_TL_MLX5_TUNE=inf --map-by ppr:2:node -np 4 test/mpi/ucc_test_mpi -c alltoall -O 1

The tests pass but I am getting the following errors:

tl_mlx5_mcast_coll.c:41   TL_MLX5 ERROR mcast_coll_do_bcast failed:-1
ucc_schedule.h:198  UCC  ERROR failure in task 0x22bea00, Operation is not supported

and

tl_mlx5_mcast_context.c:277  TL_MLX5 ERROR ibv_dealloc_pd failed errno 16

Do you have an idea of what it could be ? Could you look into it?

[samnordmann]

The recently merged PR #921 introduced a small bug here

ucc/src/components/tl/mlx5/tl_mlx5_context.c

Line 55 in c13d26c

ucc_rcache_destroy(self->rcache);

The line cleaning up rcache should be removed. Can you fix that please?

[MamziB]

@samnordmann @Sergei-Lebedev I did not see any Finalize issue after disabling the mcast flag. So, the PR is now ready. FYI, I tested it on HPCAC on multiple node with mcast flags ON/OFF and it was passed.

[MamziB]

@Sergei-Lebedev I resolved the merge conflict with master