TL/UCP: Add Sliding Window allreduce implementation

[nsarka]

This PR is the second in a series of two that will add Sliding Window allreduce to UCC. This one implements the function stubs left by the first PR (#902)

[swx-jenkins3]

Can one of the admins verify this patch?

[Sergei-Lebedev]

ok to test

[nsarka]

@Sergei-Lebedev It seems like the same ucc test is failing as the active_set. However, this one fails because of wrong cuda versions: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.1, please update your driver to a newer version, or use an earlier cuda container: unknown

[janjust]

@artemry-nv Seems like we have CI issues on this PR, unrelated to the PR

[artemry-nv]

@B-a-S please take a look

[B-a-S]

@B-a-S please take a look

I've rerun the job. Take a look http://hpc-master.lab.mtl.com:8080/job/ucc/3282/

[samnordmann]

The implementation looks nice and readable, thanks! I left some comments, most importantly about memory leaks

[nsarka]

The ucc gtest failed on

[ RUN      ] test_tl_mlx5_dm.MemcpyToDeviceMemory/4
[2024-05-16T18:13:52.173Z] [swx-clx01:196  :0:196] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))

[nsarka]

I updated the PR. The get/reduce/put phase and the barrier part of the algorithm are now run via schedule. I left the allgather phase the way it was inside of the get/reduce/put phase because once Ferrol's PR goes in, I will be removing the allgather and using that instead for key exchange.

Also, I moved some code to two new files, tl_ucp_dpu_offload.c and tl_ucp_dpu_offload.h. These have common code that will be used for the rest of the collectives we're planning on implementing.

[nsarka]

Please wait to review, there are some failures I should fix first.

[artemry-nv]

bot:retest

[nsarka]

@samnordmann The PR is ready for review, thank you. Please note that the allgather task is still part of the algorithm. Once Ferrol's PR goes in I will convert the code to use that for the import/allgather phase of the algorithm. I also left the reduction as part of the algorithm, since it would involve splitting the gets and puts into tasks of their own as well. There would be thousands of these tasks for large message sizes.

[samnordmann]

LGTM! Thanks for addressing the comments. I still have left a couple of new minor comments

[Sergei-Lebedev]

follow up changes:

1. Use single allocation instead of multiple mallocs
2. Get rid of explicit ucp request objects and use callbacks
3. Optimize ucp_worker calls

[janjust]

@Sergei-Lebedev When you say follow up changes - are you talking in a separate PR after this is merged? If so, I'll open up an issue and tag this so we don't forget.

[nsarka]

@Sergei-Lebedev @samnordmann The PR is ready to be merged