Code

[WIP] RL multinode#152

Code

Open

rafapi wants to merge 23 commits intomainServiceNow/TapeAgents:mainfrom deepspeed_multinodeServiceNow/TapeAgents:deepspeed_multinodeCopy head branch name to clipboard

+763-131

Commits

Commits on Dec 18, 2024

• Multnode changes

  rafapi

  committed
  3738d09

• Merge fixed from deepspeed pr

  rafapi

  committed
  e3c30ec

• Add multinode env vars

  rafapi

  committed
  40c211e

• Move config

  rafapi

  committed
  8382fe1

• Add 70B config

  rafapi

  committed
  cc0a37f

• Remove dead code

  rafapi

  committed
  103ca66

Commits on Dec 20, 2024

• Only convert DS weights for inference

  rafapi

  committed
  7f1f713

• Reduce batch size

  rafapi

  committed
  751f1e3

• Run only on main process for now

  rafapi

  committed
  36c853b

• Add conversion flag

  rafapi

  committed
  0f62cfb

• Compose finetune command for multinode

  rafapi

  committed
  46d0c88

• Add default adam params

  rafapi

  committed
  99415df

• DS multinode config

  rafapi

  committed
  b786d6b

• Merge branch 'main' into deepspeed_multinode

  rafapi

  committed
  2a55ec4

• Fix node sync

  rafapi

  committed
  64306e1

Commits on Jan 8, 2025

• Add some distributed training settings

  rafapi

  committed
  81144a2

• Fix details

  rafapi

  committed
  4da0ea0

• Optimize and clanup checkpoints

  rafapi

  committed
  a81b18a

• Bigger BS

  rafapi

  committed
  41ad0f7

• Distributed training

  rafapi

  committed
  dc87445

• Use dist training object

  rafapi

  committed
  9432194

• Handle convert to hf on final checkpoint

  rafapi

  committed
  e5c8b20

• Distributed utils

  rafapi

  committed
  a0cf1dc