Code

moe#162

Code

Merged

dan-garvey merged 26 commits intomainnod-ai/shark-ai:mainfrom moe-wipnod-ai/shark-ai:moe-wipCopy head branch name to clipboardSep 5, 2024

+1,412-228

Commits

Commits on Aug 28, 2024

• Add Mixtral LLM

  

  archana-ramalingam

  authored and

  IanNod

  committed
  e036804

  Copy full SHA for e036804

• Refactoring attention, moe and ffn blocks

  

  archana-ramalingam

  authored and

  IanNod

  committed
  3110119

  Copy full SHA for 3110119

• Allow _optional_int_prop to handle missing hyperparameters

  

  archana-ramalingam

  authored and

  IanNod

  committed
  d1691c3

  Copy full SHA for d1691c3

• Fixing circular dep and imports

  

  archana-ramalingam

  authored and

  IanNod

  committed
  a865ac3

  Copy full SHA for a865ac3

• Fix multiple expert layer weight handling + other issues

  

  archana-ramalingam

  authored and

  IanNod

  committed
  3496258

  Copy full SHA for 3496258

• Add ffn_moe layers and other fixes

  

  archana-ramalingam

  authored and

  IanNod

  committed
  2b32fba

  Copy full SHA for 2b32fba

• Edit theta slicing

  

  archana-ramalingam

  authored and

  IanNod

  committed
  15f2a22

  Copy full SHA for 15f2a22

• Fix ffn_moe theta parsing & wraping

  

  archana-ramalingam

  authored and

  IanNod

  committed
  0f155c5

  Copy full SHA for 0f155c5

• Extract tensor unmerging into a function

  

  archana-ramalingam

  authored and

  IanNod

  committed
  4a8bb97

  Copy full SHA for 4a8bb97

• Cleaning up debug statements

  

  archana-ramalingam

  authored and

  IanNod

  committed
  36eb868

  Copy full SHA for 36eb868

• Fix test failure

  

  archana-ramalingam

  authored and

  IanNod

  committed
  58890f9

  Copy full SHA for 58890f9

• Add rope_freq_base to llama

  

  archana-ramalingam

  authored and

  IanNod

  committed
  99186fd

  Copy full SHA for 99186fd

• Rebase and fixes

  IanNod

  committed
  c66cbe5

  Copy full SHA for c66cbe5

Commits on Aug 29, 2024

• Add missing grok layers

  archana-ramalingam

  committed
  0bc76f6

  Copy full SHA for 0bc76f6

Commits on Sep 3, 2024

• adds a test for exporting moe block

  Show description for 96de75d
  dan-garvey

  committed
  96de75d

  Copy full SHA for 96de75d

• actually add the test

  dan-garvey

  committed
  f323792

  Copy full SHA for f323792

Commits on Sep 5, 2024

• some fixes

  dan-garvey

  committed
  2cd365b

  Copy full SHA for 2cd365b

• moe moe moe

  dan-garvey

  committed
  67f112f

  Copy full SHA for 67f112f

• refactor paged llama

  dan-garvey

  committed
  12b2a7a

  Copy full SHA for 12b2a7a

• fix format

  dan-garvey

  committed
  77163aa

  Copy full SHA for 77163aa

• rope_freq

  dan-garvey

  committed
  5fba3de

  Copy full SHA for 5fba3de

• saver

  dan-garvey

  committed
  b315fa3

  Copy full SHA for b315fa3

• address rope freq

  dan-garvey

  committed
  a2df6a4

  Copy full SHA for a2df6a4

• fix llama attn

  dan-garvey

  committed
  47b14b6

  Copy full SHA for 47b14b6

• Merge branch 'main' into moe-wip

  dan-garvey

  authored
  911b3a3

  Copy full SHA for 911b3a3

• add tensor name

  dan-garvey

  committed
  6a28481

  Copy full SHA for 6a28481