Add some changes to make ds-inference work with MoE model#1

Draft

RezaYazdaniAminabadi wants to merge 5 commits intomasterfrom ds-inference-compatible

+119-76

Commits on Jan 17, 2024

Add some changes to make ds-inference work with MoE model
RezaYazdaniAminabadi
committed
remove safe_serialization checking to be able to save different model architectures
RezaYazdaniAminabadi
committed

Commits on Jan 20, 2024

make the transformers cache compatible with ds-inference
RezaYazdaniAminabadi
committed

Commits on Jan 25, 2024

add tp-support for mixtral
RezaYazdaniAminabadi
committed

Commits on Feb 6, 2024

fix generation with tp>1
RezaYazdaniAminabadi
committed