Move Translation model from transformers library for inference

[dasgoutam]

Currently the translation model is loaded into memory using the transformers library with the pipeline function.

Would it be possible to use vllm to serve the model?

[svenseeberg]

I had a look already but did not find a way to do this. AFAICT the APIs are different. This would probably be even more relevant with #174 as suggested in #170 (comment)

[svenseeberg]

vllm-project/vllm#1565

[dasgoutam]

vllm-project/vllm#1565

Okay. So even though the project has started support for encoder-decoder models (which NLLB is a type of), there seems to be no implementation yet for the architecture that NLLB-200 possesses i.e. M2M100ForConditionalGeneration

I had a look already but did not find a way to do this. AFAICT the APIs are different. This would probably be even more relevant with #174 as suggested in #170 (comment)

After a big of research, the only working way for inference and/or serving of NLLB models I could find was using Ctranslate2 (https://opennmt.net/CTranslate2/guides/transformers.html#nllb) or the triton inference server (https://github.com/triton-inference-server/server).

There is an (now old) article about it here - https://blog.speechmatics.com/huggingface-translation-triton , specifically addressing serving of the nllb model

[svenseeberg]

We also need a GPU server powerful enough to load the model. For the full 53B version, we would need roughly 100 GB of GPU memory. Alternatively, we could investigate if there is a quantized version. With Int 4 quantization, the model would shrink to a quarter of its size. Maybe this would work in CPU with sufficiently large RAM (32GB) and file caching. We do have servers with powerful enough CPUs available (Dual Socket AMD EPYC 9684X 96-Core Processor).

[freinold]

Alternatively, we could investigate if there is a quantized version. With Int 4 quantization, the model would shrink to a quarter of its size.

https://huggingface.co/KnutJaegersberg/nllb-moe-54b-4bit

[freinold]

I also found an alternative framework called LibreTranslate:
#50 (comment)