Unnecessary warning in Huggingface LLM when tokenizer is passed as a function argument.

[CuriousMonkey7]

This warning gets logged even if you pass the correct tokenizer for the model.

Should I create an issue for this?

I think the following change would work:

if not tokenizer and tokenizer_name != model_name:
    logger.warning(
        f"The model `{model_name}` and tokenizer `{tokenizer_name}` "
        f"are different, please ensure that they are compatible."
    )