When I use the ppl indicator, it shows nan

[MrZhang1996]

orgn-GSM8K-train Average_ppl_accuracy: nan
PPL of xxxx
GSM8K_rewritten-test-1: nan
GSM8K_rewritten-test-2: nan
GSM8K_rewritten-test-3: nan
GSM8K_rewritten-train-1: nan
GSM8K_rewritten-train-2: nan
GSM8K_rewritten-train-3: nan
orgn-GSM8K-test: nan
orgn-GSM8K-train: nan

why this happens?

[MrZhang1996]

orgn-GSM8K-train Average_ppl_accuracy: nan PPL of xxxx GSM8K_rewritten-test-1: nan GSM8K_rewritten-test-2: nan GSM8K_rewritten-test-3: nan GSM8K_rewritten-train-1: nan GSM8K_rewritten-train-2: nan GSM8K_rewritten-train-3: nan orgn-GSM8K-test: nan orgn-GSM8K-train: nan

why this happens?

The find_subsequence() function returns -1

[VagueWine]

Hello! We've found that different tokenizers have different strategies. Please try to add your model name in the 'if' structure here. See if 'find_subsequence()' could return the proper index.

[MrZhang1996]

Hello! We've found that different tokenizers have different strategies. Please try to add your model name in the 'if' structure here. See if 'find_subsequence()' could return the proper index.

Thank you very much for your reply. I have noticed this after reading the code.