Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an end-to-end quality test #1004

Open
eu9ene opened this issue Jan 22, 2025 · 0 comments
Open

Add an end-to-end quality test #1004

eu9ene opened this issue Jan 22, 2025 · 0 comments

Comments

@eu9ene
Copy link
Collaborator

eu9ene commented Jan 22, 2025

Due to the recent bugs in inference, it would be nice to have a unit test that runs translation on a test set end-to-end using a browser and checks translation quality.

For example, we could,

  1. Take our regular flore-test test set for one or several language pairs
  2. Create a web page where each line in the dataset would be a separate paragraph
  3. Run translation in the browser
  4. Take the output page and split it back
  5. Calculate the COMET or BLEU score
  6. Compare the score to the evaluation results. It should be similar with a small error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant