Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix calculation error when ref is empty #92

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,30 @@ glove2word2vec.py

.idea/
venv/

# General
.DS_Store
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove changes to your gitignore? They seem rather specific to your setup.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I can remove it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@juharris Thank you for your advice. It is what I need.

.AppleDouble
.LSOverride

# Icon must end with two \r
Icon

# Thumbnails
._*

# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent

# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
38 changes: 38 additions & 0 deletions nlgeval/tests/test_nlgeval.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,36 @@ def test_compute_metrics_omit(self):
self.assertAlmostEqual(0.960771, scores['GreedyMatchingScore'], places=5)
self.assertEqual(7, len(scores))

def test_compute_metricsa(self):
n = NLGEval()

# # Individual Metrics
scores = n.compute_individual_metrics(ref=["Chocolate , a New Way to Get Cancer "],
hyp="Chocolate , a Healthy Food")
print("Chocolate , a New Way to Get Cancer ", scores)

scores = n.compute_individual_metrics(ref=["Chocolate and an Interview "],
hyp="Chocolate , a Healthy Food")
print(scores)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you assert something here, so that tests can detect regressions?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that this case is not necessary. I will remove it

#
# # Individual Metrics
# scores = n.compute_individual_metrics(ref=["old people are helping in fact "],
# hyp="old people are easy to get along with ")
# print("old people are helping in fact ",scores)
#
# scores = n.compute_individual_metrics(ref=["old people are always happy "],
# hyp="old people are easy to get along with ")
# print("old people are always happy ",scores)

# Individual Metrics
# scores = n.compute_individual_metrics(ref=["help us to become a shy person"],
# hyp="block our chances for a successful life")
# print("help us to become a shy person", scores)
#
# scores = n.compute_individual_metrics(ref=["bring us great changes"],
# hyp="block our chances for a successful life")
# print(scores)

def test_compute_metrics_empty(self):
n = NLGEval()

Expand Down Expand Up @@ -160,3 +190,11 @@ def test_compute_metrics(self):
self.assertAlmostEqual(0.568696, scores['VectorExtremaCosineSimilarity'], places=5)
self.assertAlmostEqual(0.784205, scores['GreedyMatchingScore'], places=5)
self.assertEqual(12, len(scores))

def test_compute_metricss(self):
# The example from the README.
root_dir = os.path.join(os.path.dirname(__file__), '..', '..')
hypothesis = os.path.join(root_dir, 'examples/6.csv')
references = os.path.join(root_dir, 'examples/7.pt_beam_1.csv')
scores = nlgeval.compute_metrics(hypothesis, [references])
print(scores)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, better to assert the result you expect.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that this case is not necessary. I will remove it