Skip to content

Commit

Permalink
Fix typos
Browse files Browse the repository at this point in the history
  • Loading branch information
nielstron committed Nov 18, 2024
1 parent defa1d6 commit 815924b
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 4 deletions.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ python3 report.py run_instance_swt_logs/swea__gpt-4-1106-preview/gpt4__SWE-bench
# | Unresolved Coverage | 19.14736127176707 |
```

In order to see a coverage delta reported, you need to have the gold evaluation included in the same evaluation path, i.e. download the golden results into `run_instance_swt_logs` from the downloads section below.

## ⬇️ Downloads

### Datasets
Expand Down
8 changes: 4 additions & 4 deletions report.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ def main(
name = instance_log_path.parent

fields = (
[r"{$\mathcal{W}$ \up{}}", r"{$\suc$ \up{}}", r"{\ftx \up{}}", r"{\ftp \up{}}", r"{\ptp}"]
[r"{$\mathcal{W}$}", r"{$\suc$}", r"{\ftx}", r"{\ftp}", r"{\ptp}"]
if format.startswith("latex") else
["Applicability (W)", "Success Rate (S)", "F->X", "F->P", "P->P", "P->F"]
["Applicability (W)", "Success Rate (S)", "F->X", "F->P", "P->P"]
)
rows = [["Method", name]]
run_id = instance_log_path.parent.name
Expand All @@ -49,9 +49,9 @@ def main(

if gold_reports:
fields = (
[r"{$\dc^{\text{all}}$ }", r"{$\dc^{\ftp}$}", r"{$\dc^{\neg(\ftp)}$}"]
[r"{$\dc^{\text{all}}$ }", r"{$\dc^{\suc}$}", r"{$\dc^{\neg\suc}$}"]
if format.startswith("latex") else
["Coverage", "Resolved Coverage", "Unresolved Coverage"]
["Coverage Delta", "Coverage Delta Resolved", "Coverage Delta Unresolved"]
)
total_coverage_possible = count_coverage_delta_gold(gold_reports)
resolved_reports, unresolved_reports = filtered_by_resolved(reports)
Expand Down

0 comments on commit 815924b

Please sign in to comment.