variables not in the scope of grader

[evaneill]

Describe the bug
The variables referenced in my tests don't seem to be available to the grader, so the tests (both ran during otter assign and from the student notebook that is generated) always fail. I get a NameError: name 'var' is not defined. Did you mean: 'vars'? for the reproduced example below from both tests.

To Reproduce

1. Create an example instructor notebook with a single question, as pictured in attachment below
2. From the folder where this example notebook lives, run otter assign example.ipynb dist
3. See that two tests fail: each indicating that the variable var doesn't exist.
4. Run jupyter notebook dist/student/example.ipynb to open the student version
5. Run the three cells and get the same test error when checking the answer in the student notebook

Expected behavior
The otter assign should result in all tests passing, and the resulting student notebook solution should be able to grade the single student question.

Versions
python: 3.12.5
otter: 6.0.4
dill: 0.3.9
fica: 0.4.1
IPython: 8.30.0
nbconvert: 7.16.4
nbformat: 5.10.4

[chrispyles]

@evaneill can you please run otter assign with the -vv and --debug flags and send the output?

[evaneill]

Sure, here goes:

otter assign example.ipynb dist -vv --debug

generates:

[DEBUG otter.assign.main] User-specified master path: example.ipynb
[DEBUG otter.assign.main] User-specified result path: dist
[DEBUG otter.assign.assignment.Assignment.__init__] Initializing with config: {}
[DEBUG otter.assign.main] Normalized master path: example.ipynb
[DEBUG otter.assign.main] Normalized result path: dist
[INFO otter.assign.main] Generating views
[DEBUG otter.assign.assignment.Assignment.update] Updating config: {'solutions_pdf': False, 'export_cell': False, 'generate': {'pdf': False, 'zips': False}}
[INFO otter.assign.output.write_output_directories] Assignment summary:
Total points: 1
Autograded:   1
Manual:       0

  name  points
0   q1       1
[INFO otter.assign.main] Generating autograder zipfile
[INFO otter.assign.main] Running tests against the solutions notebook
[DEBUG otter.run.run_autograder.main] Detected containerized grading (T/F): F
[DEBUG otter.run.run_autograder.main] Config file path was resolved as: /var/folders/6f/05_hkvzx1yg7sx9kd9xs88440000gn/T/tmpdjstbuab/autograder/source/otter_config.json
[DEBUG otter.run.run_autograder.main] Autograder config was created: AutograderConfig(score_threshold=None, points_possible=None, show_stdout=False, show_hidden=False, show_all_public=False, seed=None, seed_variable=None, grade_from_log=False, serialized_variables=None, pdf=False, token=None, course_id=None, assignment_id=None, filtering=False, pagebreaks=False, debug=True, autograder_dir=/var/folders/6f/05_hkvzx1yg7sx9kd9xs88440000gn/T/tmpdjstbuab/autograder, lang=python, miniconda_path=/root/miniforge3, plugins=[], logo=False, print_summary=True, print_score=True, zips=False, log_level=None, assignment_name=None, warn_missing_pdf=False, force_public_test_summary=True, submit_blank_pdf_on_export_failure=False, use_submission_pdf=False, pdf_via_html=False)
[DEBUG otter.execute.logging.start_server] Starting execution logging TCP server in background at 127.0.0.1:49831
[INFO otter.check.notebook.Notebook.check] Running check for question: q1
[DEBUG otter.check.notebook.Notebook.check] Resolved test path: ./tests/q1.py
[DEBUG otter.check.notebook.Notebook.check] Resolved test name: None
[DEBUG otter.check.notebook.Notebook.check] Collecting calling global environment
[DEBUG otter.check.notebook.Notebook.check] Calling checker
[DEBUG otter.execute.logging.stop_server] Stopping execution logging TCP server
[DEBUG otter.assign.utils.run_tests] Otter Run output:




------------------------------- GRADING SUMMARY --------------------------------

No log found with which to verify student scores.


Total Score: 0.000 / 1.000 (0.000%)

           name  score  max_score
0  Public Tests    NaN        NaN
1            q1    0.0        1.0

Traceback (most recent call last):
  File "/Users/eneill/envs/otter_ex/bin/otter", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/otter/cli.py", line 32, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/otter/cli.py", line 69, in assign_cli
    return assign(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/otter/assign/__init__.py", line 156, in main
    run_tests(assignment, debug=debug)
  File "/Users/eneill/envs/otter_ex/lib/python3.12/site-packages/otter/assign/utils.py", line 238, in run_tests
    raise RuntimeError(
RuntimeError: Some autograder tests failed in the autograder notebook:
    q1 results:
        q1 - 1 result:
            ❌ Test case failed
            Trying:
                type(var) == int
            Expecting nothing
            **********************************************************************
            Line 1, in q1 0
            Failed example:
                type(var) == int
            Exception raised:
                Traceback (most recent call last):
                  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/doctest.py", line 1368, in __run
                    exec(compile(example.source, filename, "single",
                  File "<doctest q1 0[0]>", line 1, in <module>
                    type(var) == int
                         ^^^
                NameError: name 'var' is not defined. Did you mean: 'vars'?

        q1 - 2 result:
            ❌ Test case failed
            Trying:
                var == 5
            Expecting nothing
            **********************************************************************
            Line 1, in q1 1
            Failed example:
                var == 5
            Exception raised:
                Traceback (most recent call last):
                  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/doctest.py", line 1368, in __run
                    exec(compile(example.source, filename, "single",
                  File "<doctest q1 1[0]>", line 1, in <module>
                    var == 5
                    ^^^
                NameError: name 'var' is not defined. Did you mean: 'vars'?

[chrispyles]

@evaneill I'm not able to reproduce this issue on my machine. Can you try to reproduce it in a fresh conda/virtual environment? If it persists, please send the notebook to reproduce.

[cindygr]

Hi all - I am running into the exact same problem (or rather, my students are). They are running in Python 3.13; earlier versions of python seem fine. It appears to entirely be a name scope problem (grader can't find any of the jn variables). I can send an example JN, but since the JN works just fine in anything but 3.13, and it works fine when submitted to Gradescope, I'm not sure that will help...

[timothylwarren]

I also seem to be dealing with similar issue with
Python 3.11.11, otter 6.0.4

[cindygr]

@timothylwarren That version works fine for us, but I do know there was also a change in Gradescope last spring (?) that wasn't missing variables but bad values - because the tests are now all run after the full notebook is executed, instead of in-line like they are when you run the JN yourself. So re-used variables names are a problem (the original value gets overwritten).

[chrispyles]

@cindygr @timothylwarren can you share some notebooks that reproduce this error? Feel free to PM them to me on Slack (link in README).

[cindygr]

I’ve attached the JN before running otter assign, after, and the zip file that gets created by otter-assign. I was using Python version: 3.11.7 Otter-Grader version: 6.0.4 Additional package versions: dill: 0.3.7 fica: 0.4.1 IPython: 8.20.0 nbconvert: 7.10.0 nbformat: 5.9.2 to make the files. I tried very hard to install Python 3.13 on my mac book air so I could see if I could replicate the error(s) the students were having (variables not found in the tests) but ipykernel fails to install with that version of python, so I can’t run the JN (installing with Brew, fwiw). Backing off to 3.12 seems to fix the problem (although a few students had problem with 3.12.7). It just feels like accessing the local variables from the JN via the structure created by otter.Notebook has changed. Sorry, not much help ☹ From: Chris Pyles ***@***.***> Date: Saturday, January 11, 2025 at 7:32 AM To: ucbds-infra/otter-grader ***@***.***> Cc: Grimm, Cindy ***@***.***>, Mention ***@***.***> Subject: Re: [ucbds-infra/otter-grader] variables not in the scope of grader (Issue #890) [This email originated from outside of OSU. Use caution with links and attachments.] @cindygr<https://github.com/cindygr> @timothylwarren<https://github.com/timothylwarren> can you share some notebooks that reproduce this error? Feel free to PM them to me on Slack (link in README). — Reply to this email directly, view it on GitHub<#890 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADDOR546JUS6B53CPQ2YWQL2KE2PXAVCNFSM6AAAAABTQWZEPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBVGMYDGOJUGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[evaneill]

Sorry for the delay. I have removed the old environment, and started fresh. The tests during otter assign still fail, and the student notebook still doesn't function, but inexplicably for a different reason (no longer a NameError).

I'm attaching:

• the command line history of creating the new environment and running the otter assign
• Image of failure in student notebook

I'm unable to attach the notebook here in the issue thread - what is the easiest way to send it?

Command Line History.txt

[timothylwarren]

On your example, my grader passes the grader.check line after question but not at end of notebook.