Update Otter Assign to disallow metadata tests for colab notebooks

[jl-uscn]

Tried to create an assignment so students can use it on Colab. Added runs_on: colab in configuration. Used Otter assign to generate the student file. After Otter grader is installed, executed
import otter
grader = otter.Notebook()
and got this error:

/usr/local/lib/python3.10/dist-packages/otter/check/notebook.py in init(self, nb_path, tests_dir, tests_url_prefix)
105
106 if self.interpreter is IPythonInterpreter.COLAB and not os.path.isdir(tests_dir):
--> 107 raise ValueError(f"Tests directory {tests_dir} does not exist")
108
109 cls = type(self)

ValueError: Tests directory ./tests does not exist

I tried to add the file name as this: grader = otter.Notebook('Week_3_Coding_Exercise.ipynb') and got the same error.

It is easy to reproduce the issue. It throws the same error every time the code cell is executed

I expect this works without the tests directory since the test cases are in the meta data of the notebook.

Versions
Python: 3.10.12
otter: 6.0.4

Additional context
I read the posts about colab and still don't know how to solve the problem.

[chrispyles]

This is WAI. In order to read tests from the notebook metadata, Otter needs to read the notebook file; it's not possible (AFAIK) to access the notebook metadata otherwise. Colab does not store the notebook in the file system, so there's no way for Otter to read the notebook metadata.

It would be a good idea to update Otter Assign to enforce this by disallowing a notebook with runs_on: colab to use metadata tests. And the documentation needs to be updated since I thought this was documented but it's not.

[jl-uscn]

@chrispyles Thanks for the quick reply. Could you please clarify the following:

1. How to provent metadata tests be used for Google colab? Sounded like Otter Assign update is to be implemented
2. What should I do now to use Otter grader on Colab?

I could not find relevant information on the documentation. Thanks in advance!

[jl-uscn]

I found a assignment configuration setting:
tests:
files: true
This allows the tests directory to be created. After some exprimenting. I found that I need to upload this directory to my Google drive and mount the drive before the tests can be run. But when I run grader.export(), I got the error
RuntimeError: This method is not compatible with Google Colab

Is there a way to solve this problem. Or using Colab is too much additional work that I'd better stick with Jupyter notebook locally? Thanks!

[chrispyles]

How to provent metadata tests be used for Google colab? Sounded like Otter Assign update is to be implemented

Yes, the setting tests: files: true that you found is the correct one to use, and you do need to either mount a google drive folder with the tests or add a cell downloads them (e.g. by cloning a repo with git).

But when I run grader.export(), I got the error RuntimeError: This method is not compatible with Google Colab

grader.export() is another method that requires being able to read the notebook as a file, so it's not supported on Colab. I need to update the docs to list out which methods are incompatible but for now you can look at the Notebook class source code and any methods marked with @incompatible_with(IPythonInterpreter.COLAB) can't be used on Colab.

Or using Colab is too much additional work that I'd better stick with Jupyter notebook locally?

Because of the limitation of not being able to access the notebook file, there are several features that aren't available on Colab, including generating a submission zip with grader.export. Take a look at what methods aren't available from the link above and if there are ones that are a requirement for your use case, I'd recommend switching to something other than Colab.

[jl-uscn]

@chrispyles Thank you!