Feature: Gurobi backend interface

[brynpickering]

Fixes #349
Fixes #597 (by pinning pyomo)

Summary of changes in this pull request:

• Duplicated pyomo backend interface as a pure gurobipy interface
• Added basic tests of the interface methods

How to use:

1. Install gurobipy into your calliope env: mamba install gurobi::gurobi
2. Build optimisation problem with gurobi backend:

import calliope

model = calliope.Model(...)
model.build(backend="gurobi")
model.solve()

NOTE: without a license, one can only work with very small models. Our example models are already too big (maybe if run over ~2 timesteps it would work?).

Reviewer checklist:

• Test(s) added to cover contribution
• Documentation updated
• Changelog updated
• Coverage maintained or improved

[codecov]

Codecov Report

Attention: Patch coverage is 95.45455% with 18 lines in your changes missing coverage. Please review.

Project coverage is 95.92%. Comparing base (1e35888) to head (59d6e19).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
src/calliope/backend/gurobi_backend_model.py	95.60%	4 Missing and 7 partials ⚠️
src/calliope/backend/backend_model.py	90.90%	0 Missing and 4 partials ⚠️
src/calliope/backend/pyomo_backend_model.py	97.29%	0 Missing and 2 partials ⚠️
src/calliope/backend/expression_parser.py	95.23%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #551      +/-   ##
==========================================
+ Coverage   95.30%   95.92%   +0.61%     
==========================================
  Files          25       26       +1     
  Lines        3664     3899     +235     
  Branches      803      768      -35     
==========================================
+ Hits         3492     3740     +248     
+ Misses         83       69      -14     
- Partials       89       90       +1     

Files with missing lines	Coverage Δ
src/calliope/backend/__init__.py	100.00% <100.00%> (+12.50%)	⬆️
src/calliope/backend/latex_backend_model.py	96.26% <100.00%> (ø)
src/calliope/backend/parsing.py	96.88% <100.00%> (+0.85%)	⬆️
src/calliope/model.py	96.29% <ø> (+1.55%)	⬆️
src/calliope/backend/expression_parser.py	94.00% <95.23%> (+0.46%)	⬆️
src/calliope/backend/pyomo_backend_model.py	98.07% <97.29%> (+2.46%)	⬆️
src/calliope/backend/backend_model.py	97.66% <90.90%> (-0.43%)	⬇️
src/calliope/backend/gurobi_backend_model.py	95.60% <95.60%> (ø)

... and 3 files with indirect coverage changes

[brynpickering]

I have added tests, but we don't have gurobipy installed in the CI builds so those tests are being skipped. Hence the massive coverage drop.

[brynpickering]

I have run the urban scale model for a full year using the pyomo (with gurobi solver) and gurobi interfaces and get the attached results.

Headline: Gurobi interface peaks at ~30% less memory than pyomo.
I know you got different results when running locally @sjpfenninger. This could be because I'm using a cloud license for Gurobi which I think means it offloads the actual solving onto a remote machine.

memray_flamegraphs.zip

---

gurobi run:

import calliope

calliope.set_log_verbosity("info")
m = calliope.examples.urban_scale(time_subset=None)
m.build(backend="gurobi")
m.solve(solver_options={"Threads": 1})

pyomo run:

import calliope

calliope.set_log_verbosity("info")
m = calliope.examples.urban_scale(time_subset=None)
m.build(backend="pyomo")
m.solve(
solver="gurobi", solver_options={"Threads": 1})

Profiling using memray and setting [out] and [runfile] separately for pyomo and gurobi runs:

memray run --trace-python-allocators -o [out].bin [runfile].py    
memray flamegraph [out].bin -o [out].html    

[brynpickering]

Last week's new pyomo release has caused an issue in one of our tests (see #597). I've pinned Pyomo upper bound to avoid this issue for now.

[irm-codebase]

Fairly big review. I tried to be thorough since this is an important change.
"Nice to haves" are just that, consider ignoring them if they result in too much work.

Also, I expect that the recent changes to the backend handling and the linter will trigger errors once you try to merge this into main.

[irm-codebase]

Shouldn't we just add the solver to our dev requirements to avoid this one?

Avoiding it for the base requirements makes sense. For dev, not so much, imo.

[brynpickering]

I tried, but it won't work. Our dependencies are currently pip and conda cross-compatible. On PyPI, this dependency is gurobipy, on Anaconda it is gurobi... So, this breaks our compatibility. I've still removed the check in the tests for gurobipy. I would prefer to alert the developer to the need to install gurobipy!

[irm-codebase]

Hmm... in the future we might want to switch to pixi to avoid this kind of stuff. It prefers conda-forge by default, but you can redirect dependencies in a case-by-case basis if needed...
I'm ok with leaving this as is for now though!

[brynpickering]

Thanks for the review @irm-codebase - updated!

[irm-codebase]

Very happy with the changes!
I've added a couple of things to clarify. After that, I can approve.

[irm-codebase]

Broken pyomo link?

[irm-codebase]

Great, thank you for the changes @brynpickering!

[irm-codebase]

I was going to send this PR back because pytest failed for some tests (due to some duplicated warning messages)... but re-making my mamba environment fixed this.

I think we should re-think our approach to dependencies, though, because following your instructions also leads to failures due to the lack of cbc solver, but that's been there for a while so it does not relate to this PR.

[brynpickering]

We can't provide generalised instructions for cbc in the installation guide due to Windows users not being able to install it directly (and pixi wouldn't solve that). I would rather move to using HiGHs as our default open source solver, but I haven't got it to work with the pyomo kernel library...

[sjpfenninger]

Since I had looked at this previously and Ivan did a detailed review I'm only making some minor suggestions for the documentation. Good to go!