-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Frame out the env spec cep #50
base: main
Are you sure you want to change the base?
Changes from all commits
6fb939f
fbb8b3b
f947202
97af8dc
cdb1073
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,131 @@ | ||
<table> | ||
<tr><td> Title </td><td> Environment yaml specification </td> | ||
<tr><td> Status </td><td> Draft </td></tr> | ||
<!-- <tr><td> Status </td><td> Draft | Proposed | Accepted | Rejected | Deferred | Implemented </td></tr> --> | ||
<tr><td> Author(s) </td> | ||
<td> | ||
Eric Dill | ||
Marius van Niekerk | ||
Jaime Rodríguez-Guerra | ||
ericdill marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Albert DeFusco | ||
ericdill marked this conversation as resolved.
Show resolved
Hide resolved
|
||
</td> | ||
</tr> | ||
<tr><td> Created </td><td> March 28, 2023</td></tr> | ||
<tr><td> Updated </td><td> NA </td></tr> | ||
<tr><td> Discussion </td><td> NA </td></tr> | ||
<tr><td> Implementation </td><td> NA </td></tr> | ||
</table> | ||
|
||
## Abstract | ||
|
||
This CEP aims to formalize the current state of the environment.yaml specification. | ||
It notably does not attempt to address any proposed changes to the specification. | ||
That is currently being discussed elsewhere. | ||
|
||
## Motivation | ||
|
||
There are a number of existing projects that can accept the env.yml format as input. | ||
Each project that wants to use the env.yml spec has to implement their own parser. | ||
This is a problem because it means that there are multiple implementations of the same specification. | ||
This makes it difficult to evolve the specification because right now it is implicitly defined as however `conda-env` handles the env.yml file. | ||
This CEP aims to formalize the specification so that it can be used as a reference for other projects, including `conda-env`. | ||
There are [existing docs](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) that defines how a user should create an environment.yml file. | ||
From what I can tell this is the only formal specification for this file. | ||
It is the objective of this CEP to formalize the existing specification so that other projects can standardize on what they expect as input. | ||
Evolutions to this spec are expected and welcome, but are out of the scope for the initial proposal. | ||
|
||
Issues found in the wild that result from env.yaml not having a published spec: | ||
- [vscode issue #280](https://github.com/microsoft/vscode-python/issues/280) | ||
|
||
## Specification | ||
|
||
There are four keys that the conda env spec expects: | ||
|
||
`name`, `str`, specifying the name of the conda env that will be created. | ||
|
||
`channels`, `list`, specifying the channels that the solver should find and pull packages from | ||
|
||
`dependencies`, `list`, specifying the top-level dependencies that the solver should start with | ||
ericdill marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- `dependencies` also has an optional dict with the `pip` key that expects a `list` as its value. This key tells the solver to also include the listed dependencies to pull from pypi | ||
- Each non-pip `dependencies` entry must follow the form expected by the current [MatchSpec](https://github.com/conda/conda/blob/a8e441e3c0e80b0d4e1595579f7d9eaad2b0fb2b/conda/models/match_spec.py#L92) implementation. Some examples of currently supported dependency specs (at least at the time of this CEP): | ||
``` | ||
Examples: | ||
>>> str(MatchSpec(name='foo', build='py2*', channel='conda-forge')) | ||
'conda-forge::foo[build=py2*]' | ||
>>> str(MatchSpec('foo 1.0 py27_0')) | ||
'foo==1.0=py27_0' | ||
>>> str(MatchSpec('foo=1.0=py27_0')) | ||
'foo==1.0=py27_0' | ||
>>> str(MatchSpec('conda-forge::foo[version=1.0.*]')) | ||
'conda-forge::foo=1.0' | ||
>>> str(MatchSpec('conda-forge/linux-64::foo>=1.0')) | ||
"conda-forge/linux-64::foo[version='>=1.0']" | ||
>>> str(MatchSpec('*/linux-64::foo>=1.0')) | ||
"foo[subdir=linux-64,version='>=1.0']" | ||
``` | ||
|
||
`prefix`, `str`, the full path to the environment. It's not clear to me how `name` and `prefix` interact. | ||
ericdill marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
`variables`, `dict`, {`str`: `str`} format of environment variables to set/unset when the environment is activated/deactivated. | ||
|
||
jsonschema version of the current spec: | ||
|
||
```json | ||
{ | ||
"$schema": "https://json-schema.org/draft/2020-12/schema", | ||
"type": "object", | ||
"properties": { | ||
"name": {"type": "string"}, | ||
"prefix": {"type": "string"}, | ||
"channels": { | ||
"type": "array", | ||
"items": {"type": "string"} | ||
}, | ||
"dependencies": { | ||
"type": "array", | ||
"items": { | ||
"anyOf": [ | ||
{"type": "string"}, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How far down the MatchSpec rabbit hole do we want to go with this jsonschema? jsonschema does support various formatting validators like regex, but I'm a little hesitant to try and capture MatchSpec in a regex... Thoughts? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It feels we need a JSONSchema for the MatchSpec first, and then we can refer to it if anything. Can we leave a "type hint" for now? e.g. |
||
{"$ref": "#/definitions/pip"} | ||
] | ||
} | ||
}, | ||
"variables": { | ||
"type": "object", | ||
"additionalProperties": {"type": "string"} | ||
} | ||
}, | ||
"definitions": { | ||
"pip": { | ||
"type": "object", | ||
"additionalProperties": { | ||
"type": "array", | ||
"required": ["pip"], | ||
"properties": { | ||
"pip": {"type": "array"} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
## Other sections | ||
|
||
Other relevant sections of the proposal. Common sections include: | ||
|
||
* Specification -- The technical details of the proposed change. | ||
* Motivation -- Why the proposed change is needed. | ||
* Rationale -- Why particular decisions were made in the proposal. | ||
* Backwards Compatibility -- Will the proposed change break existing | ||
packages or workflows. | ||
* Alternatives -- Any alternatives considered during the design. | ||
* Sample Implementation -- Links to prototype or a sample implementation of | ||
the proposed change. | ||
* FAQ -- Frequently asked questions (and answers to them). | ||
* Resolution -- A short summary of the decision made by the community. | ||
* Reference -- Any references used in the design of the CEP. | ||
|
||
## Copyright | ||
|
||
All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
name: stats | ||
dependencies: | ||
- numpy | ||
- pandas |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
name: stats2 | ||
channels: | ||
- javascript | ||
dependencies: | ||
- python=3.9 | ||
- bokeh=2.4.2 | ||
- conda-forge::numpy=1.21.* | ||
- nodejs=16.13.* | ||
- flask | ||
- pip | ||
- pip: | ||
- Flask-Testing |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
name: env-name | ||
channels: | ||
- conda-forge | ||
- defaults | ||
dependencies: | ||
- python=3.7 | ||
- codecov | ||
prefix: /Users/username/anaconda3/envs/env-name | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jezdez I've been poking at this
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Interesting. I've been confused by it since it started appearing. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
name: env-name | ||
channels: | ||
- conda-forge | ||
- defaults | ||
dependencies: | ||
- python=3.7 | ||
- codecov | ||
variables: | ||
VAR1: valueA | ||
VAR2: valueB |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
{ | ||
"$schema": "https://json-schema.org/draft/2020-12/schema", | ||
"type": "object", | ||
"properties": { | ||
"name": { | ||
"type": "string" | ||
}, | ||
"prefix": { | ||
"type": "string" | ||
}, | ||
"channels": { | ||
"type": "array", | ||
"items": { | ||
"type": "string" | ||
} | ||
}, | ||
"dependencies": { | ||
"type": "array", | ||
"items": { | ||
"anyOf": [ | ||
{ | ||
"type": "string" | ||
}, | ||
{ | ||
"$ref": "#/definitions/pip" | ||
} | ||
] | ||
} | ||
}, | ||
"variables": { | ||
"type": "object", | ||
"additionalProperties": { | ||
"type": "string" | ||
} | ||
} | ||
}, | ||
"definitions": { | ||
"pip": { | ||
"type": "object", | ||
"additionalProperties": { | ||
"type": "array", | ||
"required": [ | ||
"pip" | ||
], | ||
"properties": { | ||
"pip": { | ||
"type": "array" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mariusvniekerk you interested in being a co-author of this?