From 98693bd3ba8a9bce05291dd605d9191262fb26d7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Sun, 21 May 2017 11:46:03 +0200 Subject: [PATCH 1/8] Selective versions resolutions --- .../0000-selective-versions-resolutions.md | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 accepted/0000-selective-versions-resolutions.md diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md new file mode 100644 index 0000000..77b692f --- /dev/null +++ b/accepted/0000-selective-versions-resolutions.md @@ -0,0 +1,275 @@ +- Start Date: 2017-05-21 +- RFC PR: (leave this empty) +- Yarn Issue: (leave this empty) + +# Summary + +Allow to select a nested dependency version via the `resolutions` field of +the `package.json` file. + +# Motivation + +The motivation was initially discussed in +[yarnpkg/yarn#2763](https://github.com/yarnpkg/yarn/issues/2763). + +Basically, the problem with the current behaviour of yarn is that it is +not possible to force the use of a particular version for a nested dependency. + +## Example + +For example, given the following content in the `package.json`: +```json + "devDependencies": { + "@angular/cli": "1.0.3", + "typescript": "2.3.2" + } +``` + +The `yarn.lock` file will contain: +``` +"typescript@>=2.0.0 <2.3.0": + version "2.2.2" + resolved "https://registry.yarnpkg.com/typescript/-/typescript-2.2.2.tgz#606022508479b55ffa368b58fee963a03dfd7b0c" + +typescript@2.3.2: + version "2.3.2" + resolved "https://registry.yarnpkg.com/typescript/-/typescript-2.3.2.tgz#f0f045e196f69a72f06b25fd3bd39d01c3ce9984" +``` + +Also, there will be: +- `typescript@2.3.2` in `node_modules/typescript` +- `typescript@2.2.2` in `node_modules/@angular/cli/node_modules`. + +## Problem + +In this context, it is impossible to force the use of `typescript@2.3.2` for +the whole project (except by flattening the whole project, which we don't want). + +It makes sense for typescript as the user intent is clearly to use typescript +2.3.2 for compiling all its project, and with the current behaviour, the angular +CLI (responsible of compiling `.ts` files) will simply use the 2.2.2 version +from its `node_modules`. + +## Variations of the original problem + +Similarly, even using such a content for `package.json`: +```json + "devDependencies": { + "@angular/cli": "1.0.3" + } +``` + +The need could arise for forcing the use of `typescript@2.3.2` (or +`typescript@2.1.0` for that matter). + +In these example, the need does not seem very important (the user could maybe +use `typescript@2.2.2` or ask the `@angular/cli` dev team to relax its +constraints on typescript), but there could be cases where a nested dependency introduces a bug and the project developper would want to set a specific +version for it (see for example this +[comment](https://github.com/yarnpkg/yarn/issues/2763#issuecomment-302682844)). + +## Related scenario (out of scope of this document) + +An extension of this motivation is also the potential need for mapping nested dependencies to others. For example a project developper could want to map `typescript@>=2.0.0 <2.3.0` to `my-typescript-fork@2.0.0`. + +See alternatives solutions below also. + +# Detailed design + +The proposed solution is to make the `resolutions` field of the `package.json` +file to be considered all the time and on a per-package basis. + +When a nested dependency is being resolved by yarn, if the `resolutions` field +contains a version for this package, then this version is used instead. + +All the examples are given with exact dependencies, but note that putting a +non-exact specification in the `resolutions` field should be accepted and +resolved by yarn like it usually does. + +## Example + +For example with: +```json + "devDependencies": { + "@angular/cli": "1.0.3" + }, + "resolutions": { + "typescript": "2.3.2" + } +``` + +yarn will use `typescript@2.3.2` for every nested dependency to `typescript` +and will behave as expected with respect to the `node_modules` folder by not +duplicating typescript installation. + +## Relation to non-nest dependencies + +The `devDependencies` and `dependencies` fields always take precedence over the +`resolutions` field: if the user defines explicitely a dependency there, +it means that he wants that version, even if it's specified with a non-exact +specification. So the `resolutions` field only applies to nested-dependencies. + +## Relation to the `--flat` option + +The `--flat` option becomes thus a way to populate the resolutions field for +the whole project, as it already does. +But the `resolutions` field is always considered by yarn, even if `--flat` is +not specified. + +Inceidently, this resolves this strange situation when two developers would be +working on the same project, and one is using `--flat` while the other is not, +and they would get different `node_modules` contents because of that. + +## `yarn.lock` + +This design implies that it is possible to have for a given version +specification (e.g., `>=2.0.0 <2.3.0`) a resolved version that is incompatible +with it (e.g., `2.3.2`). +It is acceptable as long as it is explicitly asked by the user. + +It is currently the case that such situation would make yarn unhappy and +provoke the modification of the `yarn.lock` (see +[yarnpkg/yarn#3420](https://github.com/yarnpkg/yarn/issues/3420)). + +This feature would remove the need for this behaviour of yarn. + +## Warnings in logs + +yarn would need to warn about the following situations: +- Unused resolutions +- Incompatible resolutions: see the above section about `yarn.lock`. +Incompatible resolutions should be accepted but warned about since it could +lead to unwanted behaviour. +- ? (see open questions below) + +# How We Teach This + +This won't have much impact as it extends the current behaviour by adding +functionality. + +The only breaking change is that `resolutions` is being considered all the time, +but that won't surprise people, this will make yarn behaviour simply more +consistent than before (see the comment on `--flat` above). + +The term "resolution" has the same meaning as before, but it is not under the +sole control of yarn itself anymore, but also under the control of the user +now. + +This is an advanced use of yarn, so new users don't really have to know about +it in the beginning. + +# Drawbacks + +## Teaching + +It makes yarn behaviour a bit more complex, even though more useful. So it +can be difficult for users to wrap their head around it. The RFC submitter has +seen it happen many times with maven, which is quite complex but complete in +its dependency management. Users would get confused and it can take time to +understand the implications of manipulation the `resolutions` field (even +though, the chosen solution, compared to the alternatives below, is much +simpler). + +## Package management paradigm + +Yarn and npm users are highly used to the idea that a dependency can be +present many times in the `node_module`, depending on which package needs it. +This has advantages and inconvenients, but it is one of the specificity of the +npm ecosystem package management. + +In this light, taking such as design decision puts yarn a bit farther to such +way of doing thing, and it could be considered a bad direction to go toward. + +Some of the alternatives below actually take this into consideration, but are +a bit more complex in terms of expressivity, so were not chosen by the RFC +submitter (see open questions below too). + +# Alternatives + +There is at least one alternative to the proposed solution, more complex but +more expressive. + +## Nested dependencies resolution per dependency + +Starting from an example, this solution would take the following form in the +`package.json` file: +```json + "devDependencies": { + "@angular/cli": "1.0.3", + "typescript": "2.3.2" + }, + "resolutions": { + "@angular/cli": { + "typescript": "2.0.2" + } + } +``` + +yarn would use `typescript@2.0.2` only for `@angular/cli` (so in +`node_modules/@angular/cli/node_modules`), but keep `typescript@2.3.2` in +`node_modules/typescript`. + +Basically, this enables the user to specify versions for nested dependencies, +but only in the context of a given dependency. + +The fields of the `resolutions` field must only refer to existing entries in +`devDependencies` and `dependencies`. + +Of course, if the same version of a nested dependency is used for many +dependencies, yarn will behave as always by keeping it directly in +`node_modules`. + +## Mapping version specifications + +This is a kind of simplified solution to the "out-of-scope scenario" in the +motivations section above (it maps versions but not dependency names). + +It was proposed in this +[comment](https://github.com/yarnpkg/yarn/issues/2763#issuecomment-301896274). + +Everything is not totally clear to me, but the idea would be to map a given +version specification to another one. +This would take this form in the `package.json`: +```json +"devDependencies": { + "@angular/cli": "1.0.3", + "typescript": "2.2.2", + "more dependencies..." + }, + "mappings": { + "typescript@>=2.0.0 <2.3.0": "typescript@2.3.2" + } +``` + +yarn would then replace matching version specifications with the user's one. +What is problematic with this is that the user has to know that `@angular/cli` +is exactly expressing its dependency to `typescript` as `>=2.0.0 <2.3.0`. + +This makes such mappings hard to maintain because they can become ignored if +`@angular/cli` is upgraded and its dependency specification changes, while +the other solutions would only result in + +# Unresolved questions + +## Is this expressive enough? + +As explained in the alternative solutions section, it would be much more +expressive and coherent with the npm ecosystem package management paradigm +to use nested dependency resolutions per project dependency. +Would the loss of simplicity acceptable maybe? + +## Warnings in logs + +Should yarn warn the user about an incoherence between an explicit dependency +and a resolution. For example if the user specify a dependency to +`typescript@2.3.2` and the resolutions field contains `typescript@2.3.0`. +For sure if the above alternative solution is chosen, this wouldn't make sense. + +Should we warn if a resolutions is incompatible, but still upper-bounded? +For example, forcing version `a@2.3` while a dependency needs version `a@2.2` is +usually less problematic than forcing version `a@2.2` while a dependency needs +version `a@2.3`. +The problem with differentiating these situations is that yarn to start giving +lots of semantics to versions and it can give false certainty to the user than +a problematic situation is not problematic. So it may be better to always warn +about incompatible resolutions. From a860d64bf51fa10eea5cd7308d4ad4765a195a5d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Sun, 21 May 2017 18:48:43 +0200 Subject: [PATCH 2/8] orthograph and max line width --- .../0000-selective-versions-resolutions.md | 21 +++++++++++-------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index 77b692f..29910bb 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -64,13 +64,16 @@ The need could arise for forcing the use of `typescript@2.3.2` (or In these example, the need does not seem very important (the user could maybe use `typescript@2.2.2` or ask the `@angular/cli` dev team to relax its -constraints on typescript), but there could be cases where a nested dependency introduces a bug and the project developper would want to set a specific +constraints on typescript), but there could be cases where a nested dependency +introduces a bug and the project developer would want to set a specific version for it (see for example this [comment](https://github.com/yarnpkg/yarn/issues/2763#issuecomment-302682844)). ## Related scenario (out of scope of this document) -An extension of this motivation is also the potential need for mapping nested dependencies to others. For example a project developper could want to map `typescript@>=2.0.0 <2.3.0` to `my-typescript-fork@2.0.0`. +An extension of this motivation is also the potential need for mapping nested +dependencies to others. For example a project developer could want to map +`typescript@>=2.0.0 <2.3.0` to `my-typescript-fork@2.0.0`. See alternatives solutions below also. @@ -105,7 +108,7 @@ duplicating typescript installation. ## Relation to non-nest dependencies The `devDependencies` and `dependencies` fields always take precedence over the -`resolutions` field: if the user defines explicitely a dependency there, +`resolutions` field: if the user defines explicitly a dependency there, it means that he wants that version, even if it's specified with a non-exact specification. So the `resolutions` field only applies to nested-dependencies. @@ -116,7 +119,7 @@ the whole project, as it already does. But the `resolutions` field is always considered by yarn, even if `--flat` is not specified. -Inceidently, this resolves this strange situation when two developers would be +Incidently, this resolves this strange situation when two developers would be working on the same project, and one is using `--flat` while the other is not, and they would get different `node_modules` contents because of that. @@ -174,14 +177,14 @@ simpler). Yarn and npm users are highly used to the idea that a dependency can be present many times in the `node_module`, depending on which package needs it. -This has advantages and inconvenients, but it is one of the specificity of the +This has advantages and disadvantages, but it is one of the specificity of the npm ecosystem package management. In this light, taking such as design decision puts yarn a bit farther to such way of doing thing, and it could be considered a bad direction to go toward. Some of the alternatives below actually take this into consideration, but are -a bit more complex in terms of expressivity, so were not chosen by the RFC +a bit more complex in terms of expressiveness, so were not chosen by the RFC submitter (see open questions below too). # Alternatives @@ -266,9 +269,9 @@ and a resolution. For example if the user specify a dependency to For sure if the above alternative solution is chosen, this wouldn't make sense. Should we warn if a resolutions is incompatible, but still upper-bounded? -For example, forcing version `a@2.3` while a dependency needs version `a@2.2` is -usually less problematic than forcing version `a@2.2` while a dependency needs -version `a@2.3`. +For example, forcing version `a@2.3` while a dependency needs version `a@2.2` +is usually less problematic than forcing version `a@2.2` while a dependency +needs version `a@2.3`. The problem with differentiating these situations is that yarn to start giving lots of semantics to versions and it can give false certainty to the user than a problematic situation is not problematic. So it may be better to always warn From 3eccabe4ad6ef5f9048d86b3f0d1ee4c3b8676c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Sun, 21 May 2017 19:05:40 +0200 Subject: [PATCH 3/8] Added a section on non-exact specifications --- .../0000-selective-versions-resolutions.md | 31 +++++++++++++++++-- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index 29910bb..f4fe257 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -85,9 +85,12 @@ file to be considered all the time and on a per-package basis. When a nested dependency is being resolved by yarn, if the `resolutions` field contains a version for this package, then this version is used instead. -All the examples are given with exact dependencies, but note that putting a +Most of the examples are given with exact dependencies, but note that putting a non-exact specification in the `resolutions` field should be accepted and -resolved by yarn like it usually does. +resolved by yarn like it usually does. This subject is discussed below also. + +Any potentially counter-intuitive situation will result in a warning being +issued. This subject is discussed at the end of this section. ## Example @@ -136,13 +139,35 @@ provoke the modification of the `yarn.lock` (see This feature would remove the need for this behaviour of yarn. +## Non-exact version specifications + +If there is a non-exact specifications in the `resolutions` field, the rule is +the same: the `resolutions` field takes precedence over the specification in a +nested dependency. + +In case the `resolutions` field is broader than the nested dependency +specification, then a warning can be issued. This happens if the the exact +version resolved by yarn based on the `resolutions` specification is +incompatible with the nested dependency specification. + +For example, if `@angular/cli` depends on `typescript@>=2.0.0 <2.3.0` and the +`resolutions` field contains `typescript@>=2.0.0 <2.4.0`, then if the latest +available version for typescript is `2.2.2`, no warning is issued, and if the +latest available version for typescript is `2.3.2` then a warning is issued. + +The rational behind that is that since the `yarn.lock` file is only modified +by the user (via yarn commands), then a warning will always be issued before +such a situation happens and is written to the `yarn.lock` file. + ## Warnings in logs -yarn would need to warn about the following situations: +yarn should warn about the following situations: - Unused resolutions - Incompatible resolutions: see the above section about `yarn.lock`. Incompatible resolutions should be accepted but warned about since it could lead to unwanted behaviour. +- Broadening specifications: see above about non-exact specifications. This +actually falls under the umbrella of incompatible resolutions. - ? (see open questions below) # How We Teach This From 04dd7cce46834e365c4afc0f881fc0dba47dc6f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Thu, 25 May 2017 11:39:57 +0200 Subject: [PATCH 4/8] clarified the questions about warning --- .../0000-selective-versions-resolutions.md | 33 ++++++------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index f4fe257..9ed9e79 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -162,13 +162,16 @@ such a situation happens and is written to the `yarn.lock` file. ## Warnings in logs yarn should warn about the following situations: -- Unused resolutions -- Incompatible resolutions: see the above section about `yarn.lock`. -Incompatible resolutions should be accepted but warned about since it could -lead to unwanted behaviour. -- Broadening specifications: see above about non-exact specifications. This -actually falls under the umbrella of incompatible resolutions. -- ? (see open questions below) +1. Unused resolutions. + +2. Incompatible resolutions (see also above the sections about `yarn.lock` +and about broadening non-exact specifications). +Basically, an incompatible resolution is used because a package does not +correctly express its dependencies. In an ideal world, the package should +be fixed at one point or another and the resolution should be removed. +In that sense, incompatible resolutions should always be warned about. +Furthermore, an incompatible resolution is a potential for unwanted behaviour +and should thus never be ignored by the user. # How We Teach This @@ -285,19 +288,3 @@ As explained in the alternative solutions section, it would be much more expressive and coherent with the npm ecosystem package management paradigm to use nested dependency resolutions per project dependency. Would the loss of simplicity acceptable maybe? - -## Warnings in logs - -Should yarn warn the user about an incoherence between an explicit dependency -and a resolution. For example if the user specify a dependency to -`typescript@2.3.2` and the resolutions field contains `typescript@2.3.0`. -For sure if the above alternative solution is chosen, this wouldn't make sense. - -Should we warn if a resolutions is incompatible, but still upper-bounded? -For example, forcing version `a@2.3` while a dependency needs version `a@2.2` -is usually less problematic than forcing version `a@2.2` while a dependency -needs version `a@2.3`. -The problem with differentiating these situations is that yarn to start giving -lots of semantics to versions and it can give false certainty to the user than -a problematic situation is not problematic. So it may be better to always warn -about incompatible resolutions. From 90bf5ddb1897ee03b668d6dbf4e9ddd598aa5b62 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Thu, 25 May 2017 16:17:56 +0200 Subject: [PATCH 5/8] rewrite with glob patterns as the solution - added more examples - added more alternatives and discuss extensions. - still missing some discussion on the `check` and on `--flat`. - some open questions --- .../0000-selective-versions-resolutions.md | 275 +++++++++++++----- 1 file changed, 204 insertions(+), 71 deletions(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index 9ed9e79..51c993c 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -50,8 +50,6 @@ It makes sense for typescript as the user intent is clearly to use typescript CLI (responsible of compiling `.ts` files) will simply use the 2.2.2 version from its `node_modules`. -## Variations of the original problem - Similarly, even using such a content for `package.json`: ```json "devDependencies": { @@ -62,6 +60,8 @@ Similarly, even using such a content for `package.json`: The need could arise for forcing the use of `typescript@2.3.2` (or `typescript@2.1.0` for that matter). +## Why? + In these example, the need does not seem very important (the user could maybe use `typescript@2.2.2` or ask the `@angular/cli` dev team to relax its constraints on typescript), but there could be cases where a nested dependency @@ -80,46 +80,158 @@ See alternatives solutions below also. # Detailed design The proposed solution is to make the `resolutions` field of the `package.json` -file to be considered all the time and on a per-package basis. +file to be considered all the time and on a per-package basis (instead of +only when the `--flat` parameter is used). When a nested dependency is being resolved by yarn, if the `resolutions` field -contains a version for this package, then this version is used instead. +contains a specification for this package, then it will be used instead. + +Special attention to the specific nature of package management in the npm +ecosystem is given in this RFC: indeed, it is not unusual to have the same +package being present as a nested dependency of multiple packages with +different versions. It is thus possible within the `resolutions` field to +express versions either for the whole dependency tree or only for a subset of +it, using a syntax relying on glob patterns. -Most of the examples are given with exact dependencies, but note that putting a +Most of the examples are given with exact dependencies, but note that using a non-exact specification in the `resolutions` field should be accepted and resolved by yarn like it usually does. This subject is discussed below also. Any potentially counter-intuitive situation will result in a warning being issued. This subject is discussed at the end of this section. -## Example +## Examples + +We the following packages and their dependencies: +``` +package-a@1.0.0 and 2.0.0 + |_ package-d1@1.0.0 + |_ package-d2@1.0.0 -For example with: +package-b@1.0.0 + |_ package-d1@2.0.0 + +package-c@1.0.0 + |_ package-a@2.0.0 + |_ ... +``` + +With: ```json "devDependencies": { - "@angular/cli": "1.0.3" + "package-a": "1.0.0", + "package-b": "1.0.0" }, "resolutions": { - "typescript": "2.3.2" + "**/package-d1": "3.0.0" } ``` -yarn will use `typescript@2.3.2` for every nested dependency to `typescript` +yarn will use `package-d1@3.0.0` for every nested dependency to `package-d1` and will behave as expected with respect to the `node_modules` folder by not -duplicating typescript installation. +duplicating the `package-d1` installation. + +With: +```json + "devDependencies": { + "package-a": "1.0.0", + "package-b": "1.0.0" + }, + "resolutions": { + "package-a/package-d1": "3.0.0" + } +``` + +yarn will use `package-d1@3.0.0` oly for `package-a` and `package-b` will +still have `package-d1@2.0.0` in its own `node_modules`. + +With: +```json + "devDependencies": { + "package-a": "1.0.0", + "package-c": "1.0.0" + }, + "resolutions": { + "**/package-a": "3.0.0" + } +``` + +`package-a` will still be resolved to `1.0.0`, but `package-c` will have +`package-a@3.0.0` in its own `node_modules`. + +With: +```json + "devDependencies": { + "package-a": "1.0.0", + "package-c": "1.0.0" + }, + "resolutions": { + "package-a": "3.0.0" + } +``` + +yarn will do nothing (see below why). + +With: +```json + "devDependencies": { + "package-a": "1.0.0", + "package-c": "1.0.0" + }, + "resolutions": { + "**/package-a/package-d1": "3.0.0" + } +``` -## Relation to non-nest dependencies +yarn will use `package-d1@3.0.0` both for `package-a` and the nested +dependency `package-a` of `package-c`. + +## Resolutions + +Each sub-field of the `resolutions` field is called a *resolution*. +It is a JSON field expressed by two strings: the package designation on the +left and a version specification on the right. + +### Package designation + +A *resolution* contains on the left-hand side a glob pattern applied to +the dependency tree (and not to the `node_modules` directory tree, since the +latter is the result of yarn resolution being influenced by the *resolution*). + +- `a/b` denotes the directly nested dependency `b` of the project's +dependency `a`. +- `**/a/b` denotes the directly nested dependency `b` +of all the dependencies and nested dependencies `a` of the project. +- `a` denotes the project's dependency `a` (see below for a discussion on +the matter of specifying a *resolution* for one of the project dependency). +- `a/**/b` denotes all the nested dependencies `b` of the project's +dependency `a`. +- `**/a` denotes all the nested dependencies `a` of the project. +- `**/a-*` denotes all the nested dependencies of the project whose +name starts with `a-`. +- `**` denotes all the nested dependencies of the project (a bad idea mostly, +as well as all other designations ending with `**`). + +### Version specification + +A *resolution* contains on the right-hand side a version specification +interpreted via the `semver` package as usually done in yarn. + +## Relation to non-nested dependencies The `devDependencies` and `dependencies` fields always take precedence over the `resolutions` field: if the user defines explicitly a dependency there, it means that he wants that version, even if it's specified with a non-exact specification. So the `resolutions` field only applies to nested-dependencies. +Nevertheless, in case of incompatibility between the specification of a +non-nested dependency version and a *resolution*, a warning is issued. ## Relation to the `--flat` option The `--flat` option becomes thus a way to populate the resolutions field for -the whole project, as it already does. -But the `resolutions` field is always considered by yarn, even if `--flat` is +the whole project, as it already does (using a package designation in the +form of `**/package-name`). +And the `resolutions` field is always considered by yarn, even when `--flat` is not specified. Incidently, this resolves this strange situation when two developers would be @@ -130,8 +242,8 @@ and they would get different `node_modules` contents because of that. This design implies that it is possible to have for a given version specification (e.g., `>=2.0.0 <2.3.0`) a resolved version that is incompatible -with it (e.g., `2.3.2`). -It is acceptable as long as it is explicitly asked by the user. +with it (e.g., `2.3.2`). It is acceptable as long as it is explicitly +asked by the user via a *resolution*. It is currently the case that such situation would make yarn unhappy and provoke the modification of the `yarn.lock` (see @@ -187,7 +299,13 @@ sole control of yarn itself anymore, but also under the control of the user now. This is an advanced use of yarn, so new users don't really have to know about -it in the beginning. +it in the beginning. Still, it is meant to be used on a potential regular +basis, in particular when some packages a project depends on have problems +in their how dependencies. + +Thus it would make sense to have a bit of the documentation talking about +this use case and underlying the fact that *resolutions* are mostly here +on a temporary basis. # Drawbacks @@ -197,30 +315,11 @@ It makes yarn behaviour a bit more complex, even though more useful. So it can be difficult for users to wrap their head around it. The RFC submitter has seen it happen many times with maven, which is quite complex but complete in its dependency management. Users would get confused and it can take time to -understand the implications of manipulation the `resolutions` field (even -though, the chosen solution, compared to the alternatives below, is much -simpler). - -## Package management paradigm - -Yarn and npm users are highly used to the idea that a dependency can be -present many times in the `node_module`, depending on which package needs it. -This has advantages and disadvantages, but it is one of the specificity of the -npm ecosystem package management. +understand the implications of manipulation the `resolutions` field . -In this light, taking such as design decision puts yarn a bit farther to such -way of doing thing, and it could be considered a bad direction to go toward. +# Alternatives and/or extensions -Some of the alternatives below actually take this into consideration, but are -a bit more complex in terms of expressiveness, so were not chosen by the RFC -submitter (see open questions below too). - -# Alternatives - -There is at least one alternative to the proposed solution, more complex but -more expressive. - -## Nested dependencies resolution per dependency +## Global nested dependencies resolution Starting from an example, this solution would take the following form in the `package.json` file: @@ -230,61 +329,95 @@ Starting from an example, this solution would take the following form in the "typescript": "2.3.2" }, "resolutions": { - "@angular/cli": { - "typescript": "2.0.2" - } + "typescript": "2.0.2" } ``` -yarn would use `typescript@2.0.2` only for `@angular/cli` (so in -`node_modules/@angular/cli/node_modules`), but keep `typescript@2.3.2` in -`node_modules/typescript`. - -Basically, this enables the user to specify versions for nested dependencies, -but only in the context of a given dependency. - -The fields of the `resolutions` field must only refer to existing entries in -`devDependencies` and `dependencies`. - -Of course, if the same version of a nested dependency is used for many -dependencies, yarn will behave as always by keeping it directly in -`node_modules`. +yarn would use `typescript@2.0.2` for the whole project and that's all. +The same kind of consideration (outside of the glob pattern thing) should be +followed that in the RFC selected solution. ## Mapping version specifications -This is a kind of simplified solution to the "out-of-scope scenario" in the -motivations section above (it maps versions but not dependency names). +This is a kind of simplified solution to the "out-of-scope scenario" presented +in the motivations section above (it maps versions but not dependency names). It was proposed in this [comment](https://github.com/yarnpkg/yarn/issues/2763#issuecomment-301896274). -Everything is not totally clear to me, but the idea would be to map a given -version specification to another one. -This would take this form in the `package.json`: +It is similar to the previous alternative but with a version specification +in the package designation. This would take this form in the `package.json`: ```json "devDependencies": { "@angular/cli": "1.0.3", "typescript": "2.2.2", "more dependencies..." }, - "mappings": { + "resolutions": { "typescript@>=2.0.0 <2.3.0": "typescript@2.3.2" } ``` yarn would then replace matching version specifications with the user's one. -What is problematic with this is that the user has to know that `@angular/cli` -is exactly expressing its dependency to `typescript` as `>=2.0.0 <2.3.0`. +For example a dependency normally resolved to `typescript@2.2.2` would be +resolved in practice to `typescript@2.3.2`. + +## Mapping version specifications as well as packages name + +Same as the tow above but with a different name on the right-hand side of the +*resolution*: +```json +"devDependencies": { + "@angular/cli": "1.0.3", + "typescript": "2.2.2" + }, + "resolutions": { + "typescript@>=2.0.0 <2.3.0": "my-typescript-fork@2.3.2" + } +``` -This makes such mappings hard to maintain because they can become ignored if -`@angular/cli` is upgraded and its dependency specification changes, while -the other solutions would only result in +or even: +```json +"devDependencies": { + "@angular/cli": "1.0.3", + "typescript": "2.2.2" + }, + "resolutions": { + "typescript": "my-typescript-fork", + } +``` + +and the version specification would be conserved. # Unresolved questions -## Is this expressive enough? +## Package designation and single `*` + +I am not totally convinced that using a single `*` in a package designation +is a good idea. Mostly because it introduces much for uncertainty to what it is +going to be applied. In particular, I'm thinking about defining a *resolution* +like that and latter adding a dependency which has a dependency that matches +but is not intended to. I feel like it adds a lot of mental weight for the +user to manipulate single `*`. + +## `--flat` option + +When used with the `install` command, the behaviour of `--flat` with respect +to this RFC is clear: it will populate the `resolutions` for all packages with +`**/dependency` as package designations. + +But when used with the `add` command, should it apply only to the nested +dependencies of the added dependency? I think that it would make sense. + +## `check` command + +The `check` command will need to be modified to take resolutions into account. +This must be specified more precisely and will be added to the RFC in the +near future. + +## Extensions of the proposition -As explained in the alternative solutions section, it would be much more -expressive and coherent with the npm ecosystem package management paradigm -to use nested dependency resolutions per project dependency. -Would the loss of simplicity acceptable maybe? +Actually, the two alternatives "Mapping version specifications" and +"Mapping version specifications as well as packages name" could be adapted +to the current proposition to support these uses cases as well. +Not sure it is a good idea but it could be useful maybe... \ No newline at end of file From 1fa7fd7dd2cb4fa16cd81c5d3fd54d367e43bcf5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Thu, 25 May 2017 16:21:21 +0200 Subject: [PATCH 6/8] correct incorrect title --- accepted/0000-selective-versions-resolutions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index 51c993c..f888201 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -317,7 +317,7 @@ seen it happen many times with maven, which is quite complex but complete in its dependency management. Users would get confused and it can take time to understand the implications of manipulation the `resolutions` field . -# Alternatives and/or extensions +# Alternatives ## Global nested dependencies resolution From df19ac9e92051da624f7a3e43d59f1b89da80ed5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Sat, 24 Jun 2017 11:51:57 +0200 Subject: [PATCH 7/8] Incorporate discussed changes + questions - Clarify retro-compatibility aspects - Add `--flat` solution according to discussions - Add `check` command solution based on @bestander comment - Re-add a new question of `--flat`... --- .../0000-selective-versions-resolutions.md | 150 +++++++++++++----- 1 file changed, 111 insertions(+), 39 deletions(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index f888201..260c98f 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -19,7 +19,7 @@ not possible to force the use of a particular version for a nested dependency. For example, given the following content in the `package.json`: ```json - "devDependencies": { + "devDependencies": { "@angular/cli": "1.0.3", "typescript": "2.3.2" } @@ -52,7 +52,7 @@ from its `node_modules`. Similarly, even using such a content for `package.json`: ```json - "devDependencies": { + "devDependencies": { "@angular/cli": "1.0.3" } ``` @@ -102,38 +102,44 @@ issued. This subject is discussed at the end of this section. ## Examples -We the following packages and their dependencies: +We have the following packages and their dependencies: ``` -package-a@1.0.0 and 2.0.0 +package-a@1.0.0 |_ package-d1@1.0.0 |_ package-d2@1.0.0 +package-a@2.0.0 + |_ package-d1@2.0.0 + |_ package-d2@1.0.0 + package-b@1.0.0 |_ package-d1@2.0.0 + |_ package-d2@1.0.0 package-c@1.0.0 |_ package-a@2.0.0 - |_ ... + |_ package-d1@2.0.0 + |_ package-d2@1.0.0 ``` With: ```json - "devDependencies": { + "dependencies": { "package-a": "1.0.0", "package-b": "1.0.0" }, "resolutions": { - "**/package-d1": "3.0.0" + "**/package-d1": "2.0.0" } ``` -yarn will use `package-d1@3.0.0` for every nested dependency to `package-d1` +yarn will use `package-d1@2.0.0` for every nested dependency to `package-d1` and will behave as expected with respect to the `node_modules` folder by not duplicating the `package-d1` installation. With: ```json - "devDependencies": { + "dependencies": { "package-a": "1.0.0", "package-b": "1.0.0" }, @@ -142,12 +148,12 @@ With: } ``` -yarn will use `package-d1@3.0.0` oly for `package-a` and `package-b` will +yarn will use `package-d1@3.0.0` only for `package-a` and `package-b` will still have `package-d1@2.0.0` in its own `node_modules`. With: ```json - "devDependencies": { + "dependencies": { "package-a": "1.0.0", "package-c": "1.0.0" }, @@ -161,7 +167,7 @@ With: With: ```json - "devDependencies": { + "dependencies": { "package-a": "1.0.0", "package-c": "1.0.0" }, @@ -174,7 +180,7 @@ yarn will do nothing (see below why). With: ```json - "devDependencies": { + "dependencies": { "package-a": "1.0.0", "package-c": "1.0.0" }, @@ -207,6 +213,10 @@ the matter of specifying a *resolution* for one of the project dependency). - `a/**/b` denotes all the nested dependencies `b` of the project's dependency `a`. - `**/a` denotes all the nested dependencies `a` of the project. +- `a` is an alias for `**/a` (for retro-compatibility, see below, and because +if it wasn't such an alias, it wouldn't mean anything as it would represent +one of the non-nested project dependencies, which can't be overridden as +explained below) - `**/a-*` denotes all the nested dependencies of the project whose name starts with `a-`. - `**` denotes all the nested dependencies of the project (a bad idea mostly, @@ -219,25 +229,55 @@ interpreted via the `semver` package as usually done in yarn. ## Relation to non-nested dependencies -The `devDependencies` and `dependencies` fields always take precedence over the +The `devDependencies`, `optionalDependencies` and `dependencies` fields always +take precedence over the `resolutions` field: if the user defines explicitly a dependency there, it means that he wants that version, even if it's specified with a non-exact specification. So the `resolutions` field only applies to nested-dependencies. Nevertheless, in case of incompatibility between the specification of a non-nested dependency version and a *resolution*, a warning is issued. +This is coherent with the fact that the package designation `package-a` can be +used safely as an alias of `**/package-a`: if it wasn't the case, `package-a` +would designate one of the non-nested dependencies and would be ignored. + +## Retro compatibility for the `resolutions` field + +Until now, the`resolutions` field can contain *resolutions* of the following +form (filled by `add --flat` or `install --flat`): +```json + "resolutions": { + "package-a": "1.0.0" + } +``` + +With the current proposal, the package designation `package-a` is an alias for +`**/package-a`: this means the behaviour of yarn with a project whose +`resolutions` field contains *resolutions* filed by a pre-RFC yarn will be +as expected: the nested dependencies will have the fixed version specified. + ## Relation to the `--flat` option The `--flat` option becomes thus a way to populate the resolutions field for the whole project, as it already does (using a package designation in the -form of `**/package-name`). -And the `resolutions` field is always considered by yarn, even when `--flat` is -not specified. +form of `package-name`). + +The only breaking change is that the `resolutions` field is always considered +by yarn, even when `--flat` is not specified! Incidently, this resolves this strange situation when two developers would be working on the same project, and one is using `--flat` while the other is not, and they would get different `node_modules` contents because of that. +Note that `--flat` being related to the installation mode (it is used via +the `install` command, but also via the `add` command but pertains to the +installation itself, not the adding), it will continue to behave as before +by asking for *resolutions* of all the nested dependencies of the project even +with `add`. + +See open question below about the need for removing `--flat` from `install` +and for introducing a `flatten` command instead. + ## `yarn.lock` This design implies that it is possible to have for a given version @@ -251,6 +291,24 @@ provoke the modification of the `yarn.lock` (see This feature would remove the need for this behaviour of yarn. +## Relation to the `check` command + +The default `check` (without specific options) reads `yarn.lock` and makes +sure that all versions in it match to what is inside `node_modules`. + +We should thus get this for free without extra changes. + +### `--verify-tree` + +`--verify-tree` was built to make sure that all packages inside `node_modules` +are consistent between each other independently of yarn's resolution logic. + +If you force a version that does not match semver requirements of a package, +`--verify-tree` would throw an error. + +For now we don't need to make changes to it, but later, we can expand +`--verify-tree` to support the overrides of the `resolutions` field. + ## Non-exact version specifications If there is a non-exact specifications in the `resolutions` field, the rule is @@ -311,7 +369,7 @@ on a temporary basis. ## Teaching -It makes yarn behaviour a bit more complex, even though more useful. So it +It makes yarn behaviour a bit more complex, although more useful. So it can be difficult for users to wrap their head around it. The RFC submitter has seen it happen many times with maven, which is quite complex but complete in its dependency management. Users would get confused and it can take time to @@ -335,12 +393,14 @@ Starting from an example, this solution would take the following form in the yarn would use `typescript@2.0.2` for the whole project and that's all. The same kind of consideration (outside of the glob pattern thing) should be -followed that in the RFC selected solution. +followed as with the selected solution of this RFC. + +This is basically too simple according to discussions with yarn maintainers. ## Mapping version specifications This is a kind of simplified solution to the "out-of-scope scenario" presented -in the motivations section above (it maps versions but not dependency names). +in the Motivations section above (it maps versions but not dependency names). It was proposed in this [comment](https://github.com/yarnpkg/yarn/issues/2763#issuecomment-301896274). @@ -348,7 +408,7 @@ It was proposed in this It is similar to the previous alternative but with a version specification in the package designation. This would take this form in the `package.json`: ```json -"devDependencies": { + "devDependencies": { "@angular/cli": "1.0.3", "typescript": "2.2.2", "more dependencies..." @@ -362,12 +422,14 @@ yarn would then replace matching version specifications with the user's one. For example a dependency normally resolved to `typescript@2.2.2` would be resolved in practice to `typescript@2.3.2`. +This is too advanced and can be considered a possible extension of this RFC. + ## Mapping version specifications as well as packages name -Same as the tow above but with a different name on the right-hand side of the +Same as the two above but with a different name on the right-hand side of the *resolution*: ```json -"devDependencies": { + "devDependencies": { "@angular/cli": "1.0.3", "typescript": "2.2.2" }, @@ -378,7 +440,7 @@ Same as the tow above but with a different name on the right-hand side of the or even: ```json -"devDependencies": { + "devDependencies": { "@angular/cli": "1.0.3", "typescript": "2.2.2" }, @@ -389,6 +451,14 @@ or even: and the version specification would be conserved. +This is too advanced and can be considered a possible extension of this RFC. + +# Future extensions + +The two alternatives discussed in the section just above, "Mapping version +specifications" and "Mapping version specifications as well as packages name", +can be adapted to the current proposition to support these uses cases as well. + # Unresolved questions ## Package designation and single `*` @@ -402,22 +472,24 @@ user to manipulate single `*`. ## `--flat` option -When used with the `install` command, the behaviour of `--flat` with respect -to this RFC is clear: it will populate the `resolutions` for all packages with -`**/dependency` as package designations. - -But when used with the `add` command, should it apply only to the nested -dependencies of the added dependency? I think that it would make sense. +I wonder if the `--flat` option of `install` shouldn't be transformed to +a `flatten` command that would: +1. Fill in the *resolutions* for all nested dependencies. +2. Set the `flat` field in the `package.json`. -## `check` command +It makes no real sense to have a flattening mode for `install`: +1. `install` already follows the `resolutions` field with this RFC. +2. `install` should be only about building the `node_modules` directory, not +modifying the the `package.json` IMHO. -The `check` command will need to be modified to take resolutions into account. -This must be specified more precisely and will be added to the RFC in the -near future. +If we do that, then the `--flat` option of `add` (and the `flat` option in the +`package.json`) would apply not to the installation but to the adding, +upgrading, etc (everything that modify the `package.json`'s dependencies): +it will ensure that the project stays flattened via the populating of the +`resolutions` field. -## Extensions of the proposition +## resolutions in dependencies -Actually, the two alternatives "Mapping version specifications" and -"Mapping version specifications as well as packages name" could be adapted -to the current proposition to support these uses cases as well. -Not sure it is a good idea but it could be useful maybe... \ No newline at end of file +If a dependency contains resolutions, should it be taken into account? +In Maven, it is not the case for example. +In other words: should we restrict resolutions to `private` project? \ No newline at end of file From bc3e5180211268a92c675c9c7460a714438d999d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Victor=20No=C3=ABl?= Date: Sat, 8 Jul 2017 11:09:28 +0200 Subject: [PATCH 8/8] final changes, no more open questions --- .../0000-selective-versions-resolutions.md | 64 +++++++++---------- 1 file changed, 31 insertions(+), 33 deletions(-) diff --git a/accepted/0000-selective-versions-resolutions.md b/accepted/0000-selective-versions-resolutions.md index 260c98f..e513391 100644 --- a/accepted/0000-selective-versions-resolutions.md +++ b/accepted/0000-selective-versions-resolutions.md @@ -208,20 +208,22 @@ latter is the result of yarn resolution being influenced by the *resolution*). dependency `a`. - `**/a/b` denotes the directly nested dependency `b` of all the dependencies and nested dependencies `a` of the project. -- `a` denotes the project's dependency `a` (see below for a discussion on -the matter of specifying a *resolution* for one of the project dependency). - `a/**/b` denotes all the nested dependencies `b` of the project's dependency `a`. - `**/a` denotes all the nested dependencies `a` of the project. - `a` is an alias for `**/a` (for retro-compatibility, see below, and because if it wasn't such an alias, it wouldn't mean anything as it would represent one of the non-nested project dependencies, which can't be overridden as -explained below) -- `**/a-*` denotes all the nested dependencies of the project whose -name starts with `a-`. +explained below). - `**` denotes all the nested dependencies of the project (a bad idea mostly, as well as all other designations ending with `**`). +Note on single star: `*` is not authorized in a package resolution because it +would introduce too much non-determinism. For example, there is the risk of a +referring to `package-*` at one point to match `package-a` and `package-b`, +and later on, this would match a new nested dependency `package-c` that wasn't +intended to be matched. + ### Version specification A *resolution* contains on the right-hand side a version specification @@ -258,8 +260,15 @@ as expected: the nested dependencies will have the fixed version specified. ## Relation to the `--flat` option -The `--flat` option becomes thus a way to populate the resolutions field for -the whole project, as it already does (using a package designation in the +Before this RFC, `--flat` is both about populating resolutions field AND +taking resolutions field into account when executing the `install` command +(including installation as part of the `add` command). + +This RFC is about taking the `resolutions` field into account when executing +the `install` command (including installation as part of the `add` command). + +So with this RFC, `--flat` is now only about populating resolutions field. +I does it in the same way as before (using a package designation in the form of `package-name`). The only breaking change is that the `resolutions` field is always considered @@ -275,8 +284,8 @@ installation itself, not the adding), it will continue to behave as before by asking for *resolutions* of all the nested dependencies of the project even with `add`. -See open question below about the need for removing `--flat` from `install` -and for introducing a `flatten` command instead. +In the future, `--flat` will need to be rethought but for now we will keep +its behaviour. ## `yarn.lock` @@ -343,6 +352,11 @@ In that sense, incompatible resolutions should always be warned about. Furthermore, an incompatible resolution is a potential for unwanted behaviour and should thus never be ignored by the user. +## Locality of the *resolutions* + +The `resolutions` field only apply to the local project and not to the projects +that depends on it. It is the same as with lock files in a way. + # How We Teach This This won't have much impact as it extends the current behaviour by adding @@ -459,21 +473,12 @@ The two alternatives discussed in the section just above, "Mapping version specifications" and "Mapping version specifications as well as packages name", can be adapted to the current proposition to support these uses cases as well. -# Unresolved questions +## `flatten` -## Package designation and single `*` +Some notes on `--flat` and its future with respect to this RFC. -I am not totally convinced that using a single `*` in a package designation -is a good idea. Mostly because it introduces much for uncertainty to what it is -going to be applied. In particular, I'm thinking about defining a *resolution* -like that and latter adding a dependency which has a dependency that matches -but is not intended to. I feel like it adds a lot of mental weight for the -user to manipulate single `*`. - -## `--flat` option - -I wonder if the `--flat` option of `install` shouldn't be transformed to -a `flatten` command that would: +The `--flat` option of `install` could be transformed to a `flatten` command +that would: 1. Fill in the *resolutions* for all nested dependencies. 2. Set the `flat` field in the `package.json`. @@ -482,14 +487,7 @@ It makes no real sense to have a flattening mode for `install`: 2. `install` should be only about building the `node_modules` directory, not modifying the the `package.json` IMHO. -If we do that, then the `--flat` option of `add` (and the `flat` option in the -`package.json`) would apply not to the installation but to the adding, -upgrading, etc (everything that modify the `package.json`'s dependencies): -it will ensure that the project stays flattened via the populating of the -`resolutions` field. - -## resolutions in dependencies - -If a dependency contains resolutions, should it be taken into account? -In Maven, it is not the case for example. -In other words: should we restrict resolutions to `private` project? \ No newline at end of file +Then the `flat` option in the `package.json` (and the `--flat` option of `add`) +would apply not to the installation but to the adding, upgrading, etc +(everything that modify the `package.json`'s dependencies). It will ensure +that the project stays flattened via the populating of the `resolutions` field.