request: pub workspaces #747

lishaduck · 2024-08-07T20:56:52Z

Is there an existing feature request for this?

I have searched the existing issues.

Command

No response

Description

Once Pub Workspaces land in stable, it would be great if Melos replaced dependency_overrides generation in favor of Pub Workspaces.
I believe this is planned, but I didn't see an issue.

Reasoning

Pub Workspaces enable a single analyzer scope, which both improves performance and reduces the maintenance burden on the Melos team.

Additional context and comments

Thanks for melos! It works great!

The text was updated successfully, but these errors were encountered:

spydon · 2024-08-07T20:58:50Z

It is indeed planned, thanks for opening an issue so that we can track it!
I'll link in the Flutter design doc for it is here too: https://flutter.dev/go/pub-workspace

spydon · 2024-09-10T12:19:54Z

Related: dart-lang/pub#4376

spydon · 2024-09-18T12:20:20Z

Related (globbing in workspace definition, which would make future migrations easier): dart-lang/pub#4391

Leptopoda · 2024-11-27T09:55:25Z

I can see how globs are nice to have for this use case, but I wouldn't consider it a blocker.
I've migrated some repos to workspaces by using melos list -rp.

I'd love to use melos + workspaces sooner rather than waiting for this that isn't even such a big inconvenience.

spydon · 2024-11-27T09:58:01Z

@Leptopoda I agree, it's not a blocker, possibly we could even write a simple migration script for it.

xaethos · 2024-12-04T06:41:53Z

Do Melos and Pub workspaces work together right now? Has anybody gotten a repo migrated by hand?

A few weeks back I took a stab at migrating a repo and couldn't get everything to work. I cannot remember for sure, but I think my blocker ultimately was the Flutter toolchain not handling Pub workspaces, though believe some Melos commands were confused by the dep overrides being elsewhere.

Is there a sense of what needs to be done for Melos and Pub workspaces to coexist?

spydon · 2024-12-04T07:20:29Z

@xaethos it shouldn't be a problem for them to co-exist already afaik. With the workspaces we can simplify a lot of our code in Melos and remove the packages config from melos.yaml though, and that needs some work.

Leptopoda · 2024-12-04T07:29:01Z

What I have done so far:

remove dep overrides
add every package to the workspace in the root pubspec
change the dependency resolution for every package to "workspace"

Et voilà.
You MUST NOT use melos bootstrap as it will try to generate the overrides again. I suggest to just use dart pub get and melos for all the other commands (format, run scripts, publish, ...).

The current stable flutter version is broken for workspaces and last time I checked master it was also not working. I found 3.26.0-0.1.preto be working in some testing (just specify it in fvm) although your use case might forbid unreleased versions. A stable release should be right around the corner.

SAGARSURI · 2024-12-12T01:50:22Z

finally workspace is released to stable version 3.27:

https://dart.dev/tools/pub/workspaces

SAGARSURI · 2024-12-12T06:29:57Z

I hope melos bootstrap will not be removed but tweaked to handle the new workspace changes.

This way we can still use hooks pre/post bootstrap command execution.
Also the variable MELOS_PACKAGES might get effected which we use it in our pipeline to run test on changed packages because of this problem fix: melos overrides the applied filter for dependents/dependencies #675

spydon · 2024-12-12T20:32:44Z

I hope melos bootstrap will not be removed but tweaked to handle the new workspace changes.

* This way we can still use `hooks` pre/post `bootstrap` command execution.

* Also the variable `MELOS_PACKAGES` might get effected which we use it in our pipeline to run test on changed packages because of this problem [fix:  melos overrides the applied filter for dependents/dependencies #675](https://github.com/invertase/melos/issues/675)

Yeah, that's true. We should keep bootstrap, but as an optional step.

I'm pretty busy currently, so if there is anyone else that would like to be assigned to this issue, give me a ping!
I don't think it is thaaat much to do, mostly removing a lot of stuff that we have in melos that is now superfluous.

lishaduck · 2024-12-24T02:01:43Z

If anyone is looking for a working example of a migration, I've got this PR: lishaduck/legacy_checks#38.

SalihCanBinboga · 2025-01-03T11:57:28Z

If anyone is looking for a working example of a migration, I've got this PR: lishaduck/legacy_checks#38.

“melos bootstrap” command cannot be executed by you, right? (“melos bootstrap” command creates a ‘pubspec_overrides.yaml’ file for each module. Because of this, pub workspaces do not function properly)

lishaduck · 2025-01-03T15:43:59Z

“melos bootstrap” command cannot be executed by you, right? (“melos bootstrap” command creates a ‘pubspec_overrides.yaml’ file for each module. Because of this, pub workspaces do not function properly)

Right. You call dart pub get from the root instead. See here: lishaduck/legacy_checks#38 (comment).

SalihCanBinboga · 2025-01-03T16:34:42Z

“melos bootstrap” command cannot be executed by you, right? (“melos bootstrap” command creates a ‘pubspec_overrides.yaml’ file for each module. Because of this, pub workspaces do not function properly)

Right. You call dart pub get from the root instead. See here: lishaduck/legacy_checks#38 (comment).

Thanks @lishaduck but I used the 'melos bootstrap' command because it also generates *.iml files specific to modules in IntelliJ. Additionally, when I run build_runner in the root directory, it still doesn't detect the modules - I need to go to the relevant module directory and run build_runner there. Hopefully this will be resolved soon..

lishaduck · 2025-01-03T17:00:45Z

Thanks @lishaduck but I used the 'melos bootstrap' command because it also generates *.iml files specific to modules in IntelliJ.

Yeah, I use VS Code, which doesn't need them. What'd I'd probably do is run bootstrap, run rm -rf .dart_tool **/pubspec_overrides.yaml, and then run dart pub get.

Additionally, when I run build_runner in the root directory, it still doesn't detect the modules - I need to go to the relevant module directory and run build_runner there. Hopefully this will be resolved soon..

I'd think that's the expected behavior? Just make a melos task to run it all concurrently.

SalihCanBinboga · 2025-01-03T17:13:35Z

Thanks @lishaduck but I used the 'melos bootstrap' command because it also generates *.iml files specific to modules in IntelliJ.

Yeah, I use VS Code, which doesn't need them. What'd I'd probably do is run bootstrap, run rm -rf .dart_tool **/pubspec_overrides.yaml, and then run dart pub get.

Additionally, when I run build_runner in the root directory, it still doesn't detect the modules - I need to go to the relevant module directory and run build_runner there. Hopefully this will be resolved soon..

I'd think that's the expected behavior? Just make a melos task to run it all concurrently.

Actually, I already have the melos run command as shown below, but due to bootstrap pub workspaces, this command cannot be executed, so it doesn't apply to all modules. I'm in a dilemma in such a situation :)

 gen:all:
    name: gen
    description: Generate all files
    run: "dart run build_runner build -d --low-resources-mode"
    packageFilters:
      dependsOn:
        - "build_runner"
    exec:
      concurrency: 4
      failFast: true

lishaduck · 2025-01-03T17:50:29Z

Actually, I already have the melos run command as shown below, but due to bootstrap pub workspaces, this command cannot be executed, so it doesn't apply to all modules. I'm in a dilemma in such a situation :)

What's the error? (I haven't migrated my build_runner-dependent packages because custom_lint support is still in-progress)

SalihCanBinboga · 2025-01-03T18:08:14Z

Actually, I already have the melos run command as shown below, but due to bootstrap pub workspaces, this command cannot be executed, so it doesn't apply to all modules. I'm in a dilemma in such a situation :)

What's the error? (I haven't migrated my build_runner-dependent packages because custom_lint support is still in-progress)

Problem: Due to the issue mentioned here, Melos does not recognize my modules, so I cannot run my Melos scripts for my modules.

lishaduck · 2025-01-03T18:25:11Z

Problem: Due to the issue mentioned here, Melos does not recognize my modules, so I cannot run my Melos scripts for my modules.

Can you just copy+paste the error message? That message doesn't provide an error nor does it mention Melos failing to recognize modules.

SalihCanBinboga · 2025-01-03T19:00:29Z

Problem: Due to the issue mentioned here, Melos does not recognize my modules, so I cannot run my Melos scripts for my modules.

Can you just copy+paste the error message? That message doesn't provide an error nor does it mention Melos failing to recognize modules.

Melos needs to recognize the modules, which requires running the melos bootstrap method, right? Since I cannot run the melos bootstrap method due to the use of the pub workspace, the modules are not recognized. Therefore, when I run melos gen:all, it does not give any errors, but it does not recognize any of the 60 modules.

lishaduck · 2025-01-03T19:01:55Z

Melos needs to recognize the modules, which requires running the melos bootstrap method, right?

I don't think so... it just uses a glob in the melos.yaml, right?

spydon · 2025-01-03T19:10:06Z

I don't think so... it just uses a glob in the melos.yaml, right?

Correct, there is no caching of the package list when bootstrapping.

SalihCanBinboga · 2025-01-03T20:41:41Z

@lishaduck @spydon You are right; I noticed the issue was different. After enabling pub workspaces, I removed the build_runner dependency from the modules because it is sufficient to use it only in the root workspace. I realized that the gen:all script doesn’t recognize the modules because it depends on build_runner..

spydon · 2025-01-06T20:35:58Z

The PR to migrate Melos to Pub Workspaces is now out, feel free to review it! (#816)

Tomorrow I'll create a pre-release so that you can try it out in your projects.

spydon · 2025-01-07T19:47:36Z

I just published Melos v7.0.0-dev.1 enabling the Pub Workspaces feature, please try it out so that we can iron out any potential problems! 🙂
https://pub.dev/packages/melos

spydon · 2025-01-07T20:05:04Z

If you want to add resolution: workspace to all of your pubspec.yaml files you can do this:

find . -type f -name "pubspec.yaml" -exec sed -i '/^name:/a resolution: workspace' {} +

and to get all your pubspec.yaml paths to add to the root pubspec's workspace section, you can do this:

find . -name "pubspec.yaml" -exec dirname {} + | sed 's|^\./||'

Remember that the workspace section doesn't support globs yet, so you'll most likely have quite a long list, you can give a thumbs up to the issue suggesting to add globs to it here: dart-lang/pub#4391

spydon · 2025-01-07T20:45:27Z

If anyone is curious how a migration might look like, this is what it looked like for the Flame monorepo:
flame-engine/flame#3433

lishaduck · 2025-01-07T21:06:08Z

Oh, and in case anyone is using custom_lint: don't upgrade 😢
custom_lint is essentially unmaintained right now, as the Dart team is actively working on shipping analyzer plugins built-in... which means Remi isn't going to put in the effort to get custom_lint working in Pub Workspaces.
Alternatively, turn off custom_lint.

## Description Since we now declare the packages in the `workspace` section in the `pubspec.yaml` file we'll move the rest of the config in there too, under it's own `melos` section. This is common practice for other dart tools too and the Dart team knows about it. Continues on the #747 effort. ## Type of Change  - [x] ✨ `feat` -- New feature (non-breaking change which adds functionality) - [ ] 🛠️ `fix` -- Bug fix (non-breaking change which fixes an issue) - [x] ❌ `!` -- Breaking change (fix or feature that would cause existing functionality to change) - [ ] 🧹 `refactor` -- Code refactor - [ ] ✅ `ci` -- Build configuration change - [ ] 📝 `docs` -- Documentation - [ ] 🗑️ `chore` -- Chore

spydon · 2025-01-10T14:25:37Z

@lishaduck there was a release for custom_lint two days ago, was that one not working with workspaces either?

lishaduck · 2025-01-10T14:31:02Z

@lishaduck there was a release for custom_lint two days ago, was that one not working with workspaces either?

I haven't tested it, but it shouldn't've fixed it. He bumped analyzer, source_gen, etc, across his ecosystem (Riverpod, Freezed, custom_lint, etc), but workspace support wasn't included, and, as I noted, isn't planned (invertase/dart_custom_lint#298).

spydon · 2025-01-10T15:06:52Z

7.0.0-dev.3 is now out, removing the root melos.yaml and moving the config in under a melos section in pubspec.yaml since half of the config is in there now anyways, migration instructions are updated in the readme and in the docs.

I'll keep this issue open until we have released 7.0.0 to stable.

Do note that if you have uses-material-design: true in an example and not in the package itself and runs flutter test in the package you will get a warning, but it will soon be fixed. (dart-lang/pub#4486)

petrnymsa · 2025-01-14T13:04:47Z

Probably not related directly with melos but until Pub workspaces were introduced we used melos even in projects which are not monorepos. Why? Because every project has several "standartized" melos scripts such as setup, codegen....

However after Melos is migrated to support pub workspaces and removed melos.yaml we cant do that anymore as - . relative path in workspace setting is not allowed.

# Main pubpsec.yaml

workspace:
	- . 

.....

melos:
  scripts:
      ....

Executing melos bs


fvm exec melos bs 
fvm: Running version: "3.27.1"

Error on line 10, column 5 of pubspec.yaml: "workspace" members must be subdirectories
   ╷
10 │   - .
   │     ^
   ╵

EDIT:

Of course we can solve it by restructuring project into "monorepo"

spydon · 2025-01-14T13:59:42Z

Of course we can solve it by restructuring project into "monorepo"

Yeah, I think that is the only way currently with pub workspaces.

petrnymsa · 2025-01-22T11:56:18Z

@spydon Not sure (again) if I should create new issue for this.

After we migrated single app project to newer melos (see previous comment) we cant run melos command inside "app" folder. Melos needs to be executed in root folder, where workspace pubspec.yaml is defined along with pubspec.lock. Otherwise melos looks for pubspec.lock inside app folder.

/root
  - pubpsec.yaml // WORKSPACE pubspec
  - pubspec.lock
  - app/
        - pubspec.yaml // APP pubspec

If I run melos run inside app folder I am getting

PathNotFoundException: Cannot open file, path = 'REDACTED/app/pubspec.lock' (OS Error: No such file or directory, errno = 2)
#0      _File.throwIfError (dart:io/file_impl.dart:675:7)
#1      _File.openSync (dart:io/file_impl.dart:490:5)
#2      _File.readAsBytesSync (dart:io/file_impl.dart:574:18)
#3      _File.readAsStringSync (dart:io/file_impl.dart:624:18)
#4      ExecutableInstallation._loadPubspecLockEntry (package:cli_launcher/cli_launcher.dart:142:47)
#5      ExecutableInstallation._pubspecLockEntry (package:cli_launcher/cli_launcher.dart:138:34)
#6      ExecutableInstallation._pubspecLockEntry (package:cli_launcher/cli_launcher.dart)
#7      ExecutableInstallation._loadIsFromPath (package:cli_launcher/cli_launcher.dart:169:19)
#8      ExecutableInstallation.isFromPath (package:cli_launcher/cli_launcher.dart:127:42)
#9      launchExecutable (package:cli_launcher/cli_launcher.dart:396:29)

spydon · 2025-01-22T11:59:14Z

@petrnymsa hmm, I can't reproduce this.
Could you create a minimal reproducible example in a separate repo?

petrnymsa · 2025-01-22T12:02:52Z

@petrnymsa hmm, I can't reproduce this. Could you create a minimal reproducible example in a separate repo?

Hmm me neither in sandbox project. We are using FVM as well, maybe something is off there

spydon · 2025-01-22T12:09:48Z

@petrnymsa what about if you add FVM to the MRE, is it reproducible then?

petrnymsa · 2025-01-22T12:28:09Z

@spydon unfortunately no, will let you know If I find a way to reproduce it

petrnymsa · 2025-01-22T12:59:43Z

@spydon Got it. During migration, we forgot melos inside dev_dependcies in app/pubspec.yaml. After removal, everything works.

markbeij · 2025-01-22T13:20:55Z

I got 7.0.0-dev.4 working. What work needs to be done before going stable?

spydon · 2025-01-22T13:39:56Z

I got 7.0.0-dev.4 working. What work needs to be done before going stable?

The shared dependencies feature needs some love, because if you set dependencies that don't work in there you won't be able to run it again to fix it, so it needs some kind of rollback feature.

Other than that everything looks good so far!

markbeij · 2025-01-23T13:52:33Z

Thanks Lukes. Is there already a ticket for the required love? From: Lukas Klingsbo ***@***.***> Date: Wednesday, 22 January 2025 at 09:40 To: invertase/melos ***@***.***> Cc: Mark Beij ***@***.***>, Manual ***@***.***> Subject: Re: [invertase/melos] request: pub workspaces (Issue #747) I got 7.0.0-dev.4<https://pub.dev/packages/melos/versions/7.0.0-dev.4> working. What work needs to be done before going stable? The shared dependencies feature needs some love, because if you set dependencies that don't work in there you won't be able to run it again to fix it, so it needs some kind of rollback feature. Other than that everything looks good so far! — Reply to this email directly, view it on GitHub<#747 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE24VANDEZUV5M46EO6K6ID2L6NUHAVCNFSM6AAAAABMFFNAPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBXGI3TQOBZGU>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

spydon · 2025-01-23T14:21:19Z

@markbeij created one here now: #847

spydon · 2025-01-25T14:32:24Z

7.0.0-dev.5 is now out and it has the shared dependencies rollback feature included.

lishaduck added the feature request label Aug 7, 2024

spydon added the Type: Accepted label Aug 7, 2024

spydon self-assigned this Sep 19, 2024

spydon mentioned this issue Dec 4, 2024

request: Optimize melos bootstrap by Skipping Unchanged Packages Based on Git Commit Hashes #797

Closed

1 task

spydon mentioned this issue Dec 12, 2024

fix: melos does not work in combination with Dart workspace #801

Closed

1 task

spydon pinned this issue Dec 12, 2024

spydon mentioned this issue Jan 6, 2025

feat!: Migrate to use the Pub workspaces feature #816

Merged

7 tasks

lishaduck mentioned this issue Jan 6, 2025

request: support --no-pub #815

Open

1 task

spydon closed this as completed in #816 Jan 7, 2025

spydon closed this as completed in 3602d90 Jan 7, 2025

spydon mentioned this issue Jan 9, 2025

feat!: Remove melos.yaml in favor of the root pubspec.yaml #832

Merged

7 tasks

spydon reopened this Jan 10, 2025

request: pub workspaces #747

request: pub workspaces #747

Comments

lishaduck commented Aug 7, 2024 • edited Loading

Is there an existing feature request for this?

Command

Description

Reasoning

Additional context and comments

spydon commented Aug 7, 2024

spydon commented Sep 10, 2024

spydon commented Sep 18, 2024

Leptopoda commented Nov 27, 2024

spydon commented Nov 27, 2024

xaethos commented Dec 4, 2024

spydon commented Dec 4, 2024

Leptopoda commented Dec 4, 2024

SAGARSURI commented Dec 12, 2024

SAGARSURI commented Dec 12, 2024 • edited Loading

spydon commented Dec 12, 2024

lishaduck commented Dec 24, 2024

SalihCanBinboga commented Jan 3, 2025

lishaduck commented Jan 3, 2025

SalihCanBinboga commented Jan 3, 2025

lishaduck commented Jan 3, 2025

SalihCanBinboga commented Jan 3, 2025 • edited Loading

lishaduck commented Jan 3, 2025

SalihCanBinboga commented Jan 3, 2025

lishaduck commented Jan 3, 2025

SalihCanBinboga commented Jan 3, 2025

lishaduck commented Jan 3, 2025

spydon commented Jan 3, 2025

SalihCanBinboga commented Jan 3, 2025 • edited Loading

spydon commented Jan 6, 2025 • edited Loading

spydon commented Jan 7, 2025

spydon commented Jan 7, 2025 • edited Loading

spydon commented Jan 7, 2025

lishaduck commented Jan 7, 2025

spydon commented Jan 10, 2025

lishaduck commented Jan 10, 2025

spydon commented Jan 10, 2025

petrnymsa commented Jan 14, 2025 • edited Loading

spydon commented Jan 14, 2025

petrnymsa commented Jan 22, 2025

spydon commented Jan 22, 2025

petrnymsa commented Jan 22, 2025

spydon commented Jan 22, 2025

petrnymsa commented Jan 22, 2025

petrnymsa commented Jan 22, 2025

markbeij commented Jan 22, 2025

spydon commented Jan 22, 2025

markbeij commented Jan 23, 2025 via email

spydon commented Jan 23, 2025 • edited Loading

spydon commented Jan 25, 2025

lishaduck commented Aug 7, 2024 •

edited

Loading

SAGARSURI commented Dec 12, 2024 •

edited

Loading

SalihCanBinboga commented Jan 3, 2025 •

edited

Loading

SalihCanBinboga commented Jan 3, 2025 •

edited

Loading

spydon commented Jan 6, 2025 •

edited

Loading

spydon commented Jan 7, 2025 •

edited

Loading

petrnymsa commented Jan 14, 2025 •

edited

Loading

spydon commented Jan 23, 2025 •

edited

Loading