Support for monorepo in Gleam CLI

[ghivert]

Why monorepo in the first place?

Many Gleam users nowadays seems to be building their project as a monorepo. And it makes sense, because monorepo for Gleam seems like a natural thing. Because we need to share types between frontend and backend in the first place, but also because sometimes we need to vendor some dependencies for example. Path dependency notably helps on that subject, and it's regularly the solution proposed when newcomers experiment with code sharing.
Moreover, it's often desired to be able to bundle everything in one single repository, because it easy maintenance, it easy support for developers (only one repo to pull, one repo to keep in sync, etc.), and it easy git management.

After working on few projects on Gleam, most of them running in production daily (and as such, with some experience on the subject), I think it's the correct time to start discussion about the inclusion of some monorepo tooling in the Gleam CLI, and where to put boundaries between what should be (hypothetically) bundled in the Gleam CLI, and what should be handled from the outside world.

Some personal opinions

I have some opinions about monorepo in general & be happy to discuss about it. I think it's a nice place to start to already propose some key things to take in consideration, while of course, everything is subject to modification. 🙂

• Environment variables and stuff should be handled outside of tooling (dotenv or mise for example). It's often easier to manage it from the outside, and while it can be bundled in the tooling, a lot of toolings are already managing them in a great way.
• Commands should be runnable from anywhere in the repo source tree to save some time. You maybe need to update a dependency in a specific package, or trigger a build for one specific package in your CI. Avoiding having to move in the subfolder would be great.
• Compilers and runtimes should be shared across the monorepo & the projects to improve consistency. In TS monorepo, you can end up with multiple versions of TS across the codebase. I think it's the worst of both worlds. In the same codebase, you have some features you can use, and some features you can't. Ensuring the upgrades are widely shared across all packages removes the pain to manage the versions of runtimes & compilers.
• New packages should be easy to create, and easy to share. Removing the pain of referencing a package increase the ability to properly split by concerns. Premature splitting is bad, but not splitting at all is bad too, because it can increase compile duration, and can lead to coupled code, where everything could be uncoupled properly. Plus, allowing to simply split code by concerns does not prevent anyone to just use a simple "common" package and put everything inside.
• Packages should be simple to bundle & share individually on Hex. In free software, when you want to reduce human maintenance cost and properly split your project in multiple packages (like sketch, redraw or glitr are doing), having a way to simply manage & share them is a huge gain.

An idea to start exploration

Like many monorepo tooling, I think it would make sense to use a configuration file, at the root of the project. Something like gleam.toml, maybe with a different name if we prefer to keep gleam.toml for single projects. I like the idea to use gleam.toml though, because it allows to start the project as a simple experimentation, and then switch to a monorepo when it's needed. (For information, that's what Yarn or NPM are doing, by relying on package.json as configuration file.)

The root gleam.toml would define the Gleam version provided (but does not take care of installation or anything, let's delegate this to mise, asdf, or other package managers), and every package would use it. In case an inconsistency is found in a package or dependency, warn immediately.

I think we could define a monorepo at root with a slightly different gleam.toml. We could keep the existing fields, and add a field workspace = true. Once activated, the folder and every children folders will start to behave as monorepo. Taking inspiration from others monorepo management tools, I think it makes sense to consider the root folder acts as a "Workspace". A workspace is a collection of packages and services, that could compile, lint, execute scripts and potentially run for services.
A workspace could simply define projects, whether as a key-value pair dict, or an array of global regex to auto-detect Gleam projects. For every referenced service, Gleam build tool will consider it as workspace-wide available package.

Once the build tool is aware of the different packages in the monorepo, we could introduce a new command, workspace, to run commands in a specific sub-project. gleam workspace package add gleam_http would add gleam_http directly in the package/gleam.toml.

Having the support for scripts is important for me, because as Gleam users, we have to rely on codegen, but also on database migrations or others stuff. I think it would be a gain to have a centralized place to define everything. For example, having an interfaces package defining JSON Typedef, with a script indicating how to build it would be a way to gain time. I think a key-value dict would be enough for this simple use case as a start.
It would also be nice to let sub-projects define scripts, and as such, that would let gleam workspace command run script in sub-projects too.

How the monorepo dependencies should be handled

Once the monorepo structure is made, the Gleam CLI would centralized all build artifacts, whether they're Erlang or JS at the root of the monorepo, and all dependencies would be centralized at the root of the monorepo too. Every time we need to use a package in the monorepo as a dependency, instead of relying on path dependency, we could define it as package = {workspace = "package" }, and the build would instantly use the artifacts already compiled in the global cache. That would speed up compile time for every services, and would solve inconsistencies.

Every time a package would be modified, the build tool could also rebuild the dependent packages.

An example of what gleam.toml could look like

# Define the name of the project.
name = "project"
# Define the project as monorepo. The flag is necessary to prevent the package to be published.
# We could avoid setting the flag, and only detect if the config has [workspace] defined,
# but I think it's nice to have both to force user to be explicit.
workspace = true

# Keep the gleam flag as whole project indicator.
gleam = "1.6.1"

# Define a workspace, that could be a key-value dict, to define sub-projects.
[workspace]
package_1 = "packages/package_1"
package_2 = "packages/package_foo"
service_1 = "services/frontend"
service_2 = "services/backend"

# Or define the workspace a list of global regex, that will automatically detect sub-projects.
workspace = [
  "packages/*",
  "services/*",
]

# Define some scripts to be run from anywhere in the project.
# Could be helpful to run JSON Typedef generation for example.
[scripts]
build = "bla"

# Keep the repository information as is.
[repository]
type = "github"
user = "ghivert"
repo = "redraw"
path = "redraw"

# Keep the dependencies for monorepo-wide dependencies
[dependencies]
gleam_javascript = ">= 0.11.0 and < 1.0.0"
gleam_stdlib = ">= 0.34.0 and < 2.0.0"

[dev-dependencies]
gleeunit = ">= 1.0.0 and < 2.0.0"

An idea for extensions

I think it would also be worth it to let the Gleam CLI to accept plugins to run. That would help to support multiple targets, for example. If in a single monorepo, we support both Erlang & JavaScript and we got some TypeScript compilation, it would be nice to have a plugin managing the TypeScript compilation directly. Instead of having to manage every flavour of JavaScript (TypeScript, PureScript, ClojureScript, anything really), we could simply allow the Gleam build tool to call a plugin with some lifecycle hooks, and let the plugin work on its own.

[richard-viney]

Thoughts on making inheritance from the workspace gleam.toml more explicit in a package's gleam.toml?

Top-level workspacegleam.toml:

[workspace]
packages = [
  "package_1",
  "package_2"
]

[workspace.package]
gleam = "1.6.0"
license = "MIT"
authors = ["Author #1", "Author #2"]

[workspace.package.repository]
type = "github"
user = "ghivert"
repo = "redraw"

[workspace.dependencies]
gleam_stdlib = ">= 0.43.0 and < 2.0.0"

[workspace.dev-dependencies]
gleeunit = ">= 1.2.0 and < 2.0.0"

Then in a package's gleam.toml:

gleam.workspace = true
authors.workspace = true
license.workspace = true
repository = { workspace = true, path = "path/to/this/package" }

[dependencies]
gleam_stdlib = { workspace = true }
glint = "1.1.0"

[dev-dependencies]
gleeunit = { workspace = true }

This is the approach that Cargo takes.

[ghivert]

Isn't it a bit verbose if we have to reference every package in the main gleam.toml?

[richard-viney]

Are you referring to this part?

[workspace]
packages = [
  "packages/package_1",
  "packages/package_2"
]

If so then yes I think wildcards should be allowed there, e.g.:

[workspace]
packages = [
  "packages/*",
]

[ghivert]

My bad, I was tired yesterday. Seems like a great way to manage version and runtimes. 🙂

[lpil]

Gleam always opts for explicitness and clarify over brevity, and it's 1 line of configuration per package (which likely contains many thousands of lines) so I don't find this globbing idea very compelling presently.

[jrstrunk]

I don't have any opinions on how this should be done, but I do want to pitch in and say that some sort of monorepo management tooling would be a really nice QoL improvement for the ecosystem!

[lpil]

What QoL improvements would you expect from monorepo tooling?