Virtual Monorepo from multiple smaller repos

[wizpresso-steve-cy-fan]

We have a bunch of remote "microrepos" where we want to combine them back into a monorepo. We can easily import all those microrepos with git subtree but our team lead suggested us not to, due to inherent synchronization issue.

Right now Josh seems only working on one way, by decomposing different Git history into microrepos for one central monorepo, but the converse is not true. Can we use Josh in a way such that we can stitch multiple independent remote microrepos back together as one monorepo, without having to use git subtree or git submodules? I.e. Josh will also recursively solve the git remote for you in the monorepo, by passing your authentication information to the upstream remote git repo provider.

We need this because we have over 100 microrepos and we want to centralize the software warehouse toolings, CI/CD pipelines and formatters, yet we don't want to compromise our security with monorepo because it is giving out way too much information. We specifically slashed out Git submodules due to its finicky behavior regarding code branching and synchronization which git subtree method also suffers. Then we come across Josh and found out it is pretty good for our work case if it can work in reverse.

[LMG]

Hi @wizpresso-steve-cy-fan , there are probably multiple ways to go about that with josh, but here is what I think is simplest:

• Clone locally the repository you want to integrate in the monorepo
• Add the josh-served monorepo as a new remote, specifying a filter that tells where your imported repo should be placed: git remote add josh https://your-josh-instance.com/your/monorepo.git:/a/new/folder/for/this/project.git. Note that the folder doesn't need to exist yet.
• Push to this folder with git push josh HEAD:a-branch -o merge with a-branch probably being master I suppose.

After writing this I notice this is documented here: https://josh-project.github.io/josh/guide/importing.html

[wizpresso-steve-cy-fan]

No, we actually considered that before, but it still doesn't solve the code security problem because you are essentially creating a bunch of microrepos from a monorepo, not creating a monorepo from a bunch of microrepos.

[aschrijver]

I have a similar or related need, and it is a need that may exist anywhere monorepo projects also have a plugin architecture that allows 3rd-parties (via a SDK or otherwise) to develop extensions. These plugin repo's are created everywhere, if lucky only on Github, but often in other code forges too (Gitlab, Sourcehut, Codeberg, or self-hosted ones, etc). Then the need is there to document where they live, e.g. in a curated awesome README. Sometimes projects create an additional organization account, just for contrib. Like Yunohost vs. Yunohost-Apps.

My need is as follows:

• A monorepo with multiple Josh workspaces for core maintainers.
• An additional workspace referencing external project(s) for plugin developers.

My goal is that the plugin developer effectively works in the central monorepo, always a fix + PR away from contributing when they find a bug. In my case their dev process starts by cloning the monorepo, and from its rootdir invoke a CLI command that sets up their repo via SDK boilerplate, and they can start coding right away.