[RFC] Time to maintain a fork of Next.js?

[flybayer]

Problem

Our custom compiler approach has done fairly well thus far, but we are reaching its limits. Currently we compile a blitz codebase into a next.js code base (inside .blitz/build) and then run next from .blitz/build. This allows us to override and customize most things about next.js.

But honestly this is a bit hacky and introduces a whole slew of problems that we have to try and work around. For one, runtime file paths are wrong because they are inside .blitz/build. We currently are filtering and manipulating next.js logs so the displayed path is correct. But it's still incorrect in browser errors. Also this breaks node.js debuggers — breakpoints must instead be added to code inside .blitz/build. This also sets up a double file-watcher process where we watch for changes in root, copy changes to .blitz/build and then next.js watches .blitz/build and compiles that. We've had various issues with this that still aren't resolved.

Solution

The goal is to remove our custom compiler and have a more native approach.

Needs

We can do a lot through babel plugins, but the following are things we need outside of babel:

• Essential: Ability to turn queries/mutations files into API routes in the server build
• Essential: Multiple pages & api folders that get merged together
• Wanted: A layer in front of Next.js that allows us to do any customization. This allows us to have blitz.config.js instead of next.config.js and add features like supporting typescript for that config file.

Approaches

There's essentially two approaches we can take:

1. Change blitz to be only an add-on to next.js
   • This would require reverting a few blitz changes, chief of which would be changing blitz.config.js to next.config.js with a withBlitz wrapper inside
   • This requires collaboration and cooperation with next.js to add some extra APIs that we can use for adding the RPC layer (in theory this sounds possible, but whether we could actually reach an agreement with the Next.js team on what we need is unknown at this time)
   • This prevents a whole class of customizations/overrides that we might want to do since we can't change any next.js internals. For example one thing we couldn't do is add typescript support for the main config file (which is something a lot of us want).
   • This locks us to the very conservative nature of Next.js. We're already limited by building on top of next.js (compared to Redwood, for example), and this would only restrict us even more.
2. Fork Next.js, make changes to core, and keep up to date with the main next.js repo
   • This adds more complexity for us to maintain, but essentially removes all restrictions because we can easily change any next.js internals to improve fullstack DX. If we are smart about our changes, there will be minimal merge conflicts with upstream changes
   • This frees us from the very conservative nature of Next.js and allows us to innovate more rapidly on features. It could be some extra blitz features get back-ported to next.js itself (like first class Layout support, for example)
   • Another benefit is that some changes are WAY simpler to make in core next.js instead of a babel plugin on top.

I really think we should go with the fork approach. Especially because we are way more progressive than Next.js in adding new features and optimizing DX for a fullstack framework. Maintaining a fork opens up a lot of opportunities for us. Thus far I think more people are very happy with the improvements & changes we make on top of Next.js than folks who would rather we stay closer to native next.js.

What are your thoughts on this? (please include in your response if you have used Blitz or not)

[stevenpetryk]

A fork makes sense to me. But maybe instead of forking with the intention of adding Blitz-specific functionality to Next, the fork should be done with the intention of making it pluggable enough to build Blitz as a plug-in. That could lead to the fork getting merged into Next and no longer needing to maintain it down the road.

[MrLeebo]

I think that once the idea gets out that blitz could be a next.js addon, somebody will try to build that whether we participate or not.

[hylerrix]

Now with the Blitz Pivot RFC, Blitz is going to be an addon to not just Next.js but also Remix, SvelteKit, Nuxt, etc.

It's so exciting now!

[webdeb]

Haven't had it on the radar for some time, but yeah adoption by embracing the ecosystem is the only way to go.

[adamsoffer]

My gut says stick with approach #1 and as @stevenpetryk suggests build Blitz as a plug-in (the next team is working on a plugin system). One of the things that draws me to blitz is that it's built directly on top of Next.js and shares the same API surface.

[adamsoffer]

Perhaps you can fork as a stop gap until this plugin system is released.

[flybayer]

Fyi, they've abandoned the plugin system they originally implemented (it's being totally removed). And instead they are working on a much more conservative approach where they add one hook at a time. And these new hooks don't let you run arbitrary code.

[MrLeebo]

Since next.js would like to see blitz as an add-on, surely they would come up with an alternative for us if we went that route, right?

[flybayer]

The premise is that yes they want to work with us, but ultimately if there is a conflict of interest, they will of course choose in their interest.

[leerob]

Hi - one thing. For clarity, the plugin system is not abandoned. Next.js has the ability to add plugins (e.g. mdx) and the RFC mentioned is for improving this system and providing more hooks for folks.

[AndriusBartulis]

I think the question here is about the goal of Blitz. Which approach will help Blitz achieve its goals according to the set-out philosophy?

For instance, let's look at: "[Blitz] Includes everything from the database to your frontend all inside a single app. Only one thing to develop. Only one thing to deploy." and also "The out-of-the-box experience guides you on a path perfect for most applications. But when you need to go off the beaten path, you are totally free to do so."

Does making Blitz a Next.js plugin move it closer to those goals, or further away? What about in a year's time when Next.js becomes even more opinionated or changes direction?

I feel that if the goal of Blitz long term is to be the full-stack convention-over-configuration monolithic framework for the JS community (in the spirit of Ruby on Rails and Laravel), then it makes sense to fork Next, and in the far future, potentially no longer depend on it directly at all.

To create a truly amazing dev experience might require making quite significant modifications to how Next works, meaning that depending on Next would probably end up limiting Blitz not only today but in the future, as Blitz gets more adoption and can be more ambitious with its feature set.

True innovation and "generational leap-frogging" improvements require the freedom to move quickly, without unnecessary constraints. Depending on Next (which would be the case if Blitz was a plugin) is the quickest way to get rid of that freedom and add quite a high level of constraints from a project with a potentially different philosophy and long-term goals.

Hence I think that a fork of Next is a better choice, and I would go further and say that in my opinion maintaining compatibility with Next will become less and less relevant as Blitz develops its own ecosystem.

[hylerrix]

Nice thought. My wish to Blitz is both the full-stack convention-over-configuration monolithic framework for the JS community and toolkits for Next.js users and optionally Remix, SvelteKit, Nuxt, etc.. This two-part is not in conflict with each other and all can reach the most important goals: fullstack DX.

Monolith can totally be made of blocks (toolkit), and these blocks you can directly import to your other projects. Blitz's pivot to toolkit RFC shows us the better path we can follow.

[MrLeebo]

Any resources on how we would minimize merge conflicts as a fork? I'm worried this is downplaying the amount of effort involved. Since we'd be accessing internals, every next.js commit could be a breaking change, no?

[AndriusBartulis]

Good point... I can see how eventually it would become harder and harder to keep merging next origin into blitz-next. I do think at the end of the day the question is really on whether Blitz is best served by being built on top of next or not. If yes, then plug-in route makes sense. If not, then fork to start with, but eventually the fork might go away too and next features blitz needs get moved into blitz itself.

[flybayer]

How you do this is make our changes on the "main" next.js upstream branch which is canary. And then we have an upstream branch that is exactly upstream canary. On each new next.js release, we merge our upstream branch into our canary branch. As long as we haven't changed the same line of code as an upstream change, it will merge cleanly. And I think this should be the case for 95%+ of upstream changes.

[Skn0tt]

Judging by the changelog of patch releases of Next.js (let alone feature releases), 5% of upstream changes is still a ton of work.

[flybayer]

Ok maybe it's 99%, I don't know :) 🤷

As I said below, it's something we can try and if after a month or two it's too much work, then we can change course.

[ThePaulMcBride]

Without knowing much about the internals of Blitz, I'd be in favour of sticking as closely to Next as possible. It is battle tested and has a company backing it and paying to develop it. Blitz feels less like a risk because of its ties to Next.

I feel like moving away from Next would massively increase the work you and other maintainers would need to do to keep feature parity with Next. The fact the Blitz is a layer on top of Next is the reason I picked it over Redwood etc.

[flybayer]

I'd be in favour of sticking as closely to Next as possible.

Yes, for sure, what's on the table is not whether to depart from being next.js compatible or not. The current plan is to remain compatible with Next.js.

What's on the table is how to best implement our current features and changes that we have on top of Next.js. Going the fork route doesn't mean we will depart from next.js compatibility. Even with the plugin/addon route, we have the ability to break compatibility for some APIs if we wanted.

It is battle tested and has a company backing it and paying to develop it. Blitz feels less like a risk because of its ties to Next.

Again, we'll remain tied to next with either approach. And either approach will be more robust that what we have right now.

[hasparus]

(1) limits the scope of the project, but (2) is... ambitious, even brave.
I like @stevenpetryk's pluggable fork idea. You'd probably want to keep the change in Next small to make keeping up with a VC-funded tech company and 1.5k contributors less of a pain.

(I have used Blitz.)

[paulm17]

I'm just going to say option 2 at this point. A lot of other answers from the other comments, reflect what I want to say. 😃

[MrLeebo]

Blitz isn't just a codebase, it's also a community. Hard to imagine that community continuing to exist after approach # 1. I imagine it would get absorbed into the next.js community.

[Skn0tt]

The idea of Blitz as an Add-On to Next.js seems to have been around for quite a while now. I remember Tim Neutkens mentioning it to me as early as May of last year, when Blitz was merely Next + RPC layer.
I could see why: the RPC layer is the single most "innovative" thing that Blitz entails, it's our main selling point. People love it, and it's the main thing that makes Blitz so well-suited for fullstack applications, as opposed to content-centric Next.js.

Apart from the RPC Layer, most of the other things that Blitz currently does could easily be added to Next.js: Recipes, Auth (see next-auth), Code Generation (see Yeoman).

Once we think about future Blitz features, Next.js starts looking old: Websockets, React Native support, ...
Interestingly, none of the features we have planned are in-line with Vercel's product vision. For the foreseeable future, they won't be looking at Back-End / Fullstack / Application territories, but championing the frontend instead.
This is where Blitz / Next.js diverge. Blitz.js tries to do all the things that Next.js never will, while building up on the (stellar) work that Vercel does on Next.js.
We're standing on giant shoulders here, profiting of the immense, vc-funded efforts that Vercel puts into Next.js. This should not be discounted!

Before deciding to fork Next.js, we need to lay out the following things:

• What are we hoping to get out of forking? Can those needs be addressed otherwise?
  • @flybayer mentioned it'd "remove all restrictions" - we should come up with an exhaustive list of ways in which Next.js restricts us now and in the near future
  • "Having to compile to the filesystem" is the main one that comes to my mind
  • What are the others?
• How can we ensure minimal-invasive changes from upstream, enabling easy adoption of their work?

We should be very careful with this decision, to not end as in https://xkcd.com/927/.

xkcd.com/927

[Skn0tt]

Thanks for the list! I added some thoughts, trying to condense them down to what we want from our fork:

• Inability to customize anything in next.js core
  • Not a restriction until we have a specific thing we want to change
• Inability to inject API routes into the build
  • this is the compiler painpoint. what happened to the webpack-injection thing?
  • if we forked, I imagine this would be well-implementable using a minimally invasive hook in the API Route logic (I've already worked on that part of Next.js, yay ^^)
• Inability to have multiple pages/api folders
  • same as above
• Inability to use blitz.config.js
  • this would be if we go the Add-On route, right? we already have blitz.config.js
• Inability to use typescript for blitz.config.js or next.config.js
  • is this really that big of a problem?
• Inability to run arbitrary code at various parts of the application lifecycle
  • good one! again, I think we could do this "minimally invasive"
• Inability to customize in-browser errors
  • good one!
• Must use babel plugins to implement a lot of things like http middleware, custom Page functionality like Page.authenticate, first-class Layout support etc. Some of these may stay as babel plugins, but some would be way simpler and easier to maintain as changes to core.
  • what would we gain from first-class Layout support?
• Decrease in blitz contributor morale because of having to always work around next.js stuff via babel, etc.
  • good one!

At first glance, it seems like we want:

1. hooks into various parts of the Next lifecycle
2. hooks into API Route Logic

I just want to make sure we're absolutely sure what the things are that we're getting us into. While adding hooks sounds fine, things like "changing central behaviour in our fork" seems like a suicide mission, and we don't want that 😅

[flybayer]

this would be if we go the Add-On route, right? we already have blitz.config.js

Yes. My comments are assuming that we have to choose another approach than our current compiler approach.

• inability to use typescript for blitz.config.js or next.config.js
  • is this really that big of a problem?

This is a much bigger deal because we use blitz.config.js for middleware config which you want to import typescript code for (this comes up all the time). However we could find another approach like a separate TS config file at the root of the project. (but not ideal)

what would we gain from first-class Layout support?

Better DX

And yes, I think it's mainly adding hooks to various parts of the code.

[flybayer]

I think the big summary is Forking next.js allows us to go one step further in polishing our DX. We can find other ways to do a lot of things, but we'll never be able to really have the polish that first class changes provide.

[flybayer]

• Inability to inject API routes into the build
  • this is the compiler painpoint. what happened to the webpack-injection thing?

This is what I had mentioned before: https://github.com/sysgears/webpack-virtual-modules I haven't tried it, but I don't have much hope for it. I'm afraid, that even if it works now, it may stop working at anytime because of next.js internal webpack changes.

Edit: looks like that only works for files inside node_modules, regular source files

[leerob]

Hi - one thing to clarify 👋

Interestingly, none of the features we have planned are in-line with Vercel's product vision. For the foreseeable future, they won't be looking at Back-End / Fullstack / Application territories, but championing the frontend instead.
This is where Blitz / Next.js diverge. Blitz.js tries to do all the things that Next.js never will, while building up on the (stellar) work that Vercel does on Next.js.

This is not exactly true. You are correct that Vercel strives to be the best platform for front-end teams. This includes serverless functions as well, but it's easier to just say front-end 😄

Realistically, as a company, we can't be the best platform for front-end and back-end (yet). We are focusing on our area of expertise. Instead, we want to be a first-class integrated solution with other back-ends. For example, this integration with Supabase.

If you have questions about Vercel's vision, please message me!

[JuanM04]

I see Next.js as a quite good frontend framework; it works perfectly for making fast websites with React and even a bit more (with getStaticProps, API routes, etc.). If you try to to more (like a dashboard with multiple models or a complex systems), you'll be a long time developing it.

Then, there is Blitz.js, a incredible Full Stack framework. It's designed to quickly make a scalable app. It's overkill for a simple website, but perfect for a monolith with lots of calls to a db.

I think Blitz.js should stop being "a framework based on Next.js" and be "a fullstack framework (with Next.js to handle the frontend)". Next.js would be a tool instead of the foundation of Blitz (yet a very important tool)

[Skn0tt]

Being compatible to Next.js is part of Blitz' appeal. With Next.js having almost evolved to an industry standard, Blitz can stand on these giant shoulders and present itself as the long-known, but immensely more powerful sibling of Next.js.

Next.js is responsible for 90% of what makes Blitz so great - because it's based on it. "Using Next.js as a tool" seems like an over-estimation of what Blitz does.

[flybayer]

I could see in 5 years maybe moving more in this direction if needed to solve problems Blitz devs want. But for the foreseeable future sticking close to Next.js greatly helps adoption.

[JuanM04]

Yeah, both are good points. Nonetheless, I look forward forking Next.js

[ntgussoni]

Interesting discussion!

Forking next will inevitably mean that we'll probably diverge too much at some point.

On the other hand we are only adding hacks to make things work around next which could eventually become a roadblock.

I agree with @Skn0tt that we don't have to fix a problem we don't yet have so perhaps we can try to figure out what we really need to address and see how much of a footprint those changes might make on the short run. Perhaps little clever changes can boost blitz a lot and that doesn't mean having to solve conflicts with the upstream every day.

I guess that we all agree that going wild on the fork is just a death sentence because nobody will be able to keep up with their changes.

For the forseable future they'll probably focus on frontend DX but vercel is not stupid, they want the whole stack... So they'll eventually pick a winner from all the full stack framework out there or make their own thing.

Im pro forking next but being clever with the changes made to it.

[flybayer]

A key point that's being brought to light here is that there's a fundamental tension between Next.js and Blitz.

Next.js priorities (best I can tell from the outside):

• Runtime performance for content sites which drives leanness, tiny bundles, less features, etc
• Preventing people from shooting themselves in the foot. In other words, give people a dull knife instead of a sharp one so they don't cut themselves. This is why they aren't moving forward with that general plugin RFC.

Blitz priorities:

• Fullstack DX for web apps (vs content sites)
• Power features for advanced applications
• Give people sharp knives (same as Rails: https://rubyonrails.org/doctrine/#provide-sharp-knives)

So the challenge for us is how to navigate this? We can stick very close to next.js and sacrifice blitz priorities. Or we can fully embrace what Blitz wants which leads to some divergence from next. This is a spectrum that we can navigate back and forth in over time. But I feel that over the long term, Blitz is going to want to diverge from next at least a certain amount.

[brodycj]

Hey I was wondering if anyone has brought the list of the known inabilities to the attention of the Next.js project?

[flybayer]

@brodybits I had a meeting with Guillermo Rauch yesterday where we briefly talked about this. And I'm following up with him again. I'll report back.

[adamsoffer]

I brought this RFC to the attention one of the Next.js authors / maintainers (Joe) and I asked if he had any thoughts. He said the community should make the decision it needs to make but he expressed that Next.js would be more than happy to upstream any bug fixes (the few referenced here about stack traces) and collaborate on adding hooks where necessary for Blitz. He said that each hook Blitz needs would just need to have its own public Next.js RFC and go through due process before being added but that the new plugin system is intended exactly for this: identify hooks needed by the ecosystem and add them.

[flybayer]

Thank you @adamsoffer!

He said that each hook Blitz needs would just need to have its own public Next.js RFC and go through due process before being added

The tension I'm describing will be highlighted in that process. What the majority of the next.js community wants is likely different than what the blitz community wants. We also want to move much faster than Next.js. I think the most realistic option for us to go with a fork but also seek to get our changes adopted upstream. Otherwise we'll be sitting around twiddling our thumbs waiting for Next.js RFCs to circulate :)

[malkomalko]

Would it be worthwhile to take a step back and look at this from a roadmap/api perspective? Meaning, where does Blitz want to be at 1.0 and 2.0. What are the obstacles and motivating drivers that help us choose between options 1 and 2?

[webdeb]

I already asked how to migrate next -> blitz.

And actually I would really like just to use the non-api approach in a next app. What if blitz could be just a directory which would represent the backend? I would love to import functions directly from there and use them with useMutation/useQuery of blitz, which would then compile to the actual api communication layer.

I really like the idea of blitz, but I am unsure, if the community needs a whole new framework, as there are so many next apps out there which could benefit from the non-api technology on top of existing next.js apps.

For example it would be really cool, if we could just have a directory structure like:

pages/
  listing/
     index.tsx
     blitz/
        queries.ts
        mutations.ts

And compiling to:

api/
  listing/
     (names of the functions inside query & mutations) like
     getOne.js
     createNew.js

So, IMHO, I think that using blitz as an extension of next would make blitz smaller, but also easier to maintain and easier to adopt.

[flybayer]

Thanks for sharing your thoughts!

I am unsure, if the community needs a whole new framework

Judging by the success of Blitz so far and the passionate love which folks express for it (😅), people do want another framework. Blitz provides much more than the zero-api layer. It's the whole, integrated package that people like.

If someone doesn't like the whole integrate package, then they can certainly build on next.js itself and patch together all the libraries they need. For zero-api in next, you can try https://github.com/trpc/trpc although it's not as seamless as what blitz has.

[webdeb]

Judging by the success of Blitz so far and the passionate love which folks express for it (😅), people do want another framework. Blitz provides much more than the zero-api layer. It's the whole, integrated package that people like.

I have to admit, that I haven't tried blitz so far, but I am planning to do so for a new project. However, what attracted me the most, is actually the zero-api layer. And I would guess, that this is the major building block, which makes it so productive.

I will try it sometime in the future, to see other benefits, which are definitely there like auth and integration with forms etc. But I think that even these could be still part of blitz, even if its an addon. For example imagine we have blitz as dependency in a next project. Which is structured by some conventions, like pages && blitz/pages.

It could still be used to install whole domains like blitz install blog and you would get a scaffold of a blog:

blitz/pages/blog/
   views/{post,list,tag,tags,etc}.js
   api/{getPost,listPosts,getPostsByTag,getTags}.js

Maybe the main idea here is to decouple a part of blitz (at least the zero-api layer) so it could be adopted on existing next applications, and I am sure for new projects people would choose blitz right away.

[flybayer]

Update

I've officially forked next.js to https://github.com/blitz-js/next.js. And it's initial integration into our monorepo is happening in #2079

I had followed up with Guillermo. He said they are looking into possibility of adding import('queries/**') which we might could use for api route generation (but it's unclear if that would actually happen or when). So I told him we're moving forward with a fork for the sake of progress with an eye to remove the fork at some point.

But realistically, I think we'll have a fork forever. But we'll definitely try to merge features upstream if they want them.

And I'll reiterate that we absolutely plan to maintain nextjs feature parity and deployment compatibility.

[JuanM04]

The best that could happen is that we end up merging some features of our fork into the main Next.js repo 😄

[paulm17]

Congrats on the fork. If there is one thing that I have leant since writing software since 2000 (and a bit before). Is that control is paramount. It would be a best case scenario that you control as many pieces as possible rather than hoping things get merged into next.

As with the more control you have, the easier to execute on the vision becomes.

Anyway, excited to see what the next 12 months bring! 🚀🚀🚀