zig: commonify & bootstrap #331011

RossComputerGuy · 2024-07-30T04:39:12Z

Description of changes

This is the first step towards my goal of being able to bootstrap LLVM in Nix with Zig. A couple things this PR does is commonify Zig to make handling future updates simpler, I've been meaning to do this for a while. This PR also implements the bootstrap derivations for Zig.

Things done

Add a 👍 reaction to pull requests you find important.

alyssais · 2024-07-30T07:22:10Z

pkgs/by-name/me/meson/package.nix

-      hash = "sha256-dDOmSRBKl/gs7I3kmLXIyQk3zsOdlaYov72pPSel4+I=";
-    })
-  ];
+    ./linker-support-zig-cc.patch;


Why are we patching this in Nixpkgs at all instead of fixing it properly upstream? My read of the upstream PR was that upstream was responsive and interested, and that there was just the small issue of getting the test suite fixed left to do, and upstream had even provided a hint for how to do that.

Because there's already an upstream PR. This is just a vendored copy after reading the feedback left after #318034 was merged. I don't see anything about a test suite fix and I'm not familiar with how meson works.

I understand that this is just vendoring the patch, but I'm not thrilled that it was added in the first place — the original author of the upstream PR doesn't seem to be continuing with it, which means that we have to keep this patch around (with the implied maintenance burden on Nixpkgs' Meson maintainers) until somebody else picks it up. We can add patches to Nixpkgs when they're urgent, when it's not feasible to upstream them, or when it's doing something Nixpkgs-specific, but allowing patches like this, where none of those things are true, means that, for non-maintainers, there's never any incentive to upstream anything, resulting in packages getting more and more complicated until they collapse under their own weight. For that reason, I think this patch needs to be sorted out upstream, and that we should not keep it around in Nixpkgs in the meantime. The path to getting it upstream is clear, and doesn't sound like it would be a lot of work.

I don't see anything about a test suite fix

mesonbuild/meson#12293 (comment)

Yeah, @tomberek and I talked yesterday about this patch. I only added it so meson projects would at least configure in pkgsZig. I understand the patch isn't exactly stable but there's not really any other way around the error which occurs without this patch. The problem looks like tests in meson fail because Zig doesn't support a flag which means we likely would have to implement the -rpath-link flag.

there's not really any other way around the error which occurs without this patch.

Well, there is, and it's to help this get fixed upstream, which as far as I can see means just extending an existing check to also check for Zig.

Alright, I've made a patch mesonbuild/meson#13493

Great, thanks. For testing on NixOS I just use a nix-shell with python3 and whatever else the test suite needs.

Yeah it looks like I'll need to pull in the CC wrapper for the tests to perform correctly.

Aleksanaa · 2024-07-31T04:09:38Z

The CI is considered not correct on unused arguments in this case, as noted in #331085.

RossComputerGuy · 2024-08-01T04:53:43Z

zig.passthru.stdenv works better now.

kirillrdy · 2024-12-21T21:43:41Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 331011

`x86_64-linux`

❌ 15 packages failed to build:

backlight-auto
blackshades
colorstorm
cyber
dt
findup
linuxwave
pkgsZig.stdenv
zig_0_10
zig_0_10.doc
zig_0_11
zig_0_11.doc
zig_0_9
zig_0_9.doc
zon2nix

✅ 28 packages built:

arocc (aroccPackages.latest)
aroccPackages.latest-unwrapped
aroccStdenv
cargo-lambda
cargo-zigbuild
clipbuzz
emacsPackages.zig-mode
glsl_analyzer
ly
mepo
minizign
ncdu
pkgsArocc.stdenv
poop
river
river.man
rivercarro
superhtml
waylock
wayprompt
zf
zig (zig_0_13)
zig.doc (zig_0_13.doc)
zigStdenv
zig_0_12
zig_0_12.doc
zls
ztags

RossComputerGuy · 2024-12-22T18:06:05Z

I've figured out that Zig 0.9 and 0.10 have the postBuild hook wrong. Will fix tonight and move from staging to master.

kirillrdy · 2024-12-22T22:54:14Z

we can soon remove 0.9 benbusby/colorstorm#15

kirillrdy · 2024-12-22T23:00:37Z

zig 0.10 is also unused after #365116

pkgs/development/compilers/zig/default.nix

kirillrdy · 2025-01-05T11:22:10Z

@RossComputerGuy is it possible to extract "commonify" portion into separate PR that can be merged sooner ?

RossComputerGuy · 2025-01-06T16:34:05Z

Sorry for the late reply, yes it is possible however this PR contains fixes for pkgsZig which we should get in.

kirillrdy · 2025-01-07T13:03:10Z

Sorry for the late reply, yes it is possible however this PR contains fixes for pkgsZig which we should get in.

understood, let me know if there is anything i can do to help to get this merged

RossComputerGuy · 2025-01-07T20:15:58Z

Only reviews are required, I'm low on bandwidth so I'll be slow to update things.

kirillrdy · 2025-01-09T10:49:30Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 331011

`x86_64-linux`

❌ 13 packages failed to build:

backlight-auto
blackshades
colorstorm
cyber
dt
findup
zig_0_10
zig_0_10.doc
zig_0_11
zig_0_11.doc
zig_0_9
zig_0_9.doc
zon2nix

✅ 33 packages built:

arocc
aroccPackages.latest-unwrapped
aroccStdenv
cargo-lambda
cargo-zigbuild
clipbuzz
emacsPackages.zig-mode
ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim
glsl_analyzer
linuxwave
ly
mepo
minizign
ncdu
poop
river
river.man
rivercarro
superhtml
waylock
wayprompt
zf
zig
zig.doc
zigStdenv
zig_0_12
zig_0_12.doc
zls
ztags

m-bdf

Looking at wrapBintoolsWith and wrapCCWith's source code, it seems that ld should go in bintools instead of cc, and clang/clang++ must not be prefixed with targetPrefix to be wrapped correctly:

nixpkgs/pkgs/build-support/bintools-wrapper/default.nix

Line 233 in 33bbc2e

if [ -e ''${ld:-$ldPath/${targetPrefix}ld} ]; then

nixpkgs/pkgs/build-support/cc-wrapper/default.nix

Line 393 in 33bbc2e

elif [ -e $ccPath/clang ]; then

nixpkgs/pkgs/build-support/cc-wrapper/default.nix

Line 407 in 33bbc2e

elif [ -e $ccPath/clang++ ]; then

m-bdf · 2025-01-10T01:45:24Z

pkgs/development/compilers/zig/bintools.nix

+  }
+  ''
+    mkdir -p $out/bin
+    for tool in ar objcopy ranlib; do


Suggested change

for tool in ar objcopy ranlib; do

for tool in ar ld.lld objcopy ranlib; do

m-bdf · 2025-01-10T01:45:34Z

pkgs/development/compilers/zig/bintools.nix

+      makeWrapper "$zig/bin/zig" "$out/bin/${targetPrefix}$tool" \
+        --add-flags "$tool" \
+        --run "export ZIG_GLOBAL_CACHE_DIR=\$(mktemp -d)"
+    done


Suggested change

done

done

mv $out/bin/${targetPrefix}ld.lld $out/bin/${targetPrefix}ld

m-bdf · 2025-01-10T01:47:29Z

pkgs/development/compilers/zig/cc.nix

+    for tool in cc c++ ld.lld; do
+      makeWrapper "$zig/bin/zig" "$out/bin/${targetPrefix}$tool" \


Suggested change

for tool in cc c++ ld.lld; do

makeWrapper "$zig/bin/zig" "$out/bin/${targetPrefix}$tool" \

for tool in cc c++; do

makeWrapper "$zig/bin/zig" "$out/bin/$tool" \

m-bdf · 2025-01-10T01:47:58Z

pkgs/development/compilers/zig/cc.nix

+    mv $out/bin/${targetPrefix}c++ $out/bin/${targetPrefix}clang++
+    mv $out/bin/${targetPrefix}cc $out/bin/${targetPrefix}clang
+    mv $out/bin/${targetPrefix}ld.lld $out/bin/${targetPrefix}ld


Suggested change

mv $out/bin/${targetPrefix}c++ $out/bin/${targetPrefix}clang++

mv $out/bin/${targetPrefix}cc $out/bin/${targetPrefix}clang

mv $out/bin/${targetPrefix}ld.lld $out/bin/${targetPrefix}ld

mv $out/bin/cc $out/bin/clang

mv $out/bin/c++ $out/bin/clang++

m-bdf

Using the $TMPDIR variable (with a default to /tmp) instead of calling mktemp on each invokation of zig speeds up compilation (or even configure scripts) by an order of magnitude on my machine.

m-bdf · 2025-01-10T01:57:21Z

pkgs/development/compilers/zig/generic.nix

+    ];
+
+    preBuild = ''
+      export ZIG_GLOBAL_CACHE_DIR="$TMPDIR/zig-cache";


Suggested change

export ZIG_GLOBAL_CACHE_DIR="$TMPDIR/zig-cache";

export ZIG_GLOBAL_CACHE_DIR="$TMPDIR/zig-cache"

m-bdf · 2025-01-10T01:57:58Z

pkgs/development/compilers/zig/cc.nix

+    for tool in cc c++ ld.lld; do
+      makeWrapper "$zig/bin/zig" "$out/bin/${targetPrefix}$tool" \
+        --add-flags "$tool" \
+        --run "export ZIG_GLOBAL_CACHE_DIR=\$(mktemp -d)"


Suggested change

--run "export ZIG_GLOBAL_CACHE_DIR=\$(mktemp -d)"

--run "export ZIG_GLOBAL_CACHE_DIR=\''${TMPDIR-/tmp}/zig-cache"

m-bdf · 2025-01-10T01:58:13Z

pkgs/development/compilers/zig/bintools.nix

+    for tool in ar objcopy ranlib; do
+      makeWrapper "$zig/bin/zig" "$out/bin/${targetPrefix}$tool" \
+        --add-flags "$tool" \
+        --run "export ZIG_GLOBAL_CACHE_DIR=\$(mktemp -d)"


Suggested change

--run "export ZIG_GLOBAL_CACHE_DIR=\$(mktemp -d)"

--run "export ZIG_GLOBAL_CACHE_DIR=\''${TMPDIR-/tmp}/zig-cache"

RossComputerGuy · 2025-01-10T05:49:26Z

@kirillrdy Those errors should be fixed now, nixpkgs-review should pass.

Eveeifyeve · 2025-01-10T11:44:18Z

Would it hurt if this pr includes a patch for std.system.darwin.getSdk or does a seperate pr need to be made?
As this is causing issues to package ghostty on darwin.

kirillrdy · 2025-01-10T12:03:00Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 331011

`x86_64-linux`

❌ 10 packages failed to build:

backlight-auto
blackshades
cyber
dt
findup
zig_0_10
zig_0_10.doc
zig_0_11
zig_0_11.doc
zon2nix

✅ 36 packages built:

arocc
aroccPackages.latest-unwrapped
aroccStdenv
cargo-lambda
cargo-zigbuild
clipbuzz
colorstorm
emacsPackages.zig-mode
flow-editor
ghostty
ghostty.man
ghostty.shell_integration
ghostty.terminfo
ghostty.vim
glsl_analyzer
linuxwave
ly
mepo
minizign
ncdu
poop
river
river.man
rivercarro
superhtml
waylock
wayprompt
zf
zig
zig.doc
zigStdenv
zig_0_12
zig_0_12.doc
zig_0_9
zls
ztags

RossComputerGuy requested a review from figsoda as a code owner July 30, 2024 04:39

github-actions bot added the 6.topic: zig label Jul 30, 2024

RossComputerGuy force-pushed the feat/zig-bootstrap branch from 92ca0ed to 511678d Compare July 30, 2024 04:41

RossComputerGuy mentioned this pull request Jul 30, 2024

Add stdenv & CC support for Zig #318034

Merged

13 tasks

ofborg bot requested review from AndersonTorres and andrewrk July 30, 2024 05:50

ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Jul 30, 2024

github-actions bot added the 6.topic: policy discussion label Jul 30, 2024

alyssais reviewed Jul 30, 2024

View reviewed changes

ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ and removed 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Jul 30, 2024

RossComputerGuy force-pushed the feat/zig-bootstrap branch 2 times, most recently from c19a377 to ace18eb Compare July 31, 2024 02:52

RossComputerGuy force-pushed the feat/zig-bootstrap branch 7 times, most recently from a0c11c2 to 5af85b0 Compare August 1, 2024 04:47

RossComputerGuy force-pushed the feat/zig-bootstrap branch from 5af85b0 to 6f3ab6e Compare August 1, 2024 05:35

RossComputerGuy marked this pull request as draft December 23, 2024 02:12

RossComputerGuy force-pushed the feat/zig-bootstrap branch from 4a3852d to 56506fd Compare December 23, 2024 03:48

RossComputerGuy changed the base branch from staging to master December 23, 2024 03:48

RossComputerGuy marked this pull request as ready for review December 23, 2024 03:48

RossComputerGuy force-pushed the feat/zig-bootstrap branch 2 times, most recently from db214ed to 356a033 Compare December 23, 2024 06:48

kirillrdy reviewed Jan 5, 2025

View reviewed changes

pkgs/development/compilers/zig/default.nix Outdated Show resolved Hide resolved

kirillrdy reviewed Jan 5, 2025

View reviewed changes

pkgs/development/compilers/zig/default.nix Outdated Show resolved Hide resolved

RossComputerGuy force-pushed the feat/zig-bootstrap branch from 356a033 to 4c49050 Compare January 9, 2025 02:52

m-bdf suggested changes Jan 10, 2025

View reviewed changes

RossComputerGuy force-pushed the feat/zig-bootstrap branch from 4c49050 to b6ef265 Compare January 10, 2025 04:43

RossComputerGuy added 5 commits January 9, 2025 23:30

zig: commonify

82fdc1e

maintainers/teams: add RossComputerGuy to zig

aeaa579

CODEOWNERS: add RossComputerGuy for Zig

aad55fb

zig.cc: add bintools

7cdaf6a

zig: split cc and bintools wrappers

baebc26

RossComputerGuy force-pushed the feat/zig-bootstrap branch from b6ef265 to baebc26 Compare January 10, 2025 07:31

	for tool in ar objcopy ranlib; do
	for tool in ar ld.lld objcopy ranlib; do

		for tool in cc c++ ld.lld; do
		makeWrapper "$zig/bin/zig" "$out/bin/${targetPrefix}$tool" \

	export ZIG_GLOBAL_CACHE_DIR="$TMPDIR/zig-cache";
	export ZIG_GLOBAL_CACHE_DIR="$TMPDIR/zig-cache"

	--run "export ZIG_GLOBAL_CACHE_DIR=\$(mktemp -d)"
	--run "export ZIG_GLOBAL_CACHE_DIR=\''${TMPDIR-/tmp}/zig-cache"

zig: commonify & bootstrap #331011

Are you sure you want to change the base?

zig: commonify & bootstrap #331011

Conversation

RossComputerGuy commented Jul 30, 2024

Description of changes

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alyssais Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aleksanaa commented Jul 31, 2024

RossComputerGuy commented Aug 1, 2024

kirillrdy commented Dec 21, 2024

nixpkgs-review result

x86_64-linux

RossComputerGuy commented Dec 22, 2024

kirillrdy commented Dec 22, 2024

kirillrdy commented Dec 22, 2024

kirillrdy commented Jan 5, 2025

RossComputerGuy commented Jan 6, 2025

kirillrdy commented Jan 7, 2025

RossComputerGuy commented Jan 7, 2025

kirillrdy commented Jan 9, 2025

nixpkgs-review result

x86_64-linux

m-bdf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m-bdf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RossComputerGuy commented Jan 10, 2025

Eveeifyeve commented Jan 10, 2025 • edited Loading

kirillrdy commented Jan 10, 2025

nixpkgs-review result

x86_64-linux

alyssais Jul 30, 2024 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`x86_64-linux`

Eveeifyeve commented Jan 10, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`