install-nix: Full progress of POSIX installation

[Anton-Latukha]

New TODO round:

• - explore new master branch changes.
• - sync script with master branch.
• - do a fresh relearning of the script.
• - do small additional changes to the script.
• - rewrite macOS bridge to POSIX (I removed bridge, and old one and a second one was existing in parallel in master changes, and they seems to do the same action).
• - line wrap the script in 80 lines.
• - last bits of integration with newly developed structure. Sort off. This PR was done before it, so it is hard to retrofit it in backwards fashion. And this is POSIX code, so I can not rely on Bash functionality in that scripts. I still see it as input gates that makes checks before running installers.
• - one more review of CLI output functions.
• - testing.
• - one more full re-read of the process.
• - the n-th number of plumming the commits to the merge state.
• - more testing.

Done.

Launching merge stage.

Introduction

This is the same script from the current master.

It works the same, and process is the same, that was not touched anyhow.
I've worked on script for a while. Added structure, checks, error handling, features, output functions, so on...
And it transformed to this state.

I tested it. In testing environments. And also used to make Nix setup on my main complex Arch Linux setup.
I am sure in it.
You can test it.

I will be happy to continue work.

Video demonstration:

Current master: https://www.youtube.com/watch?v=NuyevCXBqyY
This commit: https://www.youtube.com/watch?v=yD3pgkMvuGw

Video addresses generic procedure.
Commit also advances installation in cases not shown in video.

Solves:

• Installing Nix in Docker on non-NixOS Linux fails: $USER is not set #971 'Installing Nix in Docker on non-NixOS Linux fails: $USER is not set'
• Directory $dest does not exist; creating it by running '$cmd' - is not an error, it is expected on install. On the contrary, if $dest is present - that means Nix was once already installed, that is not normal case and prints informative message now.
• Rephrased message, if running from root.
  Solves 'installing as root does not fail (or: prints an over-severe warning) installing as root does not fail (or: prints an over-severe warning)  #1125, closes Update wording on root install #1143
  Also this was mentioned in install.sh homepage instructions only work without build-user-group #697
• Solves: 'installation fails while asking sudo, after downloading the 69Mb archive installation fails while asking sudo, _after_ downloading the 69Mb archive #539'
• all this works towards solving
  'how to correctly install Nix on ArchLinux how to correctly install Nix on ArchLinux #879'

Embodies

• Update wording on root install #1143 'Update wording on root install'
• install-nix: Solve #971 USER == NULL #1549 'install-nix: Solve Installing Nix in Docker on non-NixOS Linux fails: $USER is not set #971 USER == NULL'
• install-nix: Adding HOME checks #1554 'install-nix: Adding HOME checks'
• install-nix: Solve #971 USER == NULL #1549 'install-nix: Advancement of destination folder creation means'

Features

• More mature script structure
• POSIX compatible. (referencing http://pubs.opengroup.org/onlinepubs/9699919799/)
• Documentation
• Checks all needed requirement before doing actions/installation.
• Informing user if checks not met, so user can address them.
• Verbose output of what is going on overall. So user feels in control and knows what is happening.
• Nix folder:
  • At first try to utilize rights accessible to user, to create /nix
  • If not - checks does sudo installed
    • Asks for sudo access.
  • If not - asks for su and root access.
• Detects if installation STD streams attached to terminal, or installation runs headless.
  • If file descriptor is from terminal - uses colorized output
  • Other - no colorized output, so does not injects color codes in log files and etc.
• Notice, Warning and Error messages as important messages - made so they are fall into the users eye and easy to read.
• Notice, Warning and Error messages are colorized accordingly (green, yellow, red). This tested on different platforms.
• Colorization:
  • uses terminfo and tput if available.
  • otherwise uses ASCII color codes.
• More informative error messages. Notice and Warning also.
• Inform user on what line error signal occurred in the script
• Some commands that possible to revert (nix-env --install and nix-channel --add) have revert operations. On error - script reverts those actions.
• Proper script error handling using trap.
• self resolves to absolute path.

Still:

• nixbld group and workers problem from install.sh homepage instructions only work without build-user-group #697, addressed in bugreport Nix single user install on Linux wants 'nixbld' group and workers, else - it fails #1559 'Nix single user install on Linux wants 'nixbld' group and workers, else - it fails'

It needs or to be solved, or embodied in official documentation and single-user installation (and also in this script).

Why printf used instead of echo

Shortly, - it is more portable across systems.
https://askubuntu.com/questions/467747/which-is-better-printf-or-echo,
https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo,
https://www.in-ulm.de/~mascheck/various/echo+printf/

Why trap instead of set -e

You could came across thought thatset -e is considered bad practice couple of times.
'set -e causes the shell to exit if any sub-command or pipeline returns a non-zero status.'. Also: https://stackoverflow.com/questions/19622198/what-does-set-e-mean-in-a-bash-script.

So with set -e on errors - script freaks-out and exits. And what partially was done - stays partially done on the system. So it creates stateful situation and script can not be transactional.

If Nix states to be transactional, so better to maximally work toward transactional install. I think Nix script must handle errors and revert processes, if possible, that is what traps are for.

Other

Checked with static analysis.
Tested on Arch Linux, Ubuntu and Alpine Linux in different cases:

• from user,
• from root,
• different states of system with installation of Nix, etc.

There are a lot of considerations and moments to make this script both mature, portable and understandable went to it.
If human knows shell, - he can extend this script, or troubleshoot his case from it.
It is readable and moments that probably needed explanation - documented.

I plan to mutually exchange code parts with macOS version by @grahamc.
But it needs mention, that macOS and Linux is very different platforms.
macOS is derivative of BSD.
Linux is derivative of Minix and Unix.
macOS and Linux is completely different internally, as far as possible, systems, both have more in common with Unix, then some mutual similarity internally. Please don't ask to deduplicate scripts completely, or merge them, that going to be overcomplication of process and not readable/maintainable. That requirement also going to degrade macOS part, since lowest denominator of POSIX must be used then. While macOS platform is much more homogeneous, let that be the strength of the platform.

macOS and Linux have many differences regarding commands and environment, that is why processes here should be or separate or flexible. Flexible is what I aim for. To exchange what is possible, and bring scripts to the huge level.

I am ready to maintain Linux installation of Nix. If not, - I ready to contribute this and move further.

Backlog

• Make script and installation procedure maximally transactional
• Do more in depth color state detection. Involves research on terminfo, tput colors, termcap, tput Co, $TERM and $SHELL special cases.

  if tput colors ; then
      # do color coding catch with tput
  elif
      if [ -z "$(printf '\033[1;32m')" ] ; then
          # do color coding catch with ANSI codes
      fi
  fi

• Make fully unattended script
• Discuss/think about to make Nix installation to respect XDG Base Directory specification support. If that happens - documentation everywhere need to be changed to new paths.
• If XDG Base Directory enabled - migrate old installs to respect XDG Base Directory specification (can be softlink).
• Make sourcing automatic, or much easier for users (install script does not source #555).

Testing platform

Just for this PR the testing platform was created: https://github.com/Anton-Latukha/nix-tests-terraform
, which covers the most different 9 Linux distributions:

• Current Debian
• Current CenOS
• Current OpenSUSE
• Current Ubuntu
• Current Alpine - and yes, - Alpine does not have groupadd or useradd - Alpine based on BusyBox.
• Previous version of Slackware
• Current Arch Linux
• Android (- it seems like some sort of Unix-like system)
• Trisquel (- GNU/Linux)

And it was tested that on all these distributions this installer works.

I even spent a work week on making automated BSD testing, creating them KVM machines, resolving them BSD compatibility issues, learning different BSD systems, and testing this on all major of them.

In BSD installation fails because of upstream hardcode issue: #1559, and I still didn't got any information back why is it so. If not for Linux-specific hardcoded groups - the installer would install on BSDs. The official support is the other thing, we do not claim it here. But maybe they would be interested themselves in some Nix and would start supporting/helping with it themselves.

---

This change is 

[Mic92]

No in-depth review, I just quickly scanned over the code.
Should be also tested on OS X.

[Anton-Latukha]

This is not OS X code, script only for OS X exists already.
On OS X only first pieces of script executed.

And since this is fully POSIX, - I am sure it works on macOS flawlessly.

There are bridge at the start of the script to macOS.

if [[ "$(uname -s)" = "Darwin" ]]; then

macOS script is obviously heavy macOS related, and uses Bash.

Couple of days ago I wrote to @grahamc that I work on this and I am ready to work togather.

Fitting this code to macOS one without merging to master I saw difficult. Because I saw that collaboration should happen.

Since I use Linux systems exclusively and not own macOS. I don't know anything about macOS, except that I can 100% rely on sh and POSIX.

And I going to go through macOS code he wrote and get pieces to incorporate. But it also need thought and some rewriting.

I see that if merging codebases of Linux and macOS is a tactic that should be looked on, as macOS has many differences. It is not realistic to want to smush them both together in one script, since we are going to overcomplicate things and create technical debt, and half-... solution.

Because, then TrueOS, FreeBSD comes, Windows, some other systems come, and we need to support that huge script, and smush that systems into it creating giant complicated monster, or go backwards and untie things from that script.

macOS script has a bridge inside this script.
So to do deduplication, we can talk, and or:

• separate installations completely, moving bridge to https://nixos.org/nix/install
• or create hub script in the tarball.

Since I did not export...
Oh, yeah, I done:

LC_COLLATE=C ; export LC_COLLATE
LANG=C ; export LANG
umask 022

You right, that touches macOS, it needs to be looked in, if it is potent to macOS.

As I see, we can move bridge at the top, and then we work what can be shared, and what not.

I rewrote to:

LC_COLLATE=C
LANG=C
umask 022

For now, so it does not get exported into macOS setup.
Since macOS is monolithic system, it probably should use its defaults, that are probably same on all macOS systems (UTF-8).
And since Linux is so diverse, probably better to start from C.

Now, on OS X in must be nothing to test.
Because I not exported anything to it, and bridge inside checks is in the same place as before.

[wmertens]

Good stuff! Just some typos and light refactoring…

[wmertens]

aha indeed, TIL

…

On Tue, Sep 12, 2017 at 1:02 PM Bjørn Forsman ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In scripts/install-nix-from-closure.sh <#1565 (comment)>: > +readonly self="$(dirname "$0")" +readonly ***@***.***@" +readonly ***@***.***@" +readonly appname="$0" +} +############################### +### CLI control constants +############################### +{ +# tput is not present on some systems (Alpine Linux), this trick with printf allows to store, not 'codes' - literal symbols +readonly red=$(printf '\033[1;31m') +readonly green=$(printf '\033[1;32m') +readonly yellow=$(printf '\033[1;33m') +readonly blue=$(printf '\033[1;34m') +readonly bold=$(printf '\033[1m') +readonly reset=$(printf '\033[0;m') # Reset to default output @wmertens <https://github.com/wmertens>: Better use "command -v" instead of "type -p" if you want to be compatible with POSIX sh. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1565 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWljhPSqABfkAHVm_QRyZMhHkEFM_aks5shmS1gaJpZM4PSDHd> .

[Anton-Latukha]

• Today implemented most of the suggestions/fixes.
• Additional work to correspond to POSIX.
  Special thanks to @wmertens for such strong commitment here, guidance, detailed review and suggestions.

[Anton-Latukha]

Tested in real battle on my main complex Arch Linux setup.

Knowing that you need to dodge C++ code blocker #697 #1559, that assumes you made multiuser presetup.

Install works flawlessly. I encourage everyone to try it.
Going to link Arch Linux folks here.

Note: it is beautifully coloured in reality:

~/Project/nix-1.11.15-x86_64-linux> ./install
./install: Info: Directory /nix already exists. Skipping creation of /nix
./install: Notice: 

    Performing a single-user installation of Nix:
    
copying Nix to /nix/store.................................
initialising Nix database...
./install: Info: Launching /nix/store/b4s1gxiis1ryvybnjhdjvgc5sr1nq0ys-nix-1.11.15/etc/profile.d/nix.sh
./install: Info: Left /nix/store/b4s1gxiis1ryvybnjhdjvgc5sr1nq0ys-nix-1.11.15/etc/profile.d/nix.sh
replacing old ‘nix-1.11.14’
installing ‘nix-1.11.15’
building path(s) ‘/nix/store/0scfamgyalc4kmg9943ra21a8xh3na5l-user-environment’
created 6 symlinks in user environment
./install: Info: Pulling information from channel: nixpkgs
downloading Nix expressions from ‘https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.03pre116369.14cbeaa892/nixexprs.tar.xz’...
downloading ‘https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-18.03pre116369.14cbeaa892/nixexprs.tar.xz’... [0/0 K [6911/9145 KiB, 6909.7 KiB/s]
unpacking channels...
created 2 symlinks in user environment
./install: Notice: 

    Installation finished. 8)
    
./install: Notice: 

    Launching postinstall tasks:
    
./install: Notice: 

    Installation finished!
    To ensure that necessary environment variables to use Nix binaries
    are set in current runtime environment, either log in again, or type

    . /home/pyro/.nix-profile/etc/profile.d/nix.sh

    in your shell.
    
./install: Info: 

    If you hit a bug, we would love to help.

    You can, search/open an issue at https://github.com/nixos/nix/issues

    Feel free to contact the team,
     - on IRC #nixos on irc.freenode.net
     - on twitter @nixos_org

~/Project/nix-1.11.15-x86_64-linux>

[chrissound]

Looks like a huge amount of work here. A big thank you to everyone involved in this!

[Anton-Latukha]

I've made a Docker pipeline that automatically assembles new versions of this installer into latukha/nix-install-upd image. It is a ./install-new.sh executable in there, and ./install.sh is from upstream tarball.

So anyone can always try out the current version on it in this pull request.

_{That is the side-product from my presentation at NixCon.}

... we got here a small factorio running :]...

[zimbatm]

[[ ]] is a bashism

[Anton-Latukha]

This is already part of Graham MacOS code.
MacOS by default always uses Bash. And Graham's script fully uses Bash.

I not changed this deliberately.

[grahamc]

Probably better to move to POSIX for this check too.

[Anton-Latukha]

I just can't check what:

if [[ $(($(sw_vers -productVersion | cut -d '.' -f 2))) -lt 10 ]]

Parses and does.

I do not understand this Bash:

if [[ $(($(val))) -lt 10 ]]; then

I don't get what is the sense of:

$(($(val)))

I don't know what is $(()).

[zimbatm]

$(()) is for arithmetic. Eg:

$ x=$((4 + 5));
$ echo "$x"
9

In that case I think that it can be ignored since $(sw_vers -productVersion | cut -d '.' -f 2) would return a single number. The only difference would be if it didn't return a number. Eg:

$ x=$(($(echo foo)))
$ echo $x
0

[Anton-Latukha]

That is why using logic I thrown my hands.

$(()) must calculate and return number, but I don't know what policy and history Apple has with sw_vers -productVersion on macOS, and why and how it can happen that it calculates some numbers inside it.

[Anton-Latukha]

Since I just was laid off at work.

I have time, and if someone wants something.

Since I got more Nix experience in this time, I can make second shift on this and maybe change something. Provide more clear explanation to people.

But I am interested to do single-user if it is possible now, I had a bug in Nix then.

Because, nevertheless I know much more of Nix now, I not going to forget the stepping stones they must be simple.

If person wants single-user nix - we should give him. And then he going to learn and move to multi-user himself.

I would be interested if we merge this. As it is now. Or almost as it is.

And then I can work on additions. Receive some input, and do some further collaboration.

Make options, for active and unattended install. And find a way to source current shell environment.

It was been ideal to do everything needed and merge this pull request during Ukrainian holidays.
(from now till Jan 14, when we have Old New Year 🙂 )

[Anton-Latukha]

So. What should we do with this?

It was tested and specifically presented to show that it works long time ago.
And it just stalled here.

[Anton-Latukha]

@vcunat
@grahamc

What we do about this.

I made this. It is POSIX compatible. Documentation. Comments everywhere. It is reviewed a number of times by number of people. Made testing. Live testing. Live talk. Live demonstration. Explanation. That was done purely to make people sure to merge this.

I can update it to master once more.

Or I can stop caring.

it goes out of mainline fast on every change to upstream - this is shell scripting. So far this can be synced to master. What should I do, update or close this.

[Mic92]

Maybe now that we have more a nix core maintainer team, this increases the pool of people who can review this.

[Anton-Latukha]

I went through all comments on the code, the issues mainly of output text or matter of shell style preference, or point to the situations which are not covered in master also.

And almost nobody seems to complain/fix master script issues this PR treating to.

Except for a number of newcomers in Nix issues.

I already said previously - I wanted to dedicate myself to support the installers particularly.

I am frustrated at the process of merging of this PR.

Answering the reviews now become emotional to me. I tried to be as polite as I could.

Thank you for understanding.

If this ever merged - I would start covering those suggestions.

[Anton-Latukha]

Thank you for the reviews, it mattered to me that some people really took the call and helped me.

[Anton-Latukha]

And you know that this PR is multiuser install, if I add:

groupadd -r nixbld
for n in $(seq 1 10); do useradd -c "Nix build user $n" \
    -d /var/empty -g nixbld -G nixbld -M -N -r -s "$(which nologin)" \
    nixbld$n; done

, right?

And you know in most containers where people try to install Nix - there is no systemd, right? And in many cases no bash.

And by the way - this snipped is pasted from official docs.
which is not POSIX. command -v is, present and works everywhere, and which does the same thing. So in the shell code or in official snippet better to use command -v.

[fricklerhandwerk]

@Anton-Latukha Your emphasis on the importance of a portable installer is very valuable, as is this PR. Currently I have severe - and in my opinion unnecessary - pains installing Nix on Ubuntu on a VPS, and your PR answers all of them and then some.
@shlevy @vcunat Why exactly is this PR in the backlog? I agree with @grahamc that breaking it up would have made it more manageable, but in this state it would be better to merge quickly and iterate over further details later. This thing has been standing around for more than two years now!

[stale]

I marked this as stale due to inactivity. → More info

[tomberek]

Still important to me. Not sure why this has stalled. Is there a concern about regression or behavior differences?

[fricklerhandwerk]

This is also still important for me. But this PR is huge, and if I were responsible I would loathe to review such a big change. And now there even are merge conflicts.

Maybe we can come up with a strategy to get this going again.

[admirabilis]

Are there any instructions for running this script, please? It seems to be part of the installation tarball, so I'd need to get the last working commit without merge conflicts, or open the tarball and replace install-nix-from-closure.sh, right? Sorry, I didn't look much further after it expected a .reginfo file, which seems to come from a previous installation step.

To the code reviewers: it doesn't seem too complicated reviewing the script as a whole instead of trying to review all the smallest changes... the script is big mostly due to a lot of error checking.

[Anton-Latukha]

@fricklerhandwerk

It is really simple really. The sizes of PRs in repos are proportionate to the ease to merge the improvements (which is proportional to responce times & flexibility of maintainers to hold dialog with contributors & desire to merge improvements). I had 3 PRs with tiny important fixes, they never got merged, so work ended up gathering here.

Review of this PR is not difficult at all.
Here - the contributor was very active & responcive, up to making a talk to merge the work.
People can be reasoned with, one of the first things I tend to say to people is "please, how can we not gather changes in one PR but make process gradual merging of changes". That is easier to review, that is easier to discuss, that is easier to merge. But here I had problems to get meaningful merging problems with the basic rwx file sesystem checks to checked before placing files of a system package manager that take milliseconds & run once upon installation.

Review is not difficult, it requires knowledge of sh, a widespread skill. And a knowledge of some variaty of Unix systems (widespread skill & requirement for package management work) & understanding of POSIX (which has a lot of explanations online & standard present online & describes things also succinctly, PR even has a link to it). The PR set has 46 commits - which all are tiny logically ordered, atomic & gradual changes, with a proper name & explanation, so all it can be looked & checked commit by commit - that is the easiest pass to relly check a big PR.

Again, example of the other kind of a topic, that topic I would consider complex & change is just as huge, but the most of the change is inside 1 commit - "try to check that one". When the contributor is not responding to maintainer.

In other words, the situation is really not about the code, it is of personel, if there was dialog - the work would continued.

---

@tomberek

There were some changes 1 year ago, where some fixes of things I was fixing here arrived. It was 3-2.5 years of a delay to react to installation issues, but better late then never, agree? In/from that time I know there was a couple of changes in installers & to that time I lost interest, understandably.

---

@teresaejunior

Sorry to inform you.

Since upstream not collaborated with me on this - there is no way for me to provide support.

When this work started & ongoing it was ~4->3 years ago. Since nothing of it was merged, I am not sure the script still aligns with how currently the upstream bootstrapping works.

How much time that was? Enough of time to not know Haskell and then know pretty advanced Haskell solidly & do some stuff with with it.

That people keep showing up into this thread is sad. I directly to the main upstream people several times proposed to take maintenence of inslallation&deinstallation automatization, atomicity & portability on myself. Simon Payton Jones in the recent talk used particularly a project installation/portability work as an example of a work that is hard to find volunteers for, while the work being of most importance & demanded to be done & maintained at all times. The same was & is obvious to me, so that is why I was astonished to the lack of upstream interest into reports & my propositions & not got any responce nor help. From what I observed I'd say they seem to pretended that all that discussion never existed.

So if someone would be able to be compatible with the Nix upstream & would find a foot hold to fix this situation & would want to further Nix installation agenda - I would thank that person, not because I need it, but just for doing it for others, for caring for people.

I can not maintain the Nix installer outside of the Nix repository. The effect of it would be - to provide the path to allow to keep the upstream issues unsolved. I suggest to approach those who write the upstream script. I talked to them, but they decided to hold that responcibility tight to themselves, so they bear that responcibility, responcibility is not just badges, fame & generating code. Responcibility is maintenence. They took responcibility - they are respoincible to maintain it at all times.

[Anton-Latukha]

If upstream wants - PR can be closed.

Otherwise I would close it myself soon, since the tail of a discussion of the PR started to repeat itself.

[arrowd]

Typo: comapible.

[domenkozar]

Closing as it's too big, outdated and the author said he's not interested to continue the work.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-installer-workgroup/21495/13