ch-run: add option to ignore storage operation protection

[j-ogas]

New ch-run behavior prevents manipulating images in storage, e.g.,

$ ch-run -w [...] ${CH_IMAGE_STORAGE}/img/foo -- [...]

CI pipelines that spin up CH_IMAGE_STORAGE in TMPFS are not expected to exist beyond the pipeline; thus, it is nice to be able to run some quirky build operations outside of the ch-image builder context. For example, running spack install in parallel.

    - |-
      cmd=$(cat << EOF
      . spack/share/spack/setup-env.sh
      spack env activate -d ./spenv/gukesh -p
      for i in 0 1 2 3 4 5; do
         spack install --fail-fast &> install_${i}.out &
         export p${i}=$!
      done
      wait $p0 $p1 $p2 $p3
      spack module tcl refresh --delete-tree --upstream-modules -y
      EOF
      )
    - |-
      ch-run --write \
             --unset-env='*' \
             --set-env \
             $CH_IMAGE_STORAGE/img/dumpsterkludge -- /bin/bash -c "$cmd"

[lucaudill]

We currently have an undocumented ch-run option, --unsafe, that enables manipulation of images in storage when specified alongside --write, e.g.

$ ch-run --unsafe -w [...] ${CH_IMAGE_STORAGE}/img/foo -- [...]

We could consider making this an official option, i.e. documenting it and no longer saying it's strictly for internal use.

@reidpr, thoughts?

[lucaudill]

After some offline discussion, we've decided that using --unsafe still isn't a good idea here. --unsafe corrupts your storage directory by making the build cache out of sync, so it should be left for internal use only. Instead, you can replace

      ch-run --write \
             --unset-env='*' \
             --set-env \
             $CH_IMAGE_STORAGE/img/dumpsterkludge -- /bin/bash -c "$cmd"

with

printf 'FROM dumpsterkludge\nRUN $cmd\n' | ch-image build -t foo -

In some cases this may necessitate modifying your cmd to work as a Dockerfile RUN instruction, but it's a safe alternative to modifying images in storage that is endorsed by the dev team.

[lucaudill]

@j-ogas would the feature proposed in #1408 meet your needs here? You could then do something like

echo "$cmd" | ch-image modify -s /bin/bash dumpsterkludge

or

ch-image modify -s /bin/bash dumpsterkludgea < cmd.sh

[reidpr]

After discussion, closing in favor of #1408. --unsafe really is asking for trouble; what works now may break in the future in subtle and hard to debug ways.