Python not correctly detected in pkgsrc zone on OmniOS during bootstrap

[francescocarzaniga]

I am trying to build Bazel in a pkgsrc branded zone on OmniOS. This should be the same as building in a Joyent zone on SmartOS. I follow your steps precisely but I get

ERROR: /root/bazel/bazel-2.2.0/src/BUILD:305:2: Executing genrule //src:embedded_tools_nojdk failed (Exit 1): bash failed: error executing command
  (cd /tmp/bazel_XXB4aGad/out/execroot/io_bazel && \
  exec env - \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.zip" bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.params')
Execution platform: //:default_host_platform
/tmp/Bazel.runfiles_rehthpaw/runfiles/bazel_tools/tools/python/py2wrapper.sh: line 55: PAT/tmp/Bazel.runfiles_rehthpaw/runfiles/bazel_tools/tools/python/py2wrapper.sh: line 55: : parameter not set
----------------
Note: The failure of target //src:create_embedded_tools (with exit code 1) may have been caused by the fact that it is running under Python 2 instead of Python 3. Examine the error to determine if that appears to be the problem. Since this target is built in the host configuration, the only way to change its version is to set --host_force_python=PY3, which affects the entire build.

If this error started occurring in Bazel 0.27 and later, it may be because the Python toolchain now enforces that targets analyzed as PY2 and PY3 run under a Python 2 and Python 3 interpreter, respectively. See https://github.com/bazelbuild/bazel/issues/7899 for more information.
----------------
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 96.545s, Critical Path: 84.72s
INFO: 791 processes: 791 local.
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel 

Forcing Python3 or Python2 with --host_force_python=PY3 (respectively PY2) results in the exact same error message with py3wrapper.sh and py2wrapper.sh

[siepkes]

Can you run with more debugging flags enabled to check if Python 3 is actually used? For example export EXTRA_BAZEL_ARGS="--subcommands --host_javabase=@local_jdk//:jdk".

The Python version in the SmartOS / Triton image is 3.7 btw.

[francescocarzaniga]

The Python version in the SmartOS / Triton image is 3.7 btw.

The same goes for my zone, but for good measure I tried with other versions too, including Python 2.7.

The output with subcommands (curated a bit):

🍃  Building Bazel from scratch......fgrep: illegal option -- m
usage: fgrep [-bchHilnqrRsvx] [-A num] [-B num] [-C num|-num]
             [-e pattern_list]... [-f pattern_file]... [pattern_list] [file]...

🍃  Building Bazel with Bazel.
.awk: ENVIRON is not an array
DEBUG: /tmp/bazel_XXtDaa34/out/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:59:9:
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '2.0.0'
DEBUG: /tmp/bazel_XXtDaa34/out/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:9: rbe_ubuntu1804_java11 not using checked in configs as detect_java_home was set to True
DEBUG: /tmp/bazel_XXtDaa34/out/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:59:9:
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '2.0.0'
DEBUG: /tmp/bazel_XXtDaa34/out/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:9: rbe_ubuntu1604_java8 not using checked in configs as detect_java_home was set to True
DEBUG: /tmp/bazel_XXtDaa34/out/external/build_bazel_rules_nodejs/internal/common/check_bazel_version.bzl:49:5:
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /tmp/bazel_XXtDaa34/out/external/build_bazel_rules_nodejs/internal/common/check_bazel_version.bzl:51:5: Make sure that you are running at least Bazel 0.17.1.
WARNING: /root/bazel/bazel-2.2.0/src/main/java/com/google/devtools/build/lib/syntax/BUILD:163:1: in includes attribute of cc_library rule //src/main/java/com/google/devtools/build/lib/syntax:jni: '../../../../../../../../../external/bazel_tools/tools/jdk/include' resolves to 'external/bazel_tools/tools/jdk/include' not below the relative path of its package 'src/main/java/com/google/devtools/build/lib/syntax'. This will be an error in the future
WARNING: /root/bazel/bazel-2.2.0/src/main/java/com/google/devtools/build/lib/syntax/BUILD:163:1: in includes attribute of cc_library rule //src/main/java/com/google/devtools/build/lib/syntax:jni: '../../../../../../../../../external/bazel_tools/tools/jdk/include/solaris' resolves to 'external/bazel_tools/tools/jdk/include/solaris' not below the relative path of its package 'src/main/java/com/google/devtools/build/lib/syntax'. This will be an error in the future
INFO: Analyzed target //src:bazel_nojdk (232 packages loaded, 10166 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //src:dummy_darwin_tool_src_tools_xcode_stdredirect_StdRedirect.dylib [action 'Executing genrule //src:dummy_darwin_tool_src_tools_xcode_stdredirect_StdRedirect.dylib', configuration: ac5c3061cbea031f9ae21b059c96637fceac7aafa67523c756f25aa8c0c7afd3]
(cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; touch bazel-out/illumos-opt/bin/src/dummy_darwin_tools/src/tools/xcode/stdredirect/StdRedirect.dylib')

[...]

SUBCOMMAND: # //third_party/grpc:grpc_base_c [action 'Compiling third_party/grpc/src/core/lib/json/json_string.cc', configuration: ac5c3061cbea031f9ae21b059c96637fceac7aafa67523c756f25aa8c0c7afd3]
(cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
    PWD=/proc/self/cwd \
  /opt/local/gcc7/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -Wall -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/json_string.d '-frandom-seed=bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/json_string.o' '-DGRPC_ARES=0' -iquote . -iquote bazel-out/illumos-opt/bin -isystem third_party/grpc -isystem bazel-out/illumos-opt/bin/third_party/grpc -isystem third_party/grpc/include -isystem bazel-out/illumos-opt/bin/third_party/grpc/include -isystem third_party/zlib -isystem bazel-out/illumos-opt/bin/third_party/zlib -D__illumos__ -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/grpc/src/core/lib/json/json_string.cc -o bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/json_string.o)
SUBCOMMAND: # //src:embedded_tools_nojdk [action 'Executing genrule //src:embedded_tools_nojdk', configuration: ac5c3061cbea031f9ae21b059c96637fceac7aafa67523c756f25aa8c0c7afd3]
(cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.zip" bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.params')
SUBCOMMAND: # //third_party/grpc:grpc_base_c [action 'Compiling third_party/grpc/src/core/lib/iomgr/wakeup_fd_eventfd.cc', configuration: ac5c3061cbea031f9ae21b059c96637fceac7aafa67523c756f25aa8c0c7afd3]
(cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
    PWD=/proc/self/cwd \
  /opt/local/gcc7/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -Wall -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/wakeup_fd_eventfd.d '-frandom-seed=bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/wakeup_fd_eventfd.o' '-DGRPC_ARES=0' -iquote . -iquote bazel-out/illumos-opt/bin -isystem third_party/grpc -isystem bazel-out/illumos-opt/bin/third_party/grpc -isystem third_party/grpc/include -isystem bazel-out/illumos-opt/bin/third_party/grpc/include -isystem third_party/zlib -isystem bazel-out/illumos-opt/bin/third_party/zlib -D__illumos__ -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/grpc/src/core/lib/iomgr/wakeup_fd_eventfd.cc -o bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/wakeup_fd_eventfd.o)
SUBCOMMAND: # //third_party/grpc:grpc_base_c [action 'Compiling third_party/grpc/src/core/lib/iomgr/exec_ctx.cc', configuration: ac5c3061cbea031f9ae21b059c96637fceac7aafa67523c756f25aa8c0c7afd3]
(cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
    PWD=/proc/self/cwd \
  /opt/local/gcc7/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -Wall -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/exec_ctx.d '-frandom-seed=bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/exec_ctx.o' '-DGRPC_ARES=0' -iquote . -iquote bazel-out/illumos-opt/bin -isystem third_party/grpc -isystem bazel-out/illumos-opt/bin/third_party/grpc -isystem third_party/grpc/include -isystem bazel-out/illumos-opt/bin/third_party/grpc/include -isystem third_party/zlib -isystem bazel-out/illumos-opt/bin/third_party/zlib -D__illumos__ -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/grpc/src/core/lib/iomgr/exec_ctx.cc -o bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/exec_ctx.o)
SUBCOMMAND: # //third_party/grpc:grpc_base_c [action 'Compiling third_party/grpc/src/core/lib/iomgr/wakeup_fd_nospecial.cc', configuration: ac5c3061cbea031f9ae21b059c96637fceac7aafa67523c756f25aa8c0c7afd3]
(cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
    PWD=/proc/self/cwd \
  /opt/local/gcc7/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -Wall -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/wakeup_fd_nospecial.d '-frandom-seed=bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/wakeup_fd_nospecial.o' '-DGRPC_ARES=0' -iquote . -iquote bazel-out/illumos-opt/bin -isystem third_party/grpc -isystem bazel-out/illumos-opt/bin/third_party/grpc -isystem third_party/grpc/include -isystem bazel-out/illumos-opt/bin/third_party/grpc/include -isystem third_party/zlib -isystem bazel-out/illumos-opt/bin/third_party/zlib -D__illumos__ -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/grpc/src/core/lib/iomgr/wakeup_fd_nospecial.cc -o bazel-out/illumos-opt/bin/third_party/grpc/_objs/grpc_base_c/wakeup_fd_nospecial.o)
ERROR: /root/bazel/bazel-2.2.0/src/BUILD:305:2: Executing genrule //src:embedded_tools_nojdk failed (Exit 1): bash failed: error executing command
  (cd /tmp/bazel_XXtDaa34/out/execroot/io_bazel && \
  exec env - \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.zip" bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.params')
Execution platform: //:default_host_platform
/tmp/Bazel.runfiles_s_mkn7ay/runfiles/bazel_tools/tools/python/py3wrapper.sh: line 55: PAT/tmp/Bazel.runfiles_s_mkn7ay/runfiles/bazel_tools/tools/python/py3wrapper.sh: line 55: : parameter not set
----------------
Note: The failure of target //src:create_embedded_tools (with exit code 1) may have been caused by the fact that it is running under Python 3 instead of Python 2. Examine the error to determine if that appears to be the problem. Since this target is built in the host configuration, the only way to change its version is to set --host_force_python=PY2, which affects the entire build.

If this error started occurring in Bazel 0.27 and later, it may be because the Python toolchain now enforces that targets analyzed as PY2 and PY3 run under a Python 2 and Python 3 interpreter, respectively. See https://github.com/bazelbuild/bazel/issues/7899 for more information.
----------------
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 83.634s, Critical Path: 72.32s
INFO: 567 processes: 567 local.
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

There's loads of INFO and SUBCOMMAND, so I'm showing you only the beginning and end. Let me know if you need more lines.

[siepkes]

Could you show what is in the generated py3wrapper.sh file? Especially around line 55. iirc the temporary files should still be there after running the bazel bootstrap script.

In the logs it says these generated files are in /tmp/. Do you perhaps have /tmp mounted with noexec?

[francescocarzaniga]

Could you show what is in the generated py3wrapper.sh file? Especially around line 55. iirc the temporary files should still be there after running the bazel bootstrap script.

The file gets deleted immediately so I'm unable to get its content. Is there a way to persist those temporary files?

In the logs it says these generated files are in /tmp/. Do you perhaps have /tmp mounted with noexec?

/tmp is mounted on swap with read/write/setuid/nodevices/xattr but not noexec.

[siepkes]

I think that file gets generated from tools/python/pywrapper_template.txt. If you add set -x to the top of it all the commands executed by the script generated from that template should get printed.

[francescocarzaniga]

The output now is:

ERROR: /root/bazel/bazel-2.2.0/src/BUILD:305:2: Executing genrule //src:embedded_tools_nojdk failed (Exit 1): bash failed: error executing command
  (cd /tmp/bazel_XXQUa4YA/out/execroot/io_bazel && \
  exec env - \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.zip" bazel-out/illumos-opt/bin/src/embedded_tools_nojdk.params')
Execution platform: //:default_host_platform
+ set -u
+ STRICT=1
+ [ 1 = 1 ]
+ FAILURE_HEADER='Error occurred while attempting to use the default Python toolchain (@rules_python//python:autodetecting_toolchain)
+ which python3
+ 2> /dev/null
+ PYTHON_BIN=''
+ USED_FALLBACK=0
+ [ -z '' ]
+ which python
+ 2> /dev/null
+ PYTHON_BIN=''
+ USED_FALLBACK=1
+ [ -z '' ]
/tmp/Bazel.runfiles_z35jpv5s/runfiles/bazel_tools/tools/python/py3wrapper.sh: line 57: PAT/tmp/Bazel.runfiles_z35jpv5s/runfiles/bazel_tools/tools/python/py3wrapper.sh: line 57: : parameter not set
----------------
Note: The failure of target //src:create_embedded_tools (with exit code 1) may have been caused by the fact that it is running under Python 3 instead of Python 2. Examine the error to determine if that appears to be the problem. Since this target is built in the host configuration, the only way to change its version is to set --host_force_python=PY2, which affects the entire build.

If this error started occurring in Bazel 0.27 and later, it may be because the Python toolchain now enforces that targets analyzed as PY2 and PY3 run under a Python 2 and Python 3 interpreter, respectively. See https://github.com/bazelbuild/bazel/issues/7899 for more information.
----------------
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 94.237s, Critical Path: 82.80s
INFO: 772 processes: 772 local.
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

It seems Bazel is not able to detect the path of Python's executable. If I set PYTHON_BIN='/usr/bin/python' inside pywrapper_template.txt the build runs fine.

[siepkes]

Good to hear you can get a working build!

Isn't there a Python installed in /opt/local/bin/python ? There is in my zone. That's also the Python I would expect Bazel to find.

[francescocarzaniga]

Isn't there a Python installed in /opt/local/bin/python ? There is in my zone. That's also the Python I would expect Bazel to find.

Yes there is, both in /usr/bin (default OmniOS) and in /opt/local/bin (default pkgsrc).

The problematic part of the template seems to be this:

PYTHON_BIN="$(PATH="$PATH" which python%VERSION% 2> /dev/null)"
USED_FALLBACK="0"
if [ -z "${PYTHON_BIN:-}" ]; then
  PYTHON_BIN="$(PATH="$PATH" which python 2>/dev/null)"
  USED_FALLBACK="1"
fi
if [ -z "${PYTHON_BIN:-}" ]; then

Which corresponds in execution to:

+ which python3
+ 2> /dev/null
+ PYTHON_BIN=''
+ USED_FALLBACK=0
+ [ -z '' ]
+ which python
+ 2> /dev/null
+ PYTHON_BIN=''
+ USED_FALLBACK=1
+ [ -z '' ]

As you can see PYTHON_BIN always ends up empty, but if I run PATH="$PATH" which python3 2>/dev/null or PATH="$PATH" which python 2>/dev/null in sh it outputs correctly /usr/bin/python.