WEST_TOPDIR acquisition incorrect when using PyCharm

[LukaszMrugala]

Describe the bug
I am trying to use Twister in Pycharm, in order to access easy-to-use debugging on Windows.

I have created a PyCharm Python Script Configuration, which launches a script under the path <ZEPHYR_BASE>\scripts\twister with parameters -T=<ZEPHYR_BASE>\tests\ztest -p=qemu_x86_64 -ll=DEBUG -vv. I've input the working directory as <ZEPHYR_BASE>. Environmental variables ZEPHYR_BASE and ZEPHYR_SDK_INSTALL_DIR are set.

Tests fail with CMake build error. CMake logs indicate Ninja lexer error. Inspecting the Ninja command, we can see the following parameter:

-DWEST_TOPDIR=<WEST_LOCATION>
�[0m 

We can see a newline, which normally would have been removed thanks to the OUTPUT_STRIP_TRAILING_WHITESPACE in west.cmake. However, for some reason, a terminal color resetting character quartet is appended to the result, breaking both the trailing whitespace stripping, as well as the relevant Ninja command. WEST_TOPDIR contains a newline and those characters, wherever it would be accessed.

To Reproduce

1. Setup Zephyr and Twister according to the Getting Started Guide. Twister should run normally in CMD.
2. Create aforementioned PyCharm configuration, taking care to create a proper interpreter, containing relevant requirements.
3. Init and update west above the ZEPHYR_BASE.
4. Try to run the PyCharm configuration

Expected behavior
PyCharm configuration behaves as a CMD run of Twister.

Impact
I cannot debug Twister on Windows via the Pycharm visual debugger.

Logs and console output
Full command from build.ninja:

#############################################
# Custom command for zephyr\llext-edk.tar.xz

build zephyr/llext-edk.tar.xz | ${cmake_ninja_workdir}zephyr/llext-edk.tar.xz: CUSTOM_COMMAND zephyr/zephyr.elf || app/libapp.a zephyr/arch/arch/x86/core/libarch__x86__core.a zephyr/arch/arch/x86/pagetables_bin_target zephyr/arch/arch/x86/pagetables_o zephyr/driver_validation_h_target zephyr/drivers/console/libdrivers__console.a zephyr/drivers/interrupt_controller/libdrivers__interrupt_controller.a zephyr/drivers/serial/libdrivers__serial.a zephyr/drivers/timer/libdrivers__timer.a zephyr/kernel/libkernel.a zephyr/kobj_types_h_target zephyr/lib/libc/common/liblib__libc__common.a zephyr/lib/libc/picolibc/liblib__libc__picolibc.a zephyr/libzephyr.a zephyr/linker_zephyr_final_script_target zephyr/linker_zephyr_prebuilt_script_target zephyr/offsets zephyr/offsets_h zephyr/parse_syscalls_target zephyr/subsys/random/libsubsys__random.a zephyr/subsys/testsuite/ztest/libsubsys__testsuite__ztest.a zephyr/syscall_list_h_target zephyr/version_h zephyr/zephyr.elf zephyr/zephyr_generated_headers zephyr/zephyr_pre0.elf
  COMMAND = cmd.exe /C "cd /D <ZEPHYR_BASE>\twister-out\qemu_x86_64\tests\ztest\zexpect\testing.ztest.expect_cpp\zephyr && "<CMAKE>\bin\cmake.exe" -E make_directory edk/include/generated/zephyr && <ZEPHYR_BASE>\venv\Scripts\python.exe <ZEPHYR_BASE>/scripts/build/gen_syscalls.py --json-file <ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr/misc/generated/syscalls.json --base-output edk/include/generated/zephyr/syscalls --syscall-dispatch edk/include/generated/zephyr/syscall_dispatch.c --syscall-list <ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr/edk/include/generated/zephyr/syscall_list.h --long-registers --split-type k_timeout_t --split-type k_ticks_t && "<CMAKE>\bin\cmake.exe" -DPROJECT_BINARY_DIR=<ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr -DAPPLICATION_SOURCE_DIR=<ZEPHYR_BASE>/tests/ztest/zexpect -DINTERFACE_INCLUDE_DIRECTORIES="<ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr/include/generated/zephyr <ZEPHYR_BASE>/include <ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr/include/generated <ZEPHYR_BASE>/soc/intel/atom <ZEPHYR_BASE>/lib/libc/common/include <ZEPHYR_BASE>/lib/cpp/minimal/include <ZEPHYR_BASE>/soc/intel/atom/. <ZEPHYR_BASE>/subsys/testsuite/include <ZEPHYR_BASE>/subsys/testsuite/coverage <ZEPHYR_BASE>/subsys/testsuite/ztest/include <ZEPHYR_BASE>/subsys/testsuite/include" -Dllext_edk_file=<ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr/llext-edk.tar.xz -Dllext_edk_cflags="-DKERNEL -D__ZEPHYR__=1 -D__LINUX_ERRNO_EXTENSIONS__ -DPICOLIBC_LONG_LONG_PRINTF_SCANF -DTC_RUNID=3d14a94de136562bde815a02e9d7bdfa -DK_HEAP_MEM_POOL_SIZE=0 -DLL_EXTENSION_BUILD -fno-strict-aliasing -Werror -Os -imacros <ZEPHYR_BASE>/twister-out/qemu_x86_64/tests/ztest/zexpect/testing.ztest.expect_cpp/zephyr/include/generated/zephyr/autoconf.h -fno-printf-return-value -fno-common -g -gdwarf-4 -fdiagnostics-color=always -m64 -Wa,--divide --sysroot=<ZEPHYR_SDK_INSTALL_DIR>/x86_64-zephyr-elf/x86_64-zephyr-elf -imacros <ZEPHYR_BASE>/include/zephyr/toolchain/zephyr_stdint.h -Wall -Wformat -Wformat-security -Wformat -Wno-format-zero-length -Wdouble-promotion -Wno-pointer-sign -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -Werror=implicit-int -fno-pic -fno-pie -fno-asynchronous-unwind-tables -ftls-model=local-exec -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=<ZEPHYR_BASE>/tests/ztest/zexpect=CMAKE_SOURCE_DIR -fmacro-prefix-map=<ZEPHYR_BASE>=ZEPHYR_BASE -ffunction-sections -fdata-sections -m64 -mno-red-zone -mno-sse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -mno-sse4a -specs=picolibc.specs -std=c99" -Dllext_edk_name=llext-edk -DWEST_TOPDIR=<WEST_LOCATION>
�[0m -DZEPHYR_BASE=<ZEPHYR_BASE> -DCONFIG_LLEXT_EDK_USERSPACE_ONLY= -P <ZEPHYR_BASE>/cmake/llext-edk.cmake"
  DESC = Generating llext-edk.tar.xz
  restat = 1

Environment (please complete the following information):

• OS: Windows
• Toolchain: Zephyr SDK
• Commit SHA: 82a192c

Additional context
I was able to fix the problem on my machine via addition of quotation marks to the CMake command.

Before (failing):

  execute_process(
    COMMAND ${WEST} topdir
    OUTPUT_VARIABLE WEST_TOPDIR
    ERROR_QUIET
    RESULT_VARIABLE west_topdir_result
    OUTPUT_STRIP_TRAILING_WHITESPACE
    WORKING_DIRECTORY ${ZEPHYR_BASE}
    )

After (passing):

  execute_process(
    COMMAND "${WEST} topdir"
    OUTPUT_VARIABLE WEST_TOPDIR
    ERROR_QUIET
    RESULT_VARIABLE west_topdir_result
    OUTPUT_STRIP_TRAILING_WHITESPACE
    WORKING_DIRECTORY ${ZEPHYR_BASE}
    )

I am unsure whether that's the right approach here.

[nashif]

@pdgendt can you take a look? the suggested change is in cmake/modules/west.cmake

[pdgendt]

@LukaszMrugala can you try with #79988 to see if that helps?

[pdgendt]

Hmn, it does not make sense to have colored output there, which it the root cause, I think.

[pdgendt]

I've updated #79988 for west colorized output, please verify.

[pdgendt]

More input from: tartley/colorama#307

[LukaszMrugala]

@LukaszMrugala can you try with #79988 to see if that helps?

I've run the PR commit locally and the problem seems to persist.

[pdgendt]

I've run the PR commit locally and the problem seems to persist.

Yes, it was later I found the underlying issue:

More input from: tartley/colorama#307

[pdgendt]

@LukaszMrugala can you add something like

unset PYCHARM_HOSTED

to the script you run?

[LukaszMrugala]

PYCHARM_HOSTED

I've added del os.environ['PYCHARM_HOSTED'] to the twister Python script. After doing that, I was able to run the program properly, with a 0 return code.

[pdgendt]

I've added del os.environ['PYCHARM_HOSTED'] to the twister Python script. After doing that, I was able to run the program properly, with a 0 return code.

Great, however I doubt we want to have this in tree, but you should rather integrate it into the pycharm scripts.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.