New: Cling interactive C++ REPL #5937

notinaboat · 2022-12-01T06:26:52Z

Cling is an interactive C++ interpreter built on the top of LLVM and Clang libraries

Building Cling requires CERN's patched versions of llvm and clang.

https://root.cern/cling
https://root.cern/cling/cling_build_instructions/
https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/cling.rb

giordano · 2022-12-01T11:19:07Z

C/Cling/build_tarballs.jl

+mv cling/ src/tools/
+mkdir build
+cd build/
+cmake -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} -DCMAKE_BUILD_TYPE=Release LLVM_BUILD_TOOLS=OFF ../src


Did you mean

Suggested change

cmake -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} -DCMAKE_BUILD_TYPE=Release LLVM_BUILD_TOOLS=OFF ../src

cmake -DCMAKE_INSTALL_PREFIX=$prefix \

-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} \

-DCMAKE_BUILD_TYPE=Release \

-DLLVM_BUILD_TOOLS=OFF \

../src

(note the -D before LLVM_BUILD_TOOLS)?

Yes.
(It builds for me anyway under Docker on intel macOS)

I've already changed things a bit on my local branch because of this:

Yggdrasil/M/Metal_LLVM_Tools/build_tarballs.jl

Line 35 in 86a4221

# The very first thing we need to do is to build llvm-tblgen for x86_64-linux-muslc

I currently have something like this (still a work in progress)

# The very first thing we need to do is to build llvm-tblgen for x86_64-linux-muslc # This is because LLVM's cross-compile setup is kind of borked, so we just # build the tools natively ourselves, directly. :/ # See: https://github.com/JuliaPackaging/Yggdrasil/blob/86a4221f71ee8ca5c0fd1056829a1fe2b28c879b/M/Metal_LLVM_Tools/build_tarballs.jl#L35 # Build llvm-tblgen and llvm-config mkdir ${WORKSPACE}/bootstrap pushd ${WORKSPACE}/bootstrap CMAKE_FLAGS=() CMAKE_FLAGS+=(-DLLVM_TARGETS_TO_BUILD:STRING=host) CMAKE_FLAGS+=(-DLLVM_HOST_TRIPLE=${MACHTYPE}) CMAKE_FLAGS+=(-DCMAKE_BUILD_TYPE=Release) CMAKE_FLAGS+=(-DLLVM_ENABLE_PROJECTS='llvm') CMAKE_FLAGS+=(-DCMAKE_CROSSCOMPILING=False) CMAKE_FLAGS+=(-DCMAKE_TOOLCHAIN_FILE=${CMAKE_HOST_TOOLCHAIN}) cmake -GNinja ${LLVM_SRCDIR} ${CMAKE_FLAGS[@]} ninja -j${nproc} llvm-tblgen llvm-config popd # Let's do the actual build within the `build` subdirectory mkdir ${WORKSPACE}/build && cd ${WORKSPACE}/build CMAKE_FLAGS=() # Tell LLVM where our pre-built tblgen tools are CMAKE_FLAGS+=(-DLLVM_TABLEGEN=${WORKSPACE}/bootstrap/bin/llvm-tblgen) CMAKE_FLAGS+=(-DLLVM_CONFIG_PATH=${WORKSPACE}/bootstrap/bin/llvm-config) # Install things into $prefix CMAKE_FLAGS+=(-DCMAKE_INSTALL_PREFIX=${prefix}) # Explicitly use our cmake toolchain file and tell CMake we're cross-compiling CMAKE_FLAGS+=(-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN}) CMAKE_FLAGS+=(-DCMAKE_CROSSCOMPILING:BOOL=ON) # Release build for best performance CMAKE_FLAGS+=(-DCMAKE_BUILD_TYPE=Release) CMAKE_FLAGS+=(-DLLVM_TARGETS_TO_BUILD="host;NVPTX") CMAKE_FLAGS+=(-DLLVM_BUILD_TOOLS=OFF) cmake -GNinja ${LLVM_SRCDIR} ${CMAKE_FLAGS[@]} ninja -j${nproc} \ tools/cling/install \ install-clang-resource-headers

CMAKE_FLAGS+=(-DCMAKE_CROSSCOMPILING=False)

looks like a blatant lie.

Edit: oh, that's only for the host tools? But with ${CMAKE_HOST_TOOLCHAIN} CMAKE_CROSSCOMPILING should already be true, isn't it?

I'm copying this stuff from ...

here:

Yggdrasil/L/LLVM/common.jl

Lines 72 to 74 in 73a24b8

# The very first thing we need to do is to build llvm-tblgen for x86_64-linux-muslc

# This is because LLVM's cross-compile setup is kind of borked, so we just

# build the tools natively ourselves, directly. :/

and here:

Yggdrasil/M/Metal_LLVM_Tools/build_tarballs.jl

Lines 35 to 37 in 86a4221

# The very first thing we need to do is to build llvm-tblgen for x86_64-linux-muslc

# This is because LLVM's cross-compile setup is kind of borked, so we just

# build the tools natively ourselves, directly. :/

It seems like cross compilation is broken for llvm, so these work-arounds are needed.

More than LLVM specifically being broken, the problem is that as far as I know in CMake there is no way to pass both host and target toolchains at the same time, like more sensible build systems allow to do, so you have to manually build host tools separately. I still think manually setting CMAKE_CROSSCOMPILING is unnecessary though.

giordano · 2022-12-01T11:19:27Z

C/Cling/build_tarballs.jl

+cd tools/cling/
+make install
+cd ../..
+make install-clang-resource-headers


Suggested change

cd tools/cling/

make install

cd ../..

make install-clang-resource-headers

make -C tools/cling/ install

make install-clang-resource-headers

vchuravy · 2022-12-01T12:26:31Z

Big fan of Cling, but can I ask for the reason why you want to include it in Yggdrasil?

notinaboat · 2022-12-01T12:34:36Z

Big fan of Cling, but can I ask for the reason why you want to include it in Yggdrasil?

@vchuravy, I'm using it here to resolve #define values while wrapping system C-library headers files.

Some more explanation is here: JuliaInterop/Clang.jl#309 (comment)

notinaboat · 2022-12-01T12:54:18Z

CI says "Not exactly 2 tarballs? This isn't right!"

@giordano do you understand what this means ?

[ Info: /agent/_work/1/s/C/Cling/build/aarch64-linux-gnu/O5nkyvXJ/aarch64-linux-gnu-libgfortran5-cxx11/destdir/lib/libcling.so matches our search criteria of libcling
[ Info: /agent/_work/1/s/C/Cling/build/aarch64-linux-gnu/O5nkyvXJ/aarch64-linux-gnu-libgfortran5-cxx11/destdir/bin/cling matches our search criteria of ["cling"]
[ Info: Compressing files in /agent/_work/1/s/C/Cling/build/aarch64-linux-gnu/O5nkyvXJ/aarch64-linux-gnu-libgfortran5-cxx11/destdir/logs/cling
[ Info: Tree hash of contents of cling.v0.9.0.aarch64-linux-gnu.tar.gz: 0bad315d25ef0f23e44dc724bcd2125aa6b144da
[ Info: SHA256 of cling.v0.9.0.aarch64-linux-gnu.tar.gz: 7c2c83d7efe8285ac8dd96deb829fc62d636431d15e46d4a910171d925a099ec
[ Info: Tree hash of contents of cling-logs.v0.9.0.aarch64-linux-gnu.tar.gz: a3da66b4ee8dfa48ce019bca7d5077cb32b5ce17
[ Info: SHA256 of cling-logs.v0.9.0.aarch64-linux-gnu.tar.gz: 18641523641c657a9151f7ebae9c6194395a2d6fafaa9de6ba4be2af2dcd6222
[ Info: Timings: setup: 21.55s, build: 17m 3.08s, audit: 6m 4.2s, packaging: 4.2s
Not exactly 2 tarballs?  This isn't right!

giordano · 2022-12-01T12:56:14Z

That comes from

Yggdrasil/azure-pipelines.yml

Line 299 in 8d0c4b6

echo "Not exactly 2 tarballs? This isn't right!" >&2

Quite frankly, I believe this is the first time I see this error into the wild, looks like there are too many tarballs in the products/ directory.

notinaboat · 2022-12-01T13:12:40Z

too many ...

Or maybe not enough...
It could be that $NAME in the shell script is Cling but name in the julia script is cling -- case mismatch.

notinaboat · 2022-12-01T21:23:46Z

[13:50:58] Undefined symbols for architecture x86_64:
[13:50:58]   "___isPlatformVersionAtLeast", referenced from:
[13:50:58]       __ZN4llvm3sys2fs9copy_fileERKNS_5TwineES4_ in libLLVMSupport.a(Path.cpp.o)

Full logs

Related to this ? libsdl-org/SDL#6491

Looks like this is caused by @available in ObjC, or __builtin_available() in C/C++
https://developer.apple.com/forums/thread/123003?answerId=383166022#383166022
https://clang.llvm.org/docs/LanguageExtensions.html#objective-c-available

The copy_file function that causes the problem is in ./src/lib/Support/Unix/Path.inc here:
https://github.com/llvm/llvm-project/blob/llvmorg-9.0.1/llvm/lib/Support/Unix/Path.inc#L1204

Maybe something like this would help:

Yggdrasil/L/LLVM/common.jl

Lines 220 to 222 in 73a24b8

    
               # On OSX, we need to override LLVM's looking around for our SDK 
        
               CMAKE_FLAGS+=(-DDARWIN_macosx_CACHED_SYSROOT:STRING=/opt/${target}/${target}/sys-root) 
        
               CMAKE_FLAGS+=(-DDARWIN_macosx_OVERRIDE_SDK_VERSION:STRING=10.8)

giordano · 2022-12-01T21:28:41Z

Yggdrasil/S/SDL2/build_tarballs.jl

Lines 26 to 28 in 359cb6f

    
               # Link to libclang_rt.osx to resolve the symbol `___isPlatformVersionAtLeast`: 
        
               # <https://github.com/libsdl-org/SDL/issues/6491>. 
        
               export LDFLAGS="-L${libdir}/darwin -lclang_rt.osx"

built on the top of LLVM and Clang libraries https://root.cern/cling https://root.cern/cling/cling_build_instructions/

notinaboat · 2022-12-02T12:28:01Z

Seems to be working now (tested using tarballs downloaded from CI).

Examples of using cling to resolve non-trivial #define values on macOS, and on Linux...

% uname -a
Darwin Sams-2019-Macbook.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64

% tar xzf ../x86_64-apple-darwin.tar.gz

% grep TCP_MAXOLEN `xcrun --show-sdk-path`/usr/include/netinet/tcp.h
#define TCP_MAXOLEN     (TCP_MAXHLEN - sizeof(struct tcphdr))

% ./bin/cling -isystem`xcrun --show-sdk-path`/usr/include --nologo                                           
[cling]$ #include <netinet/tcp.h>
[cling]$ TCP_MAXOLEN
(unsigned long) 40

$ uname -a
Linux debian 6.1.0-0-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1~rc6-1~exp1 (2022-11-26) x86_64 GNU/Linux

$ tar xzf ../x86_64-linux-gnu.tar.gz

$ grep ICMP_ADVLENMIN /usr/include/netinet/ip_icmp.h 
#define	ICMP_ADVLENMIN	(8 + sizeof (struct ip) + 8)	/* min */

$ ./bin/cling --nologo
[cling]$ #include <netinet/ip_icmp.h>
[cling]$ ICMP_ADVLENMIN
(unsigned long) 36

[cling]$ #include <stdio.h>
[cling]$ printf("Hello World!\n");
Hello World!

Here's another one...
What is the value of LPSETTIMEOUT ??

#define LPSETTIMEOUT_OLD 0x060f /* set parport timeout */
#define LPSETTIMEOUT_NEW \
        _IOW(0x6, 0xf, __s64[2]) /* set parport timeout */
#if __BITS_PER_LONG == 64
#define LPSETTIMEOUT LPSETTIMEOUT_OLD
#else
#define LPSETTIMEOUT (sizeof(time_t) > sizeof(__kernel_long_t) ? \
        LPSETTIMEOUT_NEW : LPSETTIMEOUT_OLD)
#endif

[cling]$ #include <linux/lp.h>
[cling]$ LPSETTIMEOUT
(int) 1551

giordano · 2022-12-03T14:52:45Z

C/Cling/build_tarballs.jl

+    Platform("x86_64", "macOS"),
+    Platform("aarch64", "linux"; libc = "glibc")


Can maybe also try aarch64 darwin?

Suggested change

Platform("x86_64", "macOS"),

Platform("aarch64", "linux"; libc = "glibc")

Platform("aarch64", "linux"; libc = "glibc"),

Platform("x86_64", "macOS"),

Platform("aarch64", "macOS"),

And what about other platforms?

C/Cling/build_tarballs.jl

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

giordano

I should have noticed this before

C/Cling/build_tarballs.jl

giordano · 2022-12-05T00:52:18Z

Thanks for your contribution! I'm curious to see how this works out 😄

notinaboat · 2022-12-05T02:02:59Z

Thanks for your contribution! I'm curious to see how this works out 😄

Thanks for your assistance @giordano

giordano reviewed Dec 1, 2022

View reviewed changes

Cling is an interactive C++ interpreter,

12d2836

built on the top of LLVM and Clang libraries https://root.cern/cling https://root.cern/cling/cling_build_instructions/

notinaboat force-pushed the Cling branch from 5f07b50 to 12d2836 Compare December 2, 2022 10:47

notinaboat marked this pull request as ready for review December 3, 2022 10:41

giordano reviewed Dec 3, 2022

View reviewed changes

notinaboat added 2 commits December 4, 2022 06:57

try building for other platforms

3d63a4f

disable platforms where build failed

113b1bc

giordano reviewed Dec 4, 2022

View reviewed changes

C/Cling/build_tarballs.jl Show resolved Hide resolved

expand_cxxstring_abis(platforms)

d85603a

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

giordano reviewed Dec 4, 2022

View reviewed changes

C/Cling/build_tarballs.jl Outdated Show resolved Hide resolved

C/Cling/build_tarballs.jl Outdated Show resolved Hide resolved

giordano added 3 commits December 5, 2022 00:39

Apply suggestions from code review

df1c2e8

Update build_tarballs.jl

ad34b33

Update build_tarballs.jl

f69b4eb

giordano enabled auto-merge (squash) December 5, 2022 00:52

giordano merged commit 2aec942 into JuliaPackaging:master Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New: Cling interactive C++ REPL #5937

New: Cling interactive C++ REPL #5937

notinaboat commented Dec 1, 2022

giordano Dec 1, 2022

notinaboat Dec 1, 2022

giordano Dec 1, 2022 •

edited

Loading

notinaboat Dec 1, 2022

giordano Dec 1, 2022

giordano Dec 1, 2022

vchuravy commented Dec 1, 2022

notinaboat commented Dec 1, 2022

notinaboat commented Dec 1, 2022

giordano commented Dec 1, 2022

notinaboat commented Dec 1, 2022 •

edited

Loading

notinaboat commented Dec 1, 2022

giordano commented Dec 1, 2022

notinaboat commented Dec 2, 2022 •

edited

Loading

giordano Dec 3, 2022 •

edited

Loading

giordano left a comment

giordano commented Dec 5, 2022

notinaboat commented Dec 5, 2022

-cmake -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} -DCMAKE_BUILD_TYPE=Release LLVM_BUILD_TOOLS=OFF ../src
+cmake -DCMAKE_INSTALL_PREFIX=$prefix \
+    -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLVM_BUILD_TOOLS=OFF \
+    ../src

	# The very first thing we need to do is to build llvm-tblgen for x86_64-linux-muslc
	# This is because LLVM's cross-compile setup is kind of borked, so we just
	# build the tools natively ourselves, directly. :/

		Platform("x86_64", "macOS"),
		Platform("aarch64", "linux"; libc = "glibc")

New: Cling interactive C++ REPL #5937

New: Cling interactive C++ REPL #5937

Conversation

notinaboat commented Dec 1, 2022

giordano Dec 1, 2022

Choose a reason for hiding this comment

notinaboat Dec 1, 2022

Choose a reason for hiding this comment

giordano Dec 1, 2022 • edited Loading

Choose a reason for hiding this comment

notinaboat Dec 1, 2022

Choose a reason for hiding this comment

giordano Dec 1, 2022

Choose a reason for hiding this comment

giordano Dec 1, 2022

Choose a reason for hiding this comment

vchuravy commented Dec 1, 2022

notinaboat commented Dec 1, 2022

notinaboat commented Dec 1, 2022

giordano commented Dec 1, 2022

notinaboat commented Dec 1, 2022 • edited Loading

notinaboat commented Dec 1, 2022

giordano commented Dec 1, 2022

notinaboat commented Dec 2, 2022 • edited Loading

giordano Dec 3, 2022 • edited Loading

Choose a reason for hiding this comment

giordano left a comment

Choose a reason for hiding this comment

giordano commented Dec 5, 2022

notinaboat commented Dec 5, 2022

giordano Dec 1, 2022 •

edited

Loading

notinaboat commented Dec 1, 2022 •

edited

Loading

notinaboat commented Dec 2, 2022 •

edited

Loading

giordano Dec 3, 2022 •

edited

Loading