Iot SDK crashing when statically linked with MUSL toolchain

[joseph-vincent]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've searched for previous similar issues and didn't find any solution
• Note : Issue seems to be similar to Issue compiling the sdk statically against musl aws-sdk-cpp#1312, but no conclusive fix there.

Describe the bug
Unable to run statically linked application with AWS SDK compiled with MUSL toolchain. SDK is throwing error "Unable to load symbols from process space". The same application can be compiled with dynamic linking and works fine.

Platform/OS/Device
Compiling the SDK and application on Alpine 3.13 container with MUSL toolchain http://musl.cc/x86_64-linux-musl-cross.tgz
Running on Ubuntu x86

SDK version number
Tried on v1.12.0 , v1.12.4 and v1.13.0

To Reproduce (observed behavior)
Steps to reproduce the behavior (please share code)

• Get this Repository. https://github.com/joseph-vincent/aws_iot_testclient
• Have Docker installed (mine is 20.10.3, but dont think it matters)
• Get Alpine 3:13 (Again, dont think version matters) and run it.
  • docker run -it --net=host -v Repo:Path alpine:3.13 sh
• Go to path where cloned repo was mounted (Or you can clone it inside itself).
• Run script called ./makeAgent.sh in the root directory within the container.
• Answer 'y' to all questions for first time run. The script is expected to do the following
  • Install dependencies on the base docker
  • Get MUSL toolchain which i am using
  • Clone AWS repo (v1.13.0, but i have tried same with v1.12.0 and v1.12.4, feel free to change the script)
  • build SDK
  • build sample application (code same as basic_pub_sub from samples/mqtt/basic_pub_sub)
  • application 'basic_pub_sub' gets generated in 'build' directory (Should be statically linked, so no dynamic linkage)
  • Run the application on Ubuntu (Mine is 18.04 hosted on VBox) and observe the error on startup (Error below)

Expected behavior
The Application will crash as soon as it starts and give below output.

Logs/output
#:~/Temp/aws_iot_testclient$ ./build/basic_pub_sub
Fatal error condition occurred in /code/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c:488: process && "Unable to load symbols from process space"
Exiting Application
No call stack information available
Aborted (core dumped)

Additional context

• Able to compile this on Ubuntu VM without MUSL and able to run it as well, but with MUSL, it gives same error even on my VM without Alpine container.
• Enabled AWS Logging to TRACE level. But there are no logs.
• Found that aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c:488 is trying to resolve lib crypto version using dlopen even though it is static and unable to do it. So I pasting the find command output if it helps on my Ubuntu VM.

#:~/Temp/aws_iot_testclient$ sudo find / -name libcrypto.so|grep -v docker
/home/devbox/Temp/libcrypto.so
/x86_64-linux-musl-cross/local/lib/libcrypto.so
/usr/local/lib/libcrypto.so
/usr/lib/x86_64-linux-gnu/libcrypto.so

[jimmyt857]

In case anyone else comes across this, I encountered this problem and was able to solve it by setting the CMAKE flag, "BYO_CRYPTO" in aws-c-cal (https://github.com/awslabs/aws-c-cal).

Specifically, add "-DBYO_CRYPTO=ON" to the cmake command for compiling aws-cpp-sdk.

This worked for me when trying to build a full static library that includes aws-cpp-sdk.

[jmklix]

@joseph-vincent have you tried adding the -DBYO_CRYPTO=ON flag? and did that fix the crash you are getting?

[bretambrose]

BYO_CRYPTO is only going to help people that are using the regular AWS SDK for C++ and don't need any CRT functionality. For IoT customers, BYO_CRYPTO requires you to implement a large suite of cryptographic and tls functionality on your own.

[joseph-vincent]

Thanks for the response,
Yes, I tried setting BYO_CRYPTO to ON, with this option the aws-c-iot, s2n and crypto libraries did not get built.
And aws-c-iot library is needed for an MQTT connection.

A dynamic link for libcrypto alone worked for me, with all other SDK libraries statically linked.

But it would be interesting to know if there is a way to static link crypto as well.

[martinkbrown]

I had the same issue. @joseph-vincent 's solution is the only on that worked for me. Remove the -static flag from and add these flags to your main executable's target_link_libraries:

-static-libgcc -static-libstdc++

[jmklix]

@joseph-vincent did this work for you?

Remove the -static flag from and add these flags to your main executable's target_link_libraries:
-static-libgcc -static-libstdc++

[BestITUserEUW]

Hey,
having the same issue here when removing the -static flag, it has an interpreter again:
with -static flag:
myProgram: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, not stripped

arm64-emulator:~$ ./myProgram Fatal error condition occurred in /work/build/third_party/aws-iot-device-sdk-cpp-v2/aws-iot-device-sdk-cpp-v2/src/aws-iot-device-sdk-cpp-v2/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c:488: process && "Unable to load symbols from process space" Exiting Application No call stack information available qemu: uncaught target signal 6 (Aborted) - core dumped
without:
myProgram: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, not stripped

arm64-emulator:~$ ./myProgramm /lib/ld-musl-aarch64.so.1: No such file or directory

[jmklix]

Changing this to a feature request as this sdk does not currently support alpine linux, but this is something that is on our backlog.

[jmklix]

We now support alpine linux and have CI tests to make sure it continues to work. Please let us know if you run into any problems when using this sdk when linked to MUSL toolchains

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.