ASAN issue with margo-elasticity unit test

[mdorier]

I got a random ASAN error: heap use after free, in the margo-elasticity unit test in the github workflow, so I'm going to report it here for when I have time to look at it more closely:

/margo/add_pool_external             [ ERROR ]
=================================================================
Error:  Could not add external pool "my_pool2": pool already registered
Error:  Could not add external pool: name ("my_pool") already used
==17989==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000000148 at pc 0x7f37830d57b3 bp 0x7f377dafff10 sp 0x7f377dafff00
READ of size 8 at 0x60d000000148 thread T1
    #0 0x7f37830d57b2 in __margo_hg_progress_fn src/margo-core.c:2039
    #1 0x7f3783cd522b in ABTD_ythread_func_wrapper (/home/runner/work/mochi-margo/mochi-margo/tests/.spack-env/view/lib/libabt.so.1+0x2222b)
    #2 0x7f3783cdbb8e in ABTD_ythread_context_func_wrapper (/home/runner/work/mochi-margo/mochi-margo/tests/.spack-env/view/lib/libabt.so.1+0x28b8e)

0x60d000000148 is located 56 bytes inside of 144-byte region [0x60d000000110,0x60d0000001a0)
freed by thread T0 here:
    #0 0x7f37832b4c38 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x7f37831060fa in __margo_abt_add_external_pool src/margo-abt-config.c:1366
    #2 0x7f37830e8bf9 in margo_add_pool_external src/margo-config.c:399
    #3 0x5635a16d5732 in add_pool_external tests/unit-tests/margo-elasticity.c:99
    #4 0x5635a16cd304 in munit_test_runner_exec tests/unit-tests/munit/munit.c:1195
    #5 0x5635a16cdf19 in munit_test_runner_run_test_with_params tests/unit-tests/munit/munit.c:1356
    #6 0x5635a16cfac7 in munit_test_runner_run_test tests/unit-tests/munit/munit.c:1584
    #7 0x5635a16d0808 in munit_test_runner_run_suite tests/unit-tests/munit/munit.c:1677
    #8 0x5635a16d0ad4 in munit_test_runner_run tests/unit-tests/munit/munit.c:1696
    #9 0x5635a16d3ad3 in munit_suite_main_custom tests/unit-tests/munit/munit.c:2026
    #10 0x5635a16d3fb5 in munit_suite_main tests/unit-tests/munit/munit.c:2054
    #11 0x5635a16dd6a5 in main tests/unit-tests/margo-elasticity.c:673
    #12 0x7f3782e29d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

previously allocated by thread T0 here:
    #0 0x7f37832b4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154
    #1 0x7f3783103694 in __margo_abt_init_from_json src/margo-abt-config.c:1015
    #2 0x7f3783143ed1 in margo_init_ext src/margo-init.c:118
    #3 0x7f37830c4a10 in margo_init src/margo-core.c:77
    #4 0x5635a16d5501 in add_pool_external tests/unit-tests/margo-elasticity.c:87
    #5 0x5635a16cd304 in munit_test_runner_exec tests/unit-tests/munit/munit.c:1195
    #6 0x5635a16cdf19 in munit_test_runner_run_test_with_params tests/unit-tests/munit/munit.c:1356
    #7 0x5635a16cfac7 in munit_test_runner_run_test tests/unit-tests/munit/munit.c:1584
    #8 0x5635a16d0808 in munit_test_runner_run_suite tests/unit-tests/munit/munit.c:1677
    #9 0x5635a16d0ad4 in munit_test_runner_run tests/unit-tests/munit/munit.c:1696
    #10 0x5635a16d3ad3 in munit_suite_main_custom tests/unit-tests/munit/munit.c:2026
    #11 0x5635a16d3fb5 in munit_suite_main tests/unit-tests/munit/munit.c:2054
    #12 0x5635a16dd6a5 in main tests/unit-tests/margo-elasticity.c:673
    #13 0x7f3782e29d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

Thread T1 created by T0 here:
    #0 0x7f3783258685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x7f3783cd4d45 in ABTD_xstream_context_create (/home/runner/work/mochi-margo/mochi-margo/tests/.spack-env/view/lib/libabt.so.1+0x21d45)

SUMMARY: AddressSanitizer: heap-use-after-free src/margo-core.c:2039 in __margo_hg_progress_fn
Shadow bytes around the buggy address:
  0x0c1a7fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a7fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a7fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a7fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1a7fff8010: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
=>0x0c1a7fff8020: fa fa fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd
  0x0c1a7fff8030: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==17989==ABORTING
Error: child killed by signal 6 (Aborted)
/margo/remove_pool                   [ OK    ] [ 0.41904248 / 0.02408127 CPU ]
/margo/add_xstream_from_json         [ OK    ] [ 0.32250276 / 0.02720329 CPU ]
/margo/add_xstream_external          [ OK    ] [ 0.31920779 / 0.33086546 CPU ]
/margo/remove_xstream                [ OK    ] [ 0.41901846 / 0.02351847 CPU ]
5 of 6 (83%) tests successful, 0 (0%) test skipped.
FAIL tests/unit-tests/margo-elasticity (exit status: 1)
...
/margo/add_pool_external             [ ERROR ]
=================================================================
Error:  Could not add external pool "my_pool2": pool already registered
Error:  Could not add external pool: name ("my_pool") already used
==17989==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d000000148 at pc 0x7f37830d57b3 bp 0x7f377dafff10 sp 0x7f377dafff00
READ of size 8 at 0x60d000000148 thread T1
    #0 0x7f37830d57b2 in __margo_hg_progress_fn src/margo-core.c:2039
    #1 0x7f3783cd522b in ABTD_ythread_func_wrapper (/home/runner/work/mochi-margo/mochi-margo/tests/.spack-env/view/lib/libabt.so.1+0x2222b)
    #2 0x7f3783cdbb8e in ABTD_ythread_context_func_wrapper (/home/runner/work/mochi-margo/mochi-margo/tests/.spack-env/view/lib/libabt.so.1+0x28b8e)

0x60d000000148 is located 56 bytes inside of 144-byte region [0x60d000000110,0x60d0000001a0)
freed by thread T0 here:
    #0 0x7f37832b4c38 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x7f37831060fa in __margo_abt_add_external_pool src/margo-abt-config.c:1366
    #2 0x7f37830e8bf9 in margo_add_pool_external src/margo-config.c:399
    #3 0x5635a16d5732 in add_pool_external tests/unit-tests/margo-elasticity.c:99
    #4 0x5635a16cd304 in munit_test_runner_exec tests/unit-tests/munit/munit.c:1195
    #5 0x5635a16cdf19 in munit_test_runner_run_test_with_params tests/unit-tests/munit/munit.c:1356
    #6 0x5635a16cfac7 in munit_test_runner_run_test tests/unit-tests/munit/munit.c:1584
    #7 0x5635a16d0808 in munit_test_runner_run_suite tests/unit-tests/munit/munit.c:1677
    #8 0x5635a16d0ad4 in munit_test_runner_run tests/unit-tests/munit/munit.c:1696
    #9 0x5635a16d3ad3 in munit_suite_main_custom tests/unit-tests/munit/munit.c:2026
    #10 0x5635a16d3fb5 in munit_suite_main tests/unit-tests/munit/munit.c:2054
    #11 0x5635a16dd6a5 in main tests/unit-tests/margo-elasticity.c:673
    #12 0x7f3782e29d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

previously allocated by thread T0 here:
    #0 0x7f37832b4a57 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154
    #1 0x7f3783103694 in __margo_abt_init_from_json src/margo-abt-config.c:1015
    #2 0x7f3783143ed1 in margo_init_ext src/margo-init.c:118
    #3 0x7f37830c4a10 in margo_init src/margo-core.c:77
    #4 0x5635a16d5501 in add_pool_external tests/unit-tests/margo-elasticity.c:87
    #5 0x5635a16cd304 in munit_test_runner_exec tests/unit-tests/munit/munit.c:1195
    #6 0x5635a16cdf19 in munit_test_runner_run_test_with_params tests/unit-tests/munit/munit.c:1356
    #7 0x5635a16cfac7 in munit_test_runner_run_test tests/unit-tests/munit/munit.c:1584
    #8 0x5635a16d0808 in munit_test_runner_run_suite tests/unit-tests/munit/munit.c:1677
    #9 0x5635a16d0ad4 in munit_test_runner_run tests/unit-tests/munit/munit.c:1696
    #10 0x5635a16d3ad3 in munit_suite_main_custom tests/unit-tests/munit/munit.c:2026
    #11 0x5635a16d3fb5 in munit_suite_main tests/unit-tests/munit/munit.c:2054
    #12 0x5635a16dd6a5 in main tests/unit-tests/margo-elasticity.c:673
    #13 0x7f3782e29d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

Thread T1 created by T0 here:
    #0 0x7f3783258685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x7f3783cd4d45 in ABTD_xstream_context_create (/home/runner/work/mochi-margo/mochi-margo/tests/.spack-env/view/lib/libabt.so.1+0x21d45)

SUMMARY: AddressSanitizer: heap-use-after-free src/margo-core.c:2039 in __margo_hg_progress_fn
Shadow bytes around the buggy address:
  0x0c1a7fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a7fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a7fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a7fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1a7fff8010: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
=>0x0c1a7fff8020: fa fa fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd
  0x0c1a7fff8030: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1a7fff8070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==17989==ABORTING
Error: child killed by signal 6 (Aborted)