Not working under HX370 (890M) - Use-After-Free and Zero-Byte Allocations

[youyoulyz]

Model Details

• Name: GPT2
• Link: https://github.com/saqib1707/gpt2-from-scratch
• Framework: PyTorch
• Compute requirements: ~ 4GB memory for batch_size=4

Environment

• APU: HX370
• Chassis Type: AIPC
• RAM: 16G
• VRAM: 512 MB
• Linux Distro: Ubuntu

Results

• Issues/Limitations: The issue appears to be a memory management error in the custom allocator logic, where freed memory is accessed after deallocation, resulting in a GPU page fault. Additionally, there are problematic zero-byte allocations that could lead to undefined behavior. Below, I explain the core issue and suggest an improvement for the custom allocator.

Additional Information

GFX_VERSION=11.0.0
ROCm=6.2.1

RUN script

export LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH;
torchrun ./src/train.py --num_epochs=5 --num_layers=24 --mini_batch_size=2

ERROR LOG

Hyperparameter Configuration:
total_batch_size: 524288
mini_batch_size: 2
context_length: 1024
num_layers: 24
embd_size: 768
num_heads: 12
max_lr: 0.001
min_lr: 0.0001
warmup_steps: 715
weight_decay: 0.1
num_epochs: 5
steps_per_epoch: 19073
eval_freq: 250
seed: 1337
logdir: ./logs/
desired batch size (number of tokens): 524288
gradient accumulation steps: 256
GPU: 0, 0
found 99 shards for split train
found 1 shards for split val
Allocated 154533888 bytes at 0x748d0d800000
Allocated 3145728 bytes at 0x748db5c00000
Allocated 3072 bytes at 0x74919280c000
Allocated 3072 bytes at 0x74919280a000
Allocated 7077888 bytes at 0x748db5400000
Allocated 9216 bytes at 0x749192806000
Allocated 2359296 bytes at 0x748dcc400000
Allocated 3072 bytes at 0x749192804000
Allocated 3072 bytes at 0x749192802000
Allocated 3072 bytes at 0x749191e0e000
Allocated 9437184 bytes at 0x748db4a00000
Allocated 12288 bytes at 0x749191e0a000
Allocated 9437184 bytes at 0x748dbd800000
Allocated 3072 bytes at 0x749191e08000
Allocated 3072 bytes at 0x749191e06000
Allocated 3072 bytes at 0x749191e04000
Allocated 7077888 bytes at 0x748dbd000000
Allocated 9216 bytes at 0x74918f404000
Allocated 2359296 bytes at 0x748dbcc00000
Allocated 3072 bytes at 0x749191e02000
Allocated 3072 bytes at 0x74918f402000
Allocated 3072 bytes at 0x74918ea0a000
Allocated 9437184 bytes at 0x748db4000000
Allocated 12288 bytes at 0x74918ea06000
Allocated 9437184 bytes at 0x748db3600000
Allocated 3072 bytes at 0x74918ea04000
Allocated 3072 bytes at 0x74918ea02000
Allocated 3072 bytes at 0x74918c003000
Allocated 7077888 bytes at 0x748db2e00000
Allocated 9216 bytes at 0x74918b604000
Allocated 2359296 bytes at 0x748dbc800000
Allocated 3072 bytes at 0x74918b602000
Allocated 3072 bytes at 0x749178631000
Allocated 3072 bytes at 0x74917862f000
Allocated 9437184 bytes at 0x748db2400000
Allocated 12288 bytes at 0x7491783bc000
Allocated 9437184 bytes at 0x748db1a00000
Allocated 3072 bytes at 0x74917862d000
Allocated 3072 bytes at 0x74917862b000
Allocated 3072 bytes at 0x7491783ba000
Allocated 7077888 bytes at 0x748db1200000
Allocated 9216 bytes at 0x749176bc8000
Allocated 2359296 bytes at 0x748db0e00000
Allocated 3072 bytes at 0x7491619fe000
Allocated 3072 bytes at 0x7491619fc000
Allocated 3072 bytes at 0x7491619fa000
Allocated 9437184 bytes at 0x748db0400000
Allocated 12288 bytes at 0x7490cbbc8000
Allocated 9437184 bytes at 0x748dafa00000
Allocated 3072 bytes at 0x7491619f8000
Allocated 3072 bytes at 0x7490cbbc6000
Allocated 3072 bytes at 0x7490cbbc4000
Allocated 7077888 bytes at 0x748daf200000
Allocated 9216 bytes at 0x7490cb092000
Allocated 2359296 bytes at 0x748daee00000
Allocated 3072 bytes at 0x7490cb090000
Allocated 3072 bytes at 0x7490c96ae000
Allocated 3072 bytes at 0x7490c96ac000
Allocated 9437184 bytes at 0x748dae400000
Allocated 12288 bytes at 0x7490c96a8000
Allocated 9437184 bytes at 0x748dada00000
Allocated 3072 bytes at 0x7490c96a6000
Allocated 3072 bytes at 0x7490c96a4000
Allocated 3072 bytes at 0x7490c96a2000
Allocated 7077888 bytes at 0x748dad200000
Allocated 9216 bytes at 0x7490941dc000
Allocated 2359296 bytes at 0x748dace00000
Allocated 3072 bytes at 0x7490941da000
Allocated 3072 bytes at 0x7490941d8000
Allocated 3072 bytes at 0x7490361c8000
Allocated 9437184 bytes at 0x748dac400000
Allocated 12288 bytes at 0x7490361c4000
Allocated 9437184 bytes at 0x748daba00000
Allocated 3072 bytes at 0x7490361c2000
Allocated 3072 bytes at 0x7490361c0000
Allocated 3072 bytes at 0x7490361be000
Allocated 7077888 bytes at 0x748dab200000
Allocated 9216 bytes at 0x7490361ba000
Allocated 2359296 bytes at 0x748daae00000
Allocated 3072 bytes at 0x748feb929000
Allocated 3072 bytes at 0x748feb927000
Allocated 3072 bytes at 0x748feb925000
Allocated 9437184 bytes at 0x748daa400000
Allocated 12288 bytes at 0x748feaefc000
Allocated 9437184 bytes at 0x748da9a00000
Allocated 3072 bytes at 0x748feaefa000
Allocated 3072 bytes at 0x748feaef8000
Allocated 3072 bytes at 0x748feaef6000
Allocated 7077888 bytes at 0x748da9200000
Allocated 9216 bytes at 0x748feaef2000
Allocated 2359296 bytes at 0x748da8e00000
Allocated 3072 bytes at 0x748feaef0000
Allocated 3072 bytes at 0x748feaeee000
Allocated 3072 bytes at 0x748f8617b000
Allocated 9437184 bytes at 0x748da8400000
Allocated 12288 bytes at 0x748f86176000
Allocated 9437184 bytes at 0x748da7a00000
Allocated 3072 bytes at 0x748f86174000
Allocated 3072 bytes at 0x748f86172000
Allocated 3072 bytes at 0x748f86170000
Allocated 7077888 bytes at 0x748da7200000
Allocated 9216 bytes at 0x748f8560c000
Allocated 2359296 bytes at 0x748da6e00000
Allocated 3072 bytes at 0x748f8616e000
Allocated 3072 bytes at 0x748f8560a000
Allocated 3072 bytes at 0x748f7c355000
Allocated 9437184 bytes at 0x748da6400000
Allocated 12288 bytes at 0x748f7c344000
Allocated 9437184 bytes at 0x748da5a00000
Allocated 3072 bytes at 0x748f7c342000
Allocated 3072 bytes at 0x748f7c340000
Allocated 3072 bytes at 0x748f7c33e000
Allocated 7077888 bytes at 0x748da5200000
Allocated 9216 bytes at 0x748f7c33a000
Allocated 2359296 bytes at 0x748da4e00000
Allocated 3072 bytes at 0x748f2800a000
Allocated 3072 bytes at 0x748f28008000
Allocated 3072 bytes at 0x748f28006000
Allocated 9437184 bytes at 0x748da4400000
Allocated 12288 bytes at 0x748f27444000
Allocated 9437184 bytes at 0x748da3a00000
Allocated 3072 bytes at 0x748f28004000
Allocated 3072 bytes at 0x748f27afe000
Allocated 3072 bytes at 0x748f27afc000
Allocated 7077888 bytes at 0x748da3200000
Allocated 9216 bytes at 0x748f27440000
Allocated 2359296 bytes at 0x748da2e00000
Allocated 3072 bytes at 0x748f2743e000
Allocated 3072 bytes at 0x748f2743c000
Allocated 3072 bytes at 0x748f2743a000
Allocated 9437184 bytes at 0x748da2400000
Allocated 12288 bytes at 0x748f27436000
Allocated 9437184 bytes at 0x748da1a00000
Allocated 3072 bytes at 0x748f27434000
Allocated 3072 bytes at 0x748f27432000
Allocated 3072 bytes at 0x748f27430000
Allocated 7077888 bytes at 0x748da1200000
Allocated 9216 bytes at 0x748f26e24000
Allocated 2359296 bytes at 0x748da0e00000
Allocated 3072 bytes at 0x748f2742e000
Allocated 3072 bytes at 0x748f26e22000
Allocated 3072 bytes at 0x748f26e20000
Allocated 9437184 bytes at 0x748da0400000
Allocated 12288 bytes at 0x748f26e1c000
Allocated 9437184 bytes at 0x748d9fa00000
Allocated 3072 bytes at 0x748f26e1a000
Allocated 3072 bytes at 0x748f268fe000
Allocated 3072 bytes at 0x748f268fc000
Allocated 7077888 bytes at 0x748d9f200000
Allocated 9216 bytes at 0x748f268f8000
Allocated 2359296 bytes at 0x748d9ee00000
Allocated 3072 bytes at 0x748f268f6000
Allocated 3072 bytes at 0x748f268f4000
Allocated 3072 bytes at 0x748f268f2000
Allocated 9437184 bytes at 0x748d9e400000
Allocated 12288 bytes at 0x748f268ee000
Allocated 9437184 bytes at 0x748d9da00000
Allocated 3072 bytes at 0x748f268ec000
Allocated 3072 bytes at 0x748f268ea000
Allocated 3072 bytes at 0x748f268e8000
Allocated 7077888 bytes at 0x748d9d200000
Allocated 9216 bytes at 0x748f268e4000
Allocated 2359296 bytes at 0x748d9ce00000
Allocated 3072 bytes at 0x748f268e2000
Allocated 3072 bytes at 0x748f268e0000
Allocated 3072 bytes at 0x748f268de000
Allocated 9437184 bytes at 0x748d9c400000
Allocated 12288 bytes at 0x748f268da000
Allocated 9437184 bytes at 0x748d67600000
Allocated 3072 bytes at 0x748f268d8000
Allocated 3072 bytes at 0x748f268d6000
Allocated 3072 bytes at 0x748f268d4000
Allocated 7077888 bytes at 0x748d66e00000
Allocated 9216 bytes at 0x748f268d0000
Allocated 2359296 bytes at 0x748d66a00000
Allocated 3072 bytes at 0x748f268ce000
Allocated 3072 bytes at 0x748f268cc000
Allocated 3072 bytes at 0x748f268ca000
Allocated 9437184 bytes at 0x748d66000000
Allocated 12288 bytes at 0x748f268c6000
Allocated 9437184 bytes at 0x748d65600000
Allocated 3072 bytes at 0x748f268c4000
Allocated 3072 bytes at 0x748f268c2000
Allocated 3072 bytes at 0x748f268c0000
Allocated 7077888 bytes at 0x748d64e00000
Allocated 9216 bytes at 0x748f268bc000
Allocated 2359296 bytes at 0x748d64a00000
Allocated 3072 bytes at 0x748f268ba000
Allocated 3072 bytes at 0x748f268b8000
Allocated 3072 bytes at 0x748f268b6000
Allocated 9437184 bytes at 0x748d5f600000
Allocated 12288 bytes at 0x748f268b2000
Allocated 9437184 bytes at 0x748d5ec00000
Allocated 3072 bytes at 0x748f268b0000
Allocated 3072 bytes at 0x748f268ae000
Allocated 3072 bytes at 0x748f268ac000
Allocated 7077888 bytes at 0x748d5e400000
Allocated 9216 bytes at 0x748f268a8000
Allocated 2359296 bytes at 0x748d64600000
Allocated 3072 bytes at 0x748f268a6000
Allocated 3072 bytes at 0x748f268a4000
Allocated 3072 bytes at 0x748f268a2000
Allocated 9437184 bytes at 0x748d5da00000
Allocated 12288 bytes at 0x748f2689e000
Allocated 9437184 bytes at 0x748d5d000000
Allocated 3072 bytes at 0x748f2689c000
Allocated 3072 bytes at 0x748f2689a000
Allocated 3072 bytes at 0x748f26898000
Allocated 7077888 bytes at 0x748d5c800000
Allocated 9216 bytes at 0x748f26894000
Allocated 2359296 bytes at 0x748d5c400000
Allocated 3072 bytes at 0x748f26892000
Allocated 3072 bytes at 0x748f26890000
Allocated 3072 bytes at 0x748f2688e000
Allocated 9437184 bytes at 0x748d57600000
Allocated 12288 bytes at 0x748f2688a000
Allocated 9437184 bytes at 0x748d56c00000
Allocated 3072 bytes at 0x748f26888000
Allocated 3072 bytes at 0x748f26886000
Allocated 3072 bytes at 0x748f2587f000
Allocated 7077888 bytes at 0x748d56400000
Allocated 9216 bytes at 0x748f2587a000
Allocated 2359296 bytes at 0x748d56000000
Allocated 3072 bytes at 0x748f25878000
Allocated 3072 bytes at 0x748f25876000
Allocated 3072 bytes at 0x748f25874000
Allocated 9437184 bytes at 0x748d55600000
Allocated 12288 bytes at 0x748f25870000
Allocated 9437184 bytes at 0x748d54c00000
Allocated 3072 bytes at 0x748f2586e000
Allocated 3072 bytes at 0x748f2586c000
Allocated 3072 bytes at 0x748f2586a000
Allocated 7077888 bytes at 0x748d54400000
Allocated 9216 bytes at 0x748f25866000
Allocated 2359296 bytes at 0x748d4fc00000
Allocated 3072 bytes at 0x748f25864000
Allocated 3072 bytes at 0x748f25862000
Allocated 3072 bytes at 0x748f25860000
Allocated 9437184 bytes at 0x748d4f200000
Allocated 12288 bytes at 0x748f2585c000
Allocated 9437184 bytes at 0x748d4e800000
Allocated 3072 bytes at 0x748f2585a000
Allocated 3072 bytes at 0x748f25858000
Allocated 3072 bytes at 0x748f25856000
Allocated 7077888 bytes at 0x748d4e000000
Allocated 9216 bytes at 0x748f25852000
Allocated 2359296 bytes at 0x748d4dc00000
Allocated 3072 bytes at 0x748f25850000
Allocated 3072 bytes at 0x748f2584e000
Allocated 3072 bytes at 0x748f2584c000
Allocated 9437184 bytes at 0x748d4d200000
Allocated 12288 bytes at 0x748f25848000
Allocated 9437184 bytes at 0x748d4c800000
Allocated 3072 bytes at 0x748f25846000
Allocated 3072 bytes at 0x748f25844000
Allocated 3072 bytes at 0x748f25842000
Allocated 7077888 bytes at 0x748d47800000
Allocated 9216 bytes at 0x748f2583e000
Allocated 2359296 bytes at 0x748d4c400000
Allocated 3072 bytes at 0x748f2583c000
Allocated 3072 bytes at 0x748f2583a000
Allocated 3072 bytes at 0x748f25838000
Allocated 9437184 bytes at 0x748d46e00000
Allocated 12288 bytes at 0x748f25834000
Allocated 9437184 bytes at 0x748d46400000
Allocated 3072 bytes at 0x748f25832000
Allocated 3072 bytes at 0x748f25830000
Allocated 3072 bytes at 0x748f2582e000
Allocated 7077888 bytes at 0x748d45c00000
Allocated 9216 bytes at 0x748f2582a000
Allocated 2359296 bytes at 0x748d45800000
Allocated 3072 bytes at 0x748f25828000
Allocated 3072 bytes at 0x748f25826000
Allocated 3072 bytes at 0x748f25824000
Allocated 9437184 bytes at 0x748d44e00000
Allocated 12288 bytes at 0x748f25820000
Allocated 9437184 bytes at 0x748d44400000
Allocated 3072 bytes at 0x748f2581e000
Allocated 3072 bytes at 0x748f2581c000
Allocated 3072 bytes at 0x748f2581a000
Allocated 7077888 bytes at 0x748d3f800000
Allocated 9216 bytes at 0x748f25816000
Allocated 2359296 bytes at 0x748d3f400000
Allocated 3072 bytes at 0x748f25814000
Allocated 3072 bytes at 0x748f25812000
Allocated 3072 bytes at 0x748f25810000
Allocated 9437184 bytes at 0x748d3ea00000
Allocated 12288 bytes at 0x748f25214000
Allocated 9437184 bytes at 0x748d3e000000
Allocated 3072 bytes at 0x748f2580e000
Allocated 3072 bytes at 0x748f2580c000
Allocated 3072 bytes at 0x748f25212000
Allocated 8 bytes at 0x748f25210000
Allocated 8 bytes at 0x748f2520e000
Allocated 8 bytes at 0x748f24c34000
Freeing memory at 0x748f24c34000
Allocated 6240 bytes at 0x748f24634000
Freeing memory at 0x748f24634000
Freeing memory at 0x748f2520e000
Freeing memory at 0x748f25210000
Allocated 271082496 bytes at 0x748cafa00000
Allocated 262250496 bytes at 0x748c9fe00000
Freeing memory at 0x748cafa00000
Allocated 266978304 bytes at 0x748cafe00000
Freeing memory at 0x748c9fe00000
Allocated 37810176 bytes at 0x748d1da00000
Freeing memory at 0x748cafe00000
Freeing memory at 0x748d1da00000
Allocated 838121472 bytes at 0x748c8de00000
num decay parameter tensors: 98 with 209,289,216 parameters
num nodecay parameter tensors: 194 with 241,152 parameters
using fused AdamW optimizer: True
Allocated 16384 bytes at 0x748f24630000
Allocated 16384 bytes at 0x748f24628000
Invalid allocation size: 0
Allocated 8192 bytes at 0x748f2520e000
Attempted to free NULL pointer
Invalid allocation size: 0
Allocated 3145728 bytes at 0x748d24800000
Attempted to free NULL pointer
Invalid allocation size: 0
Allocated 6291456 bytes at 0x748d1ec00000
Attempted to free NULL pointer
Allocated 6291456 bytes at 0x748d1e400000
Allocated 6291456 bytes at 0x748d1dc00000
Allocated 8192 bytes at 0x748f24624000
Allocated 8192 bytes at 0x748f24620000
Freeing memory at 0x748f24624000
Freeing memory at 0x748f24620000
Allocated 4608 bytes at 0x748f24624000
Allocated 3538944 bytes at 0x748d1d200000
Allocated 3145728 bytes at 0x748d1ce00000
Allocated 9437184 bytes at 0x748d1c400000
Freeing memory at 0x748d1ce00000
/home/aup/miniconda3/envs/sam2/lib/python3.10/site-packages/torch/nn/modules/linear.py:125: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at ../aten/src/ATen/Context.cpp:296.)
  return F.linear(input, self.weight, self.bias)
Memory access fault by GPU node-1 (Agent handle: 0x3d50b170) on address 0x748d1ce8f000. Reason: Page not present or supervisor privilege.
E0115 12:38:10.179000 35393 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 0 (pid: 35418) of binary: /home/aup/miniconda3/envs/sam2/bin/python
Traceback (most recent call last):
  File "/home/aup/miniconda3/envs/sam2/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/aup/miniconda3/envs/sam2/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/aup/miniconda3/envs/sam2/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/aup/miniconda3/envs/sam2/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/aup/miniconda3/envs/sam2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/aup/miniconda3/envs/sam2/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
======================================================
./src/train.py FAILED
------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-01-15_12:38:10
  host      : aup-EliteMini-Series
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 35418)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 35418
======================================================

ROCm set-up

Package: rocm-libs
Version: 6.2.4.60204-139~24.04
Priority: optional
Section: devel
Maintainer: ROCm Dev Support <rocm-dev.support@amd.com>
Installed-Size: 13.3 kB
Depends: hipblas (= 2.2.0.60204-139~24.04), hipblaslt (= 0.8.0.60204-139~24.04), hipfft (= 1.0.16.60204-139~24.04), hipsolver (= 2.2.0.60204-139~24.04), hipsparse (= 3.1.1.60204-139~24.04), hiptensor (= 1.3.0.60204-139~24.04), miopen-hip (= 3.2.0.60204-139~24.04), half (= 1.12.0.60204-139~24.04), rccl (= 2.20.5.60204-139~24.04), rocalution (= 3.2.1.60204-139~24.04), rocblas (= 4.2.4.60204-139~24.04), rocfft (= 1.0.30.60204-139~24.04), rocrand (= 3.1.1.60204-139~24.04), hiprand (= 2.11.1.60204-139~24.04), rocsolver (= 3.26.2.60204-139~24.04), rocsparse (= 3.2.1.60204-139~24.04), rocm-core (= 6.2.4.60204-139~24.04), hipsparselt (= 0.2.1.60204-139~24.04), composablekernel-dev (= 1.1.0.60204-139~24.04), hipblas-dev (= 2.2.0.60204-139~24.04), hipblaslt-dev (= 0.8.0.60204-139~24.04), hipcub-dev (= 3.2.1.60204-139~24.04), hipfft-dev (= 1.0.16.60204-139~24.04), hipsolver-dev (= 2.2.0.60204-139~24.04), hipsparse-dev (= 3.1.1.60204-139~24.04), hiptensor-dev (= 1.3.0.60204-139~24.04), miopen-hip-dev (= 3.2.0.60204-139~24.04), rccl-dev (= 2.20.5.60204-139~24.04), rocalution-dev (= 3.2.1.60204-139~24.04), rocblas-dev (= 4.2.4.60204-139~24.04), rocfft-dev (= 1.0.30.60204-139~24.04), rocprim-dev (= 3.2.2.60204-139~24.04), rocrand-dev (= 3.1.1.60204-139~24.04), hiprand-dev (= 2.11.1.60204-139~24.04), rocsolver-dev (= 3.26.2.60204-139~24.04), rocsparse-dev (= 3.2.1.60204-139~24.04), rocthrust-dev (= 3.1.1.60204-139~24.04), rocwmma-dev (= 1.5.0.60204-139~24.04), hipsparselt-dev (= 0.2.1.60204-139~24.04)
Homepage: https://github.com/RadeonOpenCompute/ROCm
Download-Size: 1,060 B
APT-Sources: https://repo.radeon.com/rocm/apt/6.2.4 noble/main amd64 Packages
Description: Radeon Open Compute (ROCm) Runtime software stack

[pomoke]

Try https://github.com/segurac/force-host-alloction-APU, or other patches to PyTorch. The patch does not sense memory usage.