Test AMD RX 580 [rk3588]

[System64fumo]

Hello, This is most likely the wrong place to document this so feel free to close the issue.

I've just purchased an RX 580 from ali express to test on my rk3588 board

The card seems to work fine but it's complaining about bar allocations.

dmesg:

...
[    5.682638] [drm] amdgpu kernel modesetting enabled.
[    5.683359] amdgpu 0000:01:00.0: enabling device (0000 -> 0003)
[    5.683896] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x6FDF 0x1002:0x0B31 0xEF).
[    5.684676] [drm] register mmio base: 0xF0200000
[    5.685083] [drm] register mmio size: 262144
[    5.685670] [drm] add ip block number 0 <vi_common>
[    5.686105] [drm] add ip block number 1 <gmc_v8_0>
[    5.686523] [drm] add ip block number 2 <tonga_ih>
[    5.686945] [drm] add ip block number 3 <gfx_v8_0>
[    5.687363] [drm] add ip block number 4 <sdma_v3_0>
[    5.687788] [drm] add ip block number 5 <powerplay>
[    5.688216] [drm] add ip block number 6 <dm>
[    5.688588] [drm] add ip block number 7 <uvd_v6_0>
[    5.689008] [drm] add ip block number 8 <vce_v3_0>
[    5.909768] amdgpu 0000:01:00.0: amdgpu: Fetched VBIOS from ROM BAR
[    5.910385] amdgpu: ATOM BIOS: xxx-xxx-xxx
[    5.937496] [drm] UVD is enabled in VM mode
[    5.937877] [drm] UVD ENC is enabled in VM mode
[    5.938273] [drm] VCE enabled in VM mode
[    5.938616] amdgpu 0000:01:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[    5.939324] amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
[    5.939921] [drm] GPU posting now...
[    6.049420] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[    6.053400] amdgpu 0000:01:00.0: BAR 2 [??? 0x00000000 flags 0x0]: releasing
[    6.053407] amdgpu 0000:01:00.0: BAR 0 [??? 0x00000000 flags 0x0]: releasing
[    6.053410] [drm:amdgpu_device_resize_fb_bar [amdgpu]] *ERROR* Problem resizing BAR0 (-16).
[    6.053757] [drm:amdgpu_device_init [amdgpu]] *ERROR* sw_init of IP block <gmc_v8_0> failed -19
[    6.054067] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_init failed
[    6.054071] amdgpu 0000:01:00.0: amdgpu: Fatal error during GPU init
[    6.054073] amdgpu 0000:01:00.0: amdgpu: amdgpu: finishing device.

lspci -v:

...
0000:01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Polaris 20 XL [Radeon RX 580 2048SP] (rev ef) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b31
	Flags: fast devsel, IRQ 110
	I/O ports at 200000 [size=256]
	Memory at f0200000 (32-bit, non-prefetchable) [disabled] [size=256K]
	Expansion ROM at f0240000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Legacy Endpoint, IntMsgNum 0
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] Physical Resizable BAR
	Capabilities: [270] Secondary PCI Express
	Capabilities: [2b0] Address Translation Service (ATS)
	Capabilities: [2c0] Page Request Interface (PRI)
	Capabilities: [2d0] Process Address Space ID (PASID)
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [370] L1 PM Substates
	Kernel modules: amdgpu
...

After attempting to increase the allocations in the device tree it still shows the same errors, I'm likely not changing it properly, Guidance would be appreciated.
This was done on an Orange Pi 5 Plus running linux 6.13-rc0 with some minor patches for my board.

[System64fumo]

Since jeff isn't quite a fan of rockchip shenanigans i'd be happy to provide a kernel (and modules) if requested.

[System64fumo]

Update: Got the bar space figured out, It now kernel panics

...
[    3.866120] [drm] amdgpu kernel modesetting enabled.
[    3.866842] amdgpu 0000:01:00.0: enabling device (0000 -> 0003)
[    3.867374] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x6FDF 0x1002:0x0B31 0xEF).
[    3.868159] [drm] register mmio base: 0xF0200000
[    3.868563] [drm] register mmio size: 262144
[    3.869101] [drm] add ip block number 0 <vi_common>
[    3.869528] [drm] add ip block number 1 <gmc_v8_0>
[    3.869953] [drm] add ip block number 2 <tonga_ih>
[    3.870371] [drm] add ip block number 3 <gfx_v8_0>
[    3.870792] [drm] add ip block number 4 <sdma_v3_0>
[    3.871217] [drm] add ip block number 5 <powerplay>
[    3.871642] [drm] add ip block number 6 <dm>
[    3.872018] [drm] add ip block number 7 <uvd_v6_0>
[    3.872436] [drm] add ip block number 8 <vce_v3_0>
[    4.093198] amdgpu 0000:01:00.0: amdgpu: Fetched VBIOS from ROM BAR
[    4.093747] amdgpu: ATOM BIOS: xxx-xxx-xxx
[    4.113422] [drm] UVD is enabled in VM mode
[    4.113794] [drm] UVD ENC is enabled in VM mode
[    4.114190] [drm] VCE enabled in VM mode
[    4.114534] amdgpu 0000:01:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[    4.115241] amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
[    4.115838] [drm] GPU posting now...
[    4.225266] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[    4.230331] amdgpu 0000:01:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[    4.231172] amdgpu 0000:01:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[    4.231902] [drm] Detected VRAM RAM=8192M, BAR=0M
[    4.232311] [drm] RAM width 256bits GDDR5
[    4.232986] [drm] amdgpu: 8192M of VRAM memory ready
[    4.233423] [drm] amdgpu: 7905M of GTT memory ready.
[    4.233895] [drm] GART: num cpu pages 65536, num gpu pages 65536
[    4.234431] ------------[ cut here ]------------
[    4.234832] WARNING: CPU: 4 PID: 227 at arch/arm64/mm/ioremap.c:28 ioremap_prot+0x90/0xcc
[    4.234839] Modules linked in: amdgpu(+) amdxcp drm_buddy drm_suballoc_helper i2c_algo_bit drm_ttm_helper ttm
[    4.234848] CPU: 4 UID: 0 PID: 227 Comm: modprobe Not tainted 6.13.0-rc1-baka+ #10
[    4.234851] Hardware name: Xunlong Orange Pi 5 Plus (DT)
[    4.234853] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    4.234856] pc : ioremap_prot+0x90/0xcc
[    4.234858] lr : ioremap_prot+0x48/0xcc
[    4.234860] sp : ffff800083aa3560
[    4.234861] x29: ffff800083aa3570 x28: 0000000000000001 x27: 0000000000000001
[    4.240113] x26: 0000000000000000 x25: 0000000000000004 x24: ffff00010a300000
[    4.240733] x23: ffff000109a62618 x22: 0000000000000000 x21: 0000000000000080
[    4.241354] x20: 0000000000080000 x19: 00000001fff80000 x18: 0000000000000006
[    4.241974] x17: 00000000fc5c1d6a x16: 000000000000029a x15: ffffffffffffffff
[    4.242594] x14: 0000000000000000 x13: 00000000000002f6 x12: 00000000000002f6
[    4.243215] x11: ffff00010a30ef68 x10: 0000000000000900 x9 : 0000000000001000
[    4.243836] x8 : 00000002fc000000 x7 : 0000000000000018 x6 : ffff800081b97e50
[    4.243839] x5 : ffff800081b97e08 x4 : 0000000000000004 x3 : 0000000000000006
[    4.246062] x2 : 00000003fc000000 x1 : 0000000000000003 x0 : 0000000000000001
[    4.246682] Call trace:
[    4.246896]  ioremap_prot+0x90/0xcc (P)
[    4.247231]  ioremap_prot+0x48/0xcc (L)
[    4.247565]  ttm_bo_kmap+0x1fc/0x2b4 [ttm]
[    4.247925]  amdgpu_bo_create_reserved+0x228/0x430 [amdgpu]
[    4.248735]  amdgpu_bo_create_kernel+0x14/0xa0 [amdgpu]
[    4.249478]  amdgpu_gart_table_vram_alloc+0x3c/0x44 [amdgpu]
[    4.250253]  gmc_v8_0_sw_init+0x4dc/0x630 [amdgpu]
[    4.250945]  amdgpu_device_init+0x1690/0x229c [amdgpu]
[    4.251175] Adding 16777212k swap on /dev/zram0.  Priority:10 extents:1 across:16777212k SS
[    4.252388]  amdgpu_driver_load_kms+0x14/0xf8 [amdgpu]
[    4.253100]  amdgpu_pci_probe+0x170/0x568 [amdgpu]
[    4.253779]  pci_device_probe+0xdc/0x258
[    4.254124]  really_probe+0xb8/0x354
[    4.254439]  __driver_probe_device+0x74/0x110
[    4.254820]  driver_probe_device+0x38/0x154
[    4.255186]  __driver_attach+0xf8/0x1d4
[    4.255522]  bus_for_each_dev+0x74/0xdc
[    4.255858]  driver_attach+0x20/0x28
[    4.256172]  bus_add_driver+0xdc/0x200
[    4.256500]  driver_register+0x64/0x114
[    4.256836]  __pci_register_driver+0x40/0x48
[    4.258155]  amdgpu_init+0x84/0x1000 [amdgpu]
[    4.258797]  do_one_initcall+0x4c/0x1b8
[    4.259132]  do_init_module+0x58/0x218
[    4.259461]  load_module+0x2554/0x2ad8
[    4.259790]  init_module_from_file+0x84/0xc4
[    4.260162]  __arm64_sys_finit_module+0x160/0x3c0
[    4.260574]  invoke_syscall+0x44/0x100
[    4.260902]  do_el0_svc+0x44/0xe0
[    4.261192]  el0_svc+0x2c/0x100
[    4.261469]  el0t_64_sync_handler+0x104/0x130
[    4.261850]  el0t_64_sync+0x154/0x158
[    4.262170] ---[ end trace 0000000000000000 ]---
[    4.262583] amdgpu 0000:01:00.0: amdgpu: (-12) kernel bo map failed
[    4.263159] [drm:amdgpu_device_init [amdgpu]] *ERROR* sw_init of IP block <gmc_v8_0> failed -12
[    4.264186] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_init failed
[    4.264748] amdgpu 0000:01:00.0: amdgpu: Fatal error during GPU init
[    4.265308] amdgpu 0000:01:00.0: amdgpu: amdgpu: finishing device.
[    4.265876] amdgpu 0000:01:00.0: probe with driver amdgpu failed with error -12

I assume i need @Coreforge's patches?

[Coreforge]

You need to adapt them a little bit for newer kernels (there's one or two places that got added that either needed a memcpy/memset changed, or needed structs changed to volatile).
I've tried it with 6.10 (armbian) on an rk3588s and while I didn't get any Oops, it looked like the GPU wasn't able to access memory properly or something. That could be something that can be fixed in the device tree, I haven't tried much further so far (messing with device tree overlays is a bit more difficult when booting over the network).

[System64fumo]

If you could tell me where those places are i'll take a crack at it.

I've tried changing a few memsets to memset_io in some logical places but that didn't do anything..

[Coreforge]

I don't exactly remember unfortunately (and there's too many modified files in my repo for git to be helpful too).
If you apply the commits from the pi branch, you can see where it then goes wrong.

[System64fumo]

I've done exactly that, went through each commit, built it, ran it, no change.

[System64fumo]

Just so we're on the same page memset and memcpy are the culprits behind this correct?
Replacing them (wherever necessary) with the _io _fromio _toio counterparts ought to fix the kernel panic correct?

[geerlingguy]

Since jeff isn't quite a fan of rockchip shenanigans i'd be happy to provide a kernel (and modules) if requested.

I love Rockchip shenanigans (and RISC-V, whatever... making hardware do things that were not intended is one of the best things to do with computers).

What I gave up on with the RK3588 in particular, was not being able to work with newer Linux kernels easily, to work on the latest amdgpu drivers. It seems like there are some ways to get newer kernels now, though I though there were still hardware quirks that limited what GPUs could run on RK3588? Or maybe that's outdated info!

[Coreforge]

Replacing them (wherever necessary) with the _io _fromio _toio counterparts ought to fix the kernel panic correct?

Generally yes, though some issues also come from struct accesses (those can so far be fixed by marking those structs as volatile).
BAR=0M looks odd to me, but I'll have to compare with my orange pi 5 to see if that's normal. I didn't have to do anything special regarding BARs so far on armbian, but it's also possible not all settings got applied like they usually would, I'd have to boot from an SD card to check.

[System64fumo]

Since jeff isn't quite a fan of rockchip shenanigans i'd be happy to provide a kernel (and modules) if requested.

I love Rockchip shenanigans (and RISC-V, whatever... making hardware do things that were not intended is one of the best things to do with computers).

What I gave up on with the RK3588 in particular, was not being able to work with newer Linux kernels easily, to work on the latest amdgpu drivers. It seems like there are some ways to get newer kernels now, though I though there were still hardware quirks that limited what GPUs could run on RK3588? Or maybe that's outdated info!

Hi Jeff! Appologies for the misunderstanding, It's actually quite simple to get newer linux on rockchip devices (Especially these days)
You could clone this repo and build it.
Or
Get it straight from torvalds himself (HDMI does not seem to work for me on linux-next since a few months ago)

I'd be happy to help you get mainline on one of your devices.

[System64fumo]

Replacing them (wherever necessary) with the _io _fromio _toio counterparts ought to fix the kernel panic correct?

Generally yes, though some issues also come from struct accesses (those can so far be fixed by marking those structs as volatile). BAR=0M looks odd to me, but I'll have to compare with my orange pi 5 to see if that's normal. I didn't have to do anything special regarding BARs so far on armbian, but it's also possible not all settings got applied like they usually would, I'd have to boot from an SD card to check.

I see, I'll try tomorrow hopefully i can get a different error or something, Thanks!

[geerlingguy]

I'd be happy to help you get mainline on one of your devices.

@System64fumo - I would like to, at some point. I'm just flooded with stuff to test right now, so if someone does get this stuff set up, if someone could leave a very point-A-to-point-B guide, that'd make it a lot easier to replicate. A lot of times someone says "just grab this thing and that thing, then boom! Done!" (But there are in reality 30 or 40 steps in there) — all these GitHub issues, I try to document (usually in excruciating detail) every single command I run, and where the sources are. Would rather anyone can follow in my footsteps!

[System64fumo]

Very well, I planned on releasing documentation for a lot of my linux shenanigans in 2025.

I will link you to that when i'm done writing everything and making stuff easier for others to use.

[System64fumo]

Update: I got a little farther today after changing ioremap_wc to phys_to_virt in ttm_bo_util.c and changing a few memsets to memset_io in other places, But the driver still fails to load properly:

...
[    3.984669] [drm] amdgpu kernel modesetting enabled.
[    3.985406] amdgpu 0000:01:00.0: enabling device (0000 -> 0003)
[    3.985940] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x6FDF 0x1002:0x0B31 0xEF).
[    3.986725] [drm] register mmio base: 0xF0200000
[    3.987131] [drm] register mmio size: 262144
[    3.987644] [drm] add ip block number 0 <vi_common>
[    3.988073] [drm] add ip block number 1 <gmc_v8_0>
[    3.988500] [drm] add ip block number 2 <tonga_ih>
[    3.988919] [drm] add ip block number 3 <gfx_v8_0>
[    3.989340] [drm] add ip block number 4 <sdma_v3_0>
[    3.989765] [drm] add ip block number 5 <powerplay>
[    3.990193] [drm] add ip block number 6 <dm>
[    3.990566] [drm] add ip block number 7 <uvd_v6_0>
[    3.990984] [drm] add ip block number 8 <vce_v3_0>
[    4.211768] amdgpu 0000:01:00.0: amdgpu: Fetched VBIOS from ROM BAR
[    4.212372] amdgpu: ATOM BIOS: xxx-xxx-xxx
[    4.220987] [drm] UVD is enabled in VM mode
[    4.221374] [drm] UVD ENC is enabled in VM mode
[    4.221770] [drm] VCE enabled in VM mode
[    4.222155] random: crng init done
[    4.222180] amdgpu 0000:01:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[    4.223158] amdgpu 0000:01:00.0: amdgpu: PCIE atomic ops is not supported
[    4.223755] [drm] GPU posting now...
[    4.334707] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[    4.342698] amdgpu 0000:01:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[    4.343539] amdgpu 0000:01:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[    4.344366] [drm] Detected VRAM RAM=8192M, BAR=0M
[    4.344775] [drm] RAM width 256bits GDDR5
[    4.345300] [drm] amdgpu: 8192M of VRAM memory ready
[    4.345732] [drm] amdgpu: 7886M of GTT memory ready.
[    4.346194] [drm] GART: num cpu pages 65536, num gpu pages 65536
[    4.352581] [drm] PCIE GART of 256M enabled (table at 0x000000F5FFF80000).
[    4.382870] [drm] Chained IB support enabled!
[    4.422135] [drm] Found UVD firmware Version: 1.130 Family ID: 16
[    4.430626] [drm] Found VCE firmware Version: 53.26 Binary ID: 3
[    4.499820] [drm:amdgpu_mm_wdoorbell [amdgpu]] *ERROR* writing beyond doorbell aperture: 0x000001e8!
[    8.585424] amdgpu: SMU load firmware failed
[    8.585427] amdgpu: fw load failed
[    8.585429] amdgpu: smu firmware loading failed
[    8.585431] amdgpu 0000:01:00.0: amdgpu: amdgpu_device_ip_init failed
[    8.585435] amdgpu 0000:01:00.0: amdgpu: Fatal error during GPU init
[    8.585496] amdgpu 0000:01:00.0: amdgpu: amdgpu: finishing device.
[    8.585572] ------------[ cut here ]------------
[    8.585574] WARNING: CPU: 5 PID: 230 at drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c:631 amdgpu_irq_put+0x98/0xac [amdgpu]
[    8.586063] Modules linked in: amdgpu(+) amdxcp drm_buddy drm_suballoc_helper i2c_algo_bit drm_ttm_helper ttm
[    8.586077] CPU: 5 UID: 0 PID: 230 Comm: modprobe Not tainted 6.13.0-rc1-baka+ #24
[    8.586083] Hardware name: Xunlong Orange Pi 5 Plus (DT)
[    8.586085] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    8.586090] pc : amdgpu_irq_put+0x98/0xac [amdgpu]
[    8.586519] lr : amdgpu_fence_driver_hw_fini+0x120/0x170 [amdgpu]
[    8.586933] sp : ffff80008441b760
[    8.586935] x29: ffff80008441b770 x28: ffff80008441bcc8 x27: ffff80008441bcc8
[    8.586941] x26: ffff80007a689b70 x25: ffff00010a000000 x24: 0000000004ffffff
[    8.586946] x23: ffff00010a000000 x22: ffff00010a000010 x21: ffff00010a0105f8
[    8.586951] x20: ffff00010a010218 x19: ffff00010a018868 x18: ffff80008441b348
[    8.586956] x17: 00000001ffbc4792 x16: 00000000000002d5 x15: ffffffffff80cb00
[    8.586961] x14: 0000000000000000 x13: ffff0004fd1037c0 x12: 00000000fffc0000
[    8.586967] x11: 00000000ffffffea x10: 00000000fffbffff x9 : ffff0004fbb037c0
[    8.586972] x8 : ffff800081f9e000 x7 : 303a31303a303030 x6 : 0000000000000002
[    8.586976] x5 : 0000000000000000 x4 : ffff00010a000000 x3 : 0000000000000000
[    8.586981] x2 : 0000000000000000 x1 : ffff00010a0254a8 x0 : ffff0001003c9c40
[    8.586986] Call trace:
[    8.586988]  amdgpu_irq_put+0x98/0xac [amdgpu] (P)
[    8.587395]  amdgpu_fence_driver_hw_fini+0x120/0x170 [amdgpu] (L)
[    8.587798]  amdgpu_device_fini_hw+0xd8/0x360 [amdgpu]
[    8.588197]  amdgpu_driver_load_kms+0xa4/0xf8 [amdgpu]
[    8.588592]  amdgpu_pci_probe+0x170/0x568 [amdgpu]
[    8.588986]  pci_device_probe+0xdc/0x258
[    8.588996]  really_probe+0xb8/0x354
[    8.589003]  __driver_probe_device+0x74/0x110
[    8.589008]  driver_probe_device+0x38/0x154
[    8.589013]  __driver_attach+0xf8/0x1d4
[    8.589018]  bus_for_each_dev+0x74/0xdc
[    8.589023]  driver_attach+0x20/0x28
[    8.589028]  bus_add_driver+0xdc/0x200
[    8.589032]  driver_register+0x64/0x114
[    8.589038]  __pci_register_driver+0x40/0x48
[    8.589044]  amdgpu_init+0x84/0x1000 [amdgpu]
[    8.589438]  do_one_initcall+0x4c/0x1b8
[    8.589445]  do_init_module+0x58/0x218
[    8.589451]  load_module+0x2554/0x2ad8
[    8.589456]  init_module_from_file+0x84/0xc4
[    8.589461]  __arm64_sys_finit_module+0x160/0x3c0
[    8.589466]  invoke_syscall+0x44/0x100
[    8.589472]  do_el0_svc+0x44/0xe0
[    8.589477]  el0_svc+0x2c/0x100
[    8.589483]  el0t_64_sync_handler+0x104/0x130
[    8.589487]  el0t_64_sync+0x154/0x158
[    8.589491] ---[ end trace 0000000000000000 ]---
...

@Coreforge Any ideas?

[System64fumo]

Alright @geerlingguy i've started work on my documentation project
Hopefully you will find my generic mainline guide and script in here useful.
Pure mainline linux should work out of the box on the radxa rock 5b now

I do plan on adding mainline u-boot eventually but until i get to it you might find this related guide helpful even if it's for the OP 5 Plus.

If you have any feedback feel free to open an issue or mr.
And feel free to hide this message since it's offtopic.