Ioctl handling seems to be a stub

[cheery]

Hi,

Having fun and playing with the idea of generating graphics with linux framebuffer and riscv assembly code of my computer. I already got my random generator to flush the screen with noise. But it's handcoded with the buffer size. I stumbled upon the ioctl calls not seeming to work.

Peeked into the rv8 src/abi/. It looks like the ioctl only supports one command and for everything else it's been programmed to give EINVAL.

The ioctl commands I tried to get to work:

FBIOGET_VSCREENINFO = 0x4600
FBIOGET_FSCREENINFO = 0x4602

The code I tried:

addi sp, sp, -fb_fix_screeninfo_size
mv a0, gp                           # fbfd
li a1, FBIOGET_VSCREENINFO
mv a2, sp                           # The screeninfo size pointer. 
li a7, sys_ioctl
ecall
blt a0, zero, fail

Could add few lines into the rv8 to make this work. But should I? It feels laborious to add some check for every single one ioctl that's been used by Linux.

[michaeljclark]

Feel free to make a patch.

Firstly, if you do make a patch, you should add checks for every single ioctl you implement as ioctl is a dangerous system call. The size of the message pointed to is not known ahead of time. It's a bad design. We should be using datagram (vs stream) size preserving messages to drivers, but Linux/POSIX is a mess. There have been many bugs in ioctl calling copy_to_user/copy_from_user for buffers that have been used as vectors for privilege escalation. There are application level syscall firewalls such as gvisor that you can look at for an examples on how to do ioctl proxy:

• https://github.com/google/gvisor
• https://github.com/google/gvisor/blob/master/pkg/abi/linux/ioctl.go
• https://github.com/google/gvisor/blob/master/pkg/sentry/fs/host/tty.go

Note, the main purpose of the user-mode simulator is to test the JIT engine, as such, the syscall translation layer only contains enough system calls to run benchmark programs. The purpose of the proof-of-concept JIT is to demonstrate OoO performance using the RISC-V ISA on commodity x86. We are able to get a peak of approximately 6 billion RISC-V instructions per second on a single thread of a low-end Intel processor, and we quantify work done per instruction as approximately equivalent to x86 on some tasks. See the slide deck, paper, and benchmarks here:

• https://anarch128.org/~mjc/rv8-slides.pdf
• https://anarch128.org/~mjc/rv8-carrv.pdf
• https://rv8.io/bench

The original plan was to add "shadow paging" i.e. MMU acceleration to the JIT to make a full system emulator that can perform equivalently to the user-mode simulator for "hot code", unlike QEMU, whose full system emulator uses a soft MMU which is extremely slow performing loads and stores. The other missing piece is an improved register allocator. If you look at the slide deck, you can see we have built tools to quantify performance degradation against native, and that is primarily due to the register allocator.

From the slide deck, you can see we have outlined a server container like scenario. There are several ways to achieve this. None of them require a framebuffer. We would require something like (Authenticated) UEFI capsule update over IPMI to remove the necessity for KVM (Keyboard/Video/Mouse) when updating firmware. This is possibly the only reason the hyper-scalers want custom firmware, because standard firmware still requires meatware (human operator) with DOS on a floppy to update itself. The network equipment vendors and RISC vendors prior to x86 domination achieved this simply with RS-232. No framebuffer required.

However, if we were going to do a client with graphics, this century, I would suggest specifying and adding VirtioVulkan to a full-system emulator. This in essence would be relatively simple (if not somewhat laborious, as there are many messages to translate), as it would equate to serializing Vulkan command queues and buffer DMA operations over VirtIO Virtqueue. This would technically be much easier than VirtGL as there is no need to define a Virtual GPU interface. Vulkan is a GPU driver model, and the Vulkan shader byte-code is now standard (SPIR-V), so there is no need for a complex state tracker. Essentially, a VirtioVulkan DRM driver would proxy Vulkan methods over the drm ioctl to the virtual GPU which would just be a proxy to the host GPU. There would need to be a GEM driver so that the DRM driver has access to host side buffer objects. With some SLAT magic (second level-address translation) using Hypervisor.framework, or HyperV, etc, the Vulkan proxy driver could be map the guest's host-side buffer objects coherently in its own address space (zero-copy) for submission the actual Vulkan driver.

VirtioVulkan would be neat! The idea behind rv8 is performance and standards, so it fits. VirtioVulkan would possibly be the highest performance standard mechanism for a RISC-V guest to access a framebuffer, so I like it.

That said, I still like my RISC-V standardized server container idea better. Much easier, as most codes on servers are either open source, or the container operator has its own source code. Client-side is going to be much tougher as there are many more proprietary codes on clients.

[cheery]

I happen to have done vkstruct. They are a system to produce automated bindings for Vulkan. Haven't touched these for a while but I guess they could be perhaps repurposed for generating the laborious message translator straight out of Vulkan's machine spec. If you like the idea, we can try that out. I just need to know what I exactly got to generate here.

I wrote an SDL app that's bringing a shared file buffer. I think it works better than the ioctl approach and is easier to try out in general. It also doesn't require any changes to the rv-jit.

[michaeljclark]

Oh wow! vkstruct and chartparser look really useful.

Now I see what you have done. Thanks for trying this experiment with rv8. This is an experimental repo so I'm happy to merge interesting demos, or point to forks containing interesting demos. Ideally using a somewhat standard interfaces e.g. fd = open("/dev/fb0", ...); addr = mmap(..., fd, ...). I like standards.

I'm definately interested in helping with VirtioVulkan, however, it seems like a big chunk of work, so it is a long-term objective. Perhaps the KVM developers might already have this in mind as it seems much cleaner than GL/GLES, as GL/GLES can be implemented as an API layer on top of Vulkan. I will have to check to see what the Mesa developers are up to. The machine readable spec will obviously make it much easier. I would be happy to collaborate on this as time permits... Thanks for logging this issue.

It makes a lot of sense for a Vulkan proxy to be able to verify messages, and the spec metadata seems like it would also help us with message constraints. As you pointed out, we could generate the message translator straight out of Vulkan's machine spec. I do, however, suspect the attack surface for a Vulkan driver is quite large. In my experiments, I have been able to freeze the display driver on Linux with unprivileged user code. That said, the drivers are relatively fresh code, so they should become harder as they mature. It would be interesting to find out what a Vulkan firewall would need to do. Sharing a GPU between host and guest will likely need validation enabled and some hardening and possibly even filtering and SPIR-V validation.

The things this project needs to turn it from proof of concept into a usable tool would be:

• MMU Acceleration using shadow paging (so we can port the JIT to the full system emulator)
• Register allocation (so we can eeek out remaining performance)
• VirtioVulkan (so we have accelerated display) would be absolutely amazing.
• DebugGUI (so we can do slowmo, fast-forward and rewind for crash replays, with asm display, etc)

Of course there are lots of details. We don't have multi-threading and atomics implemented yet. We can borrow code from RISCVEMU and Spike as they have compatible licenses. QEMU has quite a bit of GPL code so its not compatible with rv8. Interestingly, Fabrice's original code and his Virtio code is open under a friendly MIT license. Red Hat advised SiFive to GPL their code. It means we can't use it. SiFive could grant use of some of their QEMU device emulations under MIT if they wanted to help us, because its their code. QEMU/Linaro upstream don't like me for some reason so I'm no longer working on QEMU. I was under lots of pressure to upstream the code. We might end up having to write the PLIC and CLINT, device emulations again without looking at the code I wrote before 😞 . We have some of the emulations in Spike. It's just annoying to have to re-write code. The full-system mode in rv8 as still from quite an old spec and its before there proper PLIC drivers in Linux.