Keyboard and Mouse request

[Crimsn64]

I'd like to request that a mouse movement can be used as right stick and zl/zr and a keyboard with wasd as left stick. Also, if possible, the less the mouse moves, the less tilt the right stick recieves (vice versa with nore movement) (maybe even implement this using usb?) I know fortnite utilizes keyboard and mouse, but id like it on minecraft (sorry for requesting so much...)

[ndeadly]

Yeah, keyboard and mouse support is already on my list of planned features. The current issue is that as far as I can tell, most Bluetooth keyboards and mice are using Bluetooth LE, which I haven't managed to figure out yet. The mappings you describe are more or less how I plan to do it, just need to figure out how to get them connected first. I don't currently have any USB support either, but theoretically both of these options could be supported in the future.

[Crimsn64]

YAHS! Thank you SO much!

[Dstr8ction]

Just wanted to share that more ppl are very interested in this, no matter if Bluetooth or USB connection.
If I have to play just one more level in Ghostrunner with a controller/JoyCon I'm going to lose it!

Thank you for your hard work so far :-)

[einekratzekatze]

Is this still planned? Maybe you could implement parts of https://github.com/cathery/sys-con for USB support, to support keyboard and mouse easier (and make this tool more powerful in general).
Also, with Mouse to Stick movements, it's really hard to translate between both. To optimize it, you would ideally need a config Window like this:

Although for games like Splatoon which support motion/gyro controls, translating Mouse to gyro controls will be way more fluent.
For supporting motion controls, a look at this might help (it's a script for a device that lets you change inputs from any device to any device): Maybe, this StackOverflow question could help as well.
Also, if I can help with the development of the issue, let me know.

[ndeadly]

Yes, it's still planned, but isn't super high priority for me right now due to the amount of other problems that need to be solved first in order to make it possible.

Firstly, there's the issue of recognising devices. In order to correctly identify a device I need to have to know its vid and pid. This means that there can be no such thing as generic keyboard an mouse support. Every new device will need to have these identifiers added to the codebase. I'm also not sure if there's any standard reporting format for these classes of device, or whether different devices will also need to be mapped individually.

Secondly, there are multiple possible standards for modern mouse and keyboard. Some are USB, some legacy Bluetooth, and some the newer Bluetooth LE. Currently, Mission Control only supports legacy Bluetooth devices, further limiting the device support.

Maybe you could implement parts of https://github.com/cathery/sys-con for USB support, to support keyboard and mouse easier (and make this tool more powerful in general).

I have already done some work on USB support, but I want to do it via MITM instead of the way sys-con does it because it is more powerful. Currently USB support has hit a bit of a roadblock though, and I'm waiting for some feedback from other scene members regarding how to continue.

Then there's the issue of keyboard and mouse being two separate devices. In order for them to be seen as one controller by the console, I will need to "fuse" them into a single "virtual" controller. This can either be done by leveraging existing functionality in the OS and spoofing them as left and right joycons, or by manually managing everything and exposing the inputs via the debug pad API (sys-con does this). I have done some work on joycon spoofing with some success, but there are still some issues there that need to be overcome in order to create a pleasant user experience that I would consider ready for release. Manually managing could be made work, but is just a bunch of additional work I'd rather not do unless absolutely necessary.

Finally, like you say, there's the issue of how to reliably convert mouse movements into something useful. Any form of configurability offered needs to be backed up by by a nice gui, or at the very least some kind of file based framework.

None of this is infeasible, but as you can probably tell by now, it's actually a reasonable amount of work to pull off smoothly and would distract from developing other areas of the project. I don't have time for it right now, but I would welcome anyone with the necessary skills wanting to investigate any of the underlying issues further.

[einekratzekatze]

Is there something specific I can work on?

[ndeadly]

I'm also not sure if there's any standard reporting format for these classes of device, or whether different devices will also need to be mapped individually.

A good starting point might be finding an answer to this question. If it turns out that there's a standard data format, that's one less thing to worry about. It could be possible to instead check the device class instead of the vid/pid and assume that anything we haven't explicitly whitelisted (some gamepads identify as a mouse or keyboard) is the real deal. This would at least mean that legacy bluetooth devices could potentially be supported generically.

The other things would be a bit more challenging. You would need to have some degree of experience in developing Switch homebrew, familiarity with the OS, or potentially some experience in reverse engineering binaries in order to meaningfully contribute. This isn't really something I can spoonfeed someone starting from scratch.

[einekratzekatze]

Firstly, there's the issue of recognising devices. In order to correctly identify a device I need to have to know its vid and pid. This means that there can be no such thing as generic keyboard an mouse support. Every new device will need to have these identifiers added to the codebase. I'm also not sure if there's any standard reporting format for these classes of device, or whether different devices will also need to be mapped individually.

I'd guess that it probably isn't that hard, a lot of devices support all mice and keyboards, but not all controllers, so there is probably a generic way. I'll look into that, even though I can probably not test it out?
Also, the switch supports Keyboard and Mouse already, you can use them in the UI, and even in some games (being mainly or even only doom). Not sure if that's helps or is actually a bigger Issue.
For a very basic answer: Yes, all HIDs (https://en.wikipedia.org/wiki/USB#Device_classes) should be easy to support generically. Although, any custom features of these devices, obviously won't work.

Secondly, there are multiple possible standards for modern mouse and keyboard. Some are USB, some legacy Bluetooth, and some the newer Bluetooth LE. Currently, Mission Control only supports legacy Bluetooth devices, further limiting the device support.

Of course, implementing all types takes longer, but I don't see this as a huge issue.

Then there's the issue of keyboard and mouse being two separate devices. In order for them to be seen as one controller by the console, I will need to "fuse" them into a single "virtual" controller. This can either be done by leveraging existing functionality in the OS and spoofing them as left and right joycons, or by manually managing everything and exposing the inputs via the debug pad API (sys-con does this). I have done some work on joycon spoofing with some success, but there are still some issues there that need to be overcome in order to create a pleasant user experience that I would consider ready for release. Manually managing could be made work, but is just a bunch of additional work I'd rather not do unless absolutely necessary.

Making it 2 Joy Cons is probably quite limiting with keybinding and stuff, so I don't think that that's really an option, so to have a nice result, there would be probably a lot of work. I think that a bit more complex UI is probably required.

Finally, like you say, there's the issue of how to reliably convert mouse movements into something useful. Any form of configurability offered needs to be backed up by by a nice gui, or at the very least some kind of file based framework.

I think this probably isn't that tough. Even though the translation isn't perfect, this isn't a huge deal, as for most games the default is enough.

The other things would be a bit more challenging. You would need to have some degree of experience in developing Switch homebrew, familiarity with the OS, or potentially some experience in reverse engineering binaries in order to meaningfully contribute. This isn't really something I can spoonfeed someone starting from scratch.

I mean, I have to start somewhere, everyone has to. And if I can start here, with something I have use for, that would be great.
How did you start out?

As the next step to progress would probably be to implement USB support, IG I need to wait? Are there any issues that you could recommend for me to work on in general for the time being? I've looked at a few, but I am not really sure what to pick.

[ndeadly]

I'd guess that it probably isn't that hard, a lot of devices support all mice and keyboards, but not all controllers, so there is probably a generic way.

Usually what happens is a device sends an HID descriptor that describes the reporting formats that the driver can then use to interpret the data. On the Switch OS, this information gets discarded (at least for bluetooth) and cannot be relied upon. This is why I need to manually map the controls from one format to the other for each device.

Also, the switch supports Keyboard and Mouse already, you can use them in the UI, and even in some games (being mainly or even only doom). Not sure if that's helps or is actually a bigger Issue.

I'm aware of the keyboard and mouse support, however I don't think it's useful as it's treated as a separate input to gamepads and requires games to explicitly support it as far as I know. For universal support we will need to make it look like the data originates from a gamepad.

For a very basic answer: Yes, all HIDs (https://en.wikipedia.org/wiki/USB#Device_classes) should be easy to support generically. Although, any custom features of these devices, obviously won't work.

As above, yes we can determine the class of device, but this says nothing about the format of the data that it sends. That is what the descriptor is for.

Of course, implementing all types takes longer, but I don't see this as a huge issue.

The issue is that this is not simply a programming problem, and requires further research/reverse engineering into how the console handles these things internally before any implementation can be completed. This takes time. As I stated, USB is underway but has hit a roadblock. Bluetooth LE is also underway but there are several issues I haven't yet found a solution for. Sure, if we just consider the legacy bluetooth devices for now this isn't an issue, but I don't know what fraction of devices actually use this. My wireless keyboard/mouse uses LE, and everything else is USB so I don't even have a device I could test with.

Making it 2 Joy Cons is probably quite limiting with keybinding and stuff, so I don't think that that's really an option, so to have a nice result, there would be probably a lot of work. I think that a bit more complex UI is probably required.

The keybinding issue may be solvable by refactoring the controller classes to have some concept of a linked internal state or something. I wouldn't rule it out as an option, it's probably still less work than managing everything manually. I agree that a complex UI is likely required to make this a nice experience for the end user. Investing all the time and effort only for people not to use the feature because of a poor UX is not the way I want to go.

I think this probably isn't that tough. Even though the translation isn't perfect, this isn't a huge deal, as for most games the default is enough.

I didn't mean to imply that this was super difficult in and of itself, more that it may likely require some degree of configurabilty that would need to be backed up by a nice gui.

I mean, I have to start somewhere, everyone has to. And if I can start here, with something I have use for, that would be great.
How did you start out?

Sure, I don't mean to put you off contributing. Just making it clear that this is not beginners stuff and I don't have the time to walk somebody with no experience through all the different aspects to get them up to speed. But if you're willing to learn I'm happy to point you in the right direction. Just don't expect this to be something you can get done in a couple of evenings. I'm going on several years of playing around in the Switch OS with controller related stuff and still don't understand how many things work.

As the next step to progress would probably be to implement USB support, IG I need to wait?

USB support isn't absolutely necessary. The other aspects would function independently to the interface used. It would however increase the number of devices that could be used for testing. Right now I don't have anything compatible. I think once the problem I've hit with USB has been solved, most of the work is already done and it should be pretty quick to integrate. Can't really give a timeline for that though because I'm relying on the research of others.

Are there any issues that you could recommend for me to work on in general for the time being? I've looked at a few, but I am not really sure what to pick.

Hard to say without knowing your background/skillset. Are you talking about issues related to keyboard/mouse support or are you simply looking to contribute to the project in general?

[einekratzekatze]

Usually what happens is a devices sends an HID descriptor that describes the reporting formats that the driver can then use to interpret the data. On the Switch OS, this information gets discarded (at least for bluetooth) and cannot be relied upon. This is why I need to manually map the controls from one format to the other for each device.

Ok, that's annoying, maybe you can let the user press enter or right click, which should give a unique code?

Sure, I don't mean to put you off contributing. Just making it clear that this is not beginners stuff and I don't have the time to walk somebody with no experience through all the different aspects to get them up to speed. But if you're willing to learn I'm happy to point you in the right direction. Just don't expect this to be something you can get done in a couple of evenings. I'm going on several years of playing around in the Switch OS with controller related stuff and still don't understand how many things work.

I realized that it won't just take a few evenings. If you think that 100 hours won't be enough for at least a part of the above things, I can't promise that I won't lose my motivation, though.

USB support isn't absolutely necessary. The other aspects would function independently to the interface used. It would however increase the number of devices that could be used for testing. Right now I don't have anything compatible. I think once the problem I've hit with USB has been solved, most of the work is already done and it should be pretty quick to integrate. Can't really give a timeline for that though because I'm relying on the research of others.

I also don't have a Bluetooth keyboard or mouse, either, so ig it kinda is a requirement. https://github.com/kcolford/hidclient maybe this can be used in the meantime?

Hard to say without knowing your background/skillset. Are you talking about issues related to keyboard/mouse support or are you simply looking to contribute to the project in general?

Ofc I would like to work on what I need, but if it helps generally when I do simpler stuff first, possibly also to get familiar with the project, I'd also do general stuff. I didn't do any homebrew yet, and not a lot of cpp, but at least the later won't be a huge Issue.

(I left some points out as this gets too big, and these discussions probably won't be of much use later)

[einekratzekatze]

As you did not answer yet, I've taken a look at the current Issues myself.
Firstly, the once I am interested in potentially solving, to get familiar with the project:
#511 #485 #484 #483 #456 #419 #432 #407 #322 #306 #298 #203 #199 #108 #87 #43 #34
Can you recommend some to start out?

Then some Issues you could…
… close: #335 #327 #323 #313 #125
… add the BLE label to: #283
… add the feature request label to: #231 #111
… add a debug build to: #252 #52
… revisit if something changed: #178 #149 #123 #481
These aren't all issues that could be updated, just some stuff I noticed while I was looking for issues.

Also, to get a better overview, you could add a “No response” Label? Or close em if they are old?

[ndeadly]

As you did not answer yet, I've taken a look at the current Issues myself. Firstly, the once I am interested in potentially solving, to get familiar with the project: #511 #485 #484 #483 #481 #456 #419 #432 #407 #322 #306 #298 #203 #199 #108 #87 #43 #34 Can you recommend some to start out?

Then some Issues you could… … close: #335 #327 #323 #313 #125 … add the BLE label to: #283 … add the feature request label to: #231 #111 … add a debug build to: #252 #52 … revisit if something changed: #178 #149 #123 These aren't all issues that could be updated, just some stuff I noticed while I was looking for issues.

Also, to get a better overview, you could add a “No response” Label? Or close em if they are old?

Sorry, been a bit busy the past days. This conversation is starting to deviate from the original issue. If you want to chat about things, reach out to me on discord @ ndeadly#5317