Improve osu!mania playability on mobile devices

[frenzibyte]

• Requires testing on Android and ensuring AndroidOrientationManager works as it should (see Improve osu!mania playability on mobile devices #31368 (comment))
• Depends on Implement system-provided file selection display on iOS osu-framework#6445 for iOS window API (would've split it out but the PR is already approved so)
• Depends on Expose game host from iOS app delegate class osu-framework#6477
• Depends on framework bump
• Closes Disable screen rotation while in active gameplay #5331
• Addresses osu!mania isn't that comfortable to play in landscape mode on mobile #16783

RFC. This is quite a packed PR, but the changes made here are closely tied to each other and cannot be separated as they cannot be merged / reviewed on their own (except for maybe the mania judgement positioning changes).

The aim of this PR is to lay down a path forward for supporting osu!mania gameplay on mobile devices and discuss the approach taken here and whether it's good / ethical / does not violate any existing gameplay rules or mechanics.

Preview of final product:

IMG_1845-converted.MOV

IMG_0460.mov

Allow locking orientation on iOS in certain circumstances

Adds support for making OsuScreens able to force the device to a "portrait" orientation, to be used for mania-based Player screen sessions.

Improve osu!mania gameplay scaling on portrait orientation

The main change of this PR. Naturally the entire game scene is surrounded by a scaling container that enforces the game to fit to the window size in a 4:3 aspect ratio. Since osu!mania is generally vertically based, this scaling behaviour is completely unfriendly on screens in portrait orientations.

To improve on this, I've added scaling logic in ManiaPlayfieldAdjustmentContainer that scales up the osu!mania stage just enough to be playable with a touchscreen on phones and tablet devices, and try to accommodate as much horizontal space as possible before scaling down on high key numbers / wide columns

Back-to-back comparison between master and this PR:

CleanShot.2024-12-31.at.12.30.37.mp4

Make mania judgements relative to the hit target position

Following the commit above, since the playfield can be roughly scaled up to 125% its normal size, the judgements shift position, and they do so incorrectly, shifting down to below the hit target and look broken. This commit changes the anchoring of the judgements so they're tied to the hit target, so that when the playfield is scaled up, the judgements shift to the top rather than to the bottom.

Before:

CleanShot.2024-12-31.at.12.36.14.mp4

After:

CleanShot.2024-12-31.at.12.37.22.mp4

Replace ManiaTouchInputArea with touchable columns

Also following the scaling changes, which ensure that the playfield is scaled enough for the player to hit the notes with their fingers, I cannot find any use for ManiaTouchInputArea anymore, so I chose to remove it completely. This may be debatable, but I'm not sure refactoring it would make any sense and it's adding very bad visual clutter when displayed on top of the columns (it looks mispositioned from each column and basically broken to the user).

Due to this, I've looked into applying the portrait orientation changes to Android as well so osu!mania doesn't become broken there.

[Susko3]

Haven't tested android yet. Do you have a set of tests that should be done? E.g. open screen X in landscape then rotate to portrait, is should/shouldn't rotate.

CI failures need attention.

[Susko3]

Should use osu.Framework.Utils.Precision, or just compare with 1f. Otherwise a 4:3 tablet in landscape might be considered portrait by float imprecision.

Another reason to use 1f is that older 1280x1024 displays (5:4) are properly considered as landscape.

[frenzibyte]

This didn't matter at first because I didn't want to apply a general upscale of the playfield, but after further considerations I've changed it as such. I'll change the boolean to check against 1f instead.

[frenzibyte]

64e557d

[frenzibyte]

Tests I've done:

• On phone
  • Game is locked in landscape by default, can be left or right regardless of user rotation lock setting (or so is what I understand by "sensor")
  • Running osu! and in gameplay period, orientation is completely locked
  • Pausing unlocks orientation momentarily, resuming locks again.
  • Running osu!mania switches orientation to "portrait" and locks it there, nothing unlocks it until exiting.
  • Restarting does not reset back orientation to landscape, unless user exits.
• On tablet
  • Game is allowed to rotate to any orientation by default, unless locked by auto-rotate lock.
  • Running osu! and in gameplay period, orientation is completely locked
  • Pausing unlocks orientation momentarily, resuming locks again.
  • Running osu!mania does nothing special to the orientation (i.e. it's locked when in gameplay period, and otherwise you can rotate to anything).

[PercyDan54]

I get this behavior on my Xiaomi Pad
Not sure if it's some weird MIUI-specific stuff

out.mp4

[frenzibyte]

@PercyDan54 Cannot reproduce on my iPad. Try debugging through the AndroidOrientationManager class and trace down what orientation values are being fed to the activity. This looks like when you paused gameplay, RequestedOrientation got set to a value and that value caused the game to appear in that broken state.

[PercyDan54]

@PercyDan54 Cannot reproduce on my iPad. Try debugging through the AndroidOrientationManager class and trace down what orientation values are being fed to the activity. This looks like when you paused gameplay, RequestedOrientation got set to a value and that value caused the game to appear in that broken state.

It's not only when pausing the game once. If I put my tablet in Landscape it immediately enters that broken state when gameplay starts.

The Orientation in gameplay is FullPortrait and then Locked

[PercyDan54]

I had a quick look in Android docs and seems the game is put into Letterboxing mode

https://developer.android.com/guide/topics/manifest/activity-element#screen

On Android 12 (API level 31) and higher, device manufacturers can configure individual device screens (such as the tablet-sized screen of a foldable) to ignore this suggestion, and instead force an activity to be letterboxed within the user's preferred orientation of the device.

[frenzibyte]

Sounds like it's working correctly then.

[PercyDan54]

Sounds like it's working correctly then.

I guess my case might fall into the "device manufacturers configured". Imo it's better to test on more tablets to know whether most manufacturers do this, since it's a bit counter-intuitve

[peppy]

I'm not sure this approach is going to work well. My original thoughts were that we don't change the OS-level orientation locking and just adjust the ruleset's playfield to render according to a user's preference. As per the reported issues above, I have no idea how solid mobile APIs are to have an application decide whether the orientation should be switched on the fly at an OS level, disregarding how the user is holding the device.

[frenzibyte]

I have considered make the playfield render in a portrait-like mode without changing the OS orientation, but there are multiple things to take into consideration when doing this.

• The change should be applied to the entirety of Player, otherwise the pause screen will be displayed incorrectly.
• The skin editor has to somehow know the Player screen is in portrait mode and somehow flip itself to display correctly, otherwise skin editing will be completely broken on iOS / iPads.
• All game overlays will also have to handle this somehow if the player connects a hardware keyboard and pops in the top toolbar or otherwise.

For the reasons above, I thought I can change the orientation at an OS level and tick all the three points above without fuss.

UIKit define explicit API to let the device know that the orientation needs to be updated, see setNeedsUpdateOfSupportedInterfaceOrientations (or the old version of this, attemptRotationToDeviceOrientation). I would say that's enough indication that UIKit supports this solidly.

I'm not sure what are the conditions for making Android switch to letterbox mode as in the reported issue above. Need someone that can investigate that and see how bad it is and/or if letterboxing can be avoided.

[Susko3]

• On tablet
  • Running osu!mania switches orientation to "portrait". When in gameplay period, orientation is locked, and when paused or in break you can switch orientation between portrait and upside-down portrait.

I (and the mentioned android xiaomi tablet) disagree with this. There is no good reason to force portrait orientation on tablets. People may want to play osu!mania with a keyboard, and they will usually want to have their tablet in landscape mode.

[Susko3]

Long screenshot

The last column in mania 7K is blocked by the pause button. Would 7K be more playable in portrait or landscape?

[frenzibyte]

• On tablet

  • Running osu!mania switches orientation to "portrait". When in gameplay period, orientation is locked, and when paused or in break you can switch orientation between portrait and upside-down portrait.

I (and the mentioned android xiaomi tablet) disagree with this. There is no good reason to force portrait orientation on tablets. People may want to play osu!mania with a keyboard, and they will usually want to have their tablet in landscape mode.

Landscape orientation for mania on tablets can be brought back. It felt unplayable at first with the idea of "touchable columns" and no touch input overlay, but upon testing again it seems roughly fine. However I find it slightly weird to allow landscape orientation when portrait orientation works and works much better than landscape. I would rather recommend tablet users to always be on portrait if they intend to play mania. @peppy thoughts?

[frenzibyte]

Long screenshot

The last column in mania 7K is blocked by the pause button. Would 7K be more playable in portrait or landscape?

1. this is 10K mania
2. that screenshot looks like you're playing on a phone. Landscape will be extremely worse, and 10K mania on phone is practically impossible to have. I doubt it's even playable with touchable columns. It cannot be salvaged in this PR.

[peppy]

I would rather recommend tablet users to always be on portrait if they intend to play mania.

On the flip side, can we not force the orientation change and leave it up to the user to do this? Or does that not work well?

[frenzibyte]

I wouldn't say it's totally unplayable on landscape orientation on my iPad, but it feels bad with default skins due to the default column width (the column is barely wide enough to cover up my finger). Users will most certainly go through making legacy skins with wider columns to be able to play on it (and I assume it's something we want to expose as a skin setting to support default skins in the future as well).

I can still allow landscape orientation on iPads/tablets. I'll just do that.

[peppy]

On a read of the code and a test of how this looks in portrait (on PC, but that counts right?) I'm pretty on board with this direction.

e5713e5 caught me off-guard – is this something you could PR separately along with a visual explanation? I think bundling it with this PR makes things a bit hard to swallow.

[peppy]

@Susko3 are you able to give this a go/no-go for android (remaining blocking bullet point)?

[peppy]

Initial pass, no major issues.

[Susko3]

@Susko3 are you able to give this a go/no-go for android (remaining blocking bullet point)?

It's good from my side. I've tested as set out in #31368 (comment). I've only tested on a phone, but I've also simulated a tablet by forcing IsTablet = true. Works well in both cases. 4K mania is comfortable to play in portrait.

Tagging @PercyDan54 to give it a test on a real tablet, as things are updated and should work better.

[PercyDan54]

The column is a bit too thin to me in landscape, I'm not a mania player so can't really judge.

And a small visual bug is the re-layout happens after the resume countdown which is kinda weird to see the playfield going offscreen for 3 seconds

[frenzibyte]

The column is a bit too thin to me in landscape, I'm not a mania player so can't really judge.

I'll say this is the expectation of attempting to play on landscape, it won't always feel good. Switch to portrait for better experience. Allowing landscape was a choice made few comments above yours #31368 (comment).

And a small visual bug is the re-layout happens after the resume countdown which is kinda weird to see the playfield going offscreen for 3 seconds

This is to do with the playfield not being updated while the game is paused, it is out of scope in this PR and most relevant to #7285

[PercyDan54]

The column is a bit too thin to me in landscape, I'm not a mania player so can't really judge.

I'll say this is the expectation of attempting to play on landscape, it won't always feel good. Switch to portrait for better experience. Allowing landscape was a choice made few comments above yours #31368 (comment).

And a small visual bug is the re-layout happens after the resume countdown which is kinda weird to see the playfield going offscreen for 3 seconds

This is to do with the playfield not being updated while the game is paused, it is out of scope in this PR and most relevant to #7285

Alright.
Portrait 4K looks pretty good to me, until 7K it reaches just larger than the width of my finger. And 8K is around (still larger) what the above photo (landscape 4K). But I understand that such high key count is not a focus now and I would say the pr is good to go as a first step.

[peppy]

Given the above responses, I'm going to do a code-level review pass this week with the intention of getting it in 👍 .

[frenzibyte]

On a read of the code and a test of how this looks in portrait (on PC, but that counts right?) I'm pretty on board with this direction.

e5713e5 caught me off-guard – is this something you could PR separately along with a visual explanation? I think bundling it with this PR makes things a bit hard to swallow.

I would have to specifically revert changes to the DrawableManiaJudgement class as well, which is part of the 1e08b3d commit...I think it will take less time if I briefly explain it here alongside some visual showcasing (especially since merging this without this change will just break triangles skin completely).

This is happening because in 1e08b3d the anchoring of the judgement container has changed so that judgements get moved upwards proportional to the upscaling factor of the playfield.

And due to that change, the positioning of triangles skin mania judgements is completely broken as they're used to display close to the top of the playfield. Therefore both the DrawableManiaJudgement part of 1e08b3d as well as e5713e5 change the positioning of said judgements to become similar to argon and work correctly.

Before:

CleanShot.2025-01-21.at.05.10.08.mp4

After:

CleanShot.2025-01-21.at.05.10.51.mp4