Guidance on RPSI implementation

[aboba]

Fix for Issue #13

[aboba]

@fippo PTAL

[aboba]

Feedback from Stephan Wenger: https://mailarchive.ietf.org/arch/msg/avt/mrQ53xo_UI_8nR80MQQnH6CtAlY/

The proposed text is as follows:

[RFC7798] Section 8.3 specifies the use of the Reference Picture Selection
Indication (RPSI) in H.265. As noted in "Media Transport and Use of RTP in WebRTC"
[RFC8834] Section 5.1.4:

"Receivers that detect that encoder-decoder synchronization has been lost SHOULD
generate an RPSI feedback message if the codec being used supports reference-picture
selection.

[SW] Procedurally, based on recent experience, a SHOULD without explanation of that level of mandation (typically, “SHOULD unless blah blah”) will raise eyebrows. Technically, in this case, a good “unless” could be something like “unless the receiver has knowledge that the transmitter does not support RPSI. Such knowledge can be established through a) the capability exchange, but also b) through previously sent RPSI requests that were not replied to by the transmitter through the use of a non-IRAP picture.”

To explain, when you have learned that you lost a picture and a fallback to a previous identified picture is desirable, a transmitter capable of using reference picture selection would, of course, use such a reference picture. However, a slightly less smart transmitter may signal at the cap exchange time that RPSI is supported, only to treat RPSI as another form of Picture Loss Indication (PLI) and respond with an IDR/IRAP. That’s totally fine and within language and intent of RFC 5104 and 8082. However, if such a behavior is observed repeatedly, it’s a smart strategy for the receiver to discontinue the use of RPSI and fallback to sending PLI, and internally label that transmitter as RPSI-incapable. That may come handy in multipoint scenarios.

An RTP packet-stream sender that receives such an RPSI message SHOULD
act on that messages to change the reference picture, if it is possible to do
so within the available bandwidth constraints and with the codec being used."

[SW] There’s first the “SHOULD” without an “unless”, which will create heartache. However, we have a deeper problem here regarding the “bandwidth constraints”. That’s tricky, as the typical IETFer probably does not understand some of what I write below without reading up on video codec technology, and getting into such details in a profiling document like yours seems questionable. Still, as written, the guidance provided is misleading. As a transmitter, you do not have a practical option to move on with business as usual once you have received an RPSI. You have to react, or the user experience will suffer—badly. Your language could be read as if doing nothing were an option if you are in a congestion situation. It’s not.

I assume that language regarding “available bandwidth constraints” is present to appease the congestion control crowd for the usual political reasons, and for that purpose it’s probably fine, though they may also complain about the “SHOULD”. They will hopefully read it, and nod, and go on with their business. A real-world implementer will instead rely on network elasticity and ignore congestion control advise and go on with their business also. Which, so far, hasn’t brought the Internet down, either, despite gazillion of bits being sent every day by real-time video codecs.

I would remove the bandwidth constraint language. If that’s not possible, I think you need more explanation that your otherwise laudably concise document offers elsewhere. That explanation could be along the lines “if you receive RPSI, you know something is wrong in the decoder’s reference picture buffer, and you need to react or watch your product losing market share due to bad user experience. That reaction will cost bits on the wire, and that may upset the congestion control mechanics and violate constraints. Still, you have to spend those bits, otherwise the user experience will be dismal.

If you choose to stay within the bandwidth budget the congestion control algorithm suggests, or if you are up to a hard limit, you have a few options, though none of them particularly palatable. An encoder capable of RPSI necessarily is a real-time encoder, which has means to reduce its sending bitrate on a per picture basis. Two common strategies are to skip source pictures (reducing the frame rate on the wire), or dial up the quantizer (hence reduce user-perceived quality, but not as badly as it would be if an RPSI were ignored). Those, techniques, in combination with the use a reference picture indicated as useful in the RPSI, are the best known reaction by a transmitter to an RPSI under congestion. Less good but still acceptable and common implementation practice in some systems is to wait for bitrate budget and then send an IR/IRAP. The user has to cope with the frozen picture. Worst case, you can terminate the video.

[aboba]

Rewrote the PR to accomodate feeback from Stephan Wenger.

@fippo PTAL

[Philipel-WebRTC]

unless the receiver has knowledge that the sender does not support RPSI

Should this be removed? If it was negotiated then the sender supports it, if it wasn't then the assumption is that the sender does not support it. I feel like the "unless" is implied in the first part of the sentence.

[aboba]

Stephan pointed out that endpoints can negotiate support for RPSI, yet respond as if it were a PLI. So that is the "unless" part of the SHOULD.

[fippo]

But that is different, the encoder may always choose to send a new keyframe in reaction to a RPSI feedback because it determines that is "better"(the more likely case, it has sent a key frame which is more recent than the picture the RPSI feedback references in which case the "MUST act" at the bottom seems inappropriate)?

I just re-read https://www.rfc-editor.org/rfc/rfc7741#section-5.1 which uses a very different notion of RPSI. There, RPSI is used a a positive acknowledgement (presumably on every frame or periodically) whereas we treat it as a PLI replacement? Probably a good thing to discuss in the working group.

One would still need PLIs when being unable to decode a frame for a too large amount of time but want to use RPSI in cases we know that a frame is missing?

Take this series of frames:

1 - 2 - 3 - <missing or incomplete> - 5 - 6

Upon detecting that frame 4 is missing or incomplete (either by receiving the last packet of the frame with the marker bit set or the first packet of frame 5) the receiver would send a RPSI acknowledging frame 3.
The sender could act on that by sending the next frame (with number 22 because of delays) based on 3 as an IRAP.
If it does not the receiver should eventually send a PLI.

[fippo]

ah, https://www.rfc-editor.org/rfc/rfc7798#section-8.3 to the rescue:

The use of the RPSI feedback message as positive acknowledgement with
HEVC is deprecated.

Should we add this? Can we strengthen it from deprecated to "must not be used" even?

[aboba]

I added clarification, copying text from RFC 7798 Section 8.3 and adding a MUST NOT for positive acknowledgement.