RFC6716

https://datatracker.ietf.org/doc/html/rfc6716#section-2.1.7

Forward Error Correction (FEC)

Another mechanism providing robustness against packet loss is the in-band Forward Error Correction (FEC). Packets that are determined to contain perceptually important speech information, such as onsets or transients, are encoded again at a lower bitrate and this re-encoded information is added to a subsequent packet.

Opus FEC

https://lists.xiph.org/pipermail/opus/2013-January/001904.html

Adding FEC seems to change the encoded audio bit-stream itself, i.e., it doesn't just add additional protection bits, but also changes the encoded bits. This is easy to show by comparing two decoded versions of the same audio file: one encoded with no FEC, and another with FEC. Is this expected or is it a bug?

If you encode with the same target rate with FEC on and off, then while FEC is on, the encoder will reduce the rate of the non-FEC portion in order to achieve the same total rate. So yes, this changes the resulting bitstream.

Related to the above, what does the FEC information consist of? How does the decoder use it? Does it simply repeat preceding packets, or interpolate preceding and subsequent packets, or does it do something smarter?

The FEC information consists of "LBRR frames" in the SILK layer. The bitstream syntax for these is largely the same as a regular SILK frame. See https://datatracker.ietf.org/doc/html/rfc6716#section-4.2.5 for some details.

The decoder uses it to replace the last Opus frame of a dropped packet. This is done by calling opus_decode() with a pointer to the next packet and the decode_fec flag set to 1 (see
https://opus-codec.org/docs/opus_api-1.3.1/group__opus__decoder.html#ga7d1111f64c36027ddcb81799df9b3fc9). If no FEC is available in the packet, normal PLC is used instead.

Although it would be legal to have it simply be a repeat of the preceding packet (as long as the preceding packet was an active frame), the encoder actually requantizes it with a reduced bitrate. This lets it do smarter rate allocation than a simple approach like RFC 2198 or RFC
6354 redundancy, which always repeats whole packets. Nothing prevents you from using one of those codec-agnostic methods with Opus, however.

Based on my tests, the FEC in opus seems to have a limit, i.e., it adds correction but only up to approx 4---10% expected packet loss, depending on the file. What are the parameters/limits of the FEC?

You can find the details in silk_setup_LBRR() in silk/control_codec.c. Basically, there is a base bitrate at which LBRR gets activated for each audio bandwidth (NB, MB, and WB). For small amounts of packet loss (less than 25%), this threshold gets boosted (by up to 25% when there is no reported packet loss). As long as the target rate is above this threshold, LBRR gets turned on.

When it's enabled, it boosts the quantization gain on LBRR frames (w.r.t. normal frames) by a factor that decreases as the packet loss percentage goes up. At a packet loss percentage of 0, this gain gets increased by a factor of approximately 3. As the packet loss percentage goes up, the increase in quantization goes down until it hits a factor of approximately 1.37 at 12.5% packet loss.

Does FEC add any algorithmic delay? i.e., does it require the decoder to wait until post-loss packets are received in order to generate a replacement for the lost packets? If so, what is the maximum delay and can it be variably set?

A decoder must wait for one future packet to fill in a missing hole using FEC. This cannot be variably controlled (i.e., the maximum additional delay is 1 packet). Of course, a decoder is always free to ignore FEC if it does not want this additional delay, or opportunistically use whatever happens to be in its jitter buffer, etc.

How to enable in-band FEC for Opus codec

In order to make OPUS encoder to add FEC a user has to set the following configuration:

• Packet time(ptime) has to be not less than 10ms otherwise OPUS works in the CELT mode and not in SILK.
• Don’t use very high bitrates. For example, I use 24 kbps.
• In order to use FEC, your bitrate should be higher. For example, if sample rate is 8kHz then bitrate has to be 12 kbps or 24 kbps. The encoder needs higher bitrate to have a room for LBRR packets containing FEC.
• FEC must be enabled via OPUS_SET_INBAND_FEC.
• The encoder must be told to expect packet loss via OPUS_SET_PACKET_LOSS_PERC.

Now, after the encoder is ready, let’s configure the decoder. First of all, you need to know if a packet is lost. The most natural way to know it is to check the sequence number of RTP packets. When you know that the previous packet is lost you are supposed to decode the current packet twice: first time with FEC turned on to reproduce the lost packet, and then with FEC turned off to decode the current packet.

https://datatracker.ietf.org/doc/html/rfc8854

3.3. Codec-Specific In-Band FEC

Some audio codecs, notably Opus [RFC6716] and Adaptive Multi-Rate
(AMR) [RFC4867], support their own in-band FEC mechanism, where
redundant data is included in the codec payload. This is similar to
the redundant encoding mechanism described above, but as it adds no
additional framing, it can be slightly more efficient.

For Opus, audio frames deemed important are re-encoded at a lower
bitrate and appended to the next payload, allowing partial recovery
of a lost packet. This scheme is fairly efficient; experiments
performed indicate that when Opus FEC is used, the overhead imposed
is only about 20-30%, depending on the amount of protection needed.
Note that this mechanism can only carry redundancy information for
the immediately preceding audio frame; thus the decoder cannot fully
recover multiple consecutive lost packets, which can be a problem on
wireless networks. See [RFC6716], Section 2.1.7, and this Opus
mailing list post [OpusFEC] for more details.

For AMR and AMR-Wideband (AMR-WB), packets can contain copies or
lower-quality encodings of multiple prior audio frames. See
[RFC4867], Section 3.7.1, for details on this mechanism.

In-band FEC mechanisms cannot recover any of the RTP header.

https://blog.mozilla.org/webrtc/audio-fec-experiments/

We collect telemetry on dropped packets. The telemetry values are permille, so even at the 75th percentile, we’re seeing around 1.854% reported packet loss. The threshold for enabling LBRR within Opus is around 1%, so if our telemetry is accurate, FEC would not have any effect on about 66% of calls made. The tests were performed at the 75th and 95th percentiles, so 1.854% simulated packet and 19.435% packet loss.