Custom dictionaries below level 5

[rachel-bousfield]

Custom dictionaries can be attached during compression and decompression using C APIs like BrotliEncoderAttachPreparedDictionary. However, it appears they aren't used below brotli level 5. This causes a kind of silent failure where the user doesn't observe the lack of improvement.

There's a few solutions to this issue

1. The attach and/or prepare methods could document the behavior.
2. These and BrotliEncoderPrepareDictionary could fail when an incompatible level is applied.
3. The dictionary format could include the minimum brotli level for compatibility & API-simplification reasons.
4. Decide this is a bug and implement the feature for lower levels (not backwards compatible)

Here's an example in case the issue isn't clear

brotli -0 -D dictionary.lz -o dict
brotli -0 no-dict
diff dict no-dict    # would use <() but this doesn't work either 

[eustas]

Thanks for reporting. I'll check and take action when I get spare cycles.
Agree, that at least CLI should let users know if dictionary is ignored.

[rachel-bousfield]

I'd also look into how the C API could be improved. That's actually how I first discovered this issue. I was rather confused why my dictionary wasn't working :)

[pmeenan]

FWIW, ZStandard allows for dictionary use down to 0 (though, practically you don't see big benefit until 2-3).

I don't know enough about brotli's actual encoding to know if it makes sense, but it could be useful to allow for dictionaries to work at the lower levels as well if that allows for lower CPU use but still decent savings.