Binary quantization - evaluate quality

[cduk]

Feature request

Is there a way of receiving the embeddings back in BQ format? Right now, I receive the full precision embedding and quantize it in the client, but wondering if I'm missing a way to get this directly from the server in binary format? If static quantization is done at zero, then there's no need to profile the data so the server can still remain stateless.

Motivation

More compact response from server.

Your contribution

N/A

[michaelfeil]

There is a '--embedding_dtype-' parameter for this cli. You can discover new parameters via --help command in the cli, they also come with a description.

For embedding_dtype, there are few other open source implementations. Let me know if the results from it are good. It uses an English dataset to quantize.