Audio clustering - how to aggregate embeddings from NNFP

[claudiolaas]

Hi all,

I have audio clips of variable length (<10sec). I want to create embeddings and use some form of unsupervised clustering to group the clips into buckets. I have had decent success with resemblyzer+Kmeans, but i thought a different approach might yield better results. I also tried speaker diarization from pyannote.audio but that took too long.

NNFP (https://towhee.io/audio-embedding/nnfp) does indeed give me embeddings for my clips but it splits the clips into segments of 1 second length and produces embedding for each of those segments.

1. Question: does NNFP even produce valid embeddings for my clustering problem?
2. Question: how would I go about to aggregate the embeddings for a single clip?

cheers

[jaelgu]

You can just concat 10 continuous clip embeddings to make one larger embedding (dim 1280).

Or you can try some other operators who only generate 1 embedding for each audio input:

• https://towhee.io/audio-embedding/data2vec
• https://towhee.io/audio-classification/panns