Don't Separate, Learn to Remix: End-to-End Neural Remixing with Joint Optimization

Open-source code for paper - H. Yang, S. Firodiya, N.J. Bryan, M. Kim, "Don't Separate, Learn to Remix: End-to-End Neural Remixing with Joint Optimization." ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022

• Paper: https://arxiv.org/abs/2107.13634
• Webpage: https://minjekim.com/research-projects/neural-remixer/

Prerequisites

Environment

• Python 3.6.0
  
• torch 1.8.0
  
• torchaudio 0.8.0
  

Data

• Download MUSDB18 from - https://sigsep.github.io/datasets/musdb.html#musdb18-compressed-stems
• Download Slakh from - http://www.slakh.com/

Conv-TasNet model package

Asteroid - https://github.com/asteroid-team/asteroid
asteroid/models/conv-tasnet.py

Training steps

1. Data processing

• Prepare MUSDB for training or validation - python mus_process.py Set Dataset to "train" or "test" per needed.
• Prepare MUSDB for evaluation - python mus_eval_process.py
• Prepare Slakh for training or validation - python slakh_process.py Set Dataset to "train" or "test" per needed.
• Prepare Slakh for evaluation - python slakh_eval_process.py

3. Model training

Important Parameters:

Symbol	Description
n_src	Number of sources contained in the mixture
weight_src	source separation loss scale
weight_mix	remix loss scale
transfer_model	The model label to start transferring training from
trainset	dataset. MUSDB or Slakh
train_loss	SDR or SDSDR
with_silent	Whether or not using the data, has actually contains less number of sources than the number the model is designed on
baseline	Whether or not training baseline
ratio_on_rep	Whether or not having ratios applying on the representation feature space

Examples:

Scenario 1: we are training the following models, on MUSDB with 4 sources, and we want the ratio of loss weights to be 1:4.

• Model 1 - python train_model.py --n_src 4 --weight_src 1 --weight_mix 4 --trainset MUSDB
• Model 2 - python train_model.py --n_src 4 --weight_src 1 --weight_mix 4 --trainset MUSDB --ratio_on_rep
• Baseline - python train_model.py --n_src 4 --weight_src 1 --weight_mix 4 --trainset MUSDB --baseline

Model Evaluation

To get the remix output of specific one track python3 eval_on_samples.py --model_name 0826_123024 --n_src 4 --dataset MUSDB --loss_f SDSDR