How to use C++ to call resnet50_trt generated "model. dali "to preprocess data?

[wangshaobobetter]

Hi!

I have build TensorRT via ONNX, and can you tell me how to use C++ to call resnet50_trt generated "model. dali "to preprocess data?

[banasraf]

Hi @wangshaobobetter.
I assume you're using Triton to perform inference. In that case to connect the DALI model with TensorRT network you should use medel ensembles.
You can see an example of resnet50 setup in our repository here.

The DALI preprocessing model is setup using the following config file:

name: "dali"
backend: "dali"
max_batch_size: 256
input [
{
    name: "DALI_INPUT_0"
    data_type: TYPE_UINT8
    dims: [ -1 ]
}
]
 
output [
{
    name: "DALI_OUTPUT_0"
    data_type: TYPE_FP32
    dims: [ 3, 224, 224 ]
}
]

The model.dali file should be placed in the model directory inside a folder named with version name, e.g. 1, so the directory structure would look like this:

model_repository/
    dali/
        config.pbtxt
        1/
            model.dali

To connect the this model together with the network you should add the ensemble model. It's configured like this:

name: "ensemble_dali_resnet50"
platform: "ensemble"
max_batch_size: 256
input [
  {
    name: "INPUT"
    data_type: TYPE_UINT8
    dims: [ -1 ]
  }
]
output [
  {
    name: "OUTPUT"
    data_type: TYPE_FP32
    dims: [ 1000 ]
  }
]
ensemble_scheduling {
  step [
    {
      model_name: "dali"
      model_version: -1
      input_map {
        key: "DALI_INPUT_0"
        value: "INPUT"
      }
      output_map {
        key: "DALI_OUTPUT_0"
        value: "preprocessed_image"
      }
    },
    {
      model_name: "resnet50_trt"
      model_version: -1
      input_map {
        key: "input"
        value: "preprocessed_image"
      }
      output_map {
        key: "output"
        value: "OUTPUT"
      }
    }
  ]
}

[wangshaobobetter]

Thank you for answering my question. As you said, I am using Triton to perform inference. I have learned the example of resnet50 setup in your repository, and I have followed the tutorial of README to build model.dali successfully. The directory structure is the same as you showed.
I want to use C++ to do the preprocessing part， can we make "Model. Dali" into something similar to a dynamic link library directly call？

@banasraf
@jantonguirao

[banasraf]

Okay, so, if I understand correctly, you have some custom piece of C++ code and you want to use it instead of DALI for preprocessing?
In that case, Triton gives a way of doing that by providing a custom backend. You can read more about implementing a custom backend here: triton-inference-server/backend. An example of such backend implemented is here.