Bool support in TRTorch

[narendasan]

We have 3 overlapping PRs regarding bool support for certain operators. I wanted to unify the discussion here so we can establish what the best approach is here.

• Add support for bool in elementwise ops  #321
• Convert the output type of bool operation to int #327
• Convert the output type of bool operation to input type #351

For context as I understand it, there are some operators that can use booleans as masks. There is a Bool type in TensorRT but I guess the operations that support it are quite limited which is why a couple PRs are casting these bools to int. Please correct me if I am wrong and add additional detail if there are particular use cases that are important to consider.

I think we need to establish the set of operators that might need this sort of support in the case of bool and make sure we have test coverage for this set. I think there is some issues regarding making sure we cast to the type dependent on input type of the network.

I just want to get on the same page before committing to a solution here.

[narendasan]

cc: @developer0hye @inocsin

[narendasan]

Relevant issue for the sake of completeness #341

[developer0hye]

@narendasan

Actually, I think that typecasting in Boolean operator seems bad idea. I tried to check the type of tensor and converting it to float or int in "mul" operator, but I failed to implemented this by now.

[inocsin]

I think we need aten::to opration in trtorch to convert type explicitly. Either cast bool to int in Boolean op or cast data type to float in Mul op may not be a good choice.

[developer0hye]

@inocsin I agree with you. Two different operation in one layer seem not good idea.
we need aten::to operation.

[narendasan]

@peri044 You are working on casting and aten::to right?

[peri044]

Yes I started working on it. In my initial trials, I see some limitations on what types can be converted. I will update this thread by end of tomorrow with more information.

[peri044]

1. Previously I and Naren had a discussion about supporting aten:to. Here is the PR https://github.com/NVIDIA/TRTorch/pull/382. I've added the testcases related to https://github.com/NVIDIA/TRTorch/pull/327. The other two PRs 321 and https://github.com/NVIDIA/TRTorch/pull/321 don't have any different testcases.
   I currently removed the lowering of aten::to due to cast support.

2. I clarified with TensorRT team that the doc hasn't been updated https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_identity_layer.html
   TRT supports other combinations of casts as well. The conversion support can be found here https://github.com/onnx/onnx-tensorrt/blob/acb08055e71d03daa40e3e67e1b99eda0eafcf43/onnx2trt_utils.cpp#L200

3. If we have a pytorch graph where some intermediate tensors are casted to fp16 or int8, TRTorch will parse the network but engine build fails with the following error

ERROR: [TRTorch Conversion Context] - fp16 precision has been set for a layer or layer output, but fp16 is not configured in the builder
ERROR: [TRTorch Conversion Context] - Network validation failed.

Even some basic test case like this would fail with the above error

# x is fp32 tensor
out = x.to(torch.fp16)

This is due to TRT limitation. Previously with lowering of aten::to, this wouldn't be a problem as we used to ignore cast. So any layer consuming FP16 inputs in pytorch graph would have been assumed to consume fp32 inputs in TRT graph (without --fp16 flag). Solution is to use --fp16 flag
Are there any available models that you know of which have intermediate fp16 or int8 casting ? @inocsin @developer0hye
We don't want to unintentionally break any models that currently work with TRTorch for the wrong reasons.

[developer0hye]

@peri044
I searched project including .to(torch.fp16) and .to(torch.int8) in github.

However, it seems that there are few cases.

torch.fp16

torch.int8