DeepAREstimator Inference takes forever

[DeastinY]

Training a DeepAREstimator, then serialize and deserialize it and running make_evaluation_predictions, even only with one sample, on new data it's incredibly slow. Stopped it after 30 minutes of trying to create a forecast (through next(forecast_it)).
Am I doing something wrong here? Data sizes are small and GPU utilization is constantly at 100% ...
Will try some other approaches and keep this updates.

[DeastinY]

I've also tried it with predictor.predict() instead. The result is the same (takes forever).
Maybe it's due to around 70ish exogenous dynamic timeseries being included, but it's still inference, right?!
If I use it immediately it doesn't take that long either :/

[DeastinY]

Works with SimpleFeedForwardEstimator without issues

[DeastinY]

in case anyone finds this: NaN in test data was the issue :)

[mbohlkeschneider]

That's weird since we handle NaN in test data explicitly. If you find time to share the code sample that causes the bug, this would be very useful for us.

[DeastinY]

Hi @mbohlkeschneider. It'd be pretty troublesome to decouple from the data, so I'll probably get to it in the near future.
Maybe it's something else and I'm creating the dataset wrong? But adding df = df.fillna(-1000000) to mask NaN was all it needed to work again :/

[moghadas76]

I have the same issue