Feature Importance via attention weights

• Added a new functionality to access feature importance via attention weights for all DL models for Tabular data except for the TabPerceiver. This functionality is accessed via the feature_importance attribute in the trainer (computed during training with a sample of observations) and at predict time via de explain method.
• Fix all restore weights capabilities in all forms of training. Such capabilities are present in two callbacks, the EarlyStopping and the ModelCheckpoint Callbacks. Prior to this release there was a bug and the weights were not restored.

pytorch-widedeep: A flexible package for multimodal-deep-learning

joss_paper_package_version_v1.2.0

Added citations file

HuggingFace model example and fixed bug related to the option of adding a FC head

1. Fixed a bug related to the option of adding a FC head on top of the "backbone" models
2. Added a notebook to illustrate how one could use a Hugginface model along with any other model in the library

Fixed the implementation of the Additive Attention

Simple minor release fixing the implementation of the additive attention (see #110 )

Self-Supervised Pre-Training for Tabular models

There are a number of changes and new features in this release, here is a summary:

1. Refactored the code related to the 3 forms of training in the library:

   • Supervised Training (via the Trainer class)
   • Self-Supervised pre-training: we have implemented two methods or routines for self-supervised pre-training. These are:
     • Encoder-Decoder Pre-Training (via the EncoderDecoderTrainer class): this is inspired by the TabNet paper
     • Constrastive-Denoising Pre-Training (via de ConstrastiveDenoising class): this is inspired by the SAINT paper
   • Bayesian or Probabilistic Training (via the BayesianTrainer: this is inspired by the paper Weight Uncertainty in Neural Networks
     

   Just as a reminder, the current deep learning models for tabular data available in the library are:

   • Wide
   • TabMlp
   • TabResNet
   • TabNet
   • TabTransformer
   • FTTransformer
   • SAINT
   • TabFastformer
   • TabPerceiver
   • BayesianWide
   • BayesianTabMlp

2. The text related component has now 3 available models, all based on RNNs. There are reasons for that although the integration with the Hugginface Transformer library is the next step in the development of the library. The 3 models available are:

   • BasicRNN
   • AttentiveRNN
   • StackedAttentiveRNN

   The last two are based on Hierarchical Attention Networks for Document Classification. See the docs for details

3. The image related component is now fully integrated with the latest torchvision release, with a new Multi-Weight Support API. Currently, the model variants supported by our library are:

   • resnet
   • shufflenet
   • resnext
   • wide_resnet
   • regnet
   • densenet
   • mobilenet
   • mnasnet
   • efficientnet
   • squeezenet

Move docs to mkdocs

Simply Update all documentation

Probabilistic Models and Label/Feature Distribution smoothing

This release fixes some minor bugs but mainly brings a couple of new functionalities:

1. New experimental Attentive models, namely: ContextAttentionMLP and SelfAttentionMLP.
2. 2 Probabilistic models based on Bayes by Backprop (BBP) as described in Weight Uncertainty in Neural Networks, namely: BayesianTabMlp and BayesianWide.
3. Label and Feature Distribution Smoothing (LDS and FDS) for Deep Imbalanced Regression (DIR) as described in Delving into Deep Imbalanced Regression
4. Better integration with torchvision for the deepimage component of a WideDeep model
5. 3 Available models for the deeptext component of a WideDeep model. Namely: BasicRNN, AttentiveRNN and StackedAttentiveRNN

Transformers without categorical data

This minor release simply fixes issue #53 related to the fact that SAINT, the FT-Transformer and the TabFasformer failed when the input data had no categorical columns

v1.0.9: The TabFormer Family Grows

Functionalities:

• Added a new functionality called Tab2Vec that given a trained model and a fitted Tabular Preprocessor it will return an input dataframe transformed into embeddings

TabFormers: Increased the Tabformer (Transformers for Tabular Data) family

• Added a proper implementation of the FT-Transformer with Linear Attention (as introduced in the Linformer paper)
• Added a TabFastFormer model, an adaptation of the FastFormer for Tabular Data
• Added a TabPerceiver model, an adaptation of the Perceiver for Tabular Data

Docs

• Refined the docs to make them cleared and fix a few typos

SAINT and the FT-Transformer

The two main additions to the library are:

• SAINT from SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training and
• FT-Transformer from Revisiting Deep Learning Models for Tabular Data

In addition

• New DataLoader for imbalanced dataset. See here.
• Integration with torchmetrics.