demo_deep_methods.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "6t4XeVORC6EZ",
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "# Demonstration of Deep Outlier Detection Models\n",
        "1. [Introduction](#1-introduction)\n",
        "2. [Demonstration on Classical Dataset](#2-demonstration-on-classical-dataset)\n",
        "    1. [Load Data](#21-load-data)\n",
        "    2. [Model Setting](#22-model-setting)\n",
        "    3. [Performance Comparation](#23-performance-comparation)\n",
        "4. [Reference](#reference)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "pJ4UJysoC6Ec"
      },
      "source": [
        "## 1. Introduction\n",
        "\n",
        "This demostration shows the performace of deep outlier detection models in several classical dataset. The models covered in this demostration includes:\n",
        "\n",
        "1. **Deep SVDD** Deep One-Class Classification. (ICML'18)\n",
        "2. **REPEN** Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection. (KDD'18)\n",
        "3. **RDP** Unsupervised Representation Learning by Predicting Random Distances. (IJCAI'20)\n",
        "4. **RCA** A Deep Collaborative Autoencoder Approach for Anomaly Detection. (IJCAI'21)\n",
        "5. **GOAD** Classification-Based Anomaly Detection for General Data. (ICLR'20)\n",
        "6. **Neutral** Neural Transformation Learning for Deep Anomaly Detection Beyond Images. (ICML'21)\n",
        "7. **ICL** Anomaly Detection for Tabular Data with Internal Contrastive Learning. (ICLR'22)\n",
        "8. **DIF** Deep Isolation Forest for Anomaly Detection. (TKDE'23)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "zygqNbtzC6Ed"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from numpy import percentile\n",
        "import pandas as pd\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.font_manager\n",
        "from tqdm.notebook import tqdm\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "wHALOvujC6Eh",
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "source": [
        "## 2. Demonstration on Classical Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "oEXhF25XC6Eh"
      },
      "outputs": [],
      "source": [
        "from pyod.utils.utility import standardizer\n",
        "from pyod.utils.utility import precision_n_scores\n",
        "from sklearn.metrics import roc_auc_score\n",
        "from scipy.io import loadmat\n",
        "from time import time\n",
        "import os"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "oa6dNX8bC6Eh"
      },
      "source": [
        "## 2.1.  Load Data\n",
        "All the following datasets are downloaded from \n",
        "Outlier Detection DataSets (ODDS): http://odds.cs.stonybrook.edu/#table1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "zxquAumxC6Eh"
      },
      "outputs": [],
      "source": [
        "mat_file_list = [\n",
        "                 'arrhythmia.mat',\n",
        "                 'cardio.mat',\n",
        "                 'ionosphere.mat',\n",
        "                 'letter.mat',\n",
        "                 'lympho.mat',\n",
        "                 'mnist.mat',\n",
        "                 'musk.mat',\n",
        "                 'optdigits.mat',\n",
        "                 'pendigits.mat',\n",
        "                 'pima.mat',\n",
        "                 'satellite.mat',\n",
        "                 'satimage-2.mat',\n",
        "                 'vertebral.mat',\n",
        "                 'vowels.mat',\n",
        "                 'wbc.mat']"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ptQ6YSUFC6Eh"
      },
      "source": [
        "## 2.2. Model Setting\n",
        "All the following models are downloaded from \n",
        "Deep learning-based Outlier Detection (deepod): https://github.com/xuhongzuo/DeepOD\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [],
      "source": [
        "from deepod.models.dsvdd import DeepSVDD\n",
        "from deepod.models.rdp import RDP\n",
        "from deepod.models.repen import REPEN\n",
        "from deepod.models.rca import RCA\n",
        "from deepod.models.goad import GOAD\n",
        "from deepod.models.dif import DeepIsolationForest\n",
        "from deepod.models.neutral import NeuTraL\n",
        "from deepod.models.icl import ICL"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### General Parameter settings\n",
        "\n",
        "#### **epochs**: int, optional (default=100)\n",
        "        Number of training epochs\n",
        "\n",
        "#### **batch size**: int, optional (default=64 for all algorithms expect Deep Isolation Forest (1000))\n",
        "        Number of samples in a mini-batch\n",
        "\n",
        "#### **lr**: float, optional (default=1e-3)\n",
        "        Learning rate\n",
        "\n",
        "#### **hidden_dims**: list, str or int, optional (default='100,50')\n",
        "        Number of neural units in hidden layers\n",
        "            - If list, each item is a layer\n",
        "            - If str, neural units of hidden layers are split by comma\n",
        "            - If int, number of neural units of single hidden layer\n",
        "\n",
        "#### **act**: str, optional ('LeakyReLU' for GOAD, NeuTral and Repen. 'Relu' for all other algorithms)\n",
        "        activation layer name\n",
        "        choice = ['ReLU', 'LeakyReLU', 'Sigmoid', 'Tanh']\n",
        "\n",
        "#### **bias**: bool, optional (default=False)\n",
        "        Additive bias in linear layer\n",
        "\n",
        "#### **epoch_steps**: int, optional (default=-1)\n",
        "        Maximum steps in an epoch\n",
        "            - If -1, all the batches will be processed\n",
        "\n",
        "#### **verbose**: int, optional (default=1)\n",
        "        Verbosity mode\n",
        "\n",
        "#### **random_state**： int, optional (default=42)\n",
        "        the seed used by the random"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Deep Isolation Forest**\n",
        "1. #### **n_estimators** int, optional (default=6)\n",
        "        The number of base estimators in the ensemble.\n",
        "\n",
        "2. #### **max_samples** int or float, optional (default=256)\n",
        "        The number of samples to draw from X to train each base estimator.\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "kjf9bq3wC6Ei"
      },
      "source": [
        "## 2.3. Performance Comparation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 662,
          "referenced_widgets": [
            "0ba3e2bdfc8c4e76aa8f8280a92d92f6",
            "55c47939ae9c4b32a207926d43c1581f",
            "9229ffcc064d479a861e70d1ce7663c2",
            "f50886559b6b460fbf761a34fe20cc8e",
            "465aab62cf8642dc993321ea2f7ff18d",
            "7396a41b53d04174ad7f4dba05aa46f9",
            "db748f475c9c485db92efe8b86bd0dc0",
            "7216dcbeaf1344b0a996e054d209dc24",
            "0fd8cf3ed87b4016b039d1d398a74bc2",
            "d8857692bc0141a0996af1fa242b7a6b",
            "81608922e1bf4ddfaa7a84ce83608358"
          ]
        },
        "id": "zrqCgyKBC6Ei",
        "outputId": "d2b65e3c-43aa-4597-91d6-b731e9056892"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "961ae59d9d22428486bd89489df64dfa",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/14 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing cardio.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 63.75it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 63.72it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing ionosphere.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 260.01it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 289.85it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing letter.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 70.43it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 70.55it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing lympho.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 575.33it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 473.86it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing mnist.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 15.19it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 15.16it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing musk.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 35.87it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 35.84it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing optdigits.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 21.74it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 21.82it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing pendigits.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 16.12it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 16.09it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing pima.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 132.75it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 133.03it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing satellite.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 17.39it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 17.40it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing satimage-2.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 19.44it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 19.47it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing vertebral.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 428.41it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 428.84it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing vowels.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 76.96it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 77.09it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "... Processing wbc.mat ...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 10/10 [00:00<00:00, 280.72it/s]\n",
            "100%|██████████| 10/10 [00:00<00:00, 282.67it/s]\n"
          ]
        }
      ],
      "source": [
        "# initialize the container for saving the results\n",
        "classifiers = [\n",
        "\t 'DeepSVDD',\n",
        "\t 'RDP',\n",
        "\t 'REPEN',\n",
        "\t 'RCA',\n",
        "\t 'GOAD',\n",
        "\t 'Neutral',\n",
        "\t 'ICL',\n",
        "     'DIF']\n",
        "\n",
        "classifiers_indices = dict(zip(classifiers, range(len(classifiers))))\n",
        "\n",
        "df_columns = ['Data', '# Samples', '# Dimensions', 'Outlier Perc'] + classifiers\n",
        "\n",
        "roc_df = pd.DataFrame(columns=df_columns)\n",
        "n_ite = 1\n",
        "n_classifiers = len(classifiers)\n",
        "\n",
        "for j in tqdm(range(len(mat_file_list))):\n",
        "    mat_file = mat_file_list[j]\n",
        "    print(\"\\n... Processing\", mat_file, '...')\n",
        "    \n",
        "    data = loadmat(os.path.join('datasets', mat_file))\n",
        "    X = data['X']\n",
        "    y = data['y'].ravel()\n",
        "    outliers_fraction = np.count_nonzero(y) / len(y)\n",
        "    outliers_percentage = round(outliers_fraction * 100, ndigits=4)\n",
        "\n",
        "    # construct containers for saving results\n",
        "    roc_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
        "    roc_mat = np.zeros(n_classifiers)\n",
        "\n",
        "    random_state = np.random.RandomState()\n",
        "    \n",
        "    classifiers_dict = {\n",
        "\t 'DeepSVDD': DeepSVDD(verbose=0),\n",
        "\t 'RDP': RDP(verbose=0),\n",
        "\t 'REPEN':REPEN(verbose=0),\n",
        "\t 'RCA': RCA(verbose=0),\n",
        "\t 'GOAD': GOAD(verbose=0),\n",
        "\t 'Neutral': NeuTraL(verbose=0),\n",
        "\t 'ICL': ICL(verbose=0),\n",
        "     'DIF': DeepIsolationForest(verbose=0),\n",
        "\t}\n",
        "\n",
        "    X_norm = standardizer(X)\n",
        "\n",
        "    for clf_name, clf in classifiers_dict.items():\n",
        "        clf.fit(X_norm)\n",
        "        test_scores = clf.decision_function(X_norm)\n",
        "\n",
        "        roc = round(roc_auc_score(y, test_scores), ndigits=4)\n",
        "        # prn = round(precision_n_scores(y, test_scores), ndigits=4)\n",
        "        roc_mat[classifiers_indices[clf_name]] = roc\n",
        "\n",
        "    roc_list = roc_list + list(roc_mat)\n",
        "    temp_df = pd.DataFrame(roc_list).transpose()\n",
        "    temp_df.columns = df_columns\n",
        "    roc_df = pd.concat([roc_df, temp_df], axis=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 582
        },
        "id": "qNkzTf2wC6Ei",
        "outputId": "a49be8cc-6c5d-4ce8-fde2-d5cd4f82ed7c"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Data</th>\n",
              "      <th># Samples</th>\n",
              "      <th># Dimensions</th>\n",
              "      <th>Outlier Perc</th>\n",
              "      <th>DeepSVDD</th>\n",
              "      <th>RDP</th>\n",
              "      <th>REPEN</th>\n",
              "      <th>RCA</th>\n",
              "      <th>GOAD</th>\n",
              "      <th>Neutral</th>\n",
              "      <th>ICL</th>\n",
              "      <th>DIF</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>cardio</td>\n",
              "      <td>1831</td>\n",
              "      <td>21</td>\n",
              "      <td>9.6122</td>\n",
              "      <td>0.5575</td>\n",
              "      <td>0.7227</td>\n",
              "      <td>0.7463</td>\n",
              "      <td>0.8647</td>\n",
              "      <td>0.1549</td>\n",
              "      <td>0.4477</td>\n",
              "      <td>0.1944</td>\n",
              "      <td>0.9122</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ionosphere</td>\n",
              "      <td>351</td>\n",
              "      <td>33</td>\n",
              "      <td>35.8974</td>\n",
              "      <td>0.3623</td>\n",
              "      <td>0.7542</td>\n",
              "      <td>0.8787</td>\n",
              "      <td>0.9036</td>\n",
              "      <td>0.8528</td>\n",
              "      <td>0.8789</td>\n",
              "      <td>0.7403</td>\n",
              "      <td>0.8995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>letter</td>\n",
              "      <td>1600</td>\n",
              "      <td>32</td>\n",
              "      <td>6.25</td>\n",
              "      <td>0.5951</td>\n",
              "      <td>0.7743</td>\n",
              "      <td>0.6695</td>\n",
              "      <td>0.7507</td>\n",
              "      <td>0.7843</td>\n",
              "      <td>0.8825</td>\n",
              "      <td>0.7996</td>\n",
              "      <td>0.658</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>lympho</td>\n",
              "      <td>148</td>\n",
              "      <td>18</td>\n",
              "      <td>4.0541</td>\n",
              "      <td>0.3286</td>\n",
              "      <td>0.5023</td>\n",
              "      <td>0.919</td>\n",
              "      <td>0.9718</td>\n",
              "      <td>0.1338</td>\n",
              "      <td>0.6866</td>\n",
              "      <td>0.5235</td>\n",
              "      <td>0.9847</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>mnist</td>\n",
              "      <td>7603</td>\n",
              "      <td>100</td>\n",
              "      <td>9.2069</td>\n",
              "      <td>0.5344</td>\n",
              "      <td>0.7715</td>\n",
              "      <td>0.7516</td>\n",
              "      <td>0.8529</td>\n",
              "      <td>0.4846</td>\n",
              "      <td>0.4825</td>\n",
              "      <td>0.728</td>\n",
              "      <td>0.8579</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>musk</td>\n",
              "      <td>3062</td>\n",
              "      <td>166</td>\n",
              "      <td>3.1679</td>\n",
              "      <td>0.9282</td>\n",
              "      <td>0.9648</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.9588</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.9778</td>\n",
              "      <td>0.9845</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>optdigits</td>\n",
              "      <td>5216</td>\n",
              "      <td>64</td>\n",
              "      <td>2.8758</td>\n",
              "      <td>0.4485</td>\n",
              "      <td>0.5283</td>\n",
              "      <td>0.5595</td>\n",
              "      <td>0.4484</td>\n",
              "      <td>0.6884</td>\n",
              "      <td>0.5609</td>\n",
              "      <td>0.5199</td>\n",
              "      <td>0.496</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>pendigits</td>\n",
              "      <td>6870</td>\n",
              "      <td>16</td>\n",
              "      <td>2.2707</td>\n",
              "      <td>0.6831</td>\n",
              "      <td>0.6637</td>\n",
              "      <td>0.9344</td>\n",
              "      <td>0.9007</td>\n",
              "      <td>0.1407</td>\n",
              "      <td>0.4686</td>\n",
              "      <td>0.5196</td>\n",
              "      <td>0.9497</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>pima</td>\n",
              "      <td>768</td>\n",
              "      <td>8</td>\n",
              "      <td>34.8958</td>\n",
              "      <td>0.4819</td>\n",
              "      <td>0.681</td>\n",
              "      <td>0.6461</td>\n",
              "      <td>0.6722</td>\n",
              "      <td>0.4549</td>\n",
              "      <td>0.6074</td>\n",
              "      <td>0.452</td>\n",
              "      <td>0.6826</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>satellite</td>\n",
              "      <td>6435</td>\n",
              "      <td>36</td>\n",
              "      <td>31.6395</td>\n",
              "      <td>0.4742</td>\n",
              "      <td>0.6824</td>\n",
              "      <td>0.7541</td>\n",
              "      <td>0.6732</td>\n",
              "      <td>0.5925</td>\n",
              "      <td>0.6162</td>\n",
              "      <td>0.5548</td>\n",
              "      <td>0.7145</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>satimage-2</td>\n",
              "      <td>5803</td>\n",
              "      <td>36</td>\n",
              "      <td>1.2235</td>\n",
              "      <td>0.5145</td>\n",
              "      <td>0.986</td>\n",
              "      <td>0.9898</td>\n",
              "      <td>0.9974</td>\n",
              "      <td>0.9938</td>\n",
              "      <td>0.7743</td>\n",
              "      <td>0.0967</td>\n",
              "      <td>0.9966</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>vertebral</td>\n",
              "      <td>240</td>\n",
              "      <td>6</td>\n",
              "      <td>12.5</td>\n",
              "      <td>0.3913</td>\n",
              "      <td>0.5517</td>\n",
              "      <td>0.3376</td>\n",
              "      <td>0.417</td>\n",
              "      <td>0.5016</td>\n",
              "      <td>0.5684</td>\n",
              "      <td>0.5094</td>\n",
              "      <td>0.3578</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>vowels</td>\n",
              "      <td>1456</td>\n",
              "      <td>12</td>\n",
              "      <td>3.4341</td>\n",
              "      <td>0.5973</td>\n",
              "      <td>0.8573</td>\n",
              "      <td>0.8671</td>\n",
              "      <td>0.8892</td>\n",
              "      <td>0.929</td>\n",
              "      <td>0.9248</td>\n",
              "      <td>0.7386</td>\n",
              "      <td>0.8331</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wbc</td>\n",
              "      <td>378</td>\n",
              "      <td>30</td>\n",
              "      <td>5.5556</td>\n",
              "      <td>0.4713</td>\n",
              "      <td>0.8483</td>\n",
              "      <td>0.9062</td>\n",
              "      <td>0.9342</td>\n",
              "      <td>0.1607</td>\n",
              "      <td>0.5018</td>\n",
              "      <td>0.3685</td>\n",
              "      <td>0.9396</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         Data # Samples # Dimensions Outlier Perc DeepSVDD     RDP   REPEN  \\\n",
              "0      cardio      1831           21       9.6122   0.5575  0.7227  0.7463   \n",
              "0  ionosphere       351           33      35.8974   0.3623  0.7542  0.8787   \n",
              "0      letter      1600           32         6.25   0.5951  0.7743  0.6695   \n",
              "0      lympho       148           18       4.0541   0.3286  0.5023   0.919   \n",
              "0       mnist      7603          100       9.2069   0.5344  0.7715  0.7516   \n",
              "0        musk      3062          166       3.1679   0.9282  0.9648     1.0   \n",
              "0   optdigits      5216           64       2.8758   0.4485  0.5283  0.5595   \n",
              "0   pendigits      6870           16       2.2707   0.6831  0.6637  0.9344   \n",
              "0        pima       768            8      34.8958   0.4819   0.681  0.6461   \n",
              "0   satellite      6435           36      31.6395   0.4742  0.6824  0.7541   \n",
              "0  satimage-2      5803           36       1.2235   0.5145   0.986  0.9898   \n",
              "0   vertebral       240            6         12.5   0.3913  0.5517  0.3376   \n",
              "0      vowels      1456           12       3.4341   0.5973  0.8573  0.8671   \n",
              "0         wbc       378           30       5.5556   0.4713  0.8483  0.9062   \n",
              "\n",
              "      RCA    GOAD Neutral     ICL     DIF  \n",
              "0  0.8647  0.1549  0.4477  0.1944  0.9122  \n",
              "0  0.9036  0.8528  0.8789  0.7403  0.8995  \n",
              "0  0.7507  0.7843  0.8825  0.7996   0.658  \n",
              "0  0.9718  0.1338  0.6866  0.5235  0.9847  \n",
              "0  0.8529  0.4846  0.4825   0.728  0.8579  \n",
              "0  0.9588     1.0  0.9778  0.9845     1.0  \n",
              "0  0.4484  0.6884  0.5609  0.5199   0.496  \n",
              "0  0.9007  0.1407  0.4686  0.5196  0.9497  \n",
              "0  0.6722  0.4549  0.6074   0.452  0.6826  \n",
              "0  0.6732  0.5925  0.6162  0.5548  0.7145  \n",
              "0  0.9974  0.9938  0.7743  0.0967  0.9966  \n",
              "0   0.417  0.5016  0.5684  0.5094  0.3578  \n",
              "0  0.8892   0.929  0.9248  0.7386  0.8331  \n",
              "0  0.9342  0.1607  0.5018  0.3685  0.9396  "
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "roc_df"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "71M6BKLFC6Ej"
      },
      "source": [
        "## Reference"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8T8dbzZgC6Ej"
      },
      "source": [
        "1. Ting et al. [**Isolation Distributional Kernel A New Tool for Point & Group Anomaly Detection**](https://ieeexplore.ieee.org/abstract/document/9573389) *IEEE Transactions on Knowledge and Data Engineering*, 2021.\n",
        "2. Bandaragoda et al. [**Isolation‐based anomaly detection using nearest‐neighbor ensembles.**](https://proceedings.neurips.cc/paper_files/paper/2022/file/cf93972b116ca5268827d575f2cc226b-Paper-Datasets_and_Benchmarks.pdf) *Computational Intelligence*, 2018.\n",
        "3. Han et al. [**Adbench: Anomaly detection benchmark**](https://proceedings.neurips.cc/paper_files/paper/2022/file/cf93972b116ca5268827d575f2cc226b-Paper-Datasets_and_Benchmarks.pdf) *Advances in Neural Information Processing Systems*, 2022.\n",
        "4. [**DeepOD** (github.com/xuhongzuo/DeepOD)](https://github.com/xuhongzuo/DeepOD/tree/main)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There are some other useful demonstrations:\n",
        "* https://github.com/dennishnf/unsupervised-anomaly-detection\n",
        "* https://github.com/tvhahn/anomaly-pyconca/blob/master/anom-detection-milling.ipynb\n",
        "* https://github.com/Apress/beginning-anomaly-detection-using-python-based-dl\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0ba3e2bdfc8c4e76aa8f8280a92d92f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_55c47939ae9c4b32a207926d43c1581f",
              "IPY_MODEL_9229ffcc064d479a861e70d1ce7663c2",
              "IPY_MODEL_f50886559b6b460fbf761a34fe20cc8e"
            ],
            "layout": "IPY_MODEL_465aab62cf8642dc993321ea2f7ff18d"
          }
        },
        "0fd8cf3ed87b4016b039d1d398a74bc2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1c01d76b74a44818949b56bd483530b7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "266bbd62c27349e0939a805e4390649d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3984d76827ae43b08f373235ce787cf8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "40cd0e71627d4d50ae9b864c983c6632": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "42772caecc8d4f09b37b601731a3fa94": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "465aab62cf8642dc993321ea2f7ff18d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "52c76771605a47618324c5661d5d74b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8e1555e31e924d4da2a8a56a95d5bfb3",
              "IPY_MODEL_dde6f0e2f54a4b57a47a46055edc55d1",
              "IPY_MODEL_8856795c89e74dfba28c5b0afa346515"
            ],
            "layout": "IPY_MODEL_40cd0e71627d4d50ae9b864c983c6632"
          }
        },
        "55c47939ae9c4b32a207926d43c1581f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7396a41b53d04174ad7f4dba05aa46f9",
            "placeholder": "​",
            "style": "IPY_MODEL_db748f475c9c485db92efe8b86bd0dc0",
            "value": "100%"
          }
        },
        "7216dcbeaf1344b0a996e054d209dc24": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7396a41b53d04174ad7f4dba05aa46f9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "81608922e1bf4ddfaa7a84ce83608358": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8856795c89e74dfba28c5b0afa346515": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_266bbd62c27349e0939a805e4390649d",
            "placeholder": "​",
            "style": "IPY_MODEL_3984d76827ae43b08f373235ce787cf8",
            "value": " 4/4 [00:37&lt;00:00,  8.07s/it]"
          }
        },
        "8e1555e31e924d4da2a8a56a95d5bfb3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f3a5a133f4f4469395853f02be0e24e2",
            "placeholder": "​",
            "style": "IPY_MODEL_f4718d1d2e834689bbd6a763b17db48e",
            "value": "100%"
          }
        },
        "9229ffcc064d479a861e70d1ce7663c2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7216dcbeaf1344b0a996e054d209dc24",
            "max": 17,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0fd8cf3ed87b4016b039d1d398a74bc2",
            "value": 17
          }
        },
        "d8857692bc0141a0996af1fa242b7a6b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "db748f475c9c485db92efe8b86bd0dc0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "dde6f0e2f54a4b57a47a46055edc55d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_42772caecc8d4f09b37b601731a3fa94",
            "max": 4,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1c01d76b74a44818949b56bd483530b7",
            "value": 4
          }
        },
        "f3a5a133f4f4469395853f02be0e24e2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f4718d1d2e834689bbd6a763b17db48e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f50886559b6b460fbf761a34fe20cc8e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d8857692bc0141a0996af1fa242b7a6b",
            "placeholder": "​",
            "style": "IPY_MODEL_81608922e1bf4ddfaa7a84ce83608358",
            "value": " 17/17 [02:23&lt;00:00,  9.90s/it]"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}