NN_2022_2012633_DiMarzo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sdnnRxTNdyW1"
      },
      "source": [
        "# Neural Networks\n",
        "## Homework 1: Implementing advanced activation functions\n",
        "\n",
        "**Name**: Gabriele G. Di Marzo\n",
        "\n",
        "**Matricola**: 2012633\n",
        "\n",
        "Upload the completed notebook **before 30/11/2022 at 23:59** on the Google Classrom page."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "PEr8qV6-nMuL"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import matplotlib.pyplot as plt\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wAEgygyPfO7b"
      },
      "source": [
        "### Objective\n",
        "\n",
        "The purpose of this homework is to implement a new layer inside PyTorch, by properly extending the `nn.Module` object. **Before proceeding**, carefully read the following documentation:\n",
        "\n",
        "+ [nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=module#torch.nn.Module)\n",
        "+ [PyTorch: Custom Module](https://pytorch.org/tutorials/beginner/examples_nn/polynomial_module.html)\n",
        "\n",
        "You can also (optionally) learn more about activation functions by reading the survey in [1]."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYYk6NnFXqYr"
      },
      "source": [
        "### Introduction: families of activation functions\n",
        "\n",
        "In the course up to now, we have seen the application of the sigmoid, the softmax, and the ReLU. However, many additional activation functions exist [1], with varying strengths and drawbacks.\n",
        "\n",
        "Several of them are designed as variants of the ReLU. The **S-Shaped ReLU** (SReLU) [2] is defined as:\n",
        "\n",
        "$$\n",
        "\\phi(s) = \\begin{cases} t^r + a^r(s - t^r) & \\text{ if } s > t^r \\\\ s & \\text{ if } t^r > s > t^l \\\\ t^l + a^l(s - t^l) & \\text{ if } s < t^l \\end{cases} \\,.\n",
        "$$\n",
        "\n",
        "The four parameters $t^r, a^r, t^l, a^l$ are trained via back-propagation, and they are **different for each unit in the layer**. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5u4aF6Z4maHd"
      },
      "source": [
        "### Exercise 1: implementing an activation function (1 point)\n",
        "\n",
        "Let us start with the simpler **[exponential linear squashing](https://paperswithcode.com/method/elish)** (ELiSH) activation function:\n",
        "\n",
        "$$\n",
        "\\phi(x) = \\begin{cases} \\sigma(x)x & \\text{ if } x \\ge 0 \\\\ \\frac{\\exp(x) - 1}{1 + \\exp(-x)} & \\text{ otherwise} \\end{cases}\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKdIjBuZgc5o"
      },
      "source": [
        "**Exercise 1**: complete the following stub."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "DfCpFg2xdug_"
      },
      "outputs": [],
      "source": [
        "def elish(x):\n",
        "  # x is a generic torch.Tensor, and this function must compute the ELiSH activation function.\n",
        "    return torch.where(x>=0, torch.sigmoid(x)*x, torch.sigmoid(x)*(torch.exp(x) - 1 ))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o8MEZxqbn8vs"
      },
      "source": [
        "**Hints for a correct implementation**:\n",
        "\n",
        "1. There are several ways of implementing an if/else operation like the one above in PyTorch. In general, the simplest implementation of \"*if a then b, else c*\" is `torch.where(a, b, c)` (see the documentation for [torch.where](https://pytorch.org/docs/stable/generated/torch.where.html)). Any working variant is accepted here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9a6dBsb_I11"
      },
      "source": [
        "Here is a simple sanity check for the correct implementation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d2rkMj9r_UJG",
        "outputId": "e7f2ac8f-92f3-4993-f310-2ba272dc1c25"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 0.1100, -0.1323]])"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "elish(torch.FloatTensor([[0.2, -0.4]])) # Should be approximately [[0.11, -0.13]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uv1-249GftP2"
      },
      "source": [
        "### Exercise 2: some visualization experiments (1 point)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RWY76mEepSgj"
      },
      "source": [
        "**Exercise 2.1**: plot the ELiSH function in [-5, +5]."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "jdK0CyscfDtC",
        "outputId": "6ed03601-207c-484f-da63-38ce243e21b3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAdz0lEQVR4nO3deXyV5Z338c8vJxtJIMgWZNGgKBVBwARIdWoJruNSO08dN8CiQNpXa7VTtdPWmXGmtdMZfR61zlgtixtbtC7jVG2ttaRqK0vCpmyyqOx7FrKf5Xr+OIFQG7LAObnv5Hzfr1deZ7s5+fZq/ObKfd/nus05h4iI+FeS1wFERKR1KmoREZ9TUYuI+JyKWkTE51TUIiI+lxyPN+3Xr5/Lzc2Nx1u3W01NDZmZmZ5m8AuNRTONRTONRTM/jEVZWdlB51z/ll6LS1Hn5uZSWloaj7dut5KSEiZNmuRpBr/QWDTTWDTTWDTzw1iY2Wcnek27PkREfK5dRW1mvc3sJTPbaGYbzOyL8Q4mIiJR7d318XPgt865G8wsFciIYyYRETlOm0VtZtnAJcB0AOdcI9AY31giInKUtbXWh5mNBWYD64ExQBlwt3Ou5nPbFQFFADk5OXnFxcVxCdxe1dXVZGVleZrBLzQWzTQWzTQWzfwwFoWFhWXOufyWXmtPUecDS4GLnXPLzOznQJVz7p9P9G/y8/OdzvrwD41FM41FM41FMz+MhZmdsKjbczBxJ7DTObes6fFLwIWxCiciIq1rs6idc3uBHWY2oumpS4nuBhERkU7Q3rM+vgMsbDrjYxtwe/wiiYjI8dpV1M651UCL+05ERCS+9MlEERGfU1GLiPicilpExOdU1CIiPqeiFhHxORW1iIjPqahFRHxORS0i4nMqahERn1NRi4j4nIpaRMTnVNQiIj6nohYR8TkVtYiIz6moRUR8TkUtIuJzKmoREZ9TUYuI+JyKWkTE51TUIiI+p6IWEfE5FbWIiM+pqEVEfE5FLSLic8nt2cjMPgWOAGEg5JzLj2coERFp1q6iblLonDsYtyQiItIi7foQEfE5c861vZHZJ0A54IBfOudmt7BNEVAEkJOTk1dcXBzjqB1TXV1NVlaWpxn8QmPRTGPRTGPRzOuxOFAb4cZrLi074W5l51ybX8DgptsBwBrgkta2z8vLc15bsmSJ1xF8Q2PRTGPRTGPRzMux2H6oxl30s3ccUOpO0Knt2vXhnNvVdLsfeBWYcOq/Q0REEtuOw7XcPHsp1Q2hVrdrs6jNLNPMeh69D1wBfBSTlCIiCWrH4VpumbOUI/VBFs6c2Oq27TnrIwd41cyObr/IOffbU48pIpKYdpZHS7qqLsjCmQWMGpzd6vZtFrVzbhswJlYBRUQS2c7y6O6OqrogC2ZOZPSQ1ksadHqeiEin2VVRd2wmvWDmRC4Y0rtd/64jH3gREZGTtKuijptnf0BFbXSfdHtLGjSjFhGJu90VddwyeykVtUEWzOhYSYOKWkQkrvZURnd3lNc0Mn/GRMYM7VhJg4paRCRu9lbWc/PspRyubuT5GRMYexIlDdpHLSISF3sr67llzlIONZX0uDNOO+n30oxaRCTG9lXVc+ucpeyvque5OyZw4SmUNKioRURian9VdCa9r6mk8848tZIG7foQEYmZA0cauGXOUvZWRks6P7dPTN5XM2oRkRg4cKSBW+csZXdFPc9MH8/4GJU0qKhFRE7ZweoGpsxdys7yOp65fTwTz+ob0/dXUYuInILDNY1MnbuM7YdrmTc9n4IYlzSoqEVETlp5TSNT5i7jk4M1zL1tPBed3S8u30cHE0VETkJFbSNT5y1j64Fq5t6Wz9+cE5+SBs2oRUQ6rLIuyLR5y9m8r5rZ0/K45Nz+cf1+KmoRkQ6oqg9y29PL2bi3iqemXcikEQPi/j1V1CIi7XSkPsjXn17Oul2V/GJKHpO/kNMp31dFLSLSDtUNIW5/ZgUf7qzkv2+9kMtHdk5Jgw4mioi0qbYxxB3PrmDVjgr+65ZxXDVqYKd+f82oRURaUdcYZsazpZR+ephHbxrL1aNP7/QMmlGLiJxAfTDMrOdLWfrJIR69cSxfGTPIkxyaUYuItKAhFOYb88v409aDPHzDGL46brBnWVTUIiKf0xiK8K0FK/njxwf42d+N5oa8IZ7mUVGLiBwnGI7wncUreWfjfh786ihunnCG15HaX9RmFjCzVWb2ejwDiYh4JRSO8N3i1by1bh8PXDeSqQVneh0J6NiM+m5gQ7yCiIh4KRxx3POrNbzx4R7uv/o8br94mNeRjmlXUZvZEOAaYG5844iIdL6Ic3z/pbW8tno39105glmXnOV1pL9gzrm2NzJ7CfgZ0BO41zl3bQvbFAFFADk5OXnFxcUxjtox1dXVZGVleZrBLzQWzTQWzTQWURHnmLu6hj/vM746PIWvDk/1JEdhYWGZcy6/pdfaPI/azK4F9jvnysxs0om2c87NBmYD5Ofnu0mTTrhppygpKcHrDH6hsWimsWimsQDnHA/87zr+vK+Wbxeezb1XjMDMvI71V9qz6+Ni4Ctm9ilQDEw2swVxTSUiEmfOOR58YwPPf/AZV+Wm+LakoR1F7Zz7oXNuiHMuF7gZ+INzbmrck4mIxIlzjofe2sS89z9h+kW53DQixbclDTqPWkQS0GO/38yTJVu5deIZPHDdSF+XNHRwrQ/nXAlQEpckIiKd4IklW/j5O5v5+7whPHj9KN+XNGhGLSIJZO5723j4rU1cP3YQ//G1C0hK8n9Jg4paRBLE8x98yoNvbODq0QP5f38/hkAXKWlQUYtIAnhhxXb+5bV1XHZeDj+/eRzJga5VfV0rrYhIB726aic/eOVDLjm3P09MGUdKFytpUFGLSDf2xto93PPiGgqG9WX2tDzSkgNeRzopKmoR6ZZ+v34fdxev4sIzTmPe9HzSU7pmSYOKWkS6oXc/PsC3Fq7k/EG9eOb28WSkdu2rDqqoRaRbWbrtEEXzSzl7QBbP3TGBnukpXkc6ZSpqEek2yj4r545nVzDktAwWzJhA7wxvVsKLNRW1iHQLH+2qZPozyxnQM41FMyfSNyvN60gxo6IWkS5v094jTJu3jF7pKSycVcCAXuleR4opFbWIdGnbDlQzZe4yUpOTWDRrIoN79/A6UsypqEWky9pxuJYpc5fhnGPhzALO7JvpdaS46NrnrIhIwtpbWc+tc5dS2ximuKiA4QO672XFNKMWkS7nYHUDU+YupbwmyPN3TOC803t5HSmuVNQi0qVU1DYyde4ydlXU8czt4xkztLfXkeJORS0iXcaR+iBff2YF2w7UMOe2fMbn9vE6UqdQUYtIl1DXGGbGs6Ws21XJL6ZcyJfO6e91pE6johYR32sIhSmaX0rpZ4d59KaxXDYyx+tInUpnfYiIrwXDEe5ctIr3Nh/koa9dwHVjBnkdqdNpRi0ivhWOOO791RreXr+Pf71uJDeOH+p1JE+oqEXEl5xz3P/qh7y2ejf3XTmC6RcP8zqSZ1TUIuI7zjkefGMDxSt28K1JZ/PtwuFeR/JUm0VtZulmttzM1pjZOjP7t84IJiKJ69Hfb2be+58w/aJc7rtyhNdxPNeeg4kNwGTnXLWZpQDvm9lvnHNL45xNRBLQ7He38vg7m7khbwj/cu1IzMzrSJ5rs6idcw6obnqY0vTl4hlKRBLTomXb+fc3N3LN6NP5z69dQFKSShrAoj3cxkZmAaAMGA484Zz7xxa2KQKKAHJycvKKi4tjHLVjqqurycrqvou0dITGopnGopnfxuKD3SFmr21gdP8Ad41LI7kTS9oPY1FYWFjmnMtv8UXnXLu/gN7AEmBUa9vl5eU5ry1ZssTrCL6hsWimsWjmp7F466M97qwfvuFufOrPrq4x1Onf3w9jAZS6E3Rqh876cM5VNBX1Vaf2u0NEJOpPWw5y56JVjBqczbzp40lPCXgdyXfac9ZHfzPr3XS/B3A5sDHewUSk+yv7rJxZz5cyrF8mz90+nqw0fVi6Je0ZldOB55r2UycBLzrnXo9vLBHp7tbvruL2povRzp/Zfa4YHg/tOetjLTCuE7KISILYdqCa255eRlZaMgtmTmRAz+51MdpY0ycTRaRT7aqoY+rcZTgHC2ZOZMhpGV5H8j3tEBKRTrP/SD1T5izlSEOI4qICzurvn9MD/UwzahHpFJW1QW6bt5x9VQ08e/t4zh+U7XWkLkNFLSJxV9MQYvqzy9l2oIbZt+WRd2ZiXEIrVrTrQ0Tiqj4YvTrL2p2VPHFrYl1CK1Y0oxaRuAmFI9y1eBV/2nKIh752AVeNGuh1pC5JRS0icRGJOL7/0lp+13R1lq/lDfE6UpelohaRmHPO8ePX1/PKql3cc/m5CX11llhQUYtIzD369sc8++dPmfWlYdw5ObGvzhILKmoRiam5723j8T9s4ab8ofzo6vO08H8MqKhFJGZeWLGdB9/YwNWjB/Lv/2e0SjpGVNQiEhNvfriHH77yIZec25/HbhpHQFdniRkVtYicsnc/PsDdxau48IzTeGrqhaQmq1piSaMpIqek7LPDfGN+GcMH9GTe9PFkpOpzdLGmohaRk7Z+dxXTn1nBwOx0nr9jAtk9UryO1C2pqEXkpHxysObYmtLzZ0ygf880ryN1WypqEemwPZXRNaUjDubP0JrS8aaiFpEOOVTdwNS5y6isC/Lc7RMYPkBrSsebilpE2u1IfZDpz6xgZ3kd876ez+ghWlO6M6ioRaRd6oNhZj5XyoY9VTw59UImntXX60gJQ+fRiEibguEI3164kuWfHuaxm8Yy+Qs5XkdKKJpRi0irIhHHvb9awzsb9/Pj60dx/djBXkdKOCpqETkh5xz/+ut1vLZ6N/ddOYJpBWd6HSkhqahF5IQeeftjnv/gM4ouOYtvTTrb6zgJq82iNrOhZrbEzNab2Tozu7szgomIt+a+t43/alqu9Id/+wWthOeh9hxMDAH3OOdWmllPoMzM3nbOrY9zNhHxyIsrdvDgGxu4ZvTpWq7UB9qcUTvn9jjnVjbdPwJsAHQ0QaSbevPDPfzglbVccm5/Hr1prJYr9QFzzrV/Y7Nc4F1glHOu6nOvFQFFADk5OXnFxcWxS3kSqqurycrSJ6ZAY3E8jUWzlsbio4MhHi1rYFh2Evflp5OWnBgl7Yefi8LCwjLnXH5Lr7W7qM0sC/gj8FPn3CutbZufn+9KS0s7HDSWSkpKmDRpkqcZ/EJj0Uxj0ezzY1H22WGmzl1Obr9MimcVkJ2ROCvh+eHnwsxOWNTtOuvDzFKAl4GFbZW0iHQ9f7VcaQKVdFfQnrM+DJgHbHDOPRL/SCLSmbRcqf+1Z0Z9MTANmGxmq5u+ro5zLhHpBLsqtFxpV9Dm6XnOufeBxDiiIJJAqhoc0+Yuo6ouyOKiAi1X6mNalEkkAVXWBfm/pfXsr4/OpEcN1nKlfqaPkIskmNrGEDOeXcGu6ghPTc1jfG4fryNJG1TUIgmkIRTmG/PLWLm9nG+OSWPSiAFeR5J20K4PkQQRCke4a/Eq3tt8kIduuIAB1Vu9jiTtpBm1SAKIRBz/+PKHvLVuHw9cN5Ib84d6HUk6QEUt0s0dXVP65ZU7+d7l53L7xcO8jiQdpKIW6eYeemvTsTWlvzN5uNdx5CSoqEW6sSeWbOHJkq1MmXiG1pTuwlTUIt3UM3/6hIff2sTfjRvMT64fpZLuwlTUIt3QCyu282+/Xs+V5+fw8A0XkKQ1pbs0FbVIN/Pa6l384JUP+fK5/Xn8lnEkB/SfeVen/wdFupG31u3ley+uYUJuH56amkdacsDrSBIDKmqRbmLJpv3cuWglowdnM2/6eHqkqqS7CxW1SDfwpy0H+eb8Ms7N6clzd0wgK00fOu5OVNQiXdzyTw4z87lScvtmMn/GRLJ76Oos3Y2KWqQLW7W9nDueXcHpvdNZMHMifTJTvY4kcaCiFumi1u6s4Lanl9MnM5VFMwt0Ca1uTEUt0gWt213JtHnLye6RwuKiAgZmp3sdSeJIRS3SxWzae4Spc5eRmRpg8awCBvfu4XUkiTMVtUgX8vG+I9w6ZympyUksmlXA0D66GG0iUFGLdBGbm0o6kGQsnlVAbr9MryNJJ1FRi3QBm/cd4ZY5S0kyY3FRAWf11xXDE4mKWsTnoiW9DDNj0awCzlZJJxwVtYiPbdobnUmbweJZExk+QCWdiNosajN72sz2m9lHnRFIRKI27q3i1qbdHcVFBQwf0NPrSOKR9syonwWuinMOETnO+t1V3DpnGcmBaElrd0dia7OonXPvAoc7IYuIAGt2VHDLnKWkJyfxQtEXdeBQMOdc2xuZ5QKvO+dGtbJNEVAEkJOTk1dcXByjiCenurqarCz9gIPG4nh+H4vN5WEeKasnK8X4/vh0+mfE7zCS38eiM/lhLAoLC8ucc/ktvRaztRCdc7OB2QD5+flu0qRJsXrrk1JSUoLXGfxCY9HMz2PxwdZDPPqHFQzsncnCmRMZFOdPHPp5LDqb38dCi9aK+MCSjfv55oIyhvbJYNHMiQzopbU7pJmKWsRjb6zdw3dfWMWIgT15/g4tVSp/rT2n5y0GPgBGmNlOM5sR/1giieHF0h18Z/FKxg7tzaJZBSppaVGbM2rn3C2dEUQk0cx5dxs/fXMDXzqnH7+clkdGqv7AlZbpJ0Okkznn+M/fbuKpP27lmtGn88hNY3S1cGmVilqkE4XCEe5/9SNeKN3BlIln8OPrRxFIMq9jic+pqEU6SV1jmO8sXsnvN+znrkvP4R8uOwczlbS0TUUt0gkOVTcw47lS1uys4CfXn8+0L+Z6HUm6EBW1SJxtP1TL159Zzu6KOp6cksdVowZ6HUm6GBW1SByVfVZO0fOlhCKOhTMnkp/bx+tI0gWpqEXi5PW1u/nei2s4PTudp6eP1wp4ctJU1CIx5pzjFyVbefitTeSfeRqzb8vXB1nklKioRWKoPhjmBy+v5X9W7+YrYwbx0A0XkJ6ic6Tl1KioRWJkb2U935hfypqdldx7xbl8u3C4Tr+TmFBRi8TAyu3lfHN+GTUNIWZPy+OK83Vmh8SOilrkFDjnWLBsOz/+9ToGZqfz/IyL+MLAXl7Hkm5GRS1ykuqDYe5/9SNeXrmTwhH9eeymcWRnpHgdS7ohFbXISdiyv5o7F61k074jfPeyc7hr8jkkac0OiRMVtUgHvbJyJ//0Px+RnhLg6enjKRwxwOtI0s2pqEXaqbohxAOvrePllTuZMKwPj988joHZumSWxJ+KWqQdyj47zD+8sIad5bXcNXk4d116DsmB+F0hXOR4KmqRVgTDER5/ZzNPLNnCoN49eOEbX2S81uuQTqaiFjmBj3ZVct9La9mwp4ob8obwwHUj6Zmuszqk86moRT6nPhjm8Xc288t3t9E3M1UfYBHPqahFjvPe5gM88No6th2s4cb8Idx/9UidGy2eU1GLEF2n4ydvrOeNtXvI7ZvB/BkT+NI5/b2OJQKoqCXB1TWGmff+Np4s2Uow4vje5edSdMlZWvFOfKVdRW1mVwE/BwLAXOfcf8Q1lUicRSKO19bs4qHfbmJPZT1XjMzh/mvO48y+mV5HE/krbRa1mQWAJ4DLgZ3ACjP7X+fc+niHE4k15xy/W7eXR97+mI17jzB6cDaP3jSWgrP6eh1N5ITaM6OeAGxxzm0DMLNi4HrghEVd2xhmzY4KkswwgyQzkpKabg3M7Nj9o9sEko4+Z033ISnJCBzdNgkCTa9pjV/pqEjE8YeN+/npB/V8UlXGsH6ZPHbTWL4yZpDW6BDfM+dc6xuY3QBc5Zyb2fR4GjDROXfn57YrAooAUgcOzzv964/FJzFESxyi5Q8EkqLPBZrKP2CAi5CSnBQtd4u+npwUfS1a+Mc9ToJkM5KbnktOMpKbtk85+rjpfkrAordJkJJkpASi91OTjNRA9PW0JEgJRH8J+UF1dTVZWYl5vb6GkOP93SHe/jTI3lpH3zTHV89J46JByQQSvKAT+efi8/wwFoWFhWXOufyWXovZwUTn3GxgNsCIUWPdk9PzCUeif2pGXPQ27BzOQcQ5IsfuR2c7kabXIxFHOBL9NxEXvd/8PMfuhyKOcCRCOBLdLhiOEHGOUDj6b3bt2Uvf/v0Jhh2hcIRQJPpaKBKJPheJUB9yBEMRgpEIobCjMRShMRwhGA7TGIoQaf13WJvSkpPokRqgR0qAHqkBMlIDZKQk0yM1QGZagIzUZLLSkslIDZCZFr1/9LZnevQ2Kz16v1d6CmnJSSf110RJSQmTJk06tf8xXUgk4lj6ySFeW7WbNz/aw5H6EGOG9uZH1w8j49AmLptc6HVEX0i0n4vW+H0s2lPUu4Chxz0e0vTcCfVMT2byF3JOJdcpiw583im9RygcLe7GUPSr4dhXtMjrgxHqQ2EagtHn6oNh6hrD1Ici0dtgmLpgmNrG6PN1wTA1DSHKaxvZWR6itjH6uKYxTLgdvxVSAkav9BR69UihV3py9LZHCtnHffXukULvjBSye6TSOyN6vzF8ir9xuoDqhhAfbD3Eux8f4O31+9hbVU9maoArzx/IlIIzufCM3pgZJSUfex1VpMPaU9QrgHPMbBjRgr4ZuDWuqXwiOZBEciCJjDhfQNo5R0MoQnVDiJqGENUNIarro7dH6kMcqQ9SVf+X9yvrglTVBdlVUUdVXZDKuiDBVgo5bclvOC0jWt6nZaTSJzN6P3qbSp/M6PPHvjJTyEpL9uXxAOccuyrqWLOjktU7ylm9o4LVOyoIhh0ZqQEuHt6P+8ecx2Xn5dAjVafZSdfXZlE750JmdifwFtHT8552zq2Le7IEYmakpwRITwnQLyvtpN7DOUddMExFbTD6VddIZW2Q8togK9dtpM/AoZTXNFJeG6SitpGNe6uO3T/RZD4lYGT3SOW0pnLPzjh+xt48m++ZnkzP9Oht5tHdOWkBUgMd31XjnKOmMUxFbSMVtUEOHGlgT2U9eyrr2Flex5b91Ww9UE1tYxiA1OQkzh/Uizv+ZhhfPrc/eWeeRlqyylm6l3bto3bOvQm8GecscgrMjIzUZDJSkxnUu8dfvDaobhuTJp3X4r+LRBxV9UEO1zRSXttIeU0welvbXOrlNdHi33G4lg+bfgnUByNtZgokGWnJSaSnREs7OWAkJ9mxg3hHj1cEw46GUPNuo5Z2AyUZDOyVztkDsrgxfyhnD8jigsHZnHd6L1KTtdyodG/6ZGKCS0oyemdEd390REMo3LT7Jbo75kjTrpmaxhC1Tfvdj+6nbwhFqA+Gjx0YDoUdWPPpmoGkpr8okgP0SE1q2teeSq8eKQzolcbp2en0z0rT+s+SsFTUclLSkgMM6BlgQE+vk4h0f5qiiIj4nIpaRMTnVNQiIj6nohYR8TkVtYiIz6moRUR8TkUtIuJzKmoREZ9rcz3qk3pTswPAZzF/447pBxz0OINfaCyaaSyaaSya+WEsznTOtXhF5bgUtR+YWemJFuFONBqLZhqLZhqLZn4fC+36EBHxORW1iIjPdeeinu11AB/RWDTTWDTTWDTz9Vh0233UIiLdRXeeUYuIdAsqahERn0uIojaze8zMmVk/r7N4xcweNrONZrbWzF41s95eZ+pMZnaVmW0ysy1m9gOv83jFzIaa2RIzW29m68zsbq8zec3MAma2ysxe9zrLiXT7ojazocAVwHavs3jsbWCUc+4C4GPghx7n6TRmFgCeAP4WGAncYmYjvU3lmRBwj3NuJFAAfDuBx+Kou4ENXodoTbcvauBR4PtAQh81dc79zjkXanq4FBjiZZ5ONgHY4pzb5pxrBIqB6z3O5Ann3B7n3Mqm+0eIFtRgb1N5x8yGANcAc73O0ppuXdRmdj2wyzm3xussPnMH8BuvQ3SiwcCO4x7vJIHL6SgzywXGAcu8TeKpx4hO5CJeB2lNl7+4rZn9HhjYwkv3Az8iutsjIbQ2Fs6515q2uZ/on78LOzOb+IuZZQEvA991zlV5nccLZnYtsN85V2Zmk7zO05ouX9TOuctaet7MRgPDgDVmBtE/9Vea2QTn3N5OjNhpTjQWR5nZdOBa4FKXWCfQ7wKGHvd4SNNzCcnMUoiW9ELn3Cte5/HQxcBXzOxqIB3oZWYLnHNTPc71VxLmAy9m9imQ75zzeoUsT5jZVcAjwJedcwe8ztOZzCyZ6AHUS4kW9ArgVufcOk+DecCis5bngMPOue96nccvmmbU9zrnrvU6S0u69T5q+Qv/DfQE3jaz1Wb2lNeBOkvTQdQ7gbeIHjx7MRFLusnFwDRgctPPweqmGaX4WMLMqEVEuirNqEVEfE5FLSLicypqERGfU1GLiPicilpExOdU1CIiPqeiFhHxuf8PTXcj+q+VcowAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "vals = torch.arange(-5., 6., 0.1)\n",
        "x = torch.arange(-5., 6., 0.1, requires_grad=True )\n",
        "plt.plot(vals, elish(vals))\n",
        "plt.xlim([-5, 5])\n",
        "plt.grid(True)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GnvkoRVAqwg1"
      },
      "source": [
        "**Exercise 2.2**: using the utilities from `torch.autograd` ([torch.autograd](https://pytorch.org/docs/stable/autograd.html)), **compute and plot** the derivative of ELiSH using automatic differentiation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "0QJKAtmWf3ZI",
        "outputId": "2e2a8051-3f1e-4048-d576-6d8621f4bb21"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8deZJTsCigS+QIWquFSrErRuXyQIioJr3ZeKG9XWar9V24JdrLXWr7ttta0V/VnF5ueKQlUESlwqKotLVRAV2SLIbkgyWWbmfP84EydiSCZkJvdm5v18PO5j7szc3PnkEN5z5sy95xprLSIi4l8BrwsQEZG2KahFRHxOQS0i4nMKahERn1NQi4j4XCgTO+3Tp48dPHhwJnadstraWoqLiz2twS/UFklqiyS1RZIf2mLhwoUbrLW7tvZcRoJ68ODBLFiwIBO7TlllZSUjR470tAa/UFskqS2S1BZJfmgLY8yK7T2noQ8REZ9LKaiNMb2MMU8YY5YYYxYbYw7LdGEiIuKkOvRxN/CCtfY0Y0weUJTBmkREpIV2g9oY0xMYAUwAsNY2Ao2ZLUtERJqZ9ub6MMYcCNwHfAAcACwErrLW1m6z3URgIkBpaWlZRUVFRgpOVU1NDSUlJZ7W4BdqiyS1RZLaIskPbVFeXr7QWju8tedSCerhwOvAEdbaN4wxdwPV1tpfbu9nhg8fbnXUh3+oLZLUFklqiyQ/tIUxZrtBncqXiauB1dbaNxL3nwCGpas4ERFpW7tBba1dC6wyxuyVeOho3DCIiIh0gVSP+vgRMDVxxMcy4MLMlSQiIi2lFNTW2reBVsdOREQks3RmooiIzymoRUR8TkEtIuJzCmoREZ9TUIuI+JyCWkTE5xTUIiI+p6AWEfE5BbWIiM8pqEVEfE5BLSLicwpqERGfU1CLiPicglpExOcU1CIiPqegFhHxOQW1iIjPKahFRHxOQS0i4nMKahERn1NQi4j4nIJaRMTnFNQiIj6noBYR8blQKhsZY5YDW4EYELXWDs9kUSIikpRSUCeUW2s3ZKwSERFplYY+RER8zlhr29/ImE+BzYAF/mqtva+VbSYCEwFKS0vLKioq0lxqx9TU1FBSUuJpDX6htkhSWySpLZK8bou1aws4++zDFm53WNla2+4CDEjc9gXeAUa0tX1ZWZn12ty5c70uwTfUFklqiyS1RZKXbfHpp9butpu1wAK7nUxNaejDWluVuF0HPA0c0vn3EBGR3LZ8OYwcCdXVbW/XblAbY4qNMT2a14FjgPc6X6KISO5avhzKy+GLL2D27La3TeWoj1LgaWNM8/aPWmtf6GyRIiK5asUKF9JbtriQHjas7e3bDWpr7TLggDTVJyKS01ascMMdW7bArFlQVtb+z+jwPBGRLrJyZbInPWsWDE/x1MGOnPAiIiI7aOVK15PetMkNd6Qa0qAetYhIxq1a5XrSmzZ1rCfdTD1qEZEMWr3ahfSGDS6kDz644/tQj1pEJEOqqtxwx/r18OKLcMgOnoGiHrWISAZUVbme9Lp1LqS/850d35eCWkQkzT77DEaNgjVrXEgfemjn9qehDxGRNFqzxoX0Z5/BCy/AYYd1fp/qUYuIpMnnn7uQXr3ahfQRR6RnvwpqEZE0aA7plSvh+efhyCPTt28NfYiIdNK6dXD00W6ipeeegxEj0rt/9ahFRDphwwYYPRqWLYN//hOOOir9r6GgFhHZQRs3upD+6COYPt0djpcJCmoRkR2waROMGQNLlsCzz7rAzhQFtYhIB23ZAsccA++/D88849YzSUEtItIBX3wBxx4L774LTz8NY8dm/jUV1CIiKaqudsG8aBE8+SSMG9c1r6vD80REUrB1Kxx/PCxYAI89Biee2HWvrR61iEg7amth/Hh4/XWoqIBTTuna11dQi4i0oa4OTjgBXn0Vpk6F007r+hoU1CIi2xGJwEknQWUlPPwwnHWWN3VojFpEpBUNDXDqqTBnDjz4IJx7rne1qEctIrKNxkY3xPHCC/C3v8EFF3hbj3rUIiItNDW5IY4ZM+DPf4ZLLvG6og4EtTEmaIx5yxgzI5MFiYh4JRp1QxxPPw133w2XXeZ1RU5HetRXAYszVYiIiJdiMTfE8fjjcNttcOWVXleUlFJQG2MGAuOA+zNbjohI14vH4eKL4dFH4aab4Oqrva7oq4y1tv2NjHkC+D3QA7jGWju+lW0mAhMBSktLyyoqKtJcasfU1NRQUlLiaQ1+obZIUlskqS2ceBxuuWUIM2fuxoQJn3LBBSs8qaO8vHyhtXZ4q09aa9tcgPHAvYn1kcCM9n6mrKzMem3u3Llel+AbaosktUWS2sLaeNzaH/7QWrB28mR33yvAArudTE1l6OMI4ERjzHKgAhhljHmk8+8fIiLesdYNcdxzD5x55kpuvBGM8bqq1rUb1NbaSdbagdbawcBZwL+stedlvDIRkQyxFiZPhjvvdF8afv/7y3wb0qDjqEUkB/3mN3DzzfD978Ndd/m3J92sQ2cmWmsrgcqMVCIi0gV+/3sX1BdeCPfe6/+QBvWoRSSH3HGHG/I45xx3anigmyRgNylTRKRz7rnHfXl42mnw0EMQDHpdUeoU1CKS9aZMgSuucFdlefRRCHWz6egU1CKS1R55BC691F2Q9rHHIBz2uqKOU1CLSNZ6/HE3f8fIkW6ipfx8ryvaMQpqEclK06e7Lw0PO8ytFxZ6XdGOU1CLSNZ58UX3peFBB8Fzz0FxsdcVdY6CWkSyyksvwcknwz77uCu07LST1xV1noJaRLLGvHkwbhwMHgyzZsHOO3tdUXooqEUkKyxaBMcdB/37uwvS7rqr1xWlj4JaRLq9996DY46BXr1cSPfv73VF6aWgFpFubelSGD3aHXo3Zw584xteV5R+3ez8HBGRpOXL4eij3VVaKith9929rigzFNQi0i1VVcGoUVBb60J67729rihzFNQi0u2sW+eGOzZscMMd3/621xVlloJaRLqVTZtgzBhYscKd2HLwwV5XlHkKahHpNqqr3SF4S5bAjBlw5JFeV9Q1FNQi0i3U1cH48e546aeecr3qXKGgFhHfa2iAU06Bf//bzSd9wgleV9S1FNQi4mtNTXDmmW48esoUt55rdMKLiPhWLAYTJsAzz8Af/gAXXeR1Rd5QUIuIL1kLl13mhjpuugl+9COvK/KOglpEfMdadyHa+++HSZPcksvaDWpjTIEx5k1jzDvGmPeNMb/pisJEJHddfz3ceafrRf/ud15X471UvkxsAEZZa2uMMWHgVWPM89ba1zNcm4jkoNtugxtucGPTd90FxnhdkffaDWprrQVqEnfDicVmsigRyU333QfXXgunn+6GPQIanAXAuBxuZyNjgsBCYA/gHmvtz1rZZiIwEaC0tLSsoqIizaV2TE1NDSUlJZ7W4BdqiyS1RZLf2mL27L7cdNM+HHLIJn772/cIh7uuP+iHtigvL19orR3e6pPW2pQXoBcwF9ivre3Kysqs1+bOnet1Cb6htkhSWyT5qS2mTbM2GLT2qKOsravr+tf3Q1sAC+x2MrVDHyystVsSQT22c+8dIiLOnDlwxhlQVgbTp0NhodcV+U8qR33saozplVgvBMYASzJdmIhkv3nz4KSTYOhQeP556NHD64r8KZWjPvoDDyXGqQPAY9baGZktS0Sy3TvvwPHHu+sbZtMVwzMhlaM+3gUO6oJaRCRHLF3qLkbbowfMng39+nldkb/p4BcR6VIrV7qrs1jrQnq33byuyP80e56IdJm1a93FaKur3XUOhw71uqLuQUEtIl1i82Y49lj47DM3Jn3ggV5X1H0oqEUk42pq3BeHzZfQOvxwryvqXhTUIpJR9fVw8skwfz48/nhuXUIrXRTUIpIx0SicfbY7qeWhh9zltKTjdNSHiGREPO6uyDJtmrs6y/e+53VF3ZeCWkTSzlr48Y/h4Yfht7/N7auzpIOCWkTS7te/hj/+0V2l5brrvK6m+1NQi0ha3XGH60VffDHceqsm/k8HBbWIpM2UKa4Xfdpp8Ne/KqTTRUEtImnxxBMwcaI7qWXqVAgGva4oeyioRaTTXnwRzjkHDjsMnnwS8vK8rii7KKhFpFNee80dH73vvu6sw+JiryvKPgpqEdlhzXNKDxgAM2dCr15eV5SdFNQiskM++ig5p/SsWVBa6nVF2UtBLSIdtnq1m7MjHnchrTmlM0tzfYhIh6xf70J60yaYOxf23tvrirKfglpEUlZdDccdB8uXuzHpsjKvK8oNCmoRSUkkAiee6L5AnDYNRozwuqLcoaAWkXY1NcEZZ8DLL7uTWcaN87qi3KKgFpE2xeMwYYI7Rvree9380tK1dNSHiGyXtXDllfDoo/C738Hll3tdUW5SUIvIdv3qV3DPPXDNNTBpktfV5K52g9oYM8gYM9cY84Ex5n1jzFVdUZiIeOuOO+DGG910pbfcopnwvJTKGHUUuNpau8gY0wNYaIyZZa39IMO1iYhHHnjATVd6+umartQP2u1RW2vXWGsXJda3AouBAZkuTES88cQTcOmlbrrSRx7RdKV+YKy1qW9szGDgZWA/a231Ns9NBCYClJaWllVUVKSvyh1QU1NDSUmJpzX4hdoiSW2R1FpbzJ/fm8mT92evvbZy663vUFgY96i6ruWHv4vy8vKF1trhrT5prU1pAUqAhcCp7W1bVlZmvTZ37lyvS/ANtUWS2iJp27b497+tLSqy9oADrN20yZuavOKHvwtggd1OpqZ01IcxJgw8CUy11j6VnvcPEfGLbacr7d3b64qkpVSO+jDAFGCxtfaOzJckIl1J05X6Xyo96iOA84FRxpi3E8vxGa5LRLrAypUwerSmK/W7dg/Ps9a+CujgHJEss3lzmDFjYMsWTVfqd5rrQyQHbdkCP/3pt6mqchemHTbM64qkLQpqkRxTWwvjx8Py5cVMnw5HHul1RdIezfUhkkMaGtwVw+fNg1/8YjFjx3pdkaRCPWqRHBGNuilKZ81yp4gPGbLe65IkRepRi+SAeBwuuQSefhruvhsuvNDriqQjFNQiWa55TumHHoIbbnDr0r0oqEWy3OTJyTmlf/ELr6uRHaGgFsliN90EN98Ml12mOaW7MwW1SJb6wx/guuvgvPNcj1oh3X0pqEWy0JQpcNVV7lC8Bx+EgP6nd2v65xPJMv/4h5v4f+xYtx7SQbjdnoJaJItMmwbnnw8jRsCTT0J+vtcVSTooqEWyxPPPwxlnwPDhMH06FBV5XZGki4JaJAvMmQOnngr77QcvvODmlpbsoaAW6eZeeQVOPBH22MPNhNerl9cVSbopqEW6sTfegHHjYNAgmD0b+vTxuiLJBAW1SDe1YAEceyzsuqsb+tAltLKXglqkG3r7bXedw9693dVZBgzwuiLJJAW1SDfz3nvuOoclJfCvf8E3vuF1RZJpCmqRbuT992HUKHd89L/+BUOGeF2RdAUFtUg38cEHLqRDITfcscceXlckXUVBLdINfPABlJdDMOhCeuhQryuSrqRZAER8rrknHQi44Y699krDTuNNhOJboW41RGshFoFYA8Qb3K2NQjzqbm0MsO4KBM1MADDu1gQTSwgCYQiEwIQhmAeBlks+BPMhWODWA4qfVKmlRHzsvffg6KOTIb333tvZMNYIkSqoWwV1n0H9GoishfrPoWEjNG6Exk3QuAWaqiEW4UiAaV34y2zLBF1oBwsgWAiBFuuhxP1QkbvfvIRarAeL2rm/zc8FCiAQ9PAX3nHtBrUx5gFgPLDOWrtf5ksSEYD//MeFdPOY9F57NsEXH8MXH0D1Eqj5BLZ+7G4jawD71R0E8qCgL+T3gbxdoNcgyOsF4Z4Q3omPlq9lz70PgFBxIigTPd5AfouecaK3jHGLMYmedbzFbQzisUTvOwrxphZLo+ulxxtb9NjrE+v1ifXmJdLiNgJNW9wbTjSSfKx52VGBcOIN4KtvDMNqG2F23+Qbx5fP5SfWW3wS+LKN8pLrLR9rXr72iSIv8fqJWxNO3LY/UXgqPer/B/wJ+PuOt46IdMR/3qrlussXcnn5In5y0Vv0/OQteGuJC79mhf2hZA/oNwaKB0PxN6BoEBQNcM+Fe7UZAlUbKtlzj5EZ/13SztqvBnq07utBHou0CPgW235t3d2PRj4DLDRu3mYYaJshoW3fDNPBBF1gt6HdoLbWvmyMGZymkkSkNXVVsO4lWP8qtStfZ5+6d3n2yph7rrYf9D4I/ut46Pkt6Lkv9NgLwiXe1uwVY1yPOFSYtl2+W1nJyJEj297IWveJoeUng3jjVz8tfLneuM3S1Mp64lOHTdxy6/Z/ZWvbf4dIBPWMtoY+jDETgYkApaWlZRUVFe3uN5NqamooKcnRP+RtqC2S/NIWwXgNvRsWsXPDfHo1vE1RbDUAjbaYVxcfyjtVwxl2zK6E+w6hMbhzRmrwS1v4gR/aory8fKG1dnhrz6UtqFsaPny4XbBgQUdqTLvKVN4hc4TaIsnTtqheCqufgarpsOE1N7Yb6gGlI6HvSBasGsmo7x5Av/5B5sxxEy1lkv4ukvzQFsaY7Qa1jvoQyRRrYct/YOVjsOopqF7sHu99IOz7M+h/LPQ5DAJhnnsOvvtdd6bhnDnQv7+3pYu/KKhF0m3rJ7D8EVhR4Y7OMAHoOxL2vBwGngjFu31l88cfh3PPhf33h5kzNVWpfF0qh+f9AxgJ9DHGrAZ+ba2dkunCRLqVpmpY8f/h07/D+lcBA32Pgr2ugkGnusPkWvHgg3DJJXD44TBjBvTs2bVlS/eQylEfZ3dFISLdjrWwYR58cr8L6Vgd7LQ3HPB7GHIeFA1s88dvvx2uucZNV/rUU1Bc3EV1S7ejoQ+RjmqqcUMbH93rxqBDJTD4XNj9YtjlkHZPYLAWJk2C//1fOP10ePhhXS1c2qagFknV1k9g6R9h2YNuqKP3gXDIfbDbWRBO7Wqy0ShcdhlMmeJu//QnN9GSSFsU1CJtsRbWvQwf3gmrn3WnVQ86DYZe4Y7YSOH032Z1dXDWWTB9OvzqV3D99R36cclhCmqR1sSj7pC6xbfApoWQvwt8azLs+QMo+q8O7279ejjhBHjzTbjnHvjBDzJQs2QtBbVIS9GIG9pYcjvULIMee8LBf4Eh57uZ3HbAsmUwdiysWgVPPgmnnJLmmiXrKahFABq/cF8OfngX1K+DXQ6Fg26DASd2amrMefPgpJPc2PTs2XDEEWmsWXKGglpyW/16F85L/+S+IOx/LOw7CfqO6PQA8mOPwfe+BwMHwj//maYJ/yUnKaglN9V9Botvg4//6qayHPRd+NYk2HlYp3dtLdx8M0ye7HrQ06bpbEPpHAW15JT86FqY/wP4ZIqbFGnwua4H3XN7l07pmEgELr0Upk6Fs8+GBx6AgoK07FpymIJackM8Bu/8nO+suxM2BOCbF8K+P4eSIWl7iaoq90Xh/Plw442uR63D7yQdFNSS/WIN8Np5sOoJPi86nv5j/gLF6Z1D9PXX4dRTYetWN9Rx0klp3b3kuIDXBYhkVNNWqBwHq56Ag27jw17XpjWkrYU//xlGjHBDHK+9ppCW9FNQS/aq+wzmlMO6Sjj0Idjn6rTuPhKBCy90J6+MGQMLF7qpSkXSTUMfkp3Wz4NXToXoVhgxDQaMT+vulyyBM890Vwq//nr45S8hoG6PZIiCWrLPx3+DBT90V+QeNQt6pXQFuZQ9/DBcfjkUFrrjo487Lq27F/ka9QEkezRVw7wJ8OZE6FsOx85Pa0hv3QoTJriTWMrK4O23FdLSNdSjluyw7hWY9z2oWwn7/RL2+3WnTv3e1muvwfnnw/LlbpjjV7+CkP73SBdRj1q6t2gtvPVTmH2Uuzbh6Ffg2zekLaSbmlww//d/QzwOL70EN9ygkJaupT836Z6shdXTYOGPXS9690tg2B0pT+CfikWL4KKL4J133JDH3XfDTjulbfciKVNQS/ez5T3Xi17zPPTaHw5/Bfoembbd19e7XvMtt0DfvjqBRbynoJbuo/oj+M+vYUWF6zkPuwOG/shddSVNZs2CK66ApUtdb/q226B377TtXmSHKKjF/zYthCV3uoAO5Ls5Ova5BvJ3TttLVFXBT37ipibdYw948UV3EouIHyioxZ9iDVA1A5b+wV2zMFQCQ6+EfX8GhaVpe5m6OrjzTjctaVOTG/K49lrNeCf+klJQG2PGAncDQeB+a+3NGa1KcpO1sHE+fPp3WPEPaNwExbvBQbfD7hdDXs+0vVQ8Do8+CpMmwerVcPLJbphj993T9hIiadNuUBtjgsA9wBhgNTDfGPOstfaDTBcnOSDW4HrMq5+BqmehbhUEC2DgyTDkAug3Oq1j0NbCM8+446DffdeduPLII3DUUWl7CZG0S+V/wCHAx9baZQDGmArgJGC7QR3ZWssHr8zHBAIYYzCBAIFAwN0PBAgEEo8FA5hAEBMIEAwGMcEAwWCAQDBIIBggEAoSTKwHE+vBkNteE/12U001sPktWPcSfF4JG/4NsXoIFkL/Y2D/38CgUyCvV1pfNh53p3tfe+0wPvwQ9tzTBfTZZ2uODvE/Y61tewNjTgPGWmsvSdw/H/iOtfaKbbabCEwEKBtC2YIbM1Nws1g8QDQWIhYPEo2HiMVDxGzQ3cYDifuhL5+L2yDReJi4DRKzzbch4oTc47j1uA0TNyEsidsvlzAEwhAMuZMpAmFMKIQJhTHBMIFwkGA4RDAcJpgfIJQXhmAecfKIm3ziJg9r0nemXEfU1NRQUlLSpa9pbIz82FqKoyspiq6guOlTejQtpSi6EkPc1RXanS35B7I5fxib84YRD6R/YDgSCTBzZj+eemogq1YV0bdvHRMmrOSYYz4nGGz7bz/befF34Vd+aIvy8vKF1trhrT2Xts+U1tr7gPsA9tljTzu/6C7i8RhYi7UWG49jv7wfx8bjQOLWxtzzNg7xWGK7ONbGvnwO6x7DRt16PPbluiEGtgmT2N4QpSGylcKCMMZGCdCEMVECuPWgaSIUiJJnat26aSIYaCIcaCQYbCIcaCIv1EA42EReqJG8UFNqjRAHGhJLK6LxII3RQpriBTTFC4jaQmIUEjOF2IBbTKgQEy4kmFdIKL+IUH4heUWFFBQVEioowoQKXe/zy6UgsRS6IyK+vJ/v7gfyqHz5VUaOHNm5f+B41J0FGK11M9I1bnZLwyao/xzq10JkrRu6qF3ubm0s+fMF/aBvGex8AexyMOzyHUoK+lACDOxcZV8vNXEG4dSp8MQT8MUXcMgh7rjoPn3mM3r0UUB6Lr3VnVVWVnb+7yJL+L0tUgnqKqDlTOsDE49tV3Gvnhx88rjO1NVplZWV/HcnGz4ahYYGqKmP0xBppLGhicZIA42RBpoaGt16fQPRxgaaGhqINtQTbagn1tRAvLGeeFM98Wg9NloPsQg21oCJRwjaCAEbIWTcEjYR8sP1FIbXU5gXoTAvQjivjvzEen64sVO/x1EEoCLxiSCQBybYYgkAJjmU9OUbYgzijS2Wdt6sAvlQUOom5d/1CPclYMk3Yad9oOc+kJfZg5G3boW5c2HmTDcGXVUFJSXu0liXXw6HHup+xcrK3O5FS/eUSlDPB/Y0xgzBBfRZwDkZrconQiG3FBcHgILEkr5TlJtZ686G27o1uVRVQ3W16w1Wb4lRW11P3dYI9TURIrUR6msjNEYiNEXqaaqPEGuMEDQNFITrKQjXkx9uID/UQH64gbxQI4X5TfQobmKn4kaKimIUFcYoLIhRWBCnIN+SXwAF+Za8giD5BUEKCgKE8/MwwTwX7sEiCBW7JdzDBW+4l7st7Afhnl32vYG1sHIlvPkmvPFGcmlqguJiGD0abr8dTjgBioq6pCSRjGo3qK21UWPMFcBM3OF5D1hr3894ZTnEGDe3cWGhO2X564JAcWJpnbXumOBNm766bNwICxYso2fPb/L+Rti4xj22cSNs2OC2icdb32c4DDvvDLvs4padd04uvXtDr17utmfP5NKjh1tKSiAvr+PZbS3U1CTrX7PGHT63apWbuW7xYjdpf22t2z4/Hw46CP7nf2DsWDj8cPeYSDZJaYzaWvsc8FyGa5FOMMb1JouLYdA2lwQcOnQlI0d+s9Wfi8dhyxYX2hs2fDXEm9c3bnSh+emnsGCBW49E2q8pGHQnjhQWuvAMh5OfUppfOx53PeFIxL3RRCIQi319X4EADBgAe+8NF1/sbocPhwMOcG8IItlMZybmuEAg2UseOjT1n2togM2bXch/8UVyqalJLrW1Lnjr691tNOpCuKnJvbEEAm4JhZKfKIqKXC+9udfevz8MHAj9+mlqUcld+tOXHZKf78KzXz+vKxHJfjrUX0TE5xTUIiI+p6AWEfE5BbWIiM8pqEVEfE5BLSLicwpqERGfU1CLiPhcu/NR79BOjVkPrEj7jjumD7DB4xr8Qm2RpLZIUlsk+aEtdrPW7traExkJaj8wxizY3iTcuUZtkaS2SFJbJPm9LTT0ISLicwpqERGfy+agvs/rAnxEbZGktkhSWyT5ui2ydoxaRCRbZHOPWkQkKyioRUR8LieC2hhztTHGGmP6eF2LV4wxtxpjlhhj3jXGPG2M6eV1TV3JGDPWGPOhMeZjY8zPva7HK8aYQcaYucaYD4wx7xtjrvK6Jq8ZY4LGmLeMMTO8rmV7sj6ojTGDgGOAlV7X4rFZwH7W2m8DS4FJHtfTZYwxQeAe4DhgX+BsY8y+3lblmShwtbV2X+BQ4Ic53BbNrgIWe11EW7I+qIE7gZ8COf2tqbX2RWttNHH3dWCgl/V0sUOAj621y6y1jUAFcJLHNXnCWrvGWrsosb4VF1ADvK3KO8aYgcA44H6va2lLVge1MeYkoMpa+47XtfjMRcDzXhfRhQYAq1rcX00Oh1MzY8xg4CDgDW8r8dRduI5c3OtC2tLtL25rjJkNtHaJ1euAybhhj5zQVltYa59JbHMd7uPv1K6sTfzFGFMCPAn82Fpb7XU9XjDGjAfWWWsXGmNGel1PW7p9UFtrR7f2uDFmf2AI8I4xBtxH/UXGmEOstWu7sMQus722aGaMmQCMB462uXUAfRUwqMX9gYnHcpIxJowL6anW2qe8rsdDR/r6WG4AAADCSURBVAAnGmOOBwqAnYwxj1hrz/O4rq/JmRNejDHLgeHWWq9nyPKEMWYscAdwlLV2vdf1dCVjTAj3BerRuICeD5xjrX3f08I8YFyv5SFgk7X2x17X4xeJHvU11trxXtfSmqweo5av+BPQA5hljHnbGPMXrwvqKokvUa8AZuK+PHssF0M64QjgfGBU4u/g7USPUnwsZ3rUIiLdlXrUIiI+p6AWEfE5BbWIiM8pqEVEfE5BLSLicwpqERGfU1CLiPjc/wEfyllPuBENngAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# TODO: plot the gradient of the ELiSH function\n",
        "x = torch.arange(-5, 6, 0.1, requires_grad=True )\n",
        "plt.plot(vals, elish(vals), 'blue')\n",
        "plt.xlim([-5, 5])\n",
        "plt.grid(True)\n",
        "\n",
        "y = torch.sum(elish(x))\n",
        "y.backward()\n",
        "\n",
        "plt.grid(True)\n",
        "plt.plot( vals, x.grad.detach().numpy(), 'orange')\n",
        "plt.xlim([-5, 5])\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yqnx7jRxf2qy"
      },
      "source": [
        "**Exercise 2.3 (sanity check)**: build a model using the previously defined activation function, and test it on a random mini-batch of data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-YLbAbBYfJ7D",
        "outputId": "e2d48d17-ceda-46d5-d907-83d851c847ca"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[-0.0776,  0.2435],\n",
            "        [-0.4077,  0.1592],\n",
            "        [-0.2564, -0.0069],\n",
            "        [-0.0840, -0.2627],\n",
            "        [-0.1061,  0.0534]], grad_fn=<AddmmBackward0>)\n"
          ]
        }
      ],
      "source": [
        "# TODO: complete the definition of the model\n",
        "class RandomModel(torch.nn.Module): \n",
        "  def __init__(self, funct): \n",
        "    super().__init__()\n",
        "    self.l1 = torch.nn.Linear(10, 10)\n",
        "    self.l2 = torch.nn.Linear(10, 2)\n",
        "    self.funct = funct\n",
        "  \n",
        "  def forward(self, x): \n",
        "    x = self.l1(x)\n",
        "    x = self.funct(x)\n",
        "    x = self.l2(x)\n",
        "    return x\n",
        "    \n",
        "\n",
        "model = RandomModel(elish)\n",
        "\n",
        "print(model(torch.randn((5, 10))))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BKMV_HkD23K_"
      },
      "source": [
        "## Exercise 3: implementing a trainable activation function (2 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3PgdzQu3p0L"
      },
      "source": [
        "**Exercise 3:** define a `torch.nn.Module` implementing the SReLU.\n",
        "\n",
        "**Hints for a correct implementation**:\n",
        "* The layer should **only** implement the activation function. Ideally, it will always be used in combination with a fully-connected layer with no activation function.\n",
        "* Think carefully about how you want to initialize the parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z2cvBI9N32SA"
      },
      "outputs": [],
      "source": [
        "from torch.nn.parameter import Parameter\n",
        "\n",
        "class SReLU(torch.nn.Module):\n",
        "  def __init__(self, units):\n",
        "    super().__init__()\n",
        "    self.units = units\n",
        "\n",
        "\n",
        "    self.alpha1 = Parameter(torch.randn((self.units), requires_grad=True))\n",
        "    self.beta1 = Parameter(torch.randn((self.units), requires_grad=True))\n",
        "\n",
        "    self.alpha2 = Parameter(torch.randn((self.units), requires_grad=True))\n",
        "    self.beta2 = Parameter(torch.randn((self.units), requires_grad=True))\n",
        "\n",
        "  \n",
        "  def forward(self, x):\n",
        "    return torch.where(x>self.beta1, self.beta1 + self.alpha1*(x - self.beta1),\n",
        "                torch.where(\n",
        "                    self.beta2>x, x,  self.beta2 + self.alpha2*(x-self.beta2))\n",
        "                 )\n",
        "    \n",
        "    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ajdzaIa4-Mt"
      },
      "source": [
        "As a sanity check, initialize a SReLU layer and count the number of parameters:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mnATD7hW49sK",
        "outputId": "b3fa4f1a-9a75-4fc5-f757-d2d3357c2751"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "8\n"
          ]
        }
      ],
      "source": [
        "# Initialize the layer\n",
        "layer = SReLU(2)\n",
        "# Count the parameters\n",
        "numb_params = sum([p.numel() for p in layer.parameters() if p.requires_grad])\n",
        "print( numb_params ) # Should print 8!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oW_Lny-j5h6A"
      },
      "source": [
        "## Exercise 4: training a model with trainable activation functions (1 point)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yC9Ip1qz5-fp"
      },
      "source": [
        "We will use the following dataset from TensorFlow Datasets:\n",
        "https://www.tensorflow.org/datasets/catalog/german_credit_numeric"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MdoRIsX66Bww"
      },
      "outputs": [],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "import torch.optim as optim \n",
        "from collections import OrderedDict\n",
        "from torch.utils.data import DataLoader\n",
        "from sklearn import preprocessing\n",
        "from numpy import mean"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yclY7yodB-on"
      },
      "outputs": [],
      "source": [
        "class dataset(torch.utils.data.Dataset):\n",
        "  def __init__(self,X,Y):\n",
        "    self.X = X                           \n",
        "    self.Y = Y       \n",
        "\n",
        "  def __len__(self):\n",
        "    return len(self.X)                  \n",
        "\n",
        "  def __getitem__(self, idx):\n",
        "    return [self.X[idx], self.Y[idx]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bYlP5pimbMH6"
      },
      "source": [
        "train_data = tfds.load('german_credit_numeric', split='train[:75%]', as_supervised=True)\n",
        "Xtrain, ytrain = train_data.batch(5000).get_single_element()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 116,
          "referenced_widgets": [
            "76cd3bf32b814994a3b62d9387b8d4d3",
            "e8cf9e329b554790975afeb4039ba757",
            "008b07fdc96a4002ac2beca1a9f7d0be",
            "ab7891e8262b4c75a4fcb572f77a2d16",
            "c6e5cf4d54da4bdd8dbab54b9d3203ea",
            "7aee557abc584b8abdb0bbf2e1419159",
            "caf10268f0bf450e9c70b472019122a4",
            "0af801d24cdd4693abb1b46eda9e0fe7",
            "e7b142fe0c354835b6b555b7e7fff1df",
            "080ad4c5b61d4d5eb3c6c26fa3130dc5",
            "de47594d5dc64ab19d39968fccfed4db",
            "851d8034a8ee4370a2f61ab79eda6647",
            "6fc44c12dfb84fed89d1f0f9ea4305e8",
            "9c5b75b8c8604ac38a35be63b77a0d97",
            "a77bcf7be0474c889610da8c32991226",
            "4feb171f0f0d469fa0d27ac591704831",
            "60cfa7df9d1f4809aef22641da82f2b1",
            "c748036e563f4952a1938a2b03fd5811",
            "653536b09edf40f78489c96feb30055f",
            "ee9683900bfe4497957fdcf28cc6f1e6",
            "ffb3c83cfaef48b097e3ae17e6d09927",
            "f6849f2fc6a042aeb179c27006200d47",
            "833dd5cdfe664dd3a6ec6b9bcb137f75",
            "afd659d3578b49cb8299ae67b9a86a02",
            "07bacd156f87482db28235eac19c5597",
            "d551c0b12d8e47eb8a203319314582c8",
            "bcd78433f56f4201b71f20b2f14acbcd",
            "834c9a34933e44d48059cd2d2b075076",
            "48197e0c63c34739811d0fc76de4e8ea",
            "c1a13466807d408b9e1575b238fb22af",
            "20d069df08884162bac03ed46a44201e",
            "e252357a61ce47f3852151542c1396fe",
            "7cf2656e58fe434ea108dcb70a6bd8c6",
            "e0f481da504f4326a906699e5404cc39",
            "5dfea2b19ef849e48d22328977476c0a",
            "67d081ca452543a28d4e6cf205b40f6a",
            "a1a8ac1fdac544e6b4d8520a878c3d8a",
            "7fe9307d8bd543a4a409b0729a7a62ba",
            "f88dbb487ef64339b5b42f99beb2fc40",
            "9cf05a6b976a436693c06e38c576e477",
            "f971b940a3094448afe017314aacaa77",
            "9837fca54727463caeeb427518bf97d3",
            "da0047e75f1f4420b2369958545d6e54",
            "32016a60ed0c428abdd103f3f7251ba2",
            "02c79bc821a2454480a2ff5b45217a16",
            "de6c910b5e2c4da698533ff97e3a5e01",
            "160911d9ec3f465cb9e130c3f7d78637",
            "7d7b03943eb244b183d06c92fd7cc596",
            "754d024b65794e17980cdf906b34d59a",
            "d9d3b1ea29a246bf838f7064377cc0fd",
            "482f445629fa47aa93074bee01ed746f",
            "5f25e2d35c2c428b8fe0f2bb7f074065",
            "acf1cc70157a419ba7c86e7817d3bc50",
            "520701f4d5464fc696f0cda9afc6fdc9",
            "c06d1ee12ad1427194fcb8e1ce45b742"
          ]
        },
        "id": "DSVqP-52bYdf",
        "outputId": "e177a34c-d8c1-4d01-8077-a2ddea463685"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading and preparing dataset 99.61 KiB (download: 99.61 KiB, generated: 58.61 KiB, total: 158.22 KiB) to ~/tensorflow_datasets/german_credit_numeric/1.0.0...\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Dl Completed...: 0 url [00:00, ? url/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "76cd3bf32b814994a3b62d9387b8d4d3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Dl Size...: 0 MiB [00:00, ? MiB/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "851d8034a8ee4370a2f61ab79eda6647"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating splits...:   0%|          | 0/1 [00:00<?, ? splits/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "833dd5cdfe664dd3a6ec6b9bcb137f75"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train examples...:   0%|          | 0/1000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e0f481da504f4326a906699e5404cc39"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Shuffling ~/tensorflow_datasets/german_credit_numeric/1.0.0.incompleteAARTBJ/german_credit_numeric-train.tfrec…"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "02c79bc821a2454480a2ff5b45217a16"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset german_credit_numeric downloaded and prepared to ~/tensorflow_datasets/german_credit_numeric/1.0.0. Subsequent calls will reuse this data.\n"
          ]
        }
      ],
      "source": [
        "train_data = tfds.load('german_credit_numeric', split='train[:75%]', as_supervised=True)\n",
        "Xtrain, ytrain = train_data.batch(5000).get_single_element()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JH2HxGZRceFv"
      },
      "outputs": [],
      "source": [
        "train_data = tfds.load('german_credit_numeric', split='train[:75%]', as_supervised=True)\n",
        "test_data = tfds.load('german_credit_numeric', split='train[75%:]', as_supervised=True)\n",
        "\n",
        "Xtrain, ytrain = train_data.batch(5000).get_single_element()\n",
        "Xtrain, ytrain = preprocessing.normalize(Xtrain.numpy()), ytrain.numpy()\n",
        "Xtrain, ytrain = torch.Tensor(Xtrain), torch.Tensor(ytrain)\n",
        "\n",
        "Xtest, ytest = test_data.batch(5000).get_single_element()\n",
        "Xtest, ytest = preprocessing.normalize(Xtest.numpy()), ytest.numpy()\n",
        "Xtest, ytest = torch.Tensor(Xtest), torch.Tensor(ytest)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rg78pcyTCakD"
      },
      "outputs": [],
      "source": [
        "train_loader = DataLoader(dataset(Xtrain, ytrain), batch_size=128, shuffle=True)\n",
        "test_loader = DataLoader(dataset(Xtest, ytest), batch_size=128, shuffle=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FK6RdPDQ6K7l"
      },
      "source": [
        "**Exercise 4**: write a `nn.Module` using the previous `SReLU`, and train it on the german_credit_numeric dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vKnj9-AV6f38"
      },
      "outputs": [],
      "source": [
        "model = torch.nn.Sequential(OrderedDict([\n",
        "          \n",
        "          ('Linear1', torch.nn.Linear(24,10)),\n",
        "          ('SReLU1', SReLU(10)),\n",
        "          \n",
        "          ('Lin2', torch.nn.Linear(10, 10)),\n",
        "          ('SReLU2', SReLU(10)),\n",
        "          ('drop2', torch.nn.Dropout(p=0.5)),\n",
        "          \n",
        "          ('Linear3', torch.nn.Linear(10, 5)), \n",
        "          #('relu3', torch.nn.ReLU()),\n",
        "          ('SReLU3', SReLU(5)),\n",
        "\n",
        "          ('Linear4', torch.nn.Linear(5,1)),\n",
        "          ('Sigmoid', torch.nn.Sigmoid())\n",
        "        ]))\n",
        "\n",
        "criterion = torch.nn.BCELoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UtdJVYnBEPSy",
        "outputId": "888b43f3-705a-41e3-c3ec-23134d164264"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (Linear1): Linear(in_features=24, out_features=10, bias=True)\n",
              "  (SReLU1): SReLU()\n",
              "  (Lin2): Linear(in_features=10, out_features=10, bias=True)\n",
              "  (SReLU2): SReLU()\n",
              "  (drop2): Dropout(p=0.5, inplace=False)\n",
              "  (Linear3): Linear(in_features=10, out_features=5, bias=True)\n",
              "  (SReLU3): SReLU()\n",
              "  (Linear4): Linear(in_features=5, out_features=1, bias=True)\n",
              "  (Sigmoid): Sigmoid()\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "losses = []\n",
        "epochs = 300\n",
        "model.train()\n",
        "for epoch in range(epochs): \n",
        "  epoch_loss = []\n",
        "  for i, data in enumerate(train_loader, 0):\n",
        "\n",
        "    inputs, labels = data\n",
        "    y_pred = model(inputs)\n",
        "    loss = criterion(y_pred, labels.unsqueeze(1))\n",
        "    \n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    epoch_loss.append(loss.detach().item())\n",
        "\n",
        "  losses.append(mean(epoch_loss))\n",
        "\n",
        "model.eval()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        },
        "id": "7TziqM6WYYXo",
        "outputId": "da31d67d-2fa8-4df3-fa6e-5db44fdb9a6a"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwV1Zn/8c/Te9NA00CzyI7gAq6IuBtXRE1csiiaRX9ZNJmf2ZOJThLjNpNkZjJxnDgmmvDL6h5jMEFxX+IGjSKyCDT7Ks0OTbN09/n98VRxq5tuuA293v6+X6/7qrqnqu49RetT5z516hwLISAiIpkrq60rICIiLUuBXkQkwynQi4hkOAV6EZEMp0AvIpLhFOhFRDKcAr10aGa21MwuaKXvKjSzp8xsi5k91hrfKdIcctq6AiIdyCeBvkCvEEJ1S32Jmf0WWBlC+EFLfYd0LmrRi6RvCLCgJYO8SEtQoJeMYWb5Zna3ma2OXnebWX60rbeZ/c3MNpvZRjN7zcyyom3fM7NVZrbNzOab2fkNfPbtwK3A1Wa23cy+YGa3mdkfE/sMNbNgZjnR+5fN7E4zez367GfNrHdi/zPN7I2oTivM7HozuwH4NPDP0fc81bL/atIZKHUjmeT7wKnACUAA/gr8APgh8G1gJVAa7XsqEMzsSOAm4OQQwmozGwpk1//gEMKPzCwAI0IInwEws9vSqNO1wMXACuBp4DvAzWY2JHp/A/A40B0YFEKYaWano9SNNCO16CWTfBq4I4SwLoRQAdwOfDbatgfoDwwJIewJIbwWfKCnGiAfGGVmuSGEpSGERc1Yp/8XQlgQQqgCHsUvQuAXgOdDCA9F9dkQQpjZjN8rspcCvWSSw4BliffLojKA/wDKgWfNbLGZ3QwQQigHvgHcBqwzs4fN7DCaz9rE+g6ga7Q+CGjOC4pIoxToJZOsxm+YxgZHZYQQtoUQvh1CGA5cBnwrzsWHEB4MIZwZHRuAn6b5fZVAl8T7fk2o6wrg8Ea2aUhZaVYK9JJJHgJ+YGal0U3PW4E/ApjZR81shJkZsAVP2dSa2ZFmdl5003YnUAXUpvl9M4GzzWywmRUDtzShrn8CLjCzq8wsx8x6mVmc1vkQGN6EzxLZLwV6ySR3AWXALOB94J2oDGAk8DywHXgT+N8Qwkt4fv4nwHo8zdKHNAN2COE54JHo+2YAf0u3oiGE5cAl+E3ijfhF4/ho82/wewabzezJdD9TpDGmiUdERDKbWvQiIhlOgV5EJMMp0IuIZDgFehGRDNfuhkDo3bt3GDp0aFtXQ0SkQ5kxY8b6EEJpQ9vaXaAfOnQoZWVlbV0NEZEOxcyWNbZNqRsRkQynQC8ikuEU6EVEMpwCvYhIhlOgFxHJcAr0IiIZToFeRCTDpRXozWxCNGlyeTwzT73tPzezmdFrgZltTmyrSWyb3JyVr2PPVph1G6yf1mJfISLSER3wgSkzywbuBS7EJ1eebmaTQwhz431CCN9M7P9V4MTER1SFEE6gpdVWw+zbIa8Eeo9r8a8TEeko0mnRjwPKQwiLQwi7gYeBy/ez/zX4TD+tK7ebL/dsbfWvFhFpz9IJ9APw+S1jK6OyfZjZEGAY8GKiuMDMyszsLTO7opHjboj2KauoqEiz6vVk5UJ2F6hWoBcRSWrum7ETgcdDCDWJsiEhhLHAtcDdZrbPhMghhPtDCGNDCGNLSxsckyc9ud3VohcRqSedQL8KGJR4PzAqa8hE6qVtQgirouVi4GXq5u+blwK9iMg+0gn004GRZjbMzPLwYL5P7xkzOwoowSdejstKzCw/Wu8NnAHMrX9ss8ntDru3tNjHi4h0RAfsdRNCqDazm4CpQDYwKYQwx8zuAMpCCHHQnwg8HOrONn408Cszq8UvKj9J9tZpdrnFytGLiNST1nj0IYQpwJR6ZbfWe39bA8e9ARx7CPVrmtzusH1dq32diEhHkFlPxipHLyKyj8wL9MrRi4jUkWGBPsrR17lNICLSuWVYoO8OoRZqdrR1TURE2o3MC/SgPL2ISEJmBnrl6UVE9sqwQF/sS7XoRUT2yrBAH7Xo9dCUiMhemRno1aIXEdkrMwO9cvQiIntlZqBXi15EZK/MCvQ5Rb6sqWzbeoiItCOZFeiz8sCyoVoPTImIxDIr0Jt5q75aLXoRkVhmBXqI5o1VoBcRiWVeoM8p0lg3IiIJmRno1aIXEdkr8wK9UjciInVkXqBXi15EpI7MDPTK0YuI7JWBgV6pGxGRpAwM9ErdiIgkZV6gzy7Sk7EiIgmZF+hzumisGxGRhAwM9EVQu8dfIiKSXqA3swlmNt/Mys3s5ga2/9zMZkavBWa2ObHtOjNbGL2ua87KNygewVLpGxERAHIOtIOZZQP3AhcCK4HpZjY5hDA33ieE8M3E/l8FTozWewI/AsYCAZgRHbupWc8iKbuLL6srIa+4xb5GRKSjSKdFPw4oDyEsDiHsBh4GLt/P/tcAD0XrFwHPhRA2RsH9OWDCoVT4gPa26JWnFxGB9AL9AGBF4v3KqGwfZjYEGAa82JRjzewGMyszs7KKiop06t24vZOPKHUjIgLNfzN2IvB4CKGmKQeFEO4PIYwNIYwtLS09tBqoRS8iUkc6gX4VMCjxfmBU1pCJpNI2TT22eSRz9CIiklagnw6MNLNhZpaHB/PJ9Xcys6OAEuDNRPFUYLyZlZhZCTA+Kms5atGLiNRxwF43IYRqM7sJD9DZwKQQwhwzuwMoCyHEQX8i8HAIISSO3Whmd+IXC4A7Qggbm/cU6lGOXkSkjgMGeoAQwhRgSr2yW+u9v62RYycBkw6yfk2Xo9SNiEhSZj4ZCwr0IiKRDAz03cGyYHfLZohERDqKzAv0WdmQXwpVa9u6JiIi7ULmBXqAgr6w88O2roWISLuQoYG+H+xUi15EBDI20KtFLyISy8xAX9jPc/SpLv0iIp1WZgb6gr5Quwv2bG3rmoiItLkMDfT9fKk8vYhIhgb6wr6+VJ5eRCRDA33coldfehGRTA30cYtegV5EJDMDfX4vyMqFqjVtXRMRkTaXmYHesqCgP1StbuuaiIi0ucwM9ACFh8GOlp3MSkSkI8jcQN9lAFQp0IuIZG6gLxyg1I2ICJkc6Lsc5k/G7tne1jUREWlTmRvoCwf4UukbEenkMjfQd4kDvdI3ItK5ZW6gLzzMl+p5IyKdXOYHeqVuRKSTy9xAn9sNcrqpRS8inV7mBnqI+tIrRy8inVtmB/rCAWrRi0inl1agN7MJZjbfzMrN7OZG9rnKzOaa2RwzezBRXmNmM6PX5OaqeFoKD1OOXkQ6vZwD7WBm2cC9wIXASmC6mU0OIcxN7DMSuAU4I4Swycz6JD6iKoRwQjPXOz1dBvgIlqHWBzoTEemE0ol+44DyEMLiEMJu4GHg8nr7fAm4N4SwCSCEsK55q3mQCgdAqIadFW1dExGRNpNOoB8ArEi8XxmVJR0BHGFmr5vZW2Y2IbGtwMzKovIrGvoCM7sh2qesoqIZg3IXdbEUEWmufEYOMBI4B7gGeMDMekTbhoQQxgLXAneb2eH1Dw4h3B9CGBtCGFtaWtpMVSIxDIJ63ohI55VOoF8FDEq8HxiVJa0EJocQ9oQQlgAL8MBPCGFVtFwMvAyceIh1Tl88DIJ63ohIJ5ZOoJ8OjDSzYWaWB0wE6veeeRJvzWNmvfFUzmIzKzGz/ET5GcBcWktBX78Ju2Nlq32liEh7c8BeNyGEajO7CZgKZAOTQghzzOwOoCyEMDnaNt7M5gI1wHdDCBvM7HTgV2ZWi19UfpLsrdPisnKhyyCoXNJqXyki0t5YCKGt61DH2LFjQ1lZWfN94AvnQc0uGP96832miEg7Y2Yzovuh+8j8zuVdh8P2xW1dCxGRNtM5Av3OtVC9o61rIiLSJjI/0BcN9+V25elFpHPK/EDfNQ70St+ISOekQC8ikuEyP9Dn94KcIqhc1tY1ERFpE5kf6M38wamdH7Z1TURE2kTmB3rwQL+rfQyoKSLS2jpPoFeLXkQ6qU4S6Pso0ItIp9VJAn1f2LUeamvauiYiIq2u8wT6UOvBviG1e6C2unXrJCLSSjpPoIfG0zd/6Q/PndF69RERaUUHHKY4IzQW6F/9OPQ+FXZt8FcI3h1TRCSDdN4WfaiF1VNg7Qupsq3z0/u8D1+C8vsPvj41u2DZo35hERFpYZ0v0O9cDzW7fR7Z2l2weVZqv3WvpPd5L5wH025M//tDgOqq1PsVf4bXr6773fX3FxFpJp0j0Od2h+xCH+/m6ePhtY/DtkW+befa1H7rXvVlzU5vdde38il45zup9+newF3+GDxR6ukhSI2709AUh5Ur4LFusPKv6X22iMgBdI4cvRmUngmLfws1O7w1v3tT3X1yu8PWaJbD1z7l0xCe/URqe3UVTPsi7Ew8YburAgr7H/j7V/4Fqith4wzoPx4ql3p51Zp9991Y5vu+egVcvQuy85pypiIi++gcLXqAwy72IG9Z0P0oWP9G3e2lZ8HWBd5KX/cSfPiit+zfvgHKvg6LJ3mQH3yVD5QGdYN+bM2z8NIEmPYVeHqMf8aHL/q2TTN9GQ+wtuFtePnSugF/x4rEZz3dPOcuIp1a5wn0/S/2Za9TYOin993e52y/EKx93lvUe7Z4wF70ACz4H5h9F/Q+Dc54GM560o9pKNDP+TGsmQrlv4RN78Lyx1P71Q/0i37tN4Tf+Vbq+OQom3F6acsHfvO4o6qtgRVP6t6DSBvpPIG++5Ew6JMw8p9gwMfqbssugJ4n+frSP6XK170CA6/01M/OtTDqlmg0zD6+vXIJ7Fid2n/3Jqj4Bwz9DBz/b142/7992eM42PyeB+zK5XW/f/WUVKu+cqn/4sjt7uvbFsHfR9WtF8C28obvI7RHa6bCa1emf7NbRJpV5wn0ZnDWYzDsMx50Afqe78uCvn4hAA+oud09+AMc830Y9CnoORYGXBrtHwX6aTfCkwOg/AF/v+pvEKrhiK/C0d+FrHzPuXcdDgMvh60fwFuf994+STW74G+j/OZs5TIoGuqvyqVQ8ToQ6gbJnevh76Oh/FepsqoPvTdRe7Qt6ra6bUHb1kOkk+o8gT7JDK7eCec+7Tn7gr5QOCDaGDxF0/NkKB4NJWPg9D/Cha/7vgC5xZCVuEla9lXP7a94AroMhF5jISsHio/27X3PgwGXQfejYcnvvCzO8w/6OJw7FfZshoo3PLgnA/2Gab7f+rdS31fxD6jdner3v3MdPDUS3r+tuf+lmkecgqp4A144X7N9ibSyzhnoAbLzvWdNQV9/mXlgBzjpf+CMB+Gcp708K6du7xczyCvx9fzeUX/892HNMzDwitQFofhYX/Y9z4P/pbP9XgBA72jIhW5HQs8xvr75fe+CWTTEX5XL/IYtwJa5sHuLr8fdQOMU0Nx/h+ptsLzeQ1jbFqW6dLal7eW+XPag35ie97O2rY9IJ9N5A33smFth5Fd8/dxn4eMV0H2kt8yLBjV+3O6Nvhz6GV9+8HPvYTPo46l9eo3zi0nfc1Nl506Fc57x4A/Q/QjI7eYXjHUveVncot+z1VM/JScCATZO9+0VUaDfsRyqd8DC+/z47YtgyxzfFgI8/xGY8c2D/IdpRnGLvnaPL5f8DpY+nHovIi0qrUBvZhPMbL6ZlZvZzY3sc5WZzTWzOWb2YKL8OjNbGL2ua66KN5uRX/aulwBdDoOC3ukdFwepodf6ctmDno4pPSu1z4gb4NK5UNgvVZZdAIddlLqIdB/ly67DYf2bvl58FHQdmjpm1PcAg4o3Yc92781jWd6iX/eK9xY68T993xV/8WXlUqha5Wke8J47G99N79yaU2116rkB8PO0HHjjGlj9TOvXR6QTOmCgN7Ns4F7gYmAUcI2Zjaq3z0jgFuCMEMJo4BtReU/gR8ApwDjgR2ZW0qxn0FZyuvmyZAx0GQyhxvPwWYln0LLzoNuIho8f8DE4+0nodbK/7zrce+TkFEHxMf4eoMfx3ne/+GjY8Jbf2Ay1fg9hz2ZP12QXwpCr/SKz7EFvza+PUj6VSzyH/+rl8MyYuk/2pqtml98APhg7lvsN6h5RGmvQJ+Gj83x9+6KD+0wRaZJ0WvTjgPIQwuIQwm7gYeDyevt8Cbg3hLAJIIQQdzC/CHguhLAx2vYcMKF5qt7GJszwNExWNhRH171k2uZAsnK9J048WmbRMF/2GucXix7HwxmPwIX/8H16neo3ZLdEQbLfhb5c/Fvoc47/Uhj2We/Zs3FGKrcP3nMn7vGy8D7/VVBd6a8PXz7wcAvv3w5PH3dwffm3LozqOz46v5OhoB9kd9m3m6mItIh0Av0AIPG4JiujsqQjgCPM7HUze8vMJjThWMzsBjMrM7OyioqK9GvflrqP9OEMwB/Cyu8N/S44+M+LW/C9T/OlGQy5CnK7psp3b4TVf/e0Tb/zUscOutKXgz/lvYGW/MEDfckYT5Msf8y3D7/e0zwrnoCXP+rDLLxwri9XPOkPdM35Sepmb2zdy97Pf/uSpp/XpihddPR34MSf+S8ZMyga7K19EWlxzXUzNgcYCZwDXAM8YGY90j04hHB/CGFsCGFsaWlpM1WpFY3+F7h0Xqrv/cHofpQvS89seHvvU3257GEoGg7dRqa2DfucL/N6eCBd8nsP9AMu9WcG4hb70M9C18Nhzr968F77fOozpn8Z3vg0vHeLj8656u+eAqrdkwrWmw4ix79xun9nYT84+lve2wk83bX5fX/6uP7w0LV74Plz6w4hLSIHLZ1AvwpIdj8ZGJUlrQQmhxD2hBCWAAvwwJ/OsR1fdl76N3EbU3oGXPAq9G8ks1U8CrpG+f7uR0JBf++qefqfUsETPH2zZzNYNoz4sl8ganb4tm4jYNTN+z64NPYXPoTzlrkw5ufe3/+Vj8Kzp3vrvman77fmGZh1K7z3g/SHM9hQ5g+b1Vc0xOuxZqo/aJa0Y4VfiFZPSe87RGS/0gn004GRZjbMzPKAicDkevs8ibfmMbPeeCpnMTAVGG9mJdFN2PFRmdRnBn3OanyGK8uCEV/y9VDt9wYueCXV6yfW/2IfUXPY57wXUfxLICvfu4wOv94Ded9zodsR3toe+U9eltcTRtwIF74KJ/03bJ4JL0bpqNxiWPQbmH2n/yJY82zD9azeAUsf8t42O9d5eia+4ZxUNDi1vmV23W3x8M3bFjb6zyUi6TvgMMUhhGozuwkP0NnApBDCHDO7AygLIUwmFdDnAjXAd0MIGwDM7E78YgFwRwhhY0ucSKcw/HoPtEfc1Pg+2XlwyWzvvQN+/wCg6zC/WFgWjH8dyPJeL6EmGh7iz7BnG+QUAoVw5Nc87fPCef5AWdFQ7/Vzyq/9Cdw5d/o9iazsut8//x5P/2yYlrqHEY8jlNQlEeg3R33/qyvhubP8hjRoyASRZpLWePQhhCnAlHpltybWA/Ct6FX/2EnApEOrpgA+xs5V2w68X37P1Hq3kd5S73p4qix+qjd+IhdSwzUk9T0Hrop66OxY7jn94Z/3tM20L3lO/7TfeU+fkuP9mJVPAgbz7/andSE1tlBS3KLPLvSHvEKt5+w3veufBz5UQm113S6rItJk+j8o05nBaX+o+9BWU+R08VdBaaplPuKLPqzB3J96+cL74OMfen/7DW/DkGu9P/+yR70rZfLCE+t1cnRzeBjMvsOf4I2Df0007WLtHh8GotvhPp7QoI/XfcpYRNKiIRA6gwGX1G29N4fBn/Llot94+mdbuefuwZ/kzS7wFn3xqIaPzymC03+fuvm84B6Y9cN991vyB59eccEvYN5/Nu85iHQSCvRycIqP9T77cet7wf/Awv+Fo74FJcelBnRrLNDHeo2D4+709ZrEBOrxk7Szb4dpN/j62hc8jSQiTaJALwcnOy8abC2y/DFvxZ/wE3/fM9p2oECflQ3H/MAfqALvHQSeJjrlN76+JuqoVbvLp3bc/H7znINIJ6FALwcv7h1jWT4+fvGxPrQDpC4C3Q8Q6GPxkA79L/Jl4QA4/PPRw2DBfz0UHua5/7n/vu/xC+6FlfV7/YoIKNDLoTj8894HP34gquSE1LYh18DxP/YHwdLR5yMw7Do49lbvehl3Cy2J7i10GwmXLfbnBDbO2Pf42XfBwl8e/LmIZDAFejl4JSfAyfemhmNIBvq8Yhh9c/pdI7Pz4bTfesrmimUwMJrXN+7p0/1I36f3Kd79ck+im2nNLp/Td+eaQz4lkUykQC+HLh55Mxnom0vJCZ62iW/u9hwLBHisO7wX9dKpikbVqFrd4EeIdHYK9HLo+o/30TVbItDndoWLpsPR3/b3yadsy+/z1nw83PHOdZq1SqQBemBKDl2fs2D8Gy33+SWJJ2sL+/k0jFn5sOZpWPWUj68Tq1q7/ykgRTohteil4zn/BfjIUz5I26JJPtplrEp5epH61KKXjikrG4ZdD3P/re6In8rTi+xDLXrpuIZf74OhrZ7iQzODB/qqNbCzg8xUJtIKFOil4+p2uI+mCT46p2V5kH/pInjrelj8e3+JdHJK3UjHdsqv/ana4qPh5Utg7XM+RELVGp9IPTsPhn+u7jHvfMe3n/GntqmzSCtToJeOzQyGTvT1nid5LxyAXev9ZVneK2fbAu96uXk2fPAz3+e036aGbBDJYAr0kjnG/gLWvebBf/cmLwu1MO3LsOxP3iXTEjNibV0APUa3TV1FWpFy9JI5igbDRdPgwn/ULV/6Byg9y+e9JcDZf/XyzbNavYoibUEtesks3aNxd7oO97TMjpU+hv3o70O3ET5GTvejwHKi4Y6v8f1XPAkby+D4u9qs6iItRYFeMtNxd3mg/+DnHuz7nld3IvPuR6Va9Kufhteu9PVjb993wnORDk6BXjLT0KilXjwqmmC8XvAuORFWTfYhE+b+NFW+fZFPgZgcU0ekg1OOXjJb8ai6Y+XERt/iM1a9fjVUvOYtfICZ/wzPnga7N7duPUVakAK9dE7FR8OYu2Hdq94zZ+Q/eXnF694Nc+3z/n7TLE//bHyn7eoqcoiUupHOa+SNULkUNr2Xmspw13pfrnnG58B95TIgeLfMc6fC/Hvg9D/58MkiHYRa9NK5nfBjOHeKD3+ctPoZWPYo5Pf2wB5q4P3bPa+//JG2qavIQUor0JvZBDObb2blZnZzA9uvN7MKM5sZvb6Y2FaTKNfszdI+5Rb7TFbgwx9XrYKVf/EJVfp8xMsrXvPlot/UPXbHSnhpAuza2Hr1FWmCAwZ6M8sG7gUuBkYB15jZqAZ2fSSEcEL0+nWivCpRflnzVFukmZlBQV9fjwdKq94OvcZB4WHRw1Z4//v1b0JlYgz8Nc/CmqkNT1ou0g6k06IfB5SHEBaHEHYDDwOXt2y1RNpAQZS+6X0q9Ih66vQ+xS8CxdFQCf0u8OWWuanjtn7gy10aGlnap3QC/QAg0XxhZVRW3yfMbJaZPW5mybncCsyszMzeMrMrGvoCM7sh2qesokL/s0gbiVv0XQbBgI95KqfnyV4WB/rBn/Ll1g98PJ0F9/pAaeBz1oq0Q83V6+Yp4KEQwi4zuxH4HXBetG1ICGGVmQ0HXjSz90MIi5IHhxDuB+4HGDt2bGimOok0TWEU6IsGw+h/gSETIS9K2fQcA4uzoP9FnsbZOg9ev9Z758QU6KWdSifQrwKSLfSBUdleIYQNibe/Bv49sW1VtFxsZi8DJwJ1Ar1Iu1B6po9hn9vd3/c4JrVt+Oc9pdNlAHQ/Esp/5eWW5f3wQakbabfSSd1MB0aa2TAzywMmAnV6z5hZ/8Tby4B5UXmJmeVH672BM4C5iLRHw6+H8a83vC07D0pO8PVuR/qy6wgYPDG1T7JFX7UW/twb1jwH0270njkibeSALfoQQrWZ3QRMBbKBSSGEOWZ2B1AWQpgMfM3MLgOqgY3A9dHhRwO/MrNa/KLykxCCAr10bNn5vhxyFfQ+A5Y95F0y40BftcZ75uzaAC+N97K8nt5nX6QNpJWjDyFMAabUK7s1sX4LcEsDx70BHHuIdRRpXw7/onepHPkVD/BXroZ3vg0b3oaVk+HVy2HItXWPydGTtNJ29GSsSFP1PgWuWOZBHvyp2oI+3qJfeJ+XrX227jFx/r52j9/E3fhu69VXOj0FepHmUFDqwxuve8Xf71oPWGoMnZ0f+nLLHE/1PH9W3eNDLQR1OJOWoUAv0hzy+/iypipV1uNYOO9Z6H16Kn+/db4vqyt9tiuA2hr46xBY9EDr1Vc6FQV6keaQ38uXXQZB33N9vTB6rrCgb6pFHz9FC7D0QV/uWOa9cjZMa526SqejQC/SHPqeC0d+HSaUQdfDvaxLI4G+aKgPlvb+j2DPVti6wLdVLm/1akvnoEAv0hzyesBJd/tN2a7DvGxvi76Pd7WsrfZA3/1oOOm/PfjPvgu2RYF+RxToQ4B3/xnKf628vTQLTTwi0tyKokCfbNETPE+/dT70ORd6nQzD/w/Mvxv6nOP7VS73wL7yrzDvP7yspgqO/Gprn4FkGLXoRZpbPHRCPA9tPFja0j964I7nsD3+33zgtLXP+fuaKu+tM+sHfmz3o3z4Y5FDpEAv0tx6HAuXLYE+URfKgqhHzszv+Tg5Q67x94X94PAv1T12yxx/DfssFB+TSuuIHAIFepGW0HVoar0osX7qb1NDKAAc9U1fxumbD1+Ojh/hF4Xti6Fmd4tVUzoH5ehFWlrRILh0jufucwrrbRsMly3yFM6Tg2DdS17ebQTU7vK5arcvhuKjWr/ekjEU6EVaQ3FDs29Gug73m7C5PaDijajscB8uAWDrXA/8WTmweqpfLPqc3fJ1loyh1I1Ie2AGfT8Codofvsor9tQNwGuf8O6WoRZengDPfwQ+fCl17Ja5sPn9tqm3dAgK9CLtRd9oUrauI3yZ18MnIwdYcA9smpna960vpFr8ZV+Dt+vd1BVJUKAXaS/ioRPiJ2sBLnkPjv9Xz9Uv+IWXjXsAKpfAjK/7fLVVK2HHin0/TySiQC/SXhSP9ukM+1+YKBuVGtt+8f/z94d/wfdbeB+U3eSzWe380AdHS9eSP8LjvVK/CiSjKdCLtBeWBRe+5lMaJnUdCj2O9zTO0d/zfP75L8HgT/mQCnu2eNI4iasAABJdSURBVIt/1/r0v2vWrbB7Y+rmr2Q0BXqRjmD8G/CprTD8c/4+K8dz+fFgaQA716T/efH8t6un7H8/yQgK9CIdQU6XBvrgD6r7vqoJgb42eghLgb5TUKAX6ai67CfQr3kO5vybj5jZkF0bfLlldmpSFMlYCvQiHdX+Av2Mr8J734fXr04NdbxlLiz+na/v3giFh/n6+rcb/vytC+DVK1IzYUmHpUAv0lEVDY5WDHK6pQJ9bTVURt0tVzwBi37j6/P+E966HrYvhd0bfD5by4YNjQT6+ff4kMkV/2jBk5DWoEAv0lHl9oCcIsjvDV0Gpm7GbpkDNTvgtD/6Q1jvfMvHuo+fnl3ye9i9yX8R9DgWPvgZzPyXut0za/fA8kd8feM7rXte0uwU6EU6KjMP1gV9fbnuVZ+VasVffHvvU+GUXwO1MO1GvwAALPgfH04hv5d326zZCXN/DOtfT332mme9u6ZlwaZ3PP2z8d1WP0VpHmkFejObYGbzzazczG5uYPv1ZlZhZjOj1xcT264zs4XR67rmrLxIp9d/AvQ7H074sQf8aV+C2bd7K7/rcJ/WcNTNsOYZn9ikeHSqv31+L5/ndtAn/P3yP6c+t/xXPo7+gMs8wH/4IjwzBjaUtf45yiE74OiVZpYN3AtcCKwEppvZ5BDC3Hq7PhJCuKnesT2BHwFjgQDMiI7d1Cy1F+nsTvp5av2SWX7DteJ1KBriLX6AoZ+BWT9Mrb93i6/n9YKeJ8JZj/tN1+WP+JO3XYfDqr/B6Fs897/ySVj/ph+z9QPoNbbx+qz6m6eLcro0/7nKQUunRT8OKA8hLA4h7AYeBi5P8/MvAp4LIWyMgvtzwISDq6qI7Jdl+TSGI2+EwxL/m3UdCr1PAwyGXpMqz++VWj/8S967ZvqX4aXxkNsdRtyYerBq1d99uX1J49+/dSG88jEfqkHalXTGox8AJEdMWgmc0sB+nzCzs4EFwDdDCCsaOXbAQdZVRA7WcXfAute8pR9LBvoBl8JV22HrPNg0y8fbye/lQysAbJzmy8qljX/H1g98uXlWs1Z9H9WVfrM4r0fLfk8Gaa6bsU8BQ0MIx+Gt9t815WAzu8HMysysrKKiopmqJCJ79bsAjru9blky0IOneopHwdCJqW1dBkNWvt+8hf0H+nh+25YeG7/sa/CSEgNNkU6gXwUkn8wYGJXtFULYEELYFb39NXBSusdGx98fQhgbQhhbWlqabt1F5GCUnunL3OID75uVDd2PSL2PUze7N++777aFvtw8O3VhqK9mF7x/x6H14Nky28fmb+w7ZB/pBPrpwEgzG2ZmecBEYHJyBzPrn3h7GTAvWp8KjDezEjMrAcZHZSLSVs55Gi6d6zn9dHSLZrrKyvVx71f8BR4v8Zu+SXGLvnobPJQNyx7d97NevRLe/xHM/Und8vn3wOpn0qtP5XKfT3fHPm1GacQB/9IhhGrgJjxAzwMeDSHMMbM7zOyyaLevmdkcM3sP+BpwfXTsRuBO/GIxHbgjKhORtpLbFYqPTn//eErDXqf4VIdvRr2k67fKty2sO2nKrB/WfQhrzzZY87SvJ8fXCcH3jSdW2Z+a3akRO7cvSv8cOrm0LukhhCkhhCNCCIeHEP41Krs1hDA5Wr8lhDA6hHB8COHcEMIHiWMnhRBGRC/djhfpaOJA32+8L6ujsW92rvUbowDVO2DHSh8jv+fJMPzz3sJf/bfU51Qu82VWPmwvhxnfgBVPer/+PVu9DPyhr5V/bbguVavwntrAtmj/3Ztgm4L+/ujJWBHZv4FXwnF3weib4dypMPr7kNcT1j4Hj3aF16+FZQ/7vqVnwoRpMO5+yC+tm76JA32/8/2iMP+/vStmHLC3L/ZfALN+CB/8vG4dlvwRFk2qO2VifGGYeQs8e1rDOfvaPamROjsxBXoR2b/crnDM9z1H3388HH+Xj5GzIepyuewhePsL0PMkOOxiL8vKhgEf9fHu4+kK4x47/S5Iffbm91MBu3YPbJ7pvxTiiwJ4aufNz/p3xIO1ZeXB3J/CrNtg03uwq8JvAidtWwRPDoK/Dqk7AmfVGph9V+NDOGcgBXoRabqiob7M6QZnPQG9xsHYX9S9wTvgY7Bnc2r0y8qlnrbpfXpqn8ol3oMmFqdsdqxM5ffjMXoAdiz3Zdfhvpx9u/f9B6h4rW4dy3/p+fzqStgwPVW+/DH/1VB//wymQC8iTRc/eNXzRBh0JVz0tg+iltTvQr8QzPsvf1+5zI+Lu2tati9XPgnZXVLr4Dd9q1b7Dd/596Q+c9N7kFcCp//RLyTgc+bCvsMpb12QGnN/w1up8ri3Tv3ZtbbOh2fG+WTrTbH2Rdi9pWnHNOTDl/3VAhToRaTp4hZ9yZjG98ntCsf8wG/IrnnWx8EvGuKButvI1CTo2xdD71Mgu6Duw1ZvftYHUlv0QKqs4lX/jJ4nwUl3p8oL+vqvgQ8SZdsW+i+N7kdBxZup8qrVvlz9dN36LrwPNk5v2rDMu7fAi+fD0yekf0xjZt/p9xtagAK9iDRd16G+LDlx//sd+XXvcjnjG1C5OHWBuHQOnPwrz/sD9D0fesW/CKLB2Na94q32c56B0/7gZVVrfGhlgKJhftEAOPMxr8u734Hlj3t//e2L/ILS+1S/2Dx1hHfrrIpa9FvmeJ988PsDSx/09eSE6wcSXzQql9ZNQSVtngM703jiP/7F0wIU6EWk6UrPghP/EwZ/cv/7ZefDmJ95Hn3XBh/ZEjzAZ2XD+S/DBa/B6H+Bs5+Akf/Xe+zERtwAh13kaaBYfHExg55jPe1TegaM+JKPzTPvPzwFVLvbA/2Qa/yYyqUw83ueuik+xj8jbtWvedZv6EITA31i+sbkMM9JL10IM77e+Ge88x1459veo0iBXkTajawcOPrb6Q1HPOAyOPl/4YJXfRydpNLToc+ZHrTzSuDkX8CIL6a2xxeGglK/kQvQM5EuOvq7Pha/ZXlQh1RvIPCy/uPh4nf818Xi33r//n4XeFCNH+Ba8nsf3yenyHv9gI/G+VjJvqmcVX/35wagbqBP9hSK7dnq+6yZWvfhsaQ1T/v4/7W7FehFpIMyg5FfgT5npX9MQT9PzcQXEstKTYZeksiH978Qjvyar8eBPqlbYpyewVen1rsMgP4Xw9rnffyeVZO95V/QL9WiX/mXqNfQG7B6qqd3Ns2CVz7q3TMhNX1jj+NSPYKS9o4NtBE2RU8S11anJmwHvxDED54p0ItIp3HZYvhovbmNug7zp3RzuzV8TH6pj6MP0Occ78ZZmBiGK3mBKDwMhl7rs25NHu6DrQ3/P1DYL9XrZk009s7yR+DlCd4tc+3zXrbofqiu8iCdXeipoIZa9NsXp9ZXT/FW/eMl8Hb0q6W6yp/sjbVQoE9nPHoRkdaVU7hv2Zife2BujJm36jfOgOPu9JRQUlYi3BUO8F8YF74Oyx6Jhm4Y4713tszzB6zi7prxcss8b5Vnd/H7Dcse9ItCYX8P0Msf9UCelZ36nrhF33OsP+BVUwXV22HxJDj1N6lfBDG16EWkU+sxev/TGEIqVdPYoG1xz56Cvr7sfapPx1gaPcQVp25W/c1TNYWJeZK2zPGeQMM+56ma+fd4oI4DfaiG6V+B9dE9gp0Vfj8gtxg+MtnvQcSjdnYb6bn+JX9KfX5ez8Z/rRwitehFJHMMvNxbzfUnVYmd/QQs+UNqoLb6Cvp6Pr38l94VdMjEVHBePcWHR+53vl9w4vTLoE9A0WBfX/SAP3F7/kvw91Gelik5wS8Gl8721E/5A37D+N3vpp7qzSlqsdY8KNCLSCYZcrW/GlPYH0b9c+Pb45b+ulfh2Nt8hq1YbTS3Ut9zPX0z64eeoy/oWzdIb/0Anj87lXuPJ2nJ6+HdUbd+AGufhdqdqWNO+0PqmYAWoNSNiEgsDvTgE6bHwzX0GufLkhOibpiFcNy/Rjta6oKQ2x2O+JqPyDn0M/4L49h6UzjG31ETBfqsPBh4BfQ9pyXOCFCLXkQkpfR074Z57G3Q5TDvhXPyfdBlILzyMX+CNzb8Or+xOvAKH+7h9Ad9mOaiQTDqu94LKDt/3+9IXkzAf2WYtehpKdCLiMQK+sCZD6feWxaM/LKPadPrFBj66brbjvxq6v3Qa1LrXQbu5zuSgd5SA6+1IAV6EZEDySuGi9468H7pSAb6MT9Ljf/TghToRURaUxzos/J8WIZ0J2k/BAr0IiKtKafQb9rml7ZKkAcFehGR1lfQt+7DWC1MgV5EpLUde1uL9puvT4FeRKS1Db22Vb9OD0yJiGQ4BXoRkQyXVqA3swlmNt/Mys3s5v3s9wkzC2Y2Nno/1MyqzGxm9Pplc1VcRETSc8AcvZllA/cCFwIrgelmNjmEMLfeft2ArwNv1/uIRSGEZpgiXUREDkY6LfpxQHkIYXEIYTfwMHB5A/vdCfwU2NnANhERaSPpBPoBwIrE+5VR2V5mNgYYFEL4ewPHDzOzd83sFTNrwqSRIiLSHA65e6WZZQH/BVzfwOY1wOAQwgYzOwl40sxGhxC21vuMG4AbAAYPHtzAx4iIyMFKp0W/ChiUeD8wKot1A44BXjazpcCpwGQzGxtC2BVC2AAQQpgBLAIS07K7EML9IYSxIYSxpaWlB3cmIiLSIAsh7H8HsxxgAXA+HuCnA9eGEOY0sv/LwHdCCGVmVgpsDCHUmNlw4DXg2BDCxv18XwXQwHTqaesNrD+E49uTTDmXTDkP0Lm0VzoXGBJCaLClfMDUTQih2sxuAqYC2cCkEMIcM7sDKAshTN7P4WcDd5jZHqAW+PL+gnz0fYfUpDezshDCAWYQ7hgy5Vwy5TxA59Je6Vz2L60cfQhhCjClXtmtjex7TmL9z8CfD6F+IiJyiPRkrIhIhsvEQH9/W1egGWXKuWTKeYDOpb3SuezHAW/GiohIx5aJLXoREUlQoBcRyXAZE+jTHWGzvTKzpWb2fjTKZ1lU1tPMnjOzhdGy9aakaQIzm2Rm68xsdqKswbqbuyf6O82Khs9oNxo5l9vMbFViFNZLEttuic5lvpld1Da1bpiZDTKzl8xsrpnNMbOvR+Ud6m+zn/PocH8XMysws2lm9l50LrdH5cPM7O2ozo+YWV5Unh+9L4+2Dz2oLw4hdPgX3r9/ETAcyAPeA0a1db2aeA5Lgd71yv4duDlavxn4aVvXs5G6nw2MAWYfqO7AJcDTgOFPUb/d1vVP41xuwx8CrL/vqOi/tXxgWPTfYHZbn0Oifv2BMdF6N/zBx1Ed7W+zn/PocH+X6N+2a7Sei4/2eyrwKDAxKv8l8JVo/Z+AX0brE4FHDuZ7M6VFn+4Imx3N5cDvovXfAVe0YV0aFUJ4Faj/IFxjdb8c+H1wbwE9zKx/69T0wBo5l8ZcDjwcfKiPJUA5/t9iuxBCWBNCeCda3wbMwwck7FB/m/2cR2Pa7d8l+rfdHr3NjV4BOA94PCqv/zeJ/1aPA+ebmTX1ezMl0B9whM0OIADPmtmMaJA3gL4hhDXR+lqgb9tU7aA0VveO+re6KUpnTEqk0DrMuUQ/+U/EW5Ad9m9T7zygA/5dzCzbzGYC64Dn8F8cm0MI1dEuyfruPZdo+xagV1O/M1MCfSY4M4QwBrgY+L9mdnZyY/Dfbh2yL2xHrnvkPuBw4AR8RNaftW11msbMuuJPqH8j1Bs5tiP9bRo4jw75dwkh1ASfjGkg/kvjqJb+zkwJ9AcaYbPdCyGsipbrgL/g/wF8GP90jpbr2q6GTdZY3Tvc3yqE8GH0P2ct8ACpNEC7Pxczy8WD459CCE9ExR3ub9PQeXTkvwtACGEz8BJwGp4mi4ekSdZ377lE24uBDU39rkwJ9NOBkdGd6zz8psX+BltrV8ysyHwqRsysCBgPzMbP4bpot+uAv7ZNDQ9KY3WfDHwu6uFxKrAlkUZol+rlqa/E/zbg5zIx6hkxDBgJTGvt+jUmyuX+BpgXQvivxKYO9bdp7Dw64t/FzErNrEe0XohP0ToPD/ifjHar/zeJ/1afBF6MfoU1TVvfhW6uF95jYAGe7/p+W9eniXUfjvcSeA+YE9cfz8W9ACwEngd6tnVdG6n/Q/hP5z14fvELjdUd73Vwb/R3eh8Y29b1T+Nc/hDVdVb0P17/xP7fj85lPnBxW9e/3rmciadlZgEzo9clHe1vs5/z6HB/F+A44N2ozrOBW6Py4fjFqBx4DMiPygui9+XR9uEH870aAkFEJMNlSupGREQaoUAvIpLhFOhFRDKcAr2ISIZToBcRyXAK9CIiGU6BXkQkw/1/LWtxArcZSr0AAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ultimo valore loss : 0.4855262537797292\n"
          ]
        }
      ],
      "source": [
        "plt.plot(range(epochs), losses, 'orange')\n",
        "plt.title('loss funct')\n",
        "plt.show()\n",
        "print(f'ultimo valore loss : {losses[-1]}')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GFMSd36rYsQm",
        "outputId": "980ff018-61ac-4022-a5e5-55c1e9599a45"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "test accuracy   : 73.2\n",
            "train accuracy  : 77.2\n"
          ]
        }
      ],
      "source": [
        "acc_test = 0\n",
        "acc_train = 0\n",
        "\n",
        "with torch.no_grad():\n",
        "    for i, data in enumerate(test_loader):\n",
        "      inputs, labels = data\n",
        "      outputs = model(inputs)\n",
        "      outputs = [1 if outputs[i]>=0.5 else 0 for i in range(len(outputs))]\n",
        "      acc_test += sum([1 if outputs[i]==labels[i] else 0 for i in range(len(labels))])\n",
        "    \n",
        "    for i, data in enumerate(train_loader):\n",
        "      inputs, labels = data\n",
        "      outputs = model(inputs)\n",
        "      outputs = [1 if outputs[i]>=0.5 else 0 for i in range(len(outputs))]\n",
        "      acc_train += sum([1 if outputs[i]==labels[i] else 0 for i in range(len(labels))])\n",
        "\n",
        "print(f'test accuracy   : {acc_test*100/len(test_data)}')\n",
        "print(f'train accuracy  : {acc_train*100/len(train_data)}')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EJFqN1rOXmML"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_b9dhJ06gjN"
      },
      "source": [
        "**Optionally**, you can plot the distribution (histogram) of the parameters after training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lMBJAWFe6vOu"
      },
      "outputs": [],
      "source": [
        "# TODO: plot the histogram\n",
        "plt.hist( ... )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nX48ZUcI9mFa"
      },
      "source": [
        "## Final checklist\n",
        "1. Carefully check all code. Insert comments when needed. Search for \"TODO\" to see if you forgot something.\n",
        "2. Run everything one final time. *Please do not send us notebooks with errors or cells that are not working.*\n",
        "3. Upload the completed notebook **before 30/11/2021 23:59** on the Google Classrom page."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rl9dZ3HambSz"
      },
      "source": [
        "### References\n",
        "\n",
        "[1] Apicella, A., Donnarumma, F., Isgrò, F. and Prevete, R., 2021. [A survey on modern trainable activation functions](https://arxiv.org/abs/2005.00817).\n",
        "\n",
        "[2] Jin, X., Xu, C., Feng, J., Wei, Y., Xiong, J. and Yan, S., 2016. [Deep learning with s-shaped rectified linear activation units](https://arxiv.org/abs/1512.07030). In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 30, No. 1)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "wAEgygyPfO7b",
        "Rl9dZ3HambSz"
      ],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "76cd3bf32b814994a3b62d9387b8d4d3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e8cf9e329b554790975afeb4039ba757",
              "IPY_MODEL_008b07fdc96a4002ac2beca1a9f7d0be",
              "IPY_MODEL_ab7891e8262b4c75a4fcb572f77a2d16"
            ],
            "layout": "IPY_MODEL_c6e5cf4d54da4bdd8dbab54b9d3203ea"
          }
        },
        "e8cf9e329b554790975afeb4039ba757": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7aee557abc584b8abdb0bbf2e1419159",
            "placeholder": "​",
            "style": "IPY_MODEL_caf10268f0bf450e9c70b472019122a4",
            "value": "Dl Completed...: 100%"
          }
        },
        "008b07fdc96a4002ac2beca1a9f7d0be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0af801d24cdd4693abb1b46eda9e0fe7",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e7b142fe0c354835b6b555b7e7fff1df",
            "value": 1
          }
        },
        "ab7891e8262b4c75a4fcb572f77a2d16": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_080ad4c5b61d4d5eb3c6c26fa3130dc5",
            "placeholder": "​",
            "style": "IPY_MODEL_de47594d5dc64ab19d39968fccfed4db",
            "value": " 1/1 [00:00&lt;00:00,  2.21 url/s]"
          }
        },
        "c6e5cf4d54da4bdd8dbab54b9d3203ea": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7aee557abc584b8abdb0bbf2e1419159": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "caf10268f0bf450e9c70b472019122a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0af801d24cdd4693abb1b46eda9e0fe7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "e7b142fe0c354835b6b555b7e7fff1df": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "080ad4c5b61d4d5eb3c6c26fa3130dc5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "de47594d5dc64ab19d39968fccfed4db": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "851d8034a8ee4370a2f61ab79eda6647": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6fc44c12dfb84fed89d1f0f9ea4305e8",
              "IPY_MODEL_9c5b75b8c8604ac38a35be63b77a0d97",
              "IPY_MODEL_a77bcf7be0474c889610da8c32991226"
            ],
            "layout": "IPY_MODEL_4feb171f0f0d469fa0d27ac591704831"
          }
        },
        "6fc44c12dfb84fed89d1f0f9ea4305e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_60cfa7df9d1f4809aef22641da82f2b1",
            "placeholder": "​",
            "style": "IPY_MODEL_c748036e563f4952a1938a2b03fd5811",
            "value": "Dl Size...: "
          }
        },
        "9c5b75b8c8604ac38a35be63b77a0d97": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_653536b09edf40f78489c96feb30055f",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ee9683900bfe4497957fdcf28cc6f1e6",
            "value": 0
          }
        },
        "a77bcf7be0474c889610da8c32991226": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ffb3c83cfaef48b097e3ae17e6d09927",
            "placeholder": "​",
            "style": "IPY_MODEL_f6849f2fc6a042aeb179c27006200d47",
            "value": " 0/0 [00:00&lt;?, ? MiB/s]"
          }
        },
        "4feb171f0f0d469fa0d27ac591704831": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "60cfa7df9d1f4809aef22641da82f2b1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c748036e563f4952a1938a2b03fd5811": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "653536b09edf40f78489c96feb30055f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "ee9683900bfe4497957fdcf28cc6f1e6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ffb3c83cfaef48b097e3ae17e6d09927": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f6849f2fc6a042aeb179c27006200d47": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "833dd5cdfe664dd3a6ec6b9bcb137f75": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_afd659d3578b49cb8299ae67b9a86a02",
              "IPY_MODEL_07bacd156f87482db28235eac19c5597",
              "IPY_MODEL_d551c0b12d8e47eb8a203319314582c8"
            ],
            "layout": "IPY_MODEL_bcd78433f56f4201b71f20b2f14acbcd"
          }
        },
        "afd659d3578b49cb8299ae67b9a86a02": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_834c9a34933e44d48059cd2d2b075076",
            "placeholder": "​",
            "style": "IPY_MODEL_48197e0c63c34739811d0fc76de4e8ea",
            "value": "Generating splits...: 100%"
          }
        },
        "07bacd156f87482db28235eac19c5597": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c1a13466807d408b9e1575b238fb22af",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_20d069df08884162bac03ed46a44201e",
            "value": 1
          }
        },
        "d551c0b12d8e47eb8a203319314582c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e252357a61ce47f3852151542c1396fe",
            "placeholder": "​",
            "style": "IPY_MODEL_7cf2656e58fe434ea108dcb70a6bd8c6",
            "value": " 1/1 [00:00&lt;00:00,  3.80 splits/s]"
          }
        },
        "bcd78433f56f4201b71f20b2f14acbcd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "834c9a34933e44d48059cd2d2b075076": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "48197e0c63c34739811d0fc76de4e8ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c1a13466807d408b9e1575b238fb22af": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20d069df08884162bac03ed46a44201e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e252357a61ce47f3852151542c1396fe": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7cf2656e58fe434ea108dcb70a6bd8c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e0f481da504f4326a906699e5404cc39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5dfea2b19ef849e48d22328977476c0a",
              "IPY_MODEL_67d081ca452543a28d4e6cf205b40f6a",
              "IPY_MODEL_a1a8ac1fdac544e6b4d8520a878c3d8a"
            ],
            "layout": "IPY_MODEL_7fe9307d8bd543a4a409b0729a7a62ba"
          }
        },
        "5dfea2b19ef849e48d22328977476c0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f88dbb487ef64339b5b42f99beb2fc40",
            "placeholder": "​",
            "style": "IPY_MODEL_9cf05a6b976a436693c06e38c576e477",
            "value": "Generating train examples...:  36%"
          }
        },
        "67d081ca452543a28d4e6cf205b40f6a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f971b940a3094448afe017314aacaa77",
            "max": 1000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9837fca54727463caeeb427518bf97d3",
            "value": 1000
          }
        },
        "a1a8ac1fdac544e6b4d8520a878c3d8a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da0047e75f1f4420b2369958545d6e54",
            "placeholder": "​",
            "style": "IPY_MODEL_32016a60ed0c428abdd103f3f7251ba2",
            "value": " 362/1000 [00:00&lt;00:00, 3616.82 examples/s]"
          }
        },
        "7fe9307d8bd543a4a409b0729a7a62ba": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "f88dbb487ef64339b5b42f99beb2fc40": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9cf05a6b976a436693c06e38c576e477": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f971b940a3094448afe017314aacaa77": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9837fca54727463caeeb427518bf97d3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "da0047e75f1f4420b2369958545d6e54": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "32016a60ed0c428abdd103f3f7251ba2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "02c79bc821a2454480a2ff5b45217a16": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_de6c910b5e2c4da698533ff97e3a5e01",
              "IPY_MODEL_160911d9ec3f465cb9e130c3f7d78637",
              "IPY_MODEL_7d7b03943eb244b183d06c92fd7cc596"
            ],
            "layout": "IPY_MODEL_754d024b65794e17980cdf906b34d59a"
          }
        },
        "de6c910b5e2c4da698533ff97e3a5e01": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d9d3b1ea29a246bf838f7064377cc0fd",
            "placeholder": "​",
            "style": "IPY_MODEL_482f445629fa47aa93074bee01ed746f",
            "value": "Shuffling ~/tensorflow_datasets/german_credit_numeric/1.0.0.incompleteAARTBJ/german_credit_numeric-train.tfrecord*...:   0%"
          }
        },
        "160911d9ec3f465cb9e130c3f7d78637": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5f25e2d35c2c428b8fe0f2bb7f074065",
            "max": 1000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_acf1cc70157a419ba7c86e7817d3bc50",
            "value": 1000
          }
        },
        "7d7b03943eb244b183d06c92fd7cc596": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_520701f4d5464fc696f0cda9afc6fdc9",
            "placeholder": "​",
            "style": "IPY_MODEL_c06d1ee12ad1427194fcb8e1ce45b742",
            "value": " 0/1000 [00:00&lt;?, ? examples/s]"
          }
        },
        "754d024b65794e17980cdf906b34d59a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "d9d3b1ea29a246bf838f7064377cc0fd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "482f445629fa47aa93074bee01ed746f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5f25e2d35c2c428b8fe0f2bb7f074065": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "acf1cc70157a419ba7c86e7817d3bc50": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "520701f4d5464fc696f0cda9afc6fdc9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c06d1ee12ad1427194fcb8e1ce45b742": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}