C1_W2_Lab_2_huber-object-loss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab: Huber Loss hyperparameter and class\n",
    "\n",
    "In this lab, we'll extend our previous Huber loss function and show how you can include hyperparameters in defining loss functions. We'll also look at how to implement a custom loss as an object by inheriting the [Loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses/Loss) class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0p84I7yFHRT2"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "As before, this model will be trained on the `xs` and `ys` below where the relationship is $y = 2x-1$. Thus, later, when we test for `x=10`, whichever version of the model gets the closest answer to `19` will be deemed more accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0pajvrhrInPa"
   },
   "outputs": [],
   "source": [
    "# inputs\n",
    "xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
    "\n",
    "# labels\n",
    "ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom loss with hyperparameter\n",
    "\n",
    "The `loss` argument in `model.compile()` only accepts functions that accepts two parameters: the ground truth (`y_true`) and the model predictions (`y_pred`). If we want to include a hyperparameter that we can tune, then we can define a wrapper function that accepts this hyperparameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FvRZpxk2tGn6"
   },
   "outputs": [],
   "source": [
    "# wrapper function that accepts the hyperparameter\n",
    "def my_huber_loss_with_threshold(threshold):\n",
    "  \n",
    "    # function that accepts the ground truth and predictions\n",
    "    def my_huber_loss(y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) <= threshold\n",
    "        small_error_loss = tf.square(error) / 2\n",
    "        big_error_loss = threshold * (tf.abs(error) - (0.5 * threshold))\n",
    "        \n",
    "        return tf.where(is_small_error, small_error_loss, big_error_loss) \n",
    "\n",
    "    # return the inner function tuned by the hyperparameter\n",
    "    return my_huber_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now specify the `loss` as the wrapper function above. Notice that we can now set the `threshold` value. Try varying this value and see the results you get."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K7rgmb5qH5QX"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model.compile(optimizer='sgd', loss=my_huber_loss_with_threshold(threshold=1.2))\n",
    "model.fit(xs, ys, epochs=500,verbose=0)\n",
    "print(model.predict([10.0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement Custom Loss as a Class\n",
    "\n",
    "We can also implement our custom loss as a class. It inherits from the Keras Loss class and the syntax and required methods are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iderl2m0saTa"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.losses import Loss\n",
    "\n",
    "class MyHuberLoss(Loss):\n",
    "  \n",
    "    # initialize instance attributes\n",
    "    def __init__(self, threshold=1):\n",
    "        super().__init__()\n",
    "        self.threshold = threshold\n",
    "\n",
    "    # compute loss\n",
    "    def call(self, y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) <= self.threshold\n",
    "        small_error_loss = tf.square(error) / 2\n",
    "        big_error_loss = self.threshold * (tf.abs(error) - (0.5 * self.threshold))\n",
    "        return tf.where(is_small_error, small_error_loss, big_error_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify the loss by instantiating an object from your custom loss class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lwKWfSmDIayP"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])\n",
    "model.compile(optimizer='sgd', loss=MyHuberLoss(threshold=1.02))\n",
    "model.fit(xs, ys, epochs=500,verbose=0)\n",
    "print(model.predict([10.0]))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "huber-object-loss.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}