C1_W5_Lab_1_exploring-callbacks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/https-deeplearning-ai/tensorflow-3-public/blob/main/Course%201%20-%20Custom%20Models%2C%20Layers%20and%20Loss%20Functions/Week%205%20-%20Callbacks/C1_W5_Lab_1_exploring-callbacks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0c_TYhQOUe1j"
   },
   "source": [
    "# Ungraded Lab: Introduction to Keras callbacks\n",
    "\n",
    "In Keras, `Callback` is a Python class meant to be subclassed to provide specific functionality, with a set of methods called at various stages of training (including batch/epoch start and ends), testing, and predicting. Callbacks are useful to get a view on internal states and statistics of the model during training. The methods of the callbacks can  be called at different stages of training/evaluating/inference. Keras has available [callbacks](https://keras.io/api/callbacks/) and we'll show how you can use it in the following sections. Please click the **Open in Colab** badge above to complete this exercise in Colab. This will allow you to take advantage of the free GPU runtime (for faster training) and compatibility with all the packages needed in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Uyl69EyRQx-f"
   },
   "source": [
    "## Model methods that take callbacks\n",
    "Users can supply a list of callbacks to the following `tf.keras.Model` methods:\n",
    "* [`fit()`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#fit), [`fit_generator()`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#fit_generator)\n",
    "Trains the model for a fixed number of epochs (iterations over a dataset, or data yielded batch-by-batch by a Python generator).\n",
    "* [`evaluate()`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#evaluate), [`evaluate_generator()`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#evaluate_generator)\n",
    "Evaluates the model for given data or data generator. Outputs the loss and metric values from the evaluation.\n",
    "* [`predict()`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#predict), [`predict_generator()`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#predict_generator)\n",
    "Generates output predictions for the input data or data generator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AlT1Kh3uA9lZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version:  2.1.0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "import matplotlib.pyplot as plt\n",
    "import io\n",
    "from PIL import Image\n",
    "\n",
    "from tensorflow.keras.callbacks import TensorBoard, EarlyStopping, LearningRateScheduler, ModelCheckpoint, CSVLogger, ReduceLROnPlateau\n",
    "%load_ext tensorboard\n",
    "\n",
    "import os\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "import math\n",
    "import datetime\n",
    "import pandas as pd\n",
    "\n",
    "print(\"Version: \", tf.__version__)\n",
    "tf.get_logger().setLevel('INFO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HnSljqtsXKfb"
   },
   "source": [
    "# Examples of Keras callback applications\n",
    "The following section will guide you through creating simple [Callback](https://keras.io/api/callbacks/) applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "spskRuxvCYQE"
   },
   "outputs": [],
   "source": [
    "# Download and prepare the horses or humans dataset\n",
    "\n",
    "# horses_or_humans 3.0.0 has already been downloaded for you\n",
    "path = \"./tensorflow_datasets\"\n",
    "splits, info = tfds.load('horses_or_humans', data_dir=path, as_supervised=True, with_info=True, split=['train[:80%]', 'train[80%:]', 'test'])\n",
    "\n",
    "(train_examples, validation_examples, test_examples) = splits\n",
    "\n",
    "num_examples = info.splits['train'].num_examples\n",
    "num_classes = info.features['label'].num_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "veIsubKTCZsN"
   },
   "outputs": [],
   "source": [
    "SIZE = 150 #@param {type:\"slider\", min:64, max:300, step:1}\n",
    "IMAGE_SIZE = (SIZE, SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "faajLlErCb1S"
   },
   "outputs": [],
   "source": [
    "def format_image(image, label):\n",
    "  image = tf.image.resize(image, IMAGE_SIZE) / 255.0\n",
    "  return  image, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AVXPuU12Cdka"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32 #@param {type:\"integer\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0lHDkFVaCe48"
   },
   "outputs": [],
   "source": [
    "train_batches = train_examples.shuffle(num_examples // 4).map(format_image).batch(BATCH_SIZE).prefetch(1)\n",
    "validation_batches = validation_examples.map(format_image).batch(BATCH_SIZE).prefetch(1)\n",
    "test_batches = test_examples.map(format_image).batch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DxsCqEIkCgUt"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([32, 150, 150, 3])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for image_batch, label_batch in train_batches.take(1):\n",
    "  pass\n",
    "\n",
    "image_batch.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iDBpWvHXCh2A"
   },
   "outputs": [],
   "source": [
    "def build_model(dense_units, input_shape=IMAGE_SIZE + (3,)):\n",
    "  model = tf.keras.models.Sequential([\n",
    "      tf.keras.layers.Conv2D(16, (3, 3), activation='relu', input_shape=input_shape),\n",
    "      tf.keras.layers.MaxPooling2D(2, 2),\n",
    "      tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2, 2),\n",
    "      tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),\n",
    "      tf.keras.layers.MaxPooling2D(2, 2),\n",
    "      tf.keras.layers.Flatten(),\n",
    "      tf.keras.layers.Dense(dense_units, activation='relu'),\n",
    "      tf.keras.layers.Dense(2, activation='softmax')\n",
    "  ])\n",
    "  return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0ZKGkjagENSw"
   },
   "source": [
    "## [TensorBoard](https://keras.io/api/callbacks/tensorboard/)\n",
    "\n",
    "Enable visualizations for TensorBoard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CeiD2WVEHbex"
   },
   "outputs": [],
   "source": [
    "!rm -rf logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PpLwPLnAEOzv"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "logdir = os.path.join(\"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\n",
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir)\n",
    "\n",
    "model.fit(train_batches, \n",
    "          epochs=10, \n",
    "          validation_data=validation_batches, \n",
    "          callbacks=[tensorboard_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iJunWOjZE0ir"
   },
   "outputs": [],
   "source": [
    "%tensorboard --logdir logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wv9H4Pc2Mfl7"
   },
   "source": [
    "## [Model Checkpoint](https://keras.io/api/callbacks/model_checkpoint/)\n",
    "\n",
    "Callback to save the Keras model or model weights at some frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PYV4FJ8iMmDq"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "model.fit(train_batches, \n",
    "          epochs=5, \n",
    "          validation_data=validation_batches, \n",
    "          verbose=2,\n",
    "          callbacks=[ModelCheckpoint('weights.{epoch:02d}-{val_loss:.2f}.h5', verbose=1),\n",
    "          ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oGvjQ8IlMmK6"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "model.fit(train_batches, \n",
    "          epochs=1, \n",
    "          validation_data=validation_batches, \n",
    "          verbose=2,\n",
    "          callbacks=[ModelCheckpoint('saved_model', verbose=1)\n",
    "          ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y1ConwoB0EjD"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "model.fit(train_batches, \n",
    "          epochs=2, \n",
    "          validation_data=validation_batches, \n",
    "          verbose=2,\n",
    "          callbacks=[ModelCheckpoint('model.h5', verbose=1)\n",
    "          ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kptNF0--Lznv"
   },
   "source": [
    "## [Early stopping](https://keras.io/api/callbacks/early_stopping/)\n",
    "\n",
    "Stop training when a monitored metric has stopped improving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KJOJTJYdCkdY"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "model.fit(train_batches, \n",
    "          epochs=50, \n",
    "          validation_data=validation_batches, \n",
    "          verbose=2,\n",
    "          callbacks=[EarlyStopping(\n",
    "              patience=3,\n",
    "              min_delta=0.05,\n",
    "              baseline=0.8,\n",
    "              mode='min',\n",
    "              monitor='val_loss',\n",
    "              restore_best_weights=True,\n",
    "              verbose=1)\n",
    "          ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8mDzWUD4Pqq5"
   },
   "source": [
    "## [CSV Logger](https://keras.io/api/callbacks/csv_logger/)\n",
    "\n",
    "Callback that streams epoch results to a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cffnMpmGPtMh"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "csv_file = 'training.csv'\n",
    "\n",
    "model.fit(train_batches, \n",
    "          epochs=5, \n",
    "          validation_data=validation_batches, \n",
    "          callbacks=[CSVLogger(csv_file)\n",
    "          ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B9tkYi03QV7R"
   },
   "outputs": [],
   "source": [
    "pd.read_csv(csv_file).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Dt9C2Y9fRBKN"
   },
   "source": [
    "## [Learning Rate Scheduler](https://keras.io/api/callbacks/learning_rate_scheduler/)\n",
    "\n",
    "Updates the learning rate during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aJi-xY2VRC03"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "def step_decay(epoch):\n",
    "\tinitial_lr = 0.01\n",
    "\tdrop = 0.5\n",
    "\tepochs_drop = 1\n",
    "\tlr = initial_lr * math.pow(drop, math.floor((1+epoch)/epochs_drop))\n",
    "\treturn lr\n",
    "\n",
    "model.fit(train_batches, \n",
    "          epochs=5, \n",
    "          validation_data=validation_batches, \n",
    "          callbacks=[LearningRateScheduler(step_decay, verbose=1),\n",
    "                    TensorBoard(log_dir='./log_dir')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M2S4n8nrbV91"
   },
   "outputs": [],
   "source": [
    "%tensorboard --logdir log_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y0wcuQyJE_UK"
   },
   "source": [
    "## [ReduceLROnPlateau](https://keras.io/api/callbacks/reduce_lr_on_plateau/)\n",
    "\n",
    "Reduce learning rate when a metric has stopped improving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4naxZ-eCFB27"
   },
   "outputs": [],
   "source": [
    "model = build_model(dense_units=256)\n",
    "model.compile(\n",
    "    optimizer='sgd',\n",
    "    loss='sparse_categorical_crossentropy', \n",
    "    metrics=['accuracy'])\n",
    "  \n",
    "model.fit(train_batches, \n",
    "          epochs=50, \n",
    "          validation_data=validation_batches, \n",
    "          callbacks=[ReduceLROnPlateau(monitor='val_loss', \n",
    "                                       factor=0.2, verbose=1,\n",
    "                                       patience=1, min_lr=0.001),\n",
    "                     TensorBoard(log_dir='./log_dir')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "isfTWP4NYudk"
   },
   "outputs": [],
   "source": [
    "%tensorboard --logdir log_dir"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "ExploringCallbacks.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}